From patchwork Thu Oct 5 19:44:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 44165 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:7a24:b0:15d:8365:d4b8 with SMTP id t36csp257210pzh; Thu, 5 Oct 2023 12:45:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHuc/Cw+rRHu38BXbfvfv7sVwYA0ChmGg/tzk5vs8IAVY7+MCq/z+Q79ClEUg9mh9/hG7fa X-Received: by 2002:a17:907:b11:b0:9ae:699d:8a29 with SMTP id h17-20020a1709070b1100b009ae699d8a29mr5155690ejl.6.1696535129193; Thu, 05 Oct 2023 12:45:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696535129; cv=none; d=google.com; s=arc-20160816; b=P9SaogJSCvMwQpN2dg1iB2C4pYZZ2FVRo9c37sHTtAVQIy1zgnRNP4W182Pit8p7uE yRRY2/tM4VFX/D2i1xsjIQ7vCoPOoIcBZ6xMBTz2PFU9DW17BpYHOEqhddFJzXsBn0Ka hED4i0C+7rz/jf8N4X6FMFGZoB5Rw9gj+gUbEt68W9vnpy4YdY2b/RA0vpiI31/tGzUq KfOj0wULmV9uX8ib16Wsp+XnRTQGzx64nHU1lIRKzPidXgqN9T5AmFf2JFpTOBvWSDZv 33ngz2jlEff4O+3TSWlyTaKdhDviAQyPQrb9PMpQncR2Rru5q3LCpqUQWqTF5uItJmdm Q8Ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:delivered-to; bh=EeAiLWtY6GSI/RyEkfXCYfop1euFGvYVQNYXpby9D0M=; fh=e5zN9xSzcxLA6bGo3lF+CqTbY/oLwzApV03EO/RBfgQ=; b=islEyAYZL3kHDiBDSmGcWrxTWyUXbbMsA4hHfqHxogy5a2yihOLzOSRXE3rXdIHT2X jQ/OYo5Hmhn4lGeIHyWQQ/FBDtxEPrESbmOkElQd1yWcIycnpZjVEfIYHpPr4TVdSYN3 EIi+cXKMi3Q9ikyurRy9+I2uBeE+7EwxGK8ky0jtCyQtT4Ud8fMeGbzvXjA4TRJ98x1y 81zTT4ZUy1KhIoynmocKQBgzlMFdRpb3r810D2QF8l9TUZag1aJQs/H51AU72tvvpT1n x9pE55bZMKXPhxiM1mWl18uk0JLc5kw9XmIrCF6miz2GeZ2MJJugFr9F01xj4BnS+ApA PifA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y11-20020a17090629cb00b009b93c77d2cesi1034955eje.378.2023.10.05.12.45.28; Thu, 05 Oct 2023 12:45:29 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C736E68CC5D; Thu, 5 Oct 2023 22:44:54 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from relay2-d.mail.gandi.net (relay2-d.mail.gandi.net [217.70.183.194]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7614B68CB94 for ; Thu, 5 Oct 2023 22:44:46 +0300 (EEST) Received: by mail.gandi.net (Postfix) with ESMTPSA id 8E3C040004 for ; Thu, 5 Oct 2023 19:44:45 +0000 (UTC) From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Thu, 5 Oct 2023 21:44:40 +0200 Message-Id: <20231005194440.1678-5-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231005194440.1678-1-michael@niedermayer.cc> References: <20231005194440.1678-1-michael@niedermayer.cc> X-GND-Sasl: michael@niedermayer.cc Subject: [FFmpeg-devel] [PATCH 5/5] avcodec/xvididct: Make c* unsigned to avoid undefined overflows X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: QJCGIvkkSWBO Fixes: signed integer overflow: 1496950099 + 728014168 cannot be represented in type 'int' Fixes: 62667/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MJPEGB_fuzzer-6511785170305024 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/xvididct.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/libavcodec/xvididct.c b/libavcodec/xvididct.c index dcea32210a2..01072d80ab7 100644 --- a/libavcodec/xvididct.c +++ b/libavcodec/xvididct.c @@ -56,13 +56,13 @@ static const int TAB35[] = { 26722, 25172, 22654, 19266, 15137, 10426, 5315 }; static int idct_row(short *in, const int *const tab, int rnd) { - const int c1 = tab[0]; - const int c2 = tab[1]; - const int c3 = tab[2]; - const int c4 = tab[3]; - const int c5 = tab[4]; - const int c6 = tab[5]; - const int c7 = tab[6]; + const unsigned c1 = tab[0]; + const unsigned c2 = tab[1]; + const unsigned c3 = tab[2]; + const unsigned c4 = tab[3]; + const unsigned c5 = tab[4]; + const unsigned c6 = tab[5]; + const unsigned c7 = tab[6]; const int right = in[5] | in[6] | in[7]; const int left = in[1] | in[2] | in[3]; @@ -102,8 +102,8 @@ static int idct_row(short *in, const int *const tab, int rnd) return 0; } } else if (!(left | right)) { - const int a0 = (rnd + c4 * (in[0] + in[4])) >> ROW_SHIFT; - const int a1 = (rnd + c4 * (in[0] - in[4])) >> ROW_SHIFT; + const int a0 = (int)(rnd + c4 * (in[0] + in[4])) >> ROW_SHIFT; + const int a1 = (int)(rnd + c4 * (in[0] - in[4])) >> ROW_SHIFT; in[0] = a0; in[3] = a0;