From patchwork Sun May 24 22:38:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 19836 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 86A8944A62D for ; Mon, 25 May 2020 01:39:52 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7622268807A; Mon, 25 May 2020 01:39:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe05-1.mx.upcmail.net (vie01a-dmta-pe05-1.mx.upcmail.net [84.116.36.11]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4B75F681871 for ; Mon, 25 May 2020 01:39:45 +0300 (EEST) Received: from [172.31.216.235] (helo=vie01a-pemc-psmtp-pe12.mail.upcmail.net) by vie01a-dmta-pe05.mx.upcmail.net with esmtp (Exim 4.92) (envelope-from ) id 1jczHU-00006z-2Z for ffmpeg-devel@ffmpeg.org; Mon, 25 May 2020 00:39:44 +0200 Received: from localhost ([213.47.68.29]) by vie01a-pemc-psmtp-pe12.mail.upcmail.net with ESMTP id czGVjavez6Jy6czGVjjwJL; Mon, 25 May 2020 00:38:43 +0200 X-Env-Mailfrom: michael@niedermayer.cc X-Env-Rcptto: ffmpeg-devel@ffmpeg.org X-SourceIP: 213.47.68.29 X-CNFS-Analysis: v=2.3 cv=GKl27dFK c=1 sm=1 tr=0 a=2hcxjKEKjp0CzLx6oWAm4g==:117 a=2hcxjKEKjp0CzLx6oWAm4g==:17 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=GEAsPZ9sns4A:10 a=ZZnuYtJkoWoA:10 a=nZOtpAppAAAA:20 a=8D7CuHAvu7ih5_Jnnv0A:9 a=1fhp2MxaeJtTNGEnv6mo:22 a=Z5ABNNGmrOfJ6cZ5bIyy:22 a=jd6J4Gguk5HxikPWLKER:22 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Mon, 25 May 2020 00:38:41 +0200 Message-Id: <20200524223843.8152-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 X-CMAE-Envelope: MS4wfMSyh/9f4egweDO8amVRYsgWLqLL04sidMWzNufdRHIEg6Cxq7WRh/7Pa8tcoQCX7502HtL4g2bwfR1+H8CooPVIcQwDzFmuP8GJ2L6l+niIOqyZ83iP MKe7Y5rilvi9r7K5jh2xXQF6MfIDmICur9SJdsEcX2kQF9UBzUKrOOc9 Subject: [FFmpeg-devel] [PATCH 1/3] avcodec/mv30: Fix multiple integer overflows in idct_1d() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: signed integer overflow: -4869937 * 473 cannot be represented in type 'int' Fixes: 21934/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MV30_fuzzer-5667289925156864 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/mv30.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/libavcodec/mv30.c b/libavcodec/mv30.c index 658f32c6ff..013a5753fe 100644 --- a/libavcodec/mv30.c +++ b/libavcodec/mv30.c @@ -107,7 +107,7 @@ static inline void idct_1d(int *blk, int step) const int t0 = blk[0 * step] + blk[4 * step]; const int t1 = blk[0 * step] - blk[4 * step]; const int t2 = blk[2 * step] + blk[6 * step]; - const int t3 = (((blk[2 * step] - blk[6 * step]) * 362) >> 8) - t2; + const int t3 = ((int)((blk[2 * step] - blk[6 * step]) * 362U) >> 8) - t2; const int t4 = t0 + t2; const int t5 = t0 - t2; const int t6 = t1 + t3; @@ -117,10 +117,10 @@ static inline void idct_1d(int *blk, int step) const int tA = blk[1 * step] + blk[7 * step]; const int tB = blk[1 * step] - blk[7 * step]; const int tC = t8 + tA; - const int tD = (tB + t9) * 473 >> 8; - const int tE = ((t9 * -669 >> 8) - tC) + tD; - const int tF = ((tA - t8) * 362 >> 8) - tE; - const int t10 = ((tB * 277 >> 8) - tD) + tF; + const int tD = (int)((tB + t9) * 473U) >> 8; + const int tE = (((int)(t9 * -669U) >> 8) - tC) + tD; + const int tF = ((int)((tA - t8) * 362U) >> 8) - tE; + const int t10 = (((int)(tB * 277U) >> 8) - tD) + tF; blk[0 * step] = t4 + tC; blk[1 * step] = t6 + tE;