From patchwork Sat Jul 25 22:16:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 21259 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id A95CF44B318 for ; Sun, 26 Jul 2020 01:23:36 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6FB1968AE26; Sun, 26 Jul 2020 01:23:36 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe05-2.mx.upcmail.net (vie01a-dmta-pe05-2.mx.upcmail.net [84.116.36.12]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2977668AC45 for ; Sun, 26 Jul 2020 01:23:30 +0300 (EEST) Received: from [172.31.216.235] (helo=vie01a-pemc-psmtp-pe12.mail.upcmail.net) by vie01a-dmta-pe05.mx.upcmail.net with esmtp (Exim 4.92) (envelope-from ) id 1jzSU6-0002LE-08 for ffmpeg-devel@ffmpeg.org; Sun, 26 Jul 2020 00:17:38 +0200 Received: from localhost ([213.47.68.29]) by vie01a-pemc-psmtp-pe12.mail.upcmail.net with ESMTP id zST7jl0cOIr7GzST8jaNMj; Sun, 26 Jul 2020 00:16:38 +0200 X-Env-Mailfrom: michael@niedermayer.cc X-Env-Rcptto: ffmpeg-devel@ffmpeg.org X-SourceIP: 213.47.68.29 X-CNFS-Analysis: v=2.3 cv=QN4WuTDL c=1 sm=1 tr=0 a=2hcxjKEKjp0CzLx6oWAm4g==:117 a=2hcxjKEKjp0CzLx6oWAm4g==:17 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=GEAsPZ9sns4A:10 a=ZZnuYtJkoWoA:10 a=nZOtpAppAAAA:20 a=B01DAx8YP5KVSq9Ru7UA:9 a=1fhp2MxaeJtTNGEnv6mo:22 a=Z5ABNNGmrOfJ6cZ5bIyy:22 a=jd6J4Gguk5HxikPWLKER:22 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Sun, 26 Jul 2020 00:16:36 +0200 Message-Id: <20200725221637.11848-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 X-CMAE-Envelope: MS4wfPzGfbjH9AtvVy4D2m+lvqEmgBcaNeZfpcF1tQf9Mj2/agHM5IN0vMyLg72gi5LyRC2A+VVqZvfMYRJIuC3t+bhxHNjaLzYSsihADBPTDjQ0I+kiyqhy bW4m4zytu5Tn0XSBtxWJ0jlxlTW/hArCVPw5NuJlvT/4KRTLP1n2bQrY Subject: [FFmpeg-devel] [PATCH 1/2] avcodec/pgxdec: Fix invalid shift in write_frame_* X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: left shift of negative value -121 Fixes: 23911/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_PGX_fuzzer-4986800258154496 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/pgxdec.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavcodec/pgxdec.c b/libavcodec/pgxdec.c index 93b9f4e7a0..e72ec84152 100644 --- a/libavcodec/pgxdec.c +++ b/libavcodec/pgxdec.c @@ -102,7 +102,7 @@ error: for (i = 0; i < height; i++) { \ PIXEL *line = (PIXEL*)frame->data[0] + i*frame->linesize[0]/sizeof(PIXEL); \ for (j = 0; j < width; j++) { \ - int val; \ + unsigned val; \ if (sign) \ val = (PIXEL)bytestream2_get_ ##suffix(g) + (1 << (depth - 1)); \ else \ From patchwork Sat Jul 25 22:16:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 21260 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id C181344B46D for ; Sun, 26 Jul 2020 01:24:22 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AB1B968AE44; Sun, 26 Jul 2020 01:24:22 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe03-3.mx.upcmail.net (vie01a-dmta-pe03-3.mx.upcmail.net [62.179.121.162]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2F05968A7D8 for ; Sun, 26 Jul 2020 01:24:16 +0300 (EEST) Received: from [172.31.216.235] (helo=vie01a-pemc-psmtp-pe12.mail.upcmail.net) by vie01a-dmta-pe03.mx.upcmail.net with esmtp (Exim 4.92) (envelope-from ) id 1jzSU7-000A2s-0F for ffmpeg-devel@ffmpeg.org; Sun, 26 Jul 2020 00:17:39 +0200 Received: from localhost ([213.47.68.29]) by vie01a-pemc-psmtp-pe12.mail.upcmail.net with ESMTP id zST8jl0dkIr7GzST8jaNMn; Sun, 26 Jul 2020 00:16:39 +0200 X-Env-Mailfrom: michael@niedermayer.cc X-Env-Rcptto: ffmpeg-devel@ffmpeg.org X-SourceIP: 213.47.68.29 X-CNFS-Analysis: v=2.3 cv=QN4WuTDL c=1 sm=1 tr=0 a=2hcxjKEKjp0CzLx6oWAm4g==:117 a=2hcxjKEKjp0CzLx6oWAm4g==:17 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=GEAsPZ9sns4A:10 a=ZZnuYtJkoWoA:10 a=nZOtpAppAAAA:20 a=4n8GpBdDNUfUF3_th3sA:9 a=1fhp2MxaeJtTNGEnv6mo:22 a=Z5ABNNGmrOfJ6cZ5bIyy:22 a=UDnyf2zBuKT2w-IlGP_r:22 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Sun, 26 Jul 2020 00:16:37 +0200 Message-Id: <20200725221637.11848-2-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200725221637.11848-1-michael@niedermayer.cc> References: <20200725221637.11848-1-michael@niedermayer.cc> X-CMAE-Envelope: MS4wfHDIm1VykxtUzn66frtKXTFD/UrgizLKAsEposx6+hfQdykK3mcc3WCDarQQIlL/gKfQp7mTC4e134LdVJarH4G6nu2NUPtcxmU+49G88tC0AMJXdnTg rTpuX6h33bki43HfZBGRfRKKblj5j2/6CN8ttpCdwQYFOytDZW2YENT/ Subject: [FFmpeg-devel] [PATCH 2/2] avcodec/mv30: Fix several integer overflows in idct_1d() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: signed integer overflow: -1846510390 + -361755993 cannot be represented in type 'int' Fixes: 23941/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MV30_fuzzer-5654696631730176 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/mv30.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/libavcodec/mv30.c b/libavcodec/mv30.c index c83ba7ffbd..f9cc85f2ac 100644 --- a/libavcodec/mv30.c +++ b/libavcodec/mv30.c @@ -104,23 +104,23 @@ static void get_qtable(int16_t *table, int quant, const uint8_t *quant_tab) static inline void idct_1d(int *blk, int step) { - const int t0 = blk[0 * step] + blk[4 * step]; - const int t1 = blk[0 * step] - blk[4 * step]; - const int t2 = blk[2 * step] + blk[6 * step]; - const int t3 = ((int)((blk[2 * step] - blk[6 * step]) * 362U) >> 8) - t2; - const int t4 = t0 + t2; - const int t5 = t0 - t2; - const int t6 = t1 + t3; - const int t7 = t1 - t3; - const int t8 = blk[5 * step] + blk[3 * step]; - const int t9 = blk[5 * step] - blk[3 * step]; - const int tA = blk[1 * step] + blk[7 * step]; - const int tB = blk[1 * step] - blk[7 * step]; - const int tC = t8 + tA; - const int tD = (int)((tB + t9) * 473U) >> 8; - const int tE = (((int)(t9 * -669U) >> 8) - tC) + tD; - const int tF = ((int)((tA - t8) * 362U) >> 8) - tE; - const int t10 = (((int)(tB * 277U) >> 8) - tD) + tF; + const unsigned t0 = blk[0 * step] + blk[4 * step]; + const unsigned t1 = blk[0 * step] - blk[4 * step]; + const unsigned t2 = blk[2 * step] + blk[6 * step]; + const unsigned t3 = ((int)((blk[2 * step] - blk[6 * step]) * 362U) >> 8) - t2; + const unsigned t4 = t0 + t2; + const unsigned t5 = t0 - t2; + const unsigned t6 = t1 + t3; + const unsigned t7 = t1 - t3; + const unsigned t8 = blk[5 * step] + blk[3 * step]; + const unsigned t9 = blk[5 * step] - blk[3 * step]; + const unsigned tA = blk[1 * step] + blk[7 * step]; + const unsigned tB = blk[1 * step] - blk[7 * step]; + const unsigned tC = t8 + tA; + const unsigned tD = (int)((tB + t9) * 473U) >> 8; + const unsigned tE = (((int)(t9 * -669U) >> 8) - tC) + tD; + const unsigned tF = ((int)((tA - t8) * 362U) >> 8) - tE; + const unsigned t10 = (((int)(tB * 277U) >> 8) - tD) + tF; blk[0 * step] = t4 + tC; blk[1 * step] = t6 + tE;