From patchwork Thu Feb 6 13:45:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 17700 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 01E2844A41C for ; Thu, 6 Feb 2020 15:52:04 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CA12B68A6E7; Thu, 6 Feb 2020 15:52:03 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe06-2.mx.upcmail.net (vie01a-dmta-pe06-2.mx.upcmail.net [84.116.36.15]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 92FD568A0FA for ; Thu, 6 Feb 2020 15:51:57 +0200 (EET) Received: from [172.31.216.235] (helo=vie01a-pemc-psmtp-pe12.mail.upcmail.net) by vie01a-dmta-pe06.mx.upcmail.net with esmtp (Exim 4.92) (envelope-from ) id 1izhUY-0001Vu-0e for ffmpeg-devel@ffmpeg.org; Thu, 06 Feb 2020 14:46:50 +0100 Received: from localhost ([213.47.68.29]) by vie01a-pemc-psmtp-pe12.mail.upcmail.net with ESMTP id zhTZikzFRwlyszhTZiqtEZ; Thu, 06 Feb 2020 14:45:50 +0100 X-Env-Mailfrom: michael@niedermayer.cc X-Env-Rcptto: ffmpeg-devel@ffmpeg.org X-SourceIP: 213.47.68.29 X-CNFS-Analysis: v=2.3 cv=E5OzWpVl c=1 sm=1 tr=0 a=2hcxjKEKjp0CzLx6oWAm4g==:117 a=2hcxjKEKjp0CzLx6oWAm4g==:17 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=GEAsPZ9sns4A:10 a=ZZnuYtJkoWoA:10 a=nZOtpAppAAAA:20 a=kuTJBYGYT-raagX6xL0A:9 a=1fhp2MxaeJtTNGEnv6mo:22 a=Z5ABNNGmrOfJ6cZ5bIyy:22 a=UDnyf2zBuKT2w-IlGP_r:22 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Thu, 6 Feb 2020 14:45:47 +0100 Message-Id: <20200206134549.29792-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 X-CMAE-Envelope: MS4wfFwA78vCgbOiv+iR/WA4X1DNwiECwNw0srYfdh4cZV8D3Zqk5869y/oqNbVEM1oareqYp5PA6vcKcI9G9UD9xUB4RFdWU20RR0puyFJZ3W9gSeg4mGhQ KYvrpSF9clzo1CQkPqvAGobWMlw76ICal6Zj6LLekL3oirAuRKbFOGNW Subject: [FFmpeg-devel] [PATCH 1/3] avcodec/cavsdsp: Fix invalid left shifts in cavs_idct8_add_c() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: left shift of negative value -107 Fixes: 20398/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_CAVS_fuzzer-5725389278412800 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/cavsdsp.c | 52 ++++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/libavcodec/cavsdsp.c b/libavcodec/cavsdsp.c index 90a67e910c..27413b9ee2 100644 --- a/libavcodec/cavsdsp.c +++ b/libavcodec/cavsdsp.c @@ -201,20 +201,20 @@ static void cavs_idct8_add_c(uint8_t *dst, int16_t *block, ptrdiff_t stride) src[0][0] += 8; for( i = 0; i < 8; i++ ) { - const int a0 = 3*src[i][1] - (src[i][7]<<1); - const int a1 = 3*src[i][3] + (src[i][5]<<1); - const int a2 = (src[i][3]<<1) - 3*src[i][5]; - const int a3 = (src[i][1]<<1) + 3*src[i][7]; + const int a0 = 3*src[i][1] - (src[i][7]*2); + const int a1 = 3*src[i][3] + (src[i][5]*2); + const int a2 = (src[i][3]*2) - 3*src[i][5]; + const int a3 = (src[i][1]*2) + 3*src[i][7]; - const int b4 = ((a0 + a1 + a3)<<1) + a1; - const int b5 = ((a0 - a1 + a2)<<1) + a0; - const int b6 = ((a3 - a2 - a1)<<1) + a3; - const int b7 = ((a0 - a2 - a3)<<1) - a2; + const int b4 = ((a0 + a1 + a3)*2) + a1; + const int b5 = ((a0 - a1 + a2)*2) + a0; + const int b6 = ((a3 - a2 - a1)*2) + a3; + const int b7 = ((a0 - a2 - a3)*2) - a2; - const int a7 = (src[i][2]<<2) - 10*src[i][6]; - const int a6 = (src[i][6]<<2) + 10*src[i][2]; - const int a5 = ((src[i][0] - src[i][4]) << 3) + 4; - const int a4 = ((src[i][0] + src[i][4]) << 3) + 4; + const int a7 = (src[i][2]*4) - 10*src[i][6]; + const int a6 = (src[i][6]*4) + 10*src[i][2]; + const int a5 = ((src[i][0] - src[i][4]) * 8) + 4; + const int a4 = ((src[i][0] + src[i][4]) * 8) + 4; const int b0 = a4 + a6; const int b1 = a5 + a7; @@ -231,20 +231,20 @@ static void cavs_idct8_add_c(uint8_t *dst, int16_t *block, ptrdiff_t stride) src[i][7] = (b0 - b4) >> 3; } for( i = 0; i < 8; i++ ) { - const int a0 = 3*src[1][i] - (src[7][i]<<1); - const int a1 = 3*src[3][i] + (src[5][i]<<1); - const int a2 = (src[3][i]<<1) - 3*src[5][i]; - const int a3 = (src[1][i]<<1) + 3*src[7][i]; - - const int b4 = ((a0 + a1 + a3)<<1) + a1; - const int b5 = ((a0 - a1 + a2)<<1) + a0; - const int b6 = ((a3 - a2 - a1)<<1) + a3; - const int b7 = ((a0 - a2 - a3)<<1) - a2; - - const int a7 = (src[2][i]<<2) - 10*src[6][i]; - const int a6 = (src[6][i]<<2) + 10*src[2][i]; - const int a5 = (src[0][i] - src[4][i]) << 3; - const int a4 = (src[0][i] + src[4][i]) << 3; + const int a0 = 3*src[1][i] - (src[7][i]*2); + const int a1 = 3*src[3][i] + (src[5][i]*2); + const int a2 = (src[3][i]*2) - 3*src[5][i]; + const int a3 = (src[1][i]*2) + 3*src[7][i]; + + const int b4 = ((a0 + a1 + a3)*2) + a1; + const int b5 = ((a0 - a1 + a2)*2) + a0; + const int b6 = ((a3 - a2 - a1)*2) + a3; + const int b7 = ((a0 - a2 - a3)*2) - a2; + + const int a7 = (src[2][i]*4) - 10*src[6][i]; + const int a6 = (src[6][i]*4) + 10*src[2][i]; + const int a5 = (src[0][i] - src[4][i]) * 8; + const int a4 = (src[0][i] + src[4][i]) * 8; const int b0 = a4 + a6; const int b1 = a5 + a7;