From patchwork Sun Jun 21 22:36:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 20545 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 9568844AC6A for ; Mon, 22 Jun 2020 01:37:27 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7751068B63E; Mon, 22 Jun 2020 01:37:27 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe02-3.mx.upcmail.net (vie01a-dmta-pe02-3.mx.upcmail.net [62.179.121.159]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9C51C68B502 for ; Mon, 22 Jun 2020 01:37:21 +0300 (EEST) Received: from [172.31.216.235] (helo=vie01a-pemc-psmtp-pe12.mail.upcmail.net) by vie01a-dmta-pe02.mx.upcmail.net with esmtp (Exim 4.92) (envelope-from ) id 1jn8aX-0009Ut-0U for ffmpeg-devel@ffmpeg.org; Mon, 22 Jun 2020 00:37:21 +0200 Received: from localhost ([213.47.68.29]) by vie01a-pemc-psmtp-pe12.mail.upcmail.net with ESMTP id n8ZZj5ULS6Jy6n8ZZjv28F; Mon, 22 Jun 2020 00:36:21 +0200 X-Env-Mailfrom: michael@niedermayer.cc X-Env-Rcptto: ffmpeg-devel@ffmpeg.org X-SourceIP: 213.47.68.29 X-CNFS-Analysis: v=2.3 cv=GKl27dFK c=1 sm=1 tr=0 a=2hcxjKEKjp0CzLx6oWAm4g==:117 a=2hcxjKEKjp0CzLx6oWAm4g==:17 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=GEAsPZ9sns4A:10 a=ZZnuYtJkoWoA:10 a=nZOtpAppAAAA:20 a=j2wtWACOIyGdxEC8yAwA:9 a=1fhp2MxaeJtTNGEnv6mo:22 a=Z5ABNNGmrOfJ6cZ5bIyy:22 a=UDnyf2zBuKT2w-IlGP_r:22 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Mon, 22 Jun 2020 00:36:19 +0200 Message-Id: <20200621223619.1797-2-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200621223619.1797-1-michael@niedermayer.cc> References: <20200621223619.1797-1-michael@niedermayer.cc> X-CMAE-Envelope: MS4wfPygHAHrGkWQKsVj121MmOMVXNeu5hy7m6KlsI6wwm/W3B7JSM1880H5leChkJIGunWF4r9DoJj+lr+XwXPe0kzNylFUrbO8CQTLlhR9ok/buU8SuSUF m/RLXrtKTMuKIADhTzvH5as+H46rzgytxf4TP0QGVdhSycwqLEpXxTeU Subject: [FFmpeg-devel] [PATCH 2/2] avcodec/vp9dsp_template: Fix integer overflow in iadst8_1d() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: signed integer overflow: 998938090 + 1169275991 cannot be represented in type 'int' Fixes: 23411/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VP9_fuzzer-4644692330545152 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/vp9dsp_template.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/libavcodec/vp9dsp_template.c b/libavcodec/vp9dsp_template.c index c6944f5ce3..8d00e77d70 100644 --- a/libavcodec/vp9dsp_template.c +++ b/libavcodec/vp9dsp_template.c @@ -1260,20 +1260,20 @@ static av_always_inline void iadst8_1d(const dctcoef *in, ptrdiff_t stride, t6 = (t2a - t6a + (1 << 13)) >> 14; t7 = (t3a - t7a + (1 << 13)) >> 14; - t4a = 15137 * t4 + 6270 * t5; - t5a = 6270 * t4 - 15137 * t5; - t6a = 15137 * t7 - 6270 * t6; - t7a = 6270 * t7 + 15137 * t6; + t4a = 15137U * t4 + 6270U * t5; + t5a = 6270U * t4 - 15137U * t5; + t6a = 15137U * t7 - 6270U * t6; + t7a = 6270U * t7 + 15137U * t6; out[0] = t0 + t2; out[7] = -(t1 + t3); t2 = t0 - t2; t3 = t1 - t3; - out[1] = -((t4a + t6a + (1 << 13)) >> 14); - out[6] = (t5a + t7a + (1 << 13)) >> 14; - t6 = (t4a - t6a + (1 << 13)) >> 14; - t7 = (t5a - t7a + (1 << 13)) >> 14; + out[1] = -((dctint)((1U << 13) + t4a + t6a) >> 14); + out[6] = (dctint)((1U << 13) + t5a + t7a) >> 14; + t6 = (dctint)((1U << 13) + t4a - t6a) >> 14; + t7 = (dctint)((1U << 13) + t5a - t7a) >> 14; out[3] = -(((t2 + t3) * 11585 + (1 << 13)) >> 14); out[4] = ((t2 - t3) * 11585 + (1 << 13)) >> 14;