From patchwork Sat May 6 20:09:59 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 3596 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.3.129 with SMTP id 123csp222340vsd; Sat, 6 May 2017 13:10:16 -0700 (PDT) X-Received: by 10.223.157.29 with SMTP id k29mr36891661wre.156.1494101416862; Sat, 06 May 2017 13:10:16 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id i16si11024913wra.111.2017.05.06.13.10.16; Sat, 06 May 2017 13:10:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E2C59680877; Sat, 6 May 2017 23:10:07 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe05-1.mx.upcmail.net (vie01a-dmta-pe08-1.mx.upcmail.net [84.116.36.20]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E75D66805EB for ; Sat, 6 May 2017 23:10:01 +0300 (EEST) Received: from [172.31.216.43] (helo=vie01a-pemc-psmtp-pe01) by vie01a-dmta-pe08.mx.upcmail.net with esmtp (Exim 4.88) (envelope-from ) id 1d761m-000634-Hv for ffmpeg-devel@ffmpeg.org; Sat, 06 May 2017 22:10:06 +0200 Received: from localhost ([213.47.41.20]) by vie01a-pemc-psmtp-pe01 with SMTP @ mailcloud.upcmail.net id H8A01v00Y0S5wYM018A1g5; Sat, 06 May 2017 22:10:01 +0200 X-SourceIP: 213.47.41.20 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Sat, 6 May 2017 22:09:59 +0200 Message-Id: <20170506200959.30688-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.11.0 Subject: [FFmpeg-devel] [PATCH] avcodec/aacdec_fixed: Fix various integer overflows X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: 1377/clusterfuzz-testcase-minimized-5487049807233024 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/aacdec_fixed.c | 2 +- libavcodec/aacdec_template.c | 4 ++-- libavcodec/sbrdsp_fixed.c | 28 +++++++++++++++------------- 3 files changed, 18 insertions(+), 16 deletions(-) diff --git a/libavcodec/aacdec_fixed.c b/libavcodec/aacdec_fixed.c index acb8178337..e3c68a9767 100644 --- a/libavcodec/aacdec_fixed.c +++ b/libavcodec/aacdec_fixed.c @@ -180,7 +180,7 @@ static void subband_scale(int *dst, int *src, int scale, int offset, int len) } else { s = s + 32; - round = 1 << (s-1); + round = 1U << (s-1); for (i=0; i> s); dst[i] = out * ssign; diff --git a/libavcodec/aacdec_template.c b/libavcodec/aacdec_template.c index 98a3240597..ae9baeee01 100644 --- a/libavcodec/aacdec_template.c +++ b/libavcodec/aacdec_template.c @@ -2792,9 +2792,9 @@ static void spectral_to_sample(AACContext *ac, int samples) int j; /* preparation for resampler */ for(j = 0; jch[0].ret[j] = (int32_t)av_clipl_int32((int64_t)che->ch[0].ret[j]<<7)+0x8000; + che->ch[0].ret[j] = (int32_t)av_clip64((int64_t)che->ch[0].ret[j]<<7, INT32_MIN, INT32_MAX-0x8000)+0x8000; if(type == TYPE_CPE) - che->ch[1].ret[j] = (int32_t)av_clipl_int32((int64_t)che->ch[1].ret[j]<<7)+0x8000; + che->ch[1].ret[j] = (int32_t)av_clip64((int64_t)che->ch[1].ret[j]<<7, INT32_MIN, INT32_MAX-0x8000)+0x8000; } } #endif /* USE_FIXED */ diff --git a/libavcodec/sbrdsp_fixed.c b/libavcodec/sbrdsp_fixed.c index f4e3de0c71..fb9aba4e8d 100644 --- a/libavcodec/sbrdsp_fixed.c +++ b/libavcodec/sbrdsp_fixed.c @@ -34,8 +34,9 @@ static SoftFloat sbr_sum_square_c(int (*x)[2], int n) { SoftFloat ret; - int64_t accu = 0; - int i, nz, round; + uint64_t accu = 0, round; + int i, nz; + unsigned u; for (i = 0; i < n; i += 2) { // Larger values are inavlid and could cause overflows of accu. @@ -49,22 +50,22 @@ static SoftFloat sbr_sum_square_c(int (*x)[2], int n) accu += (int64_t)x[i + 1][1] * x[i + 1][1]; } - i = (int)(accu >> 32); - if (i == 0) { + u = accu >> 32; + if (u == 0) { nz = 1; } else { - nz = 0; - while (FFABS(i) < 0x40000000) { - i <<= 1; + nz = -1; + while (u < 0x80000000U) { + u <<= 1; nz++; } nz = 32 - nz; } - round = 1 << (nz-1); - i = (int)((accu + round) >> nz); - i >>= 1; - ret = av_int2sf(i, 15 - nz); + round = 1ULL << (nz-1); + u = ((accu + round) >> nz); + u >>= 1; + ret = av_int2sf(u, 15 - nz); return ret; } @@ -107,7 +108,8 @@ static void sbr_qmf_deint_neg_c(int *v, const int *src) static av_always_inline SoftFloat autocorr_calc(int64_t accu) { - int nz, mant, expo, round; + int nz, mant, expo; + unsigned round; int i = (int)(accu >> 32); if (i == 0) { nz = 1; @@ -120,7 +122,7 @@ static av_always_inline SoftFloat autocorr_calc(int64_t accu) nz = 32-nz; } - round = 1 << (nz-1); + round = 1U << (nz-1); mant = (int)((accu + round) >> nz); mant = (mant + 0x40)>>7; mant <<= 6;