From patchwork Sat Jul 28 12:32:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 9832 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:104:0:0:0:0:0 with SMTP id c4-v6csp1886122jad; Sat, 28 Jul 2018 05:33:38 -0700 (PDT) X-Google-Smtp-Source: AAOMgpf+umK0GU07oWPeA6L/GFis5ddQi+LbVcc7FJUWFVhExIw5HTZ7ZWGuOTRdzUryXv6E6haC X-Received: by 2002:adf:a0f3:: with SMTP id n48-v6mr9409492wrn.23.1532781218876; Sat, 28 Jul 2018 05:33:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532781218; cv=none; d=google.com; s=arc-20160816; b=wWS9mO0oAdVXZIaS3oqsXwnJX3PG+kUj7B0gEEWseef408f7rIvE+2R8LQyxQtMvsR ZDV4s5eKTWtBuE6ghViSpo/YAC7fzKhME/OyocAhUyaa1gIbWivxSHIbsL++APu2Ty6m vrBYM7YwfD2E/WBySUgdq7BjAi5Qrnz/R9sAmfW+lilHy8thxEjVPV7UHvPe7xp1DxIE V+qm2kIx7MWAX1h3ojaIobA/XqeXCZyzuUH0HCZUFPjv8Z7g8YvDDzmZAGaoitdl0Cr8 8M9VpWuOv5gl0SadSAuaOavMM1JXrAjnsJ1hXd5kFVxKtAMHPRn1b5RIYIG37kXk9TVB jhKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:delivered-to:arc-authentication-results; bh=Qi3c4rOSoEHDxF0TWJwwlSwwV6+Pq/JgX2Sb5fnY79s=; b=yhgufjBdinbPAxFHKlXwNkyHznNB9r8C2NnI/Jj2xLlLYE5DZ7YkbObWsERYeUIe2I 5VSq3ZH5z3sqmcf8Cz7Wh1LxxOawri2Wo4qisehP4pHkaceXivGyf/MmoOrsbmLnWEuh 7yp0Y2lzm78BluDmzBGcN8YfJtFrbScNNxhflRpkhT8q0cVZnCWe8JiI8oCnHi0lxvAP XEQldoF3pVT38PxIgDVPhZGNOAZLA0MNa8olYCm5KIckyoWtUi1VxCPKLBO1pJ12spUE DxAdWqmglMgLwXEP08wKRKz//euzmKXo9/qpC55T3XRBal/JNK2wiQKaCsttT+73QCnj r/AQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t6-v6si5946283wrm.351.2018.07.28.05.33.38; Sat, 28 Jul 2018 05:33:38 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 74BB868A590; Sat, 28 Jul 2018 15:33:01 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-qmta-pe01-3.mx.upcmail.net (vie01a-qmta-pe01-3.mx.upcmail.net [62.179.121.180]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 31A2A68A483 for ; Sat, 28 Jul 2018 15:32:55 +0300 (EEST) Received: from [172.31.218.50] (helo=vie01a-dmta-pe07-2.mx.upcmail.net) by vie01a-pqmta-pe01.mx.upcmail.net with esmtp (Exim 4.88) (envelope-from ) id 1fjOPG-0000MW-MW for ffmpeg-devel@ffmpeg.org; Sat, 28 Jul 2018 14:33:10 +0200 Received: from [172.31.216.43] (helo=vie01a-pemc-psmtp-pe01) by vie01a-dmta-pe07.mx.upcmail.net with esmtp (Exim 4.88) (envelope-from ) id 1fjOPB-0000Jz-4h for ffmpeg-devel@ffmpeg.org; Sat, 28 Jul 2018 14:33:05 +0200 Received: from localhost ([213.47.41.20]) by vie01a-pemc-psmtp-pe01 with SMTP @ mailcloud.upcmail.net id GCYx1y01K0S5wYM01CYz2U; Sat, 28 Jul 2018 14:32:59 +0200 X-SourceIP: 213.47.41.20 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Sat, 28 Jul 2018 14:32:31 +0200 Message-Id: <20180728123232.6191-3-michael@niedermayer.cc> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180728123232.6191-1-michael@niedermayer.cc> References: <20180728123232.6191-1-michael@niedermayer.cc> Subject: [FFmpeg-devel] [PATCH 3/4] avcodec/simple_idct_template: Fix several integer overflows X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: simple_idct_template.c:184:30: runtime error: signed integer overflow: -1065517056 - 1392182838 cannot be represented in type 'int' Fixes: simple_idct_template.c:269:21: runtime error: signed integer overflow: 16384 * 259254 cannot be represented in type 'int' Fixes: simple_idct_template.c:164:17: runtime error: signed integer overflow: 21407 * 210162 cannot be represented in type 'int' Fixes: simple_idct_template.c:167:17: runtime error: signed integer overflow: 21407 * 210162 cannot be represented in type 'int' Fixes: simple_idct_template.c:169:19: runtime error: signed integer overflow: 22725 * 259190 cannot be represented in type 'int' Fixes: simple_idct_template.c:171:19: runtime error: signed integer overflow: 19265 * 259190 cannot be represented in type 'int' Fixes: simple_idct_template.c:173:19: runtime error: signed integer overflow: 12873 * 259190 cannot be represented in type 'int' Fixes: simple_idct_template.c:183:28: runtime error: signed integer overflow: 1860878336 + 585177665 cannot be represented in type 'int' Fixes: simple_idct_template.c:159:17: runtime error: signed integer overflow: 16384 * 189520 cannot be represented in type 'int' Fixes: simple_idct_template.c:170:22: runtime error: signed integer overflow: 19265 * 130147 cannot be represented in type 'int' Fixes: simple_idct_template.c:174:23: runtime error: signed integer overflow: -22725 * 130147 cannot be represented in type 'int' Fixes: simple_idct_template.c:183:20: runtime error: signed integer overflow: 16384 * -175206 cannot be represented in type 'int' Fixes: simple_idct_template.c:184:22: runtime error: signed integer overflow: -16384 * -175206 cannot be represented in type 'int' Fixes: simple_idct_template.c:185:22: runtime error: signed integer overflow: -16384 * -175206 cannot be represented in type 'int' Fixes: simple_idct_template.c:186:20: runtime error: signed integer overflow: 16384 * -175206 cannot be represented in type 'int' Fixes: simple_idct_template.c:195:26: runtime error: signed integer overflow: 19265 * 150747 cannot be represented in type 'int' Fixes: simple_idct_template.c:198:27: runtime error: signed integer overflow: -22725 * 150747 cannot be represented in type 'int' Fixes: simple_idct_template.c:184:37: runtime error: signed integer overflow: 21407 * -171941 cannot be represented in type 'int' Fixes: simple_idct_template.c:185:37: runtime error: signed integer overflow: 21407 * -171941 cannot be represented in type 'int' Fixes: simple_idct_template.c:192:27: runtime error: signed integer overflow: -12873 * 206341 cannot be represented in type 'int' Fixes: simple_idct_template.c:185:30: runtime error: signed integer overflow: 1196441600 + 1703756981 cannot be represented in type 'int' Fixes: simple_idct_template.c:176:23: runtime error: signed integer overflow: -12873 * 168461 cannot be represented in type 'int' Fixes: simple_idct_template.c:191:27: runtime error: signed integer overflow: -22725 * -140062 cannot be represented in type 'int' Fixes: simple_idct_template.c:197:26: runtime error: signed integer overflow: 19265 * -140062 cannot be represented in type 'int' Fixes: simple_idct_template.c:183:34: runtime error: signed integer overflow: 8867 * -243046 cannot be represented in type 'int' Fixes: simple_idct_template.c:186:34: runtime error: signed integer overflow: 8867 * -243046 cannot be represented in type 'int' Fixes: simple_idct_template.c:186:28: runtime error: signed integer overflow: -816234496 - 2139878414 cannot be represented in type 'int' Fixes: simple_idct_template.c:188:26: runtime error: signed integer overflow: 12873 * -239872 cannot be represented in type 'int' Fixes: simple_idct_template.c:165:16: runtime error: signed integer overflow: 8867 * -260084 cannot be represented in type 'int' Fixes: simple_idct_template.c:166:16: runtime error: signed integer overflow: 8867 * -260084 cannot be represented in type 'int' Fixes: 9135/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MPEG4_fuzzer-6324422955761664 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/simple_idct_template.c | 48 +++++++++++++++---------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/libavcodec/simple_idct_template.c b/libavcodec/simple_idct_template.c index 904263fc71..35c31321c6 100644 --- a/libavcodec/simple_idct_template.c +++ b/libavcodec/simple_idct_template.c @@ -101,8 +101,8 @@ #define DC_SHIFT -1 # endif -#define MUL(a, b) ((a) * (b)) -#define MAC(a, b, c) ((a) += (b) * (c)) +#define MUL(a, b) ((int)((SUINT)(a) * (b))) +#define MAC(a, b, c) ((a) += (SUINT)(b) * (c)) #else @@ -156,15 +156,15 @@ static inline void FUNC6(idctRowCondDC)(idctin *row, int extra_shift) #endif #endif - a0 = (W4 * row[0]) + (1 << (ROW_SHIFT + extra_shift - 1)); + a0 = ((SUINT)W4 * row[0]) + (1 << (ROW_SHIFT + extra_shift - 1)); a1 = a0; a2 = a0; a3 = a0; - a0 += W2 * row[2]; - a1 += W6 * row[2]; - a2 -= W6 * row[2]; - a3 -= W2 * row[2]; + a0 += (SUINT)W2 * row[2]; + a1 += (SUINT)W6 * row[2]; + a2 -= (SUINT)W6 * row[2]; + a3 -= (SUINT)W2 * row[2]; b0 = MUL(W1, row[1]); MAC(b0, W3, row[3]); @@ -180,10 +180,10 @@ static inline void FUNC6(idctRowCondDC)(idctin *row, int extra_shift) #else if (AV_RN64A(row + 4)) { #endif - a0 += W4*row[4] + W6*row[6]; - a1 += - W4*row[4] - W2*row[6]; - a2 += - W4*row[4] + W2*row[6]; - a3 += W4*row[4] - W6*row[6]; + a0 += (SUINT) W4*row[4] + (SUINT)W6*row[6]; + a1 += (SUINT)- W4*row[4] - (SUINT)W2*row[6]; + a2 += (SUINT)- W4*row[4] + (SUINT)W2*row[6]; + a3 += (SUINT) W4*row[4] - (SUINT)W6*row[6]; MAC(b0, W5, row[5]); MAC(b0, W7, row[7]); @@ -209,15 +209,15 @@ static inline void FUNC6(idctRowCondDC)(idctin *row, int extra_shift) } #define IDCT_COLS do { \ - a0 = W4 * (col[8*0] + ((1<<(COL_SHIFT-1))/W4)); \ + a0 = (SUINT)W4 * (col[8*0] + ((1<<(COL_SHIFT-1))/W4)); \ a1 = a0; \ a2 = a0; \ a3 = a0; \ \ - a0 += W2*col[8*2]; \ - a1 += W6*col[8*2]; \ - a2 += -W6*col[8*2]; \ - a3 += -W2*col[8*2]; \ + a0 += (SUINT) W2*col[8*2]; \ + a1 += (SUINT) W6*col[8*2]; \ + a2 += (SUINT)-W6*col[8*2]; \ + a3 += (SUINT)-W2*col[8*2]; \ \ b0 = MUL(W1, col[8*1]); \ b1 = MUL(W3, col[8*1]); \ @@ -230,10 +230,10 @@ static inline void FUNC6(idctRowCondDC)(idctin *row, int extra_shift) MAC(b3, -W5, col[8*3]); \ \ if (col[8*4]) { \ - a0 += W4*col[8*4]; \ - a1 += -W4*col[8*4]; \ - a2 += -W4*col[8*4]; \ - a3 += W4*col[8*4]; \ + a0 += (SUINT) W4*col[8*4]; \ + a1 += (SUINT)-W4*col[8*4]; \ + a2 += (SUINT)-W4*col[8*4]; \ + a3 += (SUINT) W4*col[8*4]; \ } \ \ if (col[8*5]) { \ @@ -244,10 +244,10 @@ static inline void FUNC6(idctRowCondDC)(idctin *row, int extra_shift) } \ \ if (col[8*6]) { \ - a0 += W6*col[8*6]; \ - a1 += -W2*col[8*6]; \ - a2 += W2*col[8*6]; \ - a3 += -W6*col[8*6]; \ + a0 += (SUINT) W6*col[8*6]; \ + a1 += (SUINT)-W2*col[8*6]; \ + a2 += (SUINT) W2*col[8*6]; \ + a3 += (SUINT)-W6*col[8*6]; \ } \ \ if (col[8*7]) { \