From patchwork Sun Nov 5 20:20:06 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 5870 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.161.90 with SMTP id m26csp2195734jah; Sun, 5 Nov 2017 12:20:36 -0800 (PST) X-Google-Smtp-Source: ABhQp+TcF0TkEOX4HQaFenwKX2lttqU4M5yQ9jGsXIQaEPBK8tle8yr5cmZdibe6e7gzECx20a2G X-Received: by 10.223.131.226 with SMTP id 89mr11441065wre.227.1509913236804; Sun, 05 Nov 2017 12:20:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1509913236; cv=none; d=google.com; s=arc-20160816; b=W65v8cG1DfB5ETTDi+GH/j0Df9kmoq6qBl2wwCBrbQ+ljerMMcl8ZGJrPWeG52i4ae cl5X1p/+HiCJgSBVNf+z9PkqJRcub+FYScmIupA6SV6pDWoNuT4RUgEbQR0gTCW6HXl9 d6aZrmpZJUkT5K3laqUipGDoqixIpldaIX+K2ErvHkhC7iQgh1QI54oqSLbAFqwXJ5wk Jiz0W/+NlxuI8rC4SCgSTuUFHmdXcTsR/ScvpRHnTQYLegRheI8LwKztaLkFeb8Fo3L/ hzbeVQSR0lFdzNYiHCOoNylfbYJBkZR0esiS79U2TXJEa/t6fWRi/BY+p+iaqrdUmq7E D+xA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:delivered-to:arc-authentication-results; bh=26wfhEQ9H+AsnsKUYgjAWVJkNPyDM0wUT5rFGI8Vg/g=; b=pL9ja1O1zpO1v5zsy8U1Qi9ajgfwud1lkJDEVFNJDs64QmWF7QwB6//SE11K/nBiXz b7VH8ThO2IL6i7HcjbObkDtloPF8pjGBS3mZROPxWYg8BjYyXAvUFQ1oCtig/iZJNggy x5eHGBYeuSnAJ1c8RbNBtW1R90fmYc1gbNugDdWDJKMpnGJgSLXrU+sh6CDPAv83Xl6b PrdYmaUc20Z4tReh6g+y0fyM/L0N88q15EKtv40TAJftGdqy2L9rUjxHWjtMgLFE1ziH NqSA3ySDZF6Weumt1zZ5IqdyZVOL7TB9uKloxfzLbN6DLAateUvyO6+NP88gisoxvPP7 Vaew== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e18si10052319wrc.496.2017.11.05.12.20.36; Sun, 05 Nov 2017 12:20:36 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CEB24689FC0; Sun, 5 Nov 2017 22:20:10 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe04-2.mx.upcmail.net (vie01a-dmta-pe04-2.mx.upcmail.net [62.179.121.164]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BF275689F52 for ; Sun, 5 Nov 2017 22:20:04 +0200 (EET) Received: from [172.31.216.43] (helo=vie01a-pemc-psmtp-pe01) by vie01a-dmta-pe04.mx.upcmail.net with esmtp (Exim 4.88) (envelope-from ) id 1eBROy-0005f4-C0 for ffmpeg-devel@ffmpeg.org; Sun, 05 Nov 2017 21:20:16 +0100 Received: from localhost ([213.47.41.20]) by vie01a-pemc-psmtp-pe01 with SMTP @ mailcloud.upcmail.net id WLLA1w00b0S5wYM01LLBo4; Sun, 05 Nov 2017 21:20:11 +0100 X-SourceIP: 213.47.41.20 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Sun, 5 Nov 2017 21:20:06 +0100 Message-Id: <20171105202008.29966-2-michael@niedermayer.cc> X-Mailer: git-send-email 2.15.0 In-Reply-To: <20171105202008.29966-1-michael@niedermayer.cc> References: <20171105202008.29966-1-michael@niedermayer.cc> Subject: [FFmpeg-devel] [PATCH 2/4] avcodec/mdct_*: Fix integer overflow in addition in RESCALE() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: runtime error: signed integer overflow: 1219998458 - -1469874012 cannot be represented in type 'int' Fixes: 3443/clusterfuzz-testcase-minimized-5369987105554432 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/mdct_fixed.c | 8 ++++---- libavcodec/mdct_template.c | 14 +++++++------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/libavcodec/mdct_fixed.c b/libavcodec/mdct_fixed.c index a32cb00ca0..aabf0c88f8 100644 --- a/libavcodec/mdct_fixed.c +++ b/libavcodec/mdct_fixed.c @@ -39,13 +39,13 @@ void ff_mdct_calcw_c(FFTContext *s, FFTDouble *out, const FFTSample *input) /* pre rotation */ for(i=0;i> 6) +# define RSCALE(x, y) ((int)((x) + (unsigned)(y) + 32) >> 6) #else /* FFT_FIXED_32 */ -# define RSCALE(x) ((x) >> 1) +# define RSCALE(x, y) ((int)((x) + (unsigned)(y)) >> 1) #endif /* FFT_FIXED_32 */ #endif @@ -181,13 +181,13 @@ void ff_mdct_calc_c(FFTContext *s, FFTSample *out, const FFTSample *input) /* pre rotation */ for(i=0;i