From patchwork Wed May 10 12:50:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 3639 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.3.129 with SMTP id 123csp194888vsd; Wed, 10 May 2017 05:50:55 -0700 (PDT) X-Received: by 10.28.236.89 with SMTP id k86mr974283wmh.69.1494420655044; Wed, 10 May 2017 05:50:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1494420654; cv=none; d=google.com; s=arc-20160816; b=YkKHWKzx6hc4z/EUzN88qJpha6iun7x76HUqjcCxwzaS8X5d96tDsiuM/1GqrCMPoP g6J8e4SvCMRjzw2BMH75ph2UCTK6eMHfsOQjBImuXBRqdRQQG0IF/Q5LR0Lss7j38uVe i7WW8kf24HKFIfNCTlc+wcZ8vmzjvhuQFC7TMcrAcWaFd3ciCaMPzFw4pq9F7FXP3KzP envwTi05lTHlqSJEkLV9AnbCVTedPbJ1c3aBruAn/65eO8d7uArIIUlV/rXAMq58Pujy pKNzeUkDURQlGT1eBmJCtS8+PYG8OAzVnSyivCmAY5EUEnQxupZCu4bSghXeahhHqyKX hjEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:delivered-to :arc-authentication-results; bh=+LJnbGwf1AS/71KTEBH7apEZ6EdFiLzAuY0eVh8p5yY=; b=0dz0u3kxRdFerOZ7CLDN6IDHPV18JRTijQN1L0zYxv3PzSg6bBQDhlSCS8Eb6jycgV KJG10g+fXENBqDZaV97neD3y5SykEYTA0o5LRxuQF9ev7Yh8zDyu+n47CwEuJVYUHVo0 yKqN9z1+1A+rIh8vFjeo6pqNjLGd3E27rHHxLmi9vzpG4AivV+gEJn/ocXLncODgt/5a Q2I22+0QDWyUgJXFXbl34WkyQr6sFMqspRRafPNxanDirrCIyGMPYuGg4M3FOJzxyXN1 URRKUas6hGtYjVScxm90ZUex4VGKGYByxPv9WsS4u9T6ByNE1PzDiZyC31gQDeTOkyZZ bgew== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x66si3428085wme.98.2017.05.10.05.50.54; Wed, 10 May 2017 05:50:54 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5F875688372; Wed, 10 May 2017 15:50:44 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe05-1.mx.upcmail.net (vie01a-dmta-pe06-1.mx.upcmail.net [84.116.36.14]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E6E5668045D for ; Wed, 10 May 2017 15:50:37 +0300 (EEST) Received: from [172.31.216.43] (helo=vie01a-pemc-psmtp-pe01) by vie01a-dmta-pe06.mx.upcmail.net with esmtp (Exim 4.88) (envelope-from ) id 1d8R4m-0001fy-BX for ffmpeg-devel@ffmpeg.org; Wed, 10 May 2017 14:50:44 +0200 Received: from localhost ([213.47.41.20]) by vie01a-pemc-psmtp-pe01 with SMTP @ mailcloud.upcmail.net id Jcqh1v00B0S5wYM01cqi7S; Wed, 10 May 2017 14:50:42 +0200 X-SourceIP: 213.47.41.20 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Wed, 10 May 2017 14:50:40 +0200 Message-Id: <20170510125040.4960-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.11.0 Subject: [FFmpeg-devel] [PATCH] avcodec/vp8dsp: vp7_luma_dc_wht_c: Fix multiple runtime error: signed integer overflow: -1366381240 + -1262413604 cannot be represented in type 'int' X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: 1440/clusterfuzz-testcase-minimized-5785716111966208 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/vp8dsp.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/libavcodec/vp8dsp.c b/libavcodec/vp8dsp.c index 7d9cfa8278..fed5c67a90 100644 --- a/libavcodec/vp8dsp.c +++ b/libavcodec/vp8dsp.c @@ -53,7 +53,8 @@ static void name ## _idct_dc_add4y_c(uint8_t *dst, int16_t block[4][16], \ #if CONFIG_VP7_DECODER static void vp7_luma_dc_wht_c(int16_t block[4][4][16], int16_t dc[16]) { - int i, a1, b1, c1, d1; + int i; + unsigned a1, b1, c1, d1; int16_t tmp[16]; for (i = 0; i < 4; i++) { @@ -61,10 +62,10 @@ static void vp7_luma_dc_wht_c(int16_t block[4][4][16], int16_t dc[16]) b1 = (dc[i * 4 + 0] - dc[i * 4 + 2]) * 23170; c1 = dc[i * 4 + 1] * 12540 - dc[i * 4 + 3] * 30274; d1 = dc[i * 4 + 1] * 30274 + dc[i * 4 + 3] * 12540; - tmp[i * 4 + 0] = (a1 + d1) >> 14; - tmp[i * 4 + 3] = (a1 - d1) >> 14; - tmp[i * 4 + 1] = (b1 + c1) >> 14; - tmp[i * 4 + 2] = (b1 - c1) >> 14; + tmp[i * 4 + 0] = (int)(a1 + d1) >> 14; + tmp[i * 4 + 3] = (int)(a1 - d1) >> 14; + tmp[i * 4 + 1] = (int)(b1 + c1) >> 14; + tmp[i * 4 + 2] = (int)(b1 - c1) >> 14; } for (i = 0; i < 4; i++) { @@ -73,10 +74,10 @@ static void vp7_luma_dc_wht_c(int16_t block[4][4][16], int16_t dc[16]) c1 = tmp[i + 4] * 12540 - tmp[i + 12] * 30274; d1 = tmp[i + 4] * 30274 + tmp[i + 12] * 12540; AV_ZERO64(dc + i * 4); - block[0][i][0] = (a1 + d1 + 0x20000) >> 18; - block[3][i][0] = (a1 - d1 + 0x20000) >> 18; - block[1][i][0] = (b1 + c1 + 0x20000) >> 18; - block[2][i][0] = (b1 - c1 + 0x20000) >> 18; + block[0][i][0] = (int)(a1 + d1 + 0x20000) >> 18; + block[3][i][0] = (int)(a1 - d1 + 0x20000) >> 18; + block[1][i][0] = (int)(b1 + c1 + 0x20000) >> 18; + block[2][i][0] = (int)(b1 - c1 + 0x20000) >> 18; } }