From patchwork Thu Aug 17 22:17:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 4734 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.46.211 with SMTP id u202csp85829vsu; Thu, 17 Aug 2017 15:18:06 -0700 (PDT) X-Received: by 10.223.199.135 with SMTP id l7mr3910167wrg.282.1503008285994; Thu, 17 Aug 2017 15:18:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1503008285; cv=none; d=google.com; s=arc-20160816; b=hjdLP3jDh0vQuuEpnoHGjnPBHar6G3PVRILdBLhC1g1nIKm92gLK60KhU+o8I1Wzuk Lx1v9JD2W/lST7OZupdYEHA0hfhL2YJK1XjpKiOyWq8GP9NudjLS/cMUYcgSGXBakvof q5SQkbDNtKJRDmz3UI430kZesoF8INPUIOHT9jyIsLost84MWuf9JWU5bLfXGgFejJ8a st0A/TwRYaBUo8Q4Fr3/GD0zO0nAsDGWsagSOoj3vNYmUgYY+ilIVL4dUMb8AK7F0FB4 bqqds5QGDoC64ddLhOwzvs5kDEaDFrIBO0mN36uNqnFrI8fXfuxlc2z6uS4fTWHNueEn tCew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:delivered-to :arc-authentication-results; bh=z43o507lkJimmMe4SEOluyo4N8h7qWAsW19f44MJO1Q=; b=XdMjwWnMoHC1xnpSXM7nIZeTiTNH6jTQXVwJ1006aIHzEIFXQlFTFfh+uiD8OQ9Hfj fhfDJeIPPV/QHUdb8HjS/t4648E5Hh2s8eOQPWgq/NmpVXR1eGcSCqc0So1jTg2YSj0z bjF6NEojgs+N6/MgIDYqQ4A0Hvkj9Hrt3CExblenydiPYF6Ns+/0zKh5Ij8uPOxq+t43 UohtFOuOuUZ3PsQEtuULLGmmhr4b+JoEN78vgUyloj0YmKBAGBJn3sIzenKmL+MbIZG/ 00qXcZdeHBBaMVhh6VrsvhduB5OnZRD9bf8rucNEqyT4cfeTHU1LTDHEz6gc9CrTnsLv UKoQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y10si35863wmd.19.2017.08.17.15.17.57; Thu, 17 Aug 2017 15:18:05 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A88D468971B; Fri, 18 Aug 2017 01:17:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-qmta-pe02-3.mx.upcmail.net (unknown [62.179.121.183]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D1620680A18 for ; Fri, 18 Aug 2017 01:17:41 +0300 (EEST) Received: from [172.31.218.33] (helo=vie01a-dmta-pe01-3.mx.upcmail.net) by vie01a-pqmta-pe02.mx.upcmail.net with esmtp (Exim 4.88) (envelope-from ) id 1diT6o-0001a2-QZ for ffmpeg-devel@ffmpeg.org; Fri, 18 Aug 2017 00:17:46 +0200 Received: from [172.31.216.43] (helo=vie01a-pemc-psmtp-pe01) by vie01a-dmta-pe01.mx.upcmail.net with esmtp (Exim 4.88) (envelope-from ) id 1diT6h-0003ns-OM for ffmpeg-devel@ffmpeg.org; Fri, 18 Aug 2017 00:17:39 +0200 Received: from localhost ([213.47.41.20]) by vie01a-pemc-psmtp-pe01 with SMTP @ mailcloud.upcmail.net id yNHc1v01a0S5wYM01NHdm2; Fri, 18 Aug 2017 00:17:37 +0200 X-SourceIP: 213.47.41.20 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Fri, 18 Aug 2017 00:17:33 +0200 Message-Id: <20170817221735.24154-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.14.1 Subject: [FFmpeg-devel] [PATCH 1/3] avcodec/fic: Fixes signed integer overflow X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: runtime error: signed integer overflow: 1037142357 + 1227025305 cannot be represented in type 'int' Fixes: 3024/clusterfuzz-testcase-minimized-5885660323905536 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/fic.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/libavcodec/fic.c b/libavcodec/fic.c index 46260ee281..d7ee370423 100644 --- a/libavcodec/fic.c +++ b/libavcodec/fic.c @@ -85,12 +85,12 @@ static const uint8_t fic_header[7] = { 0, 0, 1, 'F', 'I', 'C', 'V' }; static av_always_inline void fic_idct(int16_t *blk, int step, int shift, int rnd) { - const int t0 = 27246 * blk[3 * step] + 18405 * blk[5 * step]; - const int t1 = 27246 * blk[5 * step] - 18405 * blk[3 * step]; - const int t2 = 6393 * blk[7 * step] + 32139 * blk[1 * step]; - const int t3 = 6393 * blk[1 * step] - 32139 * blk[7 * step]; - const unsigned t4 = 5793U * (t2 + t0 + 0x800 >> 12); - const unsigned t5 = 5793U * (t3 + t1 + 0x800 >> 12); + const unsigned t0 = 27246 * blk[3 * step] + 18405 * blk[5 * step]; + const unsigned t1 = 27246 * blk[5 * step] - 18405 * blk[3 * step]; + const unsigned t2 = 6393 * blk[7 * step] + 32139 * blk[1 * step]; + const unsigned t3 = 6393 * blk[1 * step] - 32139 * blk[7 * step]; + const unsigned t4 = 5793U * ((int)(t2 + t0 + 0x800) >> 12); + const unsigned t5 = 5793U * ((int)(t3 + t1 + 0x800) >> 12); const unsigned t6 = t2 - t0; const unsigned t7 = t3 - t1; const unsigned t8 = 17734 * blk[2 * step] - 42813 * blk[6 * step];