From patchwork Sun May 24 22:38:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 19836 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 86A8944A62D for ; Mon, 25 May 2020 01:39:52 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7622268807A; Mon, 25 May 2020 01:39:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe05-1.mx.upcmail.net (vie01a-dmta-pe05-1.mx.upcmail.net [84.116.36.11]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4B75F681871 for ; Mon, 25 May 2020 01:39:45 +0300 (EEST) Received: from [172.31.216.235] (helo=vie01a-pemc-psmtp-pe12.mail.upcmail.net) by vie01a-dmta-pe05.mx.upcmail.net with esmtp (Exim 4.92) (envelope-from ) id 1jczHU-00006z-2Z for ffmpeg-devel@ffmpeg.org; Mon, 25 May 2020 00:39:44 +0200 Received: from localhost ([213.47.68.29]) by vie01a-pemc-psmtp-pe12.mail.upcmail.net with ESMTP id czGVjavez6Jy6czGVjjwJL; Mon, 25 May 2020 00:38:43 +0200 X-Env-Mailfrom: michael@niedermayer.cc X-Env-Rcptto: ffmpeg-devel@ffmpeg.org X-SourceIP: 213.47.68.29 X-CNFS-Analysis: v=2.3 cv=GKl27dFK c=1 sm=1 tr=0 a=2hcxjKEKjp0CzLx6oWAm4g==:117 a=2hcxjKEKjp0CzLx6oWAm4g==:17 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=GEAsPZ9sns4A:10 a=ZZnuYtJkoWoA:10 a=nZOtpAppAAAA:20 a=8D7CuHAvu7ih5_Jnnv0A:9 a=1fhp2MxaeJtTNGEnv6mo:22 a=Z5ABNNGmrOfJ6cZ5bIyy:22 a=jd6J4Gguk5HxikPWLKER:22 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Mon, 25 May 2020 00:38:41 +0200 Message-Id: <20200524223843.8152-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 X-CMAE-Envelope: MS4wfMSyh/9f4egweDO8amVRYsgWLqLL04sidMWzNufdRHIEg6Cxq7WRh/7Pa8tcoQCX7502HtL4g2bwfR1+H8CooPVIcQwDzFmuP8GJ2L6l+niIOqyZ83iP MKe7Y5rilvi9r7K5jh2xXQF6MfIDmICur9SJdsEcX2kQF9UBzUKrOOc9 Subject: [FFmpeg-devel] [PATCH 1/3] avcodec/mv30: Fix multiple integer overflows in idct_1d() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: signed integer overflow: -4869937 * 473 cannot be represented in type 'int' Fixes: 21934/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MV30_fuzzer-5667289925156864 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/mv30.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/libavcodec/mv30.c b/libavcodec/mv30.c index 658f32c6ff..013a5753fe 100644 --- a/libavcodec/mv30.c +++ b/libavcodec/mv30.c @@ -107,7 +107,7 @@ static inline void idct_1d(int *blk, int step) const int t0 = blk[0 * step] + blk[4 * step]; const int t1 = blk[0 * step] - blk[4 * step]; const int t2 = blk[2 * step] + blk[6 * step]; - const int t3 = (((blk[2 * step] - blk[6 * step]) * 362) >> 8) - t2; + const int t3 = ((int)((blk[2 * step] - blk[6 * step]) * 362U) >> 8) - t2; const int t4 = t0 + t2; const int t5 = t0 - t2; const int t6 = t1 + t3; @@ -117,10 +117,10 @@ static inline void idct_1d(int *blk, int step) const int tA = blk[1 * step] + blk[7 * step]; const int tB = blk[1 * step] - blk[7 * step]; const int tC = t8 + tA; - const int tD = (tB + t9) * 473 >> 8; - const int tE = ((t9 * -669 >> 8) - tC) + tD; - const int tF = ((tA - t8) * 362 >> 8) - tE; - const int t10 = ((tB * 277 >> 8) - tD) + tF; + const int tD = (int)((tB + t9) * 473U) >> 8; + const int tE = (((int)(t9 * -669U) >> 8) - tC) + tD; + const int tF = ((int)((tA - t8) * 362U) >> 8) - tE; + const int t10 = (((int)(tB * 277U) >> 8) - tD) + tF; blk[0 * step] = t4 + tC; blk[1 * step] = t6 + tE; From patchwork Sun May 24 22:38:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 19838 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id D271844B868 for ; Mon, 25 May 2020 01:45:52 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A4AA468805D; Mon, 25 May 2020 01:45:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe06-2.mx.upcmail.net (vie01a-dmta-pe06-2.mx.upcmail.net [84.116.36.15]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A4D316809FF for ; Mon, 25 May 2020 01:45:46 +0300 (EEST) Received: from [172.31.216.235] (helo=vie01a-pemc-psmtp-pe12.mail.upcmail.net) by vie01a-dmta-pe06.mx.upcmail.net with esmtp (Exim 4.92) (envelope-from ) id 1jczHU-0007MP-2b for ffmpeg-devel@ffmpeg.org; Mon, 25 May 2020 00:39:44 +0200 Received: from localhost ([213.47.68.29]) by vie01a-pemc-psmtp-pe12.mail.upcmail.net with ESMTP id czGWjavgM6Jy6czGWjjwKN; Mon, 25 May 2020 00:38:44 +0200 X-Env-Mailfrom: michael@niedermayer.cc X-Env-Rcptto: ffmpeg-devel@ffmpeg.org X-SourceIP: 213.47.68.29 X-CNFS-Analysis: v=2.3 cv=GKl27dFK c=1 sm=1 tr=0 a=2hcxjKEKjp0CzLx6oWAm4g==:117 a=2hcxjKEKjp0CzLx6oWAm4g==:17 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=GEAsPZ9sns4A:10 a=ZZnuYtJkoWoA:10 a=nZOtpAppAAAA:20 a=U6FXAhv57UxCwAqtDFgA:9 a=1fhp2MxaeJtTNGEnv6mo:22 a=Z5ABNNGmrOfJ6cZ5bIyy:22 a=UDnyf2zBuKT2w-IlGP_r:22 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Mon, 25 May 2020 00:38:42 +0200 Message-Id: <20200524223843.8152-2-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200524223843.8152-1-michael@niedermayer.cc> References: <20200524223843.8152-1-michael@niedermayer.cc> X-CMAE-Envelope: MS4wfMSyh/9f4egweDO8amVRYsgWLqLL04sidMWzNufdRHIEg6Cxq7WRh/7Pa8tcoQCX7502HtL4g2bwfR1+H8CooPVIcQwDzFmuP8GJ2L6l+niIOqyZ83iP MKe7Y5rilvi9r7K5jh2xXQF6MfIDmICur9SJdsEcX2kQF9UBzUKrOOc9 Subject: [FFmpeg-devel] [PATCH 2/3] avcodec/wmalosslessdec: Fix integer overflow in mclms_predict() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: signed integer overflow: 2147483636 + 2048 cannot be represented in type 'int' Fixes: 22016/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMALOSSLESS_fuzzer-5109395618004992 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/wmalosslessdec.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavcodec/wmalosslessdec.c b/libavcodec/wmalosslessdec.c index 7ef7c93dbd..cfdd9e9a85 100644 --- a/libavcodec/wmalosslessdec.c +++ b/libavcodec/wmalosslessdec.c @@ -676,7 +676,7 @@ static void mclms_predict(WmallDecodeCtx *s, int icoef, int *pred) for (i = 0; i < ich; i++) pred[ich] += (uint32_t)s->channel_residues[i][icoef] * s->mclms_coeffs_cur[i + num_channels * ich]; - pred[ich] += (1 << s->mclms_scaling) >> 1; + pred[ich] += (1U << s->mclms_scaling) >> 1; pred[ich] >>= s->mclms_scaling; s->channel_residues[ich][icoef] += (unsigned)pred[ich]; } From patchwork Sun May 24 22:38:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 19835 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id D851A44A62D for ; Mon, 25 May 2020 01:39:51 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B30526808A0; Mon, 25 May 2020 01:39:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe01-3.mx.upcmail.net (vie01a-dmta-pe01-3.mx.upcmail.net [62.179.121.156]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 285896808A0 for ; Mon, 25 May 2020 01:39:45 +0300 (EEST) Received: from [172.31.216.235] (helo=vie01a-pemc-psmtp-pe12.mail.upcmail.net) by vie01a-dmta-pe01.mx.upcmail.net with esmtp (Exim 4.92) (envelope-from ) id 1jczHU-0001Ud-2b for ffmpeg-devel@ffmpeg.org; Mon, 25 May 2020 00:39:44 +0200 Received: from localhost ([213.47.68.29]) by vie01a-pemc-psmtp-pe12.mail.upcmail.net with ESMTP id czGWjavgb6Jy6czGWjjwKd; Mon, 25 May 2020 00:38:44 +0200 X-Env-Mailfrom: michael@niedermayer.cc X-Env-Rcptto: ffmpeg-devel@ffmpeg.org X-SourceIP: 213.47.68.29 X-CNFS-Analysis: v=2.3 cv=GKl27dFK c=1 sm=1 tr=0 a=2hcxjKEKjp0CzLx6oWAm4g==:117 a=2hcxjKEKjp0CzLx6oWAm4g==:17 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=GEAsPZ9sns4A:10 a=ZZnuYtJkoWoA:10 a=nZOtpAppAAAA:20 a=UJzbOOpI7e6nHFDyho8A:9 a=1fhp2MxaeJtTNGEnv6mo:22 a=Z5ABNNGmrOfJ6cZ5bIyy:22 a=QOGEsqRv6VhmHaoFNykA:22 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Mon, 25 May 2020 00:38:43 +0200 Message-Id: <20200524223843.8152-3-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200524223843.8152-1-michael@niedermayer.cc> References: <20200524223843.8152-1-michael@niedermayer.cc> X-CMAE-Envelope: MS4wfMSyh/9f4egweDO8amVRYsgWLqLL04sidMWzNufdRHIEg6Cxq7WRh/7Pa8tcoQCX7502HtL4g2bwfR1+H8CooPVIcQwDzFmuP8GJ2L6l+niIOqyZ83iP MKe7Y5rilvi9r7K5jh2xXQF6MfIDmICur9SJdsEcX2kQF9UBzUKrOOc9 Subject: [FFmpeg-devel] [PATCH 3/3] avcodec/vp9dsp_template: Fix integer overflow(s) in iadst16_1d() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: signed integer overflow: 1080285923 - -1130879337 cannot be represented in type 'int' Fixes: 22002/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VP9_fuzzer-6260237310099456 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libavcodec/vp9dsp_template.c | 152 +++++++++++++++++------------------ 1 file changed, 76 insertions(+), 76 deletions(-) diff --git a/libavcodec/vp9dsp_template.c b/libavcodec/vp9dsp_template.c index c3273dd726..c6944f5ce3 100644 --- a/libavcodec/vp9dsp_template.c +++ b/libavcodec/vp9dsp_template.c @@ -1378,48 +1378,48 @@ static av_always_inline void iadst16_1d(const dctcoef *in, ptrdiff_t stride, dctint t0a, t1a, t2a, t3a, t4a, t5a, t6a, t7a; dctint t8a, t9a, t10a, t11a, t12a, t13a, t14a, t15a; - t0 = IN(15) * 16364 + IN(0) * 804; - t1 = IN(15) * 804 - IN(0) * 16364; - t2 = IN(13) * 15893 + IN(2) * 3981; - t3 = IN(13) * 3981 - IN(2) * 15893; - t4 = IN(11) * 14811 + IN(4) * 7005; - t5 = IN(11) * 7005 - IN(4) * 14811; - t6 = IN(9) * 13160 + IN(6) * 9760; - t7 = IN(9) * 9760 - IN(6) * 13160; - t8 = IN(7) * 11003 + IN(8) * 12140; - t9 = IN(7) * 12140 - IN(8) * 11003; - t10 = IN(5) * 8423 + IN(10) * 14053; - t11 = IN(5) * 14053 - IN(10) * 8423; - t12 = IN(3) * 5520 + IN(12) * 15426; - t13 = IN(3) * 15426 - IN(12) * 5520; - t14 = IN(1) * 2404 + IN(14) * 16207; - t15 = IN(1) * 16207 - IN(14) * 2404; - - t0a = (t0 + t8 + (1 << 13)) >> 14; - t1a = (t1 + t9 + (1 << 13)) >> 14; - t2a = (t2 + t10 + (1 << 13)) >> 14; - t3a = (t3 + t11 + (1 << 13)) >> 14; - t4a = (t4 + t12 + (1 << 13)) >> 14; - t5a = (t5 + t13 + (1 << 13)) >> 14; - t6a = (t6 + t14 + (1 << 13)) >> 14; - t7a = (t7 + t15 + (1 << 13)) >> 14; - t8a = (t0 - t8 + (1 << 13)) >> 14; - t9a = (t1 - t9 + (1 << 13)) >> 14; - t10a = (t2 - t10 + (1 << 13)) >> 14; - t11a = (t3 - t11 + (1 << 13)) >> 14; - t12a = (t4 - t12 + (1 << 13)) >> 14; - t13a = (t5 - t13 + (1 << 13)) >> 14; - t14a = (t6 - t14 + (1 << 13)) >> 14; - t15a = (t7 - t15 + (1 << 13)) >> 14; - - t8 = t8a * 16069 + t9a * 3196; - t9 = t8a * 3196 - t9a * 16069; - t10 = t10a * 9102 + t11a * 13623; - t11 = t10a * 13623 - t11a * 9102; - t12 = t13a * 16069 - t12a * 3196; - t13 = t13a * 3196 + t12a * 16069; - t14 = t15a * 9102 - t14a * 13623; - t15 = t15a * 13623 + t14a * 9102; + t0 = IN(15) * 16364U + IN(0) * 804U; + t1 = IN(15) * 804U - IN(0) * 16364U; + t2 = IN(13) * 15893U + IN(2) * 3981U; + t3 = IN(13) * 3981U - IN(2) * 15893U; + t4 = IN(11) * 14811U + IN(4) * 7005U; + t5 = IN(11) * 7005U - IN(4) * 14811U; + t6 = IN(9) * 13160U + IN(6) * 9760U; + t7 = IN(9) * 9760U - IN(6) * 13160U; + t8 = IN(7) * 11003U + IN(8) * 12140U; + t9 = IN(7) * 12140U - IN(8) * 11003U; + t10 = IN(5) * 8423U + IN(10) * 14053U; + t11 = IN(5) * 14053U - IN(10) * 8423U; + t12 = IN(3) * 5520U + IN(12) * 15426U; + t13 = IN(3) * 15426U - IN(12) * 5520U; + t14 = IN(1) * 2404U + IN(14) * 16207U; + t15 = IN(1) * 16207U - IN(14) * 2404U; + + t0a = (dctint)((1U << 13) + t0 + t8 ) >> 14; + t1a = (dctint)((1U << 13) + t1 + t9 ) >> 14; + t2a = (dctint)((1U << 13) + t2 + t10) >> 14; + t3a = (dctint)((1U << 13) + t3 + t11) >> 14; + t4a = (dctint)((1U << 13) + t4 + t12) >> 14; + t5a = (dctint)((1U << 13) + t5 + t13) >> 14; + t6a = (dctint)((1U << 13) + t6 + t14) >> 14; + t7a = (dctint)((1U << 13) + t7 + t15) >> 14; + t8a = (dctint)((1U << 13) + t0 - t8 ) >> 14; + t9a = (dctint)((1U << 13) + t1 - t9 ) >> 14; + t10a = (dctint)((1U << 13) + t2 - t10) >> 14; + t11a = (dctint)((1U << 13) + t3 - t11) >> 14; + t12a = (dctint)((1U << 13) + t4 - t12) >> 14; + t13a = (dctint)((1U << 13) + t5 - t13) >> 14; + t14a = (dctint)((1U << 13) + t6 - t14) >> 14; + t15a = (dctint)((1U << 13) + t7 - t15) >> 14; + + t8 = t8a * 16069U + t9a * 3196U; + t9 = t8a * 3196U - t9a * 16069U; + t10 = t10a * 9102U + t11a * 13623U; + t11 = t10a * 13623U - t11a * 9102U; + t12 = t13a * 16069U - t12a * 3196U; + t13 = t13a * 3196U + t12a * 16069U; + t14 = t15a * 9102U - t14a * 13623U; + t15 = t15a * 13623U + t14a * 9102U; t0 = t0a + t4a; t1 = t1a + t5a; @@ -1429,49 +1429,49 @@ static av_always_inline void iadst16_1d(const dctcoef *in, ptrdiff_t stride, t5 = t1a - t5a; t6 = t2a - t6a; t7 = t3a - t7a; - t8a = (t8 + t12 + (1 << 13)) >> 14; - t9a = (t9 + t13 + (1 << 13)) >> 14; - t10a = (t10 + t14 + (1 << 13)) >> 14; - t11a = (t11 + t15 + (1 << 13)) >> 14; - t12a = (t8 - t12 + (1 << 13)) >> 14; - t13a = (t9 - t13 + (1 << 13)) >> 14; - t14a = (t10 - t14 + (1 << 13)) >> 14; - t15a = (t11 - t15 + (1 << 13)) >> 14; - - t4a = t4 * 15137 + t5 * 6270; - t5a = t4 * 6270 - t5 * 15137; - t6a = t7 * 15137 - t6 * 6270; - t7a = t7 * 6270 + t6 * 15137; - t12 = t12a * 15137 + t13a * 6270; - t13 = t12a * 6270 - t13a * 15137; - t14 = t15a * 15137 - t14a * 6270; - t15 = t15a * 6270 + t14a * 15137; + t8a = (dctint)((1U << 13) + t8 + t12) >> 14; + t9a = (dctint)((1U << 13) + t9 + t13) >> 14; + t10a = (dctint)((1U << 13) + t10 + t14) >> 14; + t11a = (dctint)((1U << 13) + t11 + t15) >> 14; + t12a = (dctint)((1U << 13) + t8 - t12) >> 14; + t13a = (dctint)((1U << 13) + t9 - t13) >> 14; + t14a = (dctint)((1U << 13) + t10 - t14) >> 14; + t15a = (dctint)((1U << 13) + t11 - t15) >> 14; + + t4a = t4 * 15137U + t5 * 6270U; + t5a = t4 * 6270U - t5 * 15137U; + t6a = t7 * 15137U - t6 * 6270U; + t7a = t7 * 6270U + t6 * 15137U; + t12 = t12a * 15137U + t13a * 6270U; + t13 = t12a * 6270U - t13a * 15137U; + t14 = t15a * 15137U - t14a * 6270U; + t15 = t15a * 6270U + t14a * 15137U; out[ 0] = t0 + t2; out[15] = -(t1 + t3); t2a = t0 - t2; t3a = t1 - t3; - out[ 3] = -((t4a + t6a + (1 << 13)) >> 14); - out[12] = (t5a + t7a + (1 << 13)) >> 14; - t6 = (t4a - t6a + (1 << 13)) >> 14; - t7 = (t5a - t7a + (1 << 13)) >> 14; + out[ 3] = -((dctint)((1U << 13) + t4a + t6a) >> 14); + out[12] = (dctint)((1U << 13) + t5a + t7a) >> 14; + t6 = (dctint)((1U << 13) + t4a - t6a) >> 14; + t7 = (dctint)((1U << 13) + t5a - t7a) >> 14; out[ 1] = -(t8a + t10a); out[14] = t9a + t11a; t10 = t8a - t10a; t11 = t9a - t11a; - out[ 2] = (t12 + t14 + (1 << 13)) >> 14; - out[13] = -((t13 + t15 + (1 << 13)) >> 14); - t14a = (t12 - t14 + (1 << 13)) >> 14; - t15a = (t13 - t15 + (1 << 13)) >> 14; - - out[ 7] = ((t2a + t3a) * -11585 + (1 << 13)) >> 14; - out[ 8] = ((t2a - t3a) * 11585 + (1 << 13)) >> 14; - out[ 4] = ((t7 + t6) * 11585 + (1 << 13)) >> 14; - out[11] = ((t7 - t6) * 11585 + (1 << 13)) >> 14; - out[ 6] = ((t11 + t10) * 11585 + (1 << 13)) >> 14; - out[ 9] = ((t11 - t10) * 11585 + (1 << 13)) >> 14; - out[ 5] = ((t14a + t15a) * -11585 + (1 << 13)) >> 14; - out[10] = ((t14a - t15a) * 11585 + (1 << 13)) >> 14; + out[ 2] = (dctint)((1U << 13) + t12 + t14) >> 14; + out[13] = -((dctint)((1U << 13) + t13 + t15) >> 14); + t14a = (dctint)((1U << 13) + t12 - t14) >> 14; + t15a = (dctint)((1U << 13) + t13 - t15) >> 14; + + out[ 7] = (dctint)(-(t2a + t3a) * 11585U + (1 << 13)) >> 14; + out[ 8] = (dctint)( (t2a - t3a) * 11585U + (1 << 13)) >> 14; + out[ 4] = (dctint)( (t7 + t6) * 11585U + (1 << 13)) >> 14; + out[11] = (dctint)( (t7 - t6) * 11585U + (1 << 13)) >> 14; + out[ 6] = (dctint)( (t11 + t10) * 11585U + (1 << 13)) >> 14; + out[ 9] = (dctint)( (t11 - t10) * 11585U + (1 << 13)) >> 14; + out[ 5] = (dctint)(-(t14a + t15a) * 11585U + (1 << 13)) >> 14; + out[10] = (dctint)( (t14a - t15a) * 11585U + (1 << 13)) >> 14; } itxfm_wrap(16, 6)