diff mbox series

[FFmpeg-devel,2/2] avcodec/mv30: Fix several integer overflows in idct_1d()

Message ID 20200725221637.11848-2-michael@niedermayer.cc
State Accepted
Commit ddf2ba54979387740b0b2fb319bb5a2c9f78debe
Headers show
Series [FFmpeg-devel,1/2] avcodec/pgxdec: Fix invalid shift in write_frame_* | expand

Checks

Context Check Description
andriy/default pending
andriy/make success Make finished
andriy/make_fate success Make fate finished

Commit Message

Michael Niedermayer July 25, 2020, 10:16 p.m. UTC 
Fixes: signed integer overflow: -1846510390 + -361755993 cannot be represented in type 'int'
Fixes: 23941/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MV30_fuzzer-5654696631730176

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
---
 libavcodec/mv30.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)

Comments

Michael Niedermayer Sept. 28, 2020, 10:35 p.m. UTC | #1 
On Sun, Jul 26, 2020 at 12:16:37AM +0200, Michael Niedermayer wrote:
> Fixes: signed integer overflow: -1846510390 + -361755993 cannot be represented in type 'int'
> Fixes: 23941/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MV30_fuzzer-5654696631730176
> 
> Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
> ---
>  libavcodec/mv30.c | 34 +++++++++++++++++-----------------
>  1 file changed, 17 insertions(+), 17 deletions(-)

will apply

[...]
diff mbox series

Patch

diff --git a/libavcodec/mv30.c b/libavcodec/mv30.c
index c83ba7ffbd..f9cc85f2ac 100644
--- a/libavcodec/mv30.c
+++ b/libavcodec/mv30.c
@@ -104,23 +104,23 @@  static void get_qtable(int16_t *table, int quant, const uint8_t *quant_tab)
 
 static inline void idct_1d(int *blk, int step)
 {
-    const int t0 = blk[0 * step] + blk[4 * step];
-    const int t1 = blk[0 * step] - blk[4 * step];
-    const int t2 = blk[2 * step] + blk[6 * step];
-    const int t3 = ((int)((blk[2 * step] - blk[6 * step]) * 362U) >> 8) - t2;
-    const int t4 = t0 + t2;
-    const int t5 = t0 - t2;
-    const int t6 = t1 + t3;
-    const int t7 = t1 - t3;
-    const int t8 = blk[5 * step] + blk[3 * step];
-    const int t9 = blk[5 * step] - blk[3 * step];
-    const int tA = blk[1 * step] + blk[7 * step];
-    const int tB = blk[1 * step] - blk[7 * step];
-    const int tC = t8 + tA;
-    const int tD = (int)((tB + t9) * 473U) >> 8;
-    const int tE = (((int)(t9 * -669U) >> 8) - tC) + tD;
-    const int tF = ((int)((tA - t8) * 362U) >> 8) - tE;
-    const int t10 = (((int)(tB * 277U) >> 8) - tD) + tF;
+    const unsigned t0 = blk[0 * step] + blk[4 * step];
+    const unsigned t1 = blk[0 * step] - blk[4 * step];
+    const unsigned t2 = blk[2 * step] + blk[6 * step];
+    const unsigned t3 = ((int)((blk[2 * step] - blk[6 * step]) * 362U) >> 8) - t2;
+    const unsigned t4 = t0 + t2;
+    const unsigned t5 = t0 - t2;
+    const unsigned t6 = t1 + t3;
+    const unsigned t7 = t1 - t3;
+    const unsigned t8 = blk[5 * step] + blk[3 * step];
+    const unsigned t9 = blk[5 * step] - blk[3 * step];
+    const unsigned tA = blk[1 * step] + blk[7 * step];
+    const unsigned tB = blk[1 * step] - blk[7 * step];
+    const unsigned tC = t8 + tA;
+    const unsigned tD = (int)((tB + t9) * 473U) >> 8;
+    const unsigned tE = (((int)(t9 * -669U) >> 8) - tC) + tD;
+    const unsigned tF = ((int)((tA - t8) * 362U) >> 8) - tE;
+    const unsigned t10 = (((int)(tB * 277U) >> 8) - tD) + tF;
 
     blk[0 * step] = t4 + tC;
     blk[1 * step] = t6 + tE;