| Message ID | CAB0OVGpfP9gV-WcLVWUTs3mrb7hVxqd4s+-EAUVRvN0_Qf-pfA@mail.gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [FFmpeg-devel] lavfi/rotate: Fix undefined behaviour | expand |
| Context | Check | Description |
|---|---|---|
| yinshiyou/configure_loongarch64 | warning | Failed to apply patch |
| andriy/make_x86 | success | Make finished |
| andriy/make_fate_x86 | success | Make fate finished |
Quoting Carl Eugen Hoyos (2022-09-01 21:28:08) > Hi! > > Attached patch fixes ticket #9799. > > Please comment, Carl Eugen > > From 2cce687961c3b56a92d88184269bf9fa075ae297 Mon Sep 17 00:00:00 2001 > From: Carl Eugen Hoyos <ceffmpeg@gmail.com> > Date: Thu, 1 Sep 2022 20:55:54 +0200 > Subject: [PATCH] lavfi/rotate: Avoid undefined behaviour. > > Fixes the following integer overflows: > libavfilter/vf_rotate.c:273:13: runtime error: signed integer overflow: 92951468 + 2058533568 cannot be represented in type 'int' > libavfilter/vf_rotate.c:273:37: runtime error: signed integer overflow: 39684 * 54149 cannot be represented in type 'int' > libavfilter/vf_rotate.c:272:13: runtime error: signed integer overflow: 247587320 + 1900985032 cannot be represented in type 'int' > libavfilter/vf_rotate.c:272:37: runtime error: signed integer overflow: 42584 * 50430 cannot be represented in type 'int' > libavfilter/vf_rotate.c:272:50: runtime error: signed integer overflow: 65083 * 52912 cannot be represented in type 'int' > libavfilter/vf_rotate.c:273:50: runtime error: signed integer overflow: 65286 * 38044 cannot be represented in type 'int' > > Fixes ticket #9799, different output with different compilers. > --- > libavfilter/vf_rotate.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/libavfilter/vf_rotate.c b/libavfilter/vf_rotate.c > index 4429e3d543..d319dfe3d9 100644 > --- a/libavfilter/vf_rotate.c > +++ b/libavfilter/vf_rotate.c > @@ -269,8 +269,8 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, > int s01 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y ]); > int s10 = AV_RL16(&src[src_linestep * int_x + i + src_linesize * int_y1]); > int s11 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y1]); > - int s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); > - int s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); > + int64_t s0 = (((int64_t)(1<<16) - frac_x)*s00 + (int64_t)frac_x*s01); > + int64_t s1 = (((int64_t)(1<<16) - frac_x)*s10 + (int64_t)frac_x*s11); Given that the same casts also appear in the AV_WL16 below, it seems better to just change the types of frac_x/frac_y. Also, shouldn't the same change be done also to interpolate_bilinear8?
Am Fr., 2. Sept. 2022 um 09:04 Uhr schrieb Anton Khirnov <anton@khirnov.net>: > > Quoting Carl Eugen Hoyos (2022-09-01 21:28:08) > > Hi! > > > > Attached patch fixes ticket #9799. > > > > Please comment, Carl Eugen > > > > From 2cce687961c3b56a92d88184269bf9fa075ae297 Mon Sep 17 00:00:00 2001 > > From: Carl Eugen Hoyos <ceffmpeg@gmail.com> > > Date: Thu, 1 Sep 2022 20:55:54 +0200 > > Subject: [PATCH] lavfi/rotate: Avoid undefined behaviour. > > > > Fixes the following integer overflows: > > libavfilter/vf_rotate.c:273:13: runtime error: signed integer overflow: 92951468 + 2058533568 cannot be represented in type 'int' > > libavfilter/vf_rotate.c:273:37: runtime error: signed integer overflow: 39684 * 54149 cannot be represented in type 'int' > > libavfilter/vf_rotate.c:272:13: runtime error: signed integer overflow: 247587320 + 1900985032 cannot be represented in type 'int' > > libavfilter/vf_rotate.c:272:37: runtime error: signed integer overflow: 42584 * 50430 cannot be represented in type 'int' > > libavfilter/vf_rotate.c:272:50: runtime error: signed integer overflow: 65083 * 52912 cannot be represented in type 'int' > > libavfilter/vf_rotate.c:273:50: runtime error: signed integer overflow: 65286 * 38044 cannot be represented in type 'int' > > > > Fixes ticket #9799, different output with different compilers. > > --- > > libavfilter/vf_rotate.c | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/libavfilter/vf_rotate.c b/libavfilter/vf_rotate.c > > index 4429e3d543..d319dfe3d9 100644 > > --- a/libavfilter/vf_rotate.c > > +++ b/libavfilter/vf_rotate.c > > @@ -269,8 +269,8 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, > > int s01 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y ]); > > int s10 = AV_RL16(&src[src_linestep * int_x + i + src_linesize * int_y1]); > > int s11 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y1]); > > - int s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); > > - int s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); > > + int64_t s0 = (((int64_t)(1<<16) - frac_x)*s00 + (int64_t)frac_x*s01); > > + int64_t s1 = (((int64_t)(1<<16) - frac_x)*s10 + (int64_t)frac_x*s11); > > Given that the same casts also appear in the AV_WL16 below, it seems > better to just change the types of frac_x/frac_y. Definitely, new patch attached. > Also, shouldn't the same change be done also to interpolate_bilinear8? I was unable to reproduce with 8-bit input. Thank you, Carl Eugen From 062b8a3a202f72125fa05582c2a3c0abc8305887 Mon Sep 17 00:00:00 2001 From: Carl Eugen Hoyos <ceffmpeg@gmail.com> Date: Sat, 3 Sep 2022 22:50:19 +0200 Subject: [PATCH] lavfi/rotate: Avoid undefined behaviour. Fixes the following integer overflows: libavfilter/vf_rotate.c:273:13: runtime error: signed integer overflow: 92951468 + 2058533568 cannot be represented in type 'int' libavfilter/vf_rotate.c:273:37: runtime error: signed integer overflow: 39684 * 54149 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:13: runtime error: signed integer overflow: 247587320 + 1900985032 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:37: runtime error: signed integer overflow: 42584 * 50430 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:50: runtime error: signed integer overflow: 65083 * 52912 cannot be represented in type 'int' libavfilter/vf_rotate.c:273:50: runtime error: signed integer overflow: 65286 * 38044 cannot be represented in type 'int' Fixes ticket #9799, different output with different compilers. --- libavfilter/vf_rotate.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/libavfilter/vf_rotate.c b/libavfilter/vf_rotate.c index 4429e3d543..96c250a459 100644 --- a/libavfilter/vf_rotate.c +++ b/libavfilter/vf_rotate.c @@ -258,8 +258,8 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, { int int_x = av_clip(x>>16, 0, max_x); int int_y = av_clip(y>>16, 0, max_y); - int frac_x = x&0xFFFF; - int frac_y = y&0xFFFF; + int64_t frac_x = x&0xFFFF; + int64_t frac_y = y&0xFFFF; int i; int int_x1 = FFMIN(int_x+1, max_x); int int_y1 = FFMIN(int_y+1, max_y); @@ -269,10 +269,10 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, int s01 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y ]); int s10 = AV_RL16(&src[src_linestep * int_x + i + src_linesize * int_y1]); int s11 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y1]); - int s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); - int s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); + int64_t s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); + int64_t s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); - AV_WL16(&dst_color[i], ((int64_t)((1<<16) - frac_y)*s0 + (int64_t)frac_y*s1) >> 32); + AV_WL16(&dst_color[i], (((1<<16) - frac_y)*s0 + frac_y*s1) >> 32); } return dst_color;
>>/Also, shouldn't the same change be done also to interpolate_bilinear8? / > I was unable to reproduce with 8-bit input. When I tested it, the issue was reproducible only with 14-bit and 16-bit input. 12-bit did work.
From 2cce687961c3b56a92d88184269bf9fa075ae297 Mon Sep 17 00:00:00 2001 From: Carl Eugen Hoyos <ceffmpeg@gmail.com> Date: Thu, 1 Sep 2022 20:55:54 +0200 Subject: [PATCH] lavfi/rotate: Avoid undefined behaviour. Fixes the following integer overflows: libavfilter/vf_rotate.c:273:13: runtime error: signed integer overflow: 92951468 + 2058533568 cannot be represented in type 'int' libavfilter/vf_rotate.c:273:37: runtime error: signed integer overflow: 39684 * 54149 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:13: runtime error: signed integer overflow: 247587320 + 1900985032 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:37: runtime error: signed integer overflow: 42584 * 50430 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:50: runtime error: signed integer overflow: 65083 * 52912 cannot be represented in type 'int' libavfilter/vf_rotate.c:273:50: runtime error: signed integer overflow: 65286 * 38044 cannot be represented in type 'int' Fixes ticket #9799, different output with different compilers. --- libavfilter/vf_rotate.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavfilter/vf_rotate.c b/libavfilter/vf_rotate.c index 4429e3d543..d319dfe3d9 100644 --- a/libavfilter/vf_rotate.c +++ b/libavfilter/vf_rotate.c @@ -269,8 +269,8 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, int s01 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y ]); int s10 = AV_RL16(&src[src_linestep * int_x + i + src_linesize * int_y1]); int s11 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y1]); - int s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); - int s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); + int64_t s0 = (((int64_t)(1<<16) - frac_x)*s00 + (int64_t)frac_x*s01); + int64_t s1 = (((int64_t)(1<<16) - frac_x)*s10 + (int64_t)frac_x*s11); AV_WL16(&dst_color[i], ((int64_t)((1<<16) - frac_y)*s0 + (int64_t)frac_y*s1) >> 32); } -- 2.30.1