| Message ID | 20221115193022.2362787-1-jdorfman@google.com |
|---|---|
| State | New |
| Headers | show |
| Series | [FFmpeg-devel] swscale/input: Use unsigned intermediates in rgb64ToUV_c_template | expand |
| Context | Check | Description |
|---|---|---|
| yinshiyou/make_loongarch64 | success | Make finished |
| yinshiyou/make_fate_loongarch64 | success | Make fate finished |
| andriy/make_x86 | success | Make finished |
| andriy/make_fate_x86 | success | Make fate finished |
On Tue, Nov 15, 2022 at 02:30:23PM -0500, Jeremy Dorfman wrote: > Large rgb2yuv tables and high pixel values cause the intermediate > int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined > behavior. This causes libswscale built with LLVM -fsanitize=undefined to > assert. Using unsigned integers instead has defined behavior and > produces identical results, and makes rgb64ToUV_c_template match > rgb64ToY_c_template. > > Fixes: signed integer overflow > --- > libswscale/input.c | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) There are more cases which should be affected by this too. I sent a more complete patchset thx [...]
diff --git a/libswscale/input.c b/libswscale/input.c index 7ff7bfaa01..d7dbedd82f 100644 --- a/libswscale/input.c +++ b/libswscale/input.c @@ -65,9 +65,9 @@ rgb64ToUV_c_template(uint16_t *dstU, uint16_t *dstV, int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX]; av_assert1(src1==src2); for (i = 0; i < width; i++) { - int r_b = input_pixel(&src1[i*4+0]); - int g = input_pixel(&src1[i*4+1]); - int b_r = input_pixel(&src1[i*4+2]); + unsigned int r_b = input_pixel(&src1[i*4+0]); + unsigned int g = input_pixel(&src1[i*4+1]); + unsigned int b_r = input_pixel(&src1[i*4+2]); dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; dstV[i] = (rv*r + gv*g + bv*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT;