diff mbox series

[FFmpeg-devel,4/7] avcodec/pgxdec: Fix issue with negative linesizes

Message ID AS8PR01MB7944C68620857D69EAB2C3128FF99@AS8PR01MB7944.eurprd01.prod.exchangelabs.com
State Accepted
Commit e4a26a64a875d4376cd78f77905a58c4f0238d85
Headers show
Series [FFmpeg-devel,1/7] avcodec/pgxdec: Make better use of size check | expand


Context Check Description
yinshiyou/make_loongarch64 success Make finished
yinshiyou/make_fate_loongarch64 success Make fate finished
andriy/make_x86 success Make finished
andriy/make_fate_x86 success Make fate finished

Commit Message

Andreas Rheinhardt April 24, 2022, 4:42 a.m. UTC
The PGX decoder accesses the lines via code like
(PIXEL*)frame->data[0] + i*frame->linesize[0]/sizeof(PIXEL)
where PIXEL is a macro parameter. This code has issues with negative
linesizes, because the type of sizeof(PIXEL) is size_t, so
that on common systems i*linesize/sizeof(PIXEL) will
always be an unsigned type that is very large in case linesize is
negative. This happens to work*, but it is undefined behaviour
and e.g. leads to "src/libavcodec/pgxdec.c:114:1: runtime error:
addition of unsigned offset to 0x7efe9c2b7040 overflowed to 0x7efe9c2b6040"
errors from UBSAN.
Fix this by using (PIXEL*)(frame->data[0] + i*frame->linesize[0]).
This is allowed because linesize has to be suitably aligned.

*: Converting a negative int to size_t works by adding SIZE_MAX + 1
to the number, so that the result is off by (SIZE_MAX + 1) /
sizeof(PIXEL). Converting the pointer arithmetic (performed on PIXELs)
back to ordinary pointers is tantamount to multiplying by sizeof(PIXEL),
so that the result is off by SIZE_MAX + 1; but SIZE_MAX + 1 == 0
for the underlying pointers.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
 libavcodec/pgxdec.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


Paul B Mahol April 24, 2022, 11:15 a.m. UTC | #1
diff mbox series


diff --git a/libavcodec/pgxdec.c b/libavcodec/pgxdec.c
index c9ada5afb5..30895b51ee 100644
--- a/libavcodec/pgxdec.c
+++ b/libavcodec/pgxdec.c
@@ -97,7 +97,7 @@  error:
     {                                                                                       \
         int i, j;                                                                           \
         for (i = 0; i < height; i++) {                                                      \
-            PIXEL *line = (PIXEL*)frame->data[0] + i*frame->linesize[0]/sizeof(PIXEL);      \
+            PIXEL *line = (PIXEL*)(frame->data[0] + i * frame->linesize[0]);                \
             for (j = 0; j < width; j++) {                                                   \
                 unsigned val;                                                               \
                 if (sign)                                                                   \