[FFmpeg-devel] lavc/r210enc: Fix undefined behaviour encoding r10k

Submitted by Carl Eugen Hoyos on July 1, 2019, 11:08 a.m.

Details

Message ID CAB0OVGphTRcfpOijAWx36WvfP7SicTgDogEvU3rSe-pMETm46Q@mail.gmail.com
State New
Headers show

Commit Message

Carl Eugen Hoyos July 1, 2019, 11:08 a.m.
Hi!

Attached patch fixes an invalid left shift reported in ticket #7982.

Please comment, Carl Eugen

Comments

mypopy@gmail.com July 1, 2019, 11:31 a.m.
On Mon, Jul 1, 2019 at 7:08 PM Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>
> Hi!
>
> Attached patch fixes an invalid left shift reported in ticket #7982.
>
> Please comment, Carl Eugen
Ha, great catch:), LGTM
Michael Niedermayer July 1, 2019, 6:27 p.m.
On Mon, Jul 01, 2019 at 01:08:34PM +0200, Carl Eugen Hoyos wrote:
> Hi!
> 
> Attached patch fixes an invalid left shift reported in ticket #7982.
> 
> Please comment, Carl Eugen

>  r210enc.c |    6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 9db2195a8e6d3c90726c2bf92119ff6043d4dd99  0001-lavc-r210enc-Fix-undefined-behaviour-encoding-r10k.patch
> From cfe1ae9a1e95dee72ff0e0a45ce8caab98a1bd0b Mon Sep 17 00:00:00 2001
> From: Carl Eugen Hoyos <ceffmpeg@gmail.com>
> Date: Mon, 1 Jul 2019 13:06:02 +0200
> Subject: [PATCH] lavc/r210enc: Fix undefined behaviour encoding r10k.
> 
> Fixes the following ubsan error:
> libavcodec/r210enc.c:69:28: runtime error: left shift of 522 by 22 places cannot be represented in type 'int'
> 
> Fixes ticket #7982.
> ---
>  libavcodec/r210enc.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/libavcodec/r210enc.c b/libavcodec/r210enc.c
> index 02412f3684..95049089bc 100644
> --- a/libavcodec/r210enc.c
> +++ b/libavcodec/r210enc.c
> @@ -60,9 +60,9 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
>          uint16_t *srcb = (uint16_t *)srcb_line;
>          for (j = 0; j < avctx->width; j++) {
>              uint32_t pixel;
> -            uint16_t r = *srcr++;
> -            uint16_t g = *srcg++;
> -            uint16_t b = *srcb++;
> +            uint32_t r = *srcr++;
> +            uint32_t g = *srcg++;
> +            uint32_t b = *srcb++;
>              if (avctx->codec_id == AV_CODEC_ID_R210)
>                  pixel = (r << 20) | (g << 10) | b;

I think plain unsigned is a less confusing choice here, but either is
ok
(uint32_t can make the reader confuse the source with 32bit pointers)

Thanks

[...]

Patch hide | download patch | download mbox

From cfe1ae9a1e95dee72ff0e0a45ce8caab98a1bd0b Mon Sep 17 00:00:00 2001
From: Carl Eugen Hoyos <ceffmpeg@gmail.com>
Date: Mon, 1 Jul 2019 13:06:02 +0200
Subject: [PATCH] lavc/r210enc: Fix undefined behaviour encoding r10k.

Fixes the following ubsan error:
libavcodec/r210enc.c:69:28: runtime error: left shift of 522 by 22 places cannot be represented in type 'int'

Fixes ticket #7982.
---
 libavcodec/r210enc.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/libavcodec/r210enc.c b/libavcodec/r210enc.c
index 02412f3684..95049089bc 100644
--- a/libavcodec/r210enc.c
+++ b/libavcodec/r210enc.c
@@ -60,9 +60,9 @@  static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
         uint16_t *srcb = (uint16_t *)srcb_line;
         for (j = 0; j < avctx->width; j++) {
             uint32_t pixel;
-            uint16_t r = *srcr++;
-            uint16_t g = *srcg++;
-            uint16_t b = *srcb++;
+            uint32_t r = *srcr++;
+            uint32_t g = *srcg++;
+            uint32_t b = *srcb++;
             if (avctx->codec_id == AV_CODEC_ID_R210)
                 pixel = (r << 20) | (g << 10) | b;
             else
-- 
2.22.0