[FFmpeg-devel] lavc/r210enc: Fix undefined behaviour encoding r10k

Hi!

Attached patch fixes an invalid left shift reported in ticket #7982.

Please comment, Carl Eugen

On Mon, Jul 1, 2019 at 7:08 PM Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>
> Hi!
>
> Attached patch fixes an invalid left shift reported in ticket #7982.
>
> Please comment, Carl Eugen
Ha, great catch:), LGTM

On Mon, Jul 01, 2019 at 01:08:34PM +0200, Carl Eugen Hoyos wrote:
> Hi!
> 
> Attached patch fixes an invalid left shift reported in ticket #7982.
> 
> Please comment, Carl Eugen

>  r210enc.c |    6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 9db2195a8e6d3c90726c2bf92119ff6043d4dd99  0001-lavc-r210enc-Fix-undefined-behaviour-encoding-r10k.patch
> From cfe1ae9a1e95dee72ff0e0a45ce8caab98a1bd0b Mon Sep 17 00:00:00 2001
> From: Carl Eugen Hoyos <ceffmpeg@gmail.com>
> Date: Mon, 1 Jul 2019 13:06:02 +0200
> Subject: [PATCH] lavc/r210enc: Fix undefined behaviour encoding r10k.
> 
> Fixes the following ubsan error:
> libavcodec/r210enc.c:69:28: runtime error: left shift of 522 by 22 places cannot be represented in type 'int'
> 
> Fixes ticket #7982.
> ---
>  libavcodec/r210enc.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/libavcodec/r210enc.c b/libavcodec/r210enc.c
> index 02412f3684..95049089bc 100644
> --- a/libavcodec/r210enc.c
> +++ b/libavcodec/r210enc.c
> @@ -60,9 +60,9 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
>          uint16_t *srcb = (uint16_t *)srcb_line;
>          for (j = 0; j < avctx->width; j++) {
>              uint32_t pixel;
> -            uint16_t r = *srcr++;
> -            uint16_t g = *srcg++;
> -            uint16_t b = *srcb++;
> +            uint32_t r = *srcr++;
> +            uint32_t g = *srcg++;
> +            uint32_t b = *srcb++;
>              if (avctx->codec_id == AV_CODEC_ID_R210)
>                  pixel = (r << 20) | (g << 10) | b;

I think plain unsigned is a less confusing choice here, but either is
ok
(uint32_t can make the reader confuse the source with 32bit pointers)

Thanks

[...]

Am Mo., 1. Juli 2019 um 20:27 Uhr schrieb Michael Niedermayer
<michael@niedermayer.cc>:
>
> On Mon, Jul 01, 2019 at 01:08:34PM +0200, Carl Eugen Hoyos wrote:
> > Hi!
> >
> > Attached patch fixes an invalid left shift reported in ticket #7982.
> >
> > Please comment, Carl Eugen
>
> >  r210enc.c |    6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> > 9db2195a8e6d3c90726c2bf92119ff6043d4dd99  0001-lavc-r210enc-Fix-undefined-behaviour-encoding-r10k.patch
> > From cfe1ae9a1e95dee72ff0e0a45ce8caab98a1bd0b Mon Sep 17 00:00:00 2001
> > From: Carl Eugen Hoyos <ceffmpeg@gmail.com>
> > Date: Mon, 1 Jul 2019 13:06:02 +0200
> > Subject: [PATCH] lavc/r210enc: Fix undefined behaviour encoding r10k.
> >
> > Fixes the following ubsan error:
> > libavcodec/r210enc.c:69:28: runtime error: left shift of 522 by 22 places cannot be represented in type 'int'
> >
> > Fixes ticket #7982.
> > ---
> >  libavcodec/r210enc.c | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/libavcodec/r210enc.c b/libavcodec/r210enc.c
> > index 02412f3684..95049089bc 100644
> > --- a/libavcodec/r210enc.c
> > +++ b/libavcodec/r210enc.c
> > @@ -60,9 +60,9 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
> >          uint16_t *srcb = (uint16_t *)srcb_line;
> >          for (j = 0; j < avctx->width; j++) {
> >              uint32_t pixel;
> > -            uint16_t r = *srcr++;
> > -            uint16_t g = *srcg++;
> > -            uint16_t b = *srcb++;
> > +            uint32_t r = *srcr++;
> > +            uint32_t g = *srcg++;
> > +            uint32_t b = *srcb++;
> >              if (avctx->codec_id == AV_CODEC_ID_R210)
> >                  pixel = (r << 20) | (g << 10) | b;
>
> I think plain unsigned is a less confusing choice here, but either is
> ok

Pushed with unsigned.

Thank you, Carl Eugen