diff mbox series

[FFmpeg-devel,1/2] avcodec/j2kenc: Fix undefined shifts of negative numbers

Message ID 20200121002348.16914-1-andreas.rheinhardt@gmail.com
State Accepted
Headers show
Series [FFmpeg-devel,1/2] avcodec/j2kenc: Fix undefined shifts of negative numbers
Related show

Checks

Context Check Description
andriy/ffmpeg-patchwork pending
andriy/ffmpeg-patchwork success Applied patch
andriy/ffmpeg-patchwork success Configure finished
andriy/ffmpeg-patchwork success Make finished
andriy/ffmpeg-patchwork success Make fate finished

Commit Message

Andreas Rheinhardt Jan. 21, 2020, 12:23 a.m. UTC
Also add parentheses to some lines to make the operator precedence
clearer.

This affected the FATE-tests vsynth*-jpeg2000 and vsynth*-jpeg2000-97
(where * ranges over { 1, 2, 3, _lena }) as well as ticket #7983.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
---
Supersedes
https://ffmpeg.org/pipermail/ffmpeg-devel/2019-September/250683.html

 libavcodec/j2kenc.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

Comments

Michael Niedermayer Jan. 21, 2020, 1:53 p.m. UTC | #1
On Tue, Jan 21, 2020 at 01:23:47AM +0100, Andreas Rheinhardt wrote:
> Also add parentheses to some lines to make the operator precedence
> clearer.
> 
> This affected the FATE-tests vsynth*-jpeg2000 and vsynth*-jpeg2000-97
> (where * ranges over { 1, 2, 3, _lena }) as well as ticket #7983.
> 
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
> ---
> Supersedes
> https://ffmpeg.org/pipermail/ffmpeg-devel/2019-September/250683.html
> 
>  libavcodec/j2kenc.c | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)

will apply

thx

[...]
diff mbox series

Patch

diff --git a/libavcodec/j2kenc.c b/libavcodec/j2kenc.c
index e91d932bb7..38643c9a28 100644
--- a/libavcodec/j2kenc.c
+++ b/libavcodec/j2kenc.c
@@ -521,13 +521,13 @@  static void init_luts(void)
         mask = ~((1<<NMSEDEC_FRACBITS)-1);
 
     for (i = 0; i < (1 << NMSEDEC_BITS); i++){
-        lut_nmsedec_sig[i]  = FFMAX(6*i - (9<<NMSEDEC_FRACBITS-1) << 12-NMSEDEC_FRACBITS, 0);
+        lut_nmsedec_sig[i]  = FFMAX((3 * i << (13 - NMSEDEC_FRACBITS)) - (9 << 11), 0);
         lut_nmsedec_sig0[i] = FFMAX((i*i + (1<<NMSEDEC_FRACBITS-1) & mask) << 1, 0);
 
         a = (i >> (NMSEDEC_BITS-2)&2) + 1;
-        lut_nmsedec_ref[i]  = FFMAX((-2*i + (1<<NMSEDEC_FRACBITS) + a*i - (a*a<<NMSEDEC_FRACBITS-2))
-                                    << 13-NMSEDEC_FRACBITS, 0);
-        lut_nmsedec_ref0[i] = FFMAX(((i*i + (1-4*i << NMSEDEC_FRACBITS-1) + (1<<2*NMSEDEC_FRACBITS)) & mask)
+        lut_nmsedec_ref[i]  = FFMAX((a - 2) * (i << (13 - NMSEDEC_FRACBITS)) +
+                                    (1 << 13) - (a * a << 11), 0);
+        lut_nmsedec_ref0[i] = FFMAX(((i * i - (i << NMSEDEC_BITS) + (1 << 2 * NMSEDEC_FRACBITS) + (1 << (NMSEDEC_FRACBITS - 1))) & mask)
                                     << 1, 0);
     }
 }
@@ -927,7 +927,7 @@  static int encode_tile(Jpeg2000EncoderContext *s, Jpeg2000Tile *tile, int tileno
                             for (y = yy0; y < yy1; y++){
                                 int *ptr = t1.data + (y-yy0)*t1.stride;
                                 for (x = xx0; x < xx1; x++){
-                                    *ptr++ = comp->i_data[(comp->coord[0][1] - comp->coord[0][0]) * y + x] << NMSEDEC_FRACBITS;
+                                    *ptr++ = comp->i_data[(comp->coord[0][1] - comp->coord[0][0]) * y + x] * (1 << NMSEDEC_FRACBITS);
                                 }
                             }
                         } else{