diff mbox series

[FFmpeg-devel,2/2] avcodec/j2kenc: Simplify creation of luts

Message ID 20200121002348.16914-2-andreas.rheinhardt@gmail.com
State New
Headers show
Series [FFmpeg-devel,1/2] avcodec/j2kenc: Fix undefined shifts of negative numbers
Related show

Checks

Context Check Description
andriy/ffmpeg-patchwork pending
andriy/ffmpeg-patchwork success Applied patch
andriy/ffmpeg-patchwork success Configure finished
andriy/ffmpeg-patchwork success Make finished
andriy/ffmpeg-patchwork success Make fate finished

Commit Message

Andreas Rheinhardt Jan. 21, 2020, 12:23 a.m. UTC
1. i * i + (1 << (NMSEDC_FRACBITS - 1)) is always positive (and it
doesn't overflow), so one can remove the FFMAX in the computation of
lut_nmsedec_sig0.
2. The discriminant of the polynomial i * i - 2^(F + 1) * i + 2^(2 * F) +
2^(F - 1) is negative; hence this polynomial has no real solutions, i.e.
its sign doesn't change and is always positive. This allows to remove
the FFMAX in the computation of lut_nmsedec_ref0.
3. After that, one sees that the computation of lut_nmsedec_ref0
actually contains lut_nmsedec_sig0. This is obscured by masking the last
NMSEDEC_FRACBITS away, but the other summands are multiples of
2^NMSEDEC_FRACBITS and so masking doesn't affect them.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
---
Supersedes https://ffmpeg.org/pipermail/ffmpeg-devel/2019-September/250687.html

 libavcodec/j2kenc.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
diff mbox series

Patch

diff --git a/libavcodec/j2kenc.c b/libavcodec/j2kenc.c
index 38643c9a28..c5c04bc4bf 100644
--- a/libavcodec/j2kenc.c
+++ b/libavcodec/j2kenc.c
@@ -522,13 +522,13 @@  static void init_luts(void)
 
     for (i = 0; i < (1 << NMSEDEC_BITS); i++){
         lut_nmsedec_sig[i]  = FFMAX((3 * i << (13 - NMSEDEC_FRACBITS)) - (9 << 11), 0);
-        lut_nmsedec_sig0[i] = FFMAX((i*i + (1<<NMSEDEC_FRACBITS-1) & mask) << 1, 0);
+        lut_nmsedec_sig0[i] = ((i * i + (1 << (NMSEDEC_FRACBITS - 1))) & mask) << 1;
 
         a = (i >> (NMSEDEC_BITS-2)&2) + 1;
         lut_nmsedec_ref[i]  = FFMAX((a - 2) * (i << (13 - NMSEDEC_FRACBITS)) +
                                     (1 << 13) - (a * a << 11), 0);
-        lut_nmsedec_ref0[i] = FFMAX(((i * i - (i << NMSEDEC_BITS) + (1 << 2 * NMSEDEC_FRACBITS) + (1 << (NMSEDEC_FRACBITS - 1))) & mask)
-                                    << 1, 0);
+        lut_nmsedec_ref0[i] = lut_nmsedec_sig0[i] - (i << (NMSEDEC_BITS + 1))
+                              + (1 << (2 * NMSEDEC_FRACBITS + 1));
     }
 }