@@ -522,13 +522,13 @@ static void init_luts(void)
for (i = 0; i < (1 << NMSEDEC_BITS); i++){
lut_nmsedec_sig[i] = FFMAX((3 * i << (13 - NMSEDEC_FRACBITS)) - (9 << 11), 0);
- lut_nmsedec_sig0[i] = FFMAX((i*i + (1<<NMSEDEC_FRACBITS-1) & mask) << 1, 0);
+ lut_nmsedec_sig0[i] = ((i * i + (1 << (NMSEDEC_FRACBITS - 1))) & mask) << 1;
a = (i >> (NMSEDEC_BITS-2)&2) + 1;
lut_nmsedec_ref[i] = FFMAX((a - 2) * (i << (13 - NMSEDEC_FRACBITS)) +
(1 << 13) - (a * a << 11), 0);
- lut_nmsedec_ref0[i] = FFMAX(((i * i - (i << NMSEDEC_BITS) + (1 << 2 * NMSEDEC_FRACBITS) + (1 << (NMSEDEC_FRACBITS - 1))) & mask)
- << 1, 0);
+ lut_nmsedec_ref0[i] = lut_nmsedec_sig0[i] - (i << (NMSEDEC_BITS + 1))
+ + (1 << (2 * NMSEDEC_FRACBITS + 1));
}
}
1. i * i + (1 << (NMSEDC_FRACBITS - 1)) is always positive (and it doesn't overflow), so one can remove the FFMAX in the computation of lut_nmsedec_sig0. 2. The discriminant of the polynomial i * i - 2^(F + 1) * i + 2^(2 * F) + 2^(F - 1) is negative; hence this polynomial has no real solutions, i.e. its sign doesn't change and is always positive. This allows to remove the FFMAX in the computation of lut_nmsedec_ref0. 3. After that, one sees that the computation of lut_nmsedec_ref0 actually contains lut_nmsedec_sig0. This is obscured by masking the last NMSEDEC_FRACBITS away, but the other summands are multiples of 2^NMSEDEC_FRACBITS and so masking doesn't affect them. Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com> --- Supersedes https://ffmpeg.org/pipermail/ffmpeg-devel/2019-September/250687.html libavcodec/j2kenc.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)