diff mbox series

[FFmpeg-devel,4/4] avcodec/svq1enc: Use unsigned for parameter >= 0 to workaround GCC bug

Message ID DB6PR0101MB22143614BE7AAF962FCADE078F879@DB6PR0101MB2214.eurprd01.prod.exchangelabs.com
State Accepted
Commit 1cb7fd317c84117bbb13b14851d62f77f57bb9ce
Headers show
Series [FFmpeg-devel,1/4] avformat/asfcrypt: Fix wrong array length in function declaration | expand

Checks

Context Check Description
yinshiyou/make_loongarch64 success Make finished
yinshiyou/make_fate_loongarch64 success Make fate finished
andriy/make_x86 success Make finished
andriy/make_fate_x86 success Make fate finished
andriy/make_armv7_RPi4 success Make finished
andriy/make_fate_armv7_RPi4 success Make fate finished

Commit Message

Andreas Rheinhardt July 11, 2022, 3:05 a.m. UTC 
encode_block() in svq1enc.c looks like the following:

static int encode_block(int block[7][256], int level)
{
    int best_score = 0;

    for (unsigned x = 0; x < level; x++) {
        int v = block[1][x];
        block[level][x] = 0;
        best_score      += v * v;
    }

    if (level > 0 && best_score > 64) {
        int score = 0;

        score += encode_block(block, level - 1);
        score += encode_block(block, level - 1);

        if (score < best_score) {
            best_score = score;
        }
    }

    return best_score;
}

When called from outside of encode_block(), it is always called with
level == 5.

This triggers a bug [1] in GCC: On -O3, it creates eight clones of
encode_block with different values of level inlined into it. The clones
with negative values are of course useless*, but they also lead to
-Warray-bounds warnings, because they access block[-1].

This has been mitigated in GCC 12: It no longer creates clones
for parameters that it knows are impossible. Somehow switching levels
to unsigned makes GCC know this. Therefore this commit does this.
(For GCC 11, this changes the warning to "array subscript 4294967295 is
above array bounds" from "array subscript -1 is below array bounds".)

[1]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102513

*: These clones can actually be discarded when compiling with
-ffunction-sections.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
---
 libavcodec/svq1enc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
diff mbox series

Patch

diff --git a/libavcodec/svq1enc.c b/libavcodec/svq1enc.c
index 3c2d594632..6072f8d07d 100644
--- a/libavcodec/svq1enc.c
+++ b/libavcodec/svq1enc.c
@@ -91,7 +91,7 @@  static int ssd_int8_vs_int16_c(const int8_t *pix1, const int16_t *pix2,
 }
 
 static int encode_block(SVQ1EncContext *s, uint8_t *src, uint8_t *ref,
-                        uint8_t *decoded, int stride, int level,
+                        uint8_t *decoded, int stride, unsigned level,
                         int threshold, int lambda, int intra)
 {
     int count, y, x, i, j, split, best_mean, best_score, best_count;