diff mbox series

[FFmpeg-devel,3/9] avcodec/vlc: Skip subtable entries in multi VLC

Message ID 20231022215113.3469-3-michael@niedermayer.cc
State Accepted
Commit 356b1ba765623ec1640a366e587232f1631800f8
Headers show
Series [FFmpeg-devel,1/9] avcodec/vlc: merge lost 16bit end of array check | expand

Checks

Context Check Description
andriy/make_x86 success Make finished
andriy/make_fate_x86 success Make fate finished
yinshiyou/make_loongarch64 success Make finished
yinshiyou/make_fate_loongarch64 success Make fate finished

Commit Message

Michael Niedermayer Oct. 22, 2023, 9:51 p.m. UTC 
These entries do not correspond to VLC symbols that can be used
they do corrupt various variables like min/max bits

This also no longer assumes that there is a single non subtable
entry
Probably fixes some infinite loops too

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
---
 libavcodec/vlc.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
diff mbox series

Patch

diff --git a/libavcodec/vlc.c b/libavcodec/vlc.c
index 77860430861..65883a506ff 100644
--- a/libavcodec/vlc.c
+++ b/libavcodec/vlc.c
@@ -401,15 +401,23 @@  static int vlc_multi_gen(VLC_MULTI_ELEM *table, const VLC *single,
     int minbits, maxbits, max = nb_codes-1;
     unsigned count[VLC_MULTI_MAX_SYMBOLS-1] = { 0, };
     VLC_MULTI_ELEM info = { { 0, }, 0, 0, };
+    int count0 = 0;
 
-    minbits = buf[0].bits;
-    maxbits = buf[0].bits;
+    for (int j = 0; j < 1<<numbits; j++) {
+        if (single->table[j].len > 0) {
+            count0 ++;
+            j += (1 << (numbits - single->table[j].len)) - 1;
+        }
+    }
+
+    minbits = 32;
+    maxbits = 0;
 
-    for (int n = 1; n < nb_codes; n++) {
+    for (int n = nb_codes - count0; n < nb_codes; n++) {
         minbits = FFMIN(minbits, buf[n].bits);
         maxbits = FFMAX(maxbits, buf[n].bits);
     }
-    maxbits = FFMIN(maxbits, numbits);
+    av_assert0(maxbits <= numbits);
 
     while (max >= nb_codes/2) {
         if (buf[max].bits+minbits > maxbits)