From patchwork Mon Dec 25 17:04:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Izen X-Patchwork-Id: 45321 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6623:b0:194:e134:edd4 with SMTP id n35csp2644024pzh; Mon, 25 Dec 2023 09:04:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IFaNZwiCXhQirDkrn0QjiH8BPzWjkBD7rWKGyWsN846QuAfY6f21JgYT6Sx846a1OjJsE2U X-Received: by 2002:a50:d549:0:b0:54c:e719:64ae with SMTP id f9-20020a50d549000000b0054ce71964aemr2657847edj.94.1703523872242; Mon, 25 Dec 2023 09:04:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703523872; cv=none; d=google.com; s=arc-20160816; b=hRy6nsEPQIpVvP7JO9zfb71If/VhMCQQPHwdyyrue/cp71Ul48FICWHMn2ude4oKmj ztntY3N2JsEaz7KFpx9wRqRIbnrsfQWYtvgtOUbKvDcYV6b94WXyLJikvMBhWEnLC0Xs jtDyw8BhVlgM0DWtPMc1bsfH5k6um2MXSf8eXEYWK+HdKbO9JjKYe5qw/P+rCximhJ30 4+x7n+C8PiED6C7LNUnEUfJ9wkto5abtE+QcWH8ky+aTz3IGQ6/9swp09e4AXwAw81bu i+hJB5uQlIpd4/0TP6cavzR5p8ZTnyMVNnTVE+Q4KZ3SJOaG21gEW6a5bKPNPH364b9a wx4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=oU+bXuhnyDUdY3FhW18Ez+nfLgxjIJRHszVLSQHnwdc=; fh=+bdjGe20eEUjtjncwA1dnEEVYNfJL4vyhV+sIRR4l+g=; b=Uy4w3r3lfNK4Zwu3SRljhqzQiz1rYMxe9x0NpTYMo5SeTgDj4wD2FqyUqU9ILcC3In JLjhXNNvVKssUGGNj4aJP+n3BVGjjbEsAKLGP+fbUyFacEWIWyaWmLcLzBz/uCP/tTtR mjA5m3qG2LHI+DsNHeEE5MWXtGmPkTItumuFv/Pr4yPz1uPu1zRXS5j5HHsosVXzMGjv 5C8zv0LoRI21hG5VxWdeNKxcOi6NmuTJJdf9nEsPB68zN3HXnU81NT23jsUz6cJoUPkm 0mrupk7znl/qrnCeD3w1H2PEF2y+Pb8U7MNizcdGARPNfiuhRxiVZ+5+2IpYSY7xXShB qPWg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=fnrhw1tG; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f10-20020a0564021e8a00b0055412c8c675si4617144edf.397.2023.12.25.09.04.31; Mon, 25 Dec 2023 09:04:32 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=fnrhw1tG; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 192AD68D194; Mon, 25 Dec 2023 19:04:28 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A5B0A68CF0D for ; Mon, 25 Dec 2023 19:04:21 +0200 (EET) Received: by mail-qv1-f43.google.com with SMTP id 6a1803df08f44-67f95d97982so4746716d6.1 for ; Mon, 25 Dec 2023 09:04:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703523860; x=1704128660; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=II6ZnMdHqgR6la3vlHYeonUJNnTIgLukdelPT4E3Nkw=; b=fnrhw1tG8bsCNygcdj+kJKorOrgn6+WdSBWxFiPpYnZEjXQ/1Bp8L4auqlvtD6cYAF i9+Cb+2ARa+9StJRfcAVCQxoY23YyRkFeEPWLo5upPDAeZX8snYwVZuj2vegNLT3EDR2 76dHUsm/b47erxb/we/P3Wc3Q8KJ9xr0/gFApxu9XYIf4h2etgG2FrmeMZaI2i8L7h0/ NIJu8PeSTlLv8hrz9Cjwotu+umpQKF6QpHC8brtK9UvpM6mCGczhxB5bWYZSWgBmbizl KQhNyZVuT6aYSaeINAUeADoJ8OEhplNRGQ4rdxizS/oESYLyYrybCOJll5MzlAqse0Bg gZZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703523860; x=1704128660; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=II6ZnMdHqgR6la3vlHYeonUJNnTIgLukdelPT4E3Nkw=; b=D764g6EnVQTMWI1JH9JLfatFi3ieEg2Iv6aT5V0Yu+ZCbSaUAM6tw9ZaUS7JJBBSIl +PT2rx4rOc3TMhywLV6Q38W6w2uw+xvsfWjyPdsQf9r2IntfxWiSX62tGptWLdTzxj8s EgmX90hwS2EVV2MM1sBOY87zohnNJ77kdgWiwZ9MQo7BkfFvkEdc1AJVdQVhfus020Rb XqFJ8ZLJi+XYAa37ONAIZ0MWqZfTuU5eZPAVUk339pAGTshmG5/SAX8X/SD/kbHkZAeU jMnVpyZOahe/PGAjlYMMq79hOlEwVjdNB/Sjpj1TCFtFsUUXhkyizTvvBOH1vK+IARG9 Zivg== X-Gm-Message-State: AOJu0YwGYmwlzckH0kUWSfIReri6ISg3QJBTUTeoEfqDopnLGr1+G0GF kAF/HbxnmHV6+XbSaOIppnMnv3t4mCE= X-Received: by 2002:a05:620a:1a20:b0:780:ff63:b337 with SMTP id bk32-20020a05620a1a2000b00780ff63b337mr11642675qkb.7.1703523859790; Mon, 25 Dec 2023 09:04:19 -0800 (PST) Received: from gauss.local (c-68-56-149-176.hsd1.mi.comcast.net. [68.56.149.176]) by smtp.gmail.com with ESMTPSA id dz25-20020a05620a2b9900b007812d3b7f57sm2740038qkb.91.2023.12.25.09.04.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Dec 2023 09:04:19 -0800 (PST) From: Leo Izen To: ffmpeg-devel@ffmpeg.org Date: Mon, 25 Dec 2023 12:04:17 -0500 Message-ID: <20231225170417.153992-1-leo.izen@gmail.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2] avcodec/jpegxl_parser: check ANS cluster alphabet size vs bundle size X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Leo Izen Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 1pTkJ0lSX3+x The specification doesn't mention that clusters cannot have alphabet sizes greater than 1 << bundle->log_alphabet_size, but the reference implementation rejects these entropy streams as invalid, so we should too. Refusing to do so can overflow a stack variable on line 556 that should be large enough otherwise. Fixes #10738. Found-by: Zeng Yunxiang and Li Zeyuan Signed-off-by: Leo Izen --- libavcodec/jpegxl_parser.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/libavcodec/jpegxl_parser.c b/libavcodec/jpegxl_parser.c index 006eb6b295..f026fda9ac 100644 --- a/libavcodec/jpegxl_parser.c +++ b/libavcodec/jpegxl_parser.c @@ -64,26 +64,26 @@ typedef struct JXLSymbolDistribution { int log_bucket_size; /* this is the actual size of the alphabet */ int alphabet_size; - /* ceil(log(alphabet_size)) */ - int log_alphabet_size; /* for prefix code distributions */ VLC vlc; /* in case bits == 0 */ uint32_t default_symbol; + /* ceil(log(alphabet_size)) */ + int log_alphabet_size; /* * each (1 << log_alphabet_size) length * with log_alphabet_size <= 8 */ /* frequencies associated with this Distribution */ - uint32_t freq[258]; + uint32_t freq[256]; /* cutoffs for using the symbol table */ - uint16_t cutoffs[258]; + uint16_t cutoffs[256]; /* the symbol table for this distribution */ - uint16_t symbols[258]; + uint16_t symbols[256]; /* the offset for symbols */ - uint16_t offsets[258]; + uint16_t offsets[256]; /* if this distribution contains only one symbol this is its index */ int uniq_pos; @@ -382,13 +382,13 @@ static int populate_distribution(GetBitContext *gb, JXLSymbolDistribution *dist, int len = 0, shift, omit_log = -1, omit_pos = -1; int prev = 0, num_same = 0; uint32_t total_count = 0; - uint8_t logcounts[258] = { 0 }; - uint8_t same[258] = { 0 }; + uint8_t logcounts[256] = { 0 }; + uint8_t same[256] = { 0 }; + const int table_size = 1 << log_alphabet_size; dist->uniq_pos = -1; if (get_bits1(gb)) { /* simple code */ - dist->alphabet_size = 256; if (get_bits1(gb)) { uint8_t v1 = jxl_u8(gb); uint8_t v2 = jxl_u8(gb); @@ -398,17 +398,24 @@ static int populate_distribution(GetBitContext *gb, JXLSymbolDistribution *dist, dist->freq[v2] = (1 << 12) - dist->freq[v1]; if (!dist->freq[v1]) dist->uniq_pos = v2; + dist->alphabet_size = 1 + FFMAX(v1, v2); } else { uint8_t x = jxl_u8(gb); dist->freq[x] = 1 << 12; dist->uniq_pos = x; + dist->alphabet_size = 1 + x; } + if (dist->alphabet_size > table_size) + return AVERROR_INVALIDDATA; + return 0; } if (get_bits1(gb)) { /* flat code */ dist->alphabet_size = jxl_u8(gb) + 1; + if (dist->alphabet_size > table_size) + return AVERROR_INVALIDDATA; for (int i = 0; i < dist->alphabet_size; i++) dist->freq[i] = (1 << 12) / dist->alphabet_size; for (int i = 0; i < (1 << 12) % dist->alphabet_size; i++) @@ -426,6 +433,9 @@ static int populate_distribution(GetBitContext *gb, JXLSymbolDistribution *dist, return AVERROR_INVALIDDATA; dist->alphabet_size = jxl_u8(gb) + 3; + if (dist->alphabet_size > table_size) + return AVERROR_INVALIDDATA; + for (int i = 0; i < dist->alphabet_size; i++) { logcounts[i] = get_vlc2(gb, dist_prefix_table, 7, 1); if (logcounts[i] == 13) {