From patchwork Mon Oct 2 20:25:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Izen X-Patchwork-Id: 44109 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1204:b0:15d:8365:d4b8 with SMTP id v4csp1639549pzf; Mon, 2 Oct 2023 13:25:51 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF+lHNVz0qmr6wkhAe6RL2TELFHi77V8KHOQmKEG87JqOuOhsh4jTGP9gRogNT5OxoErM// X-Received: by 2002:a05:6402:884:b0:523:b1b0:f69f with SMTP id e4-20020a056402088400b00523b1b0f69fmr10759848edy.32.1696278351344; Mon, 02 Oct 2023 13:25:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696278351; cv=none; d=google.com; s=arc-20160816; b=npC/2pQ/3pPgfLZqjXN+GyQ203WQyAwxEhyaAzZc8aTqVDd9YgZU6zBVTEitndT4K8 e1w0RiZFxOqcq10ORyqbtrK71Hfcf1/zm1/YIIyZUmy+oOAcSxntxPPpWdvhWvUHJfEb Yd2Yi1AbBjpcpiWtwuPPEmiuu1TlFN+rAHRgL3gKAWkyslzRRZ/87P68ORPy/Su+anNl rZuu1pJCmJAT70iGnDIesCSjF85sfA0ndFaLnflBq/5FQyEihSaGJsQa4I1osyR06jK8 58cqvOFe/3Eei2qV0p2vFx/DDsMLQdv0IFMOLR/Yu5iBCxkxZERMMBqwJzvLTuTjuuMe Br2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=O2/+Z1shRTmaB+lm5h3N2qSeOkgilrrJO2gWwlkaO7E=; fh=FYzOVThyl/FlqQ7uXyPgiZ3IBa6Cm47bzOUzHuMA/IA=; b=wa1DzkSNCDFkcTm/qySFG4kVBZilptjIUS3Aeie6WndPuldFGC5d9FWsKAcVDPpbDB n3h6VEsXk7Y4AU3pSWie4Oi2CgkAoyTjEjNDQnVIZ8tScjTYQTP2Z87M/A/RVI/K2jrP 0zLlFBQz/u9MrhUtmhEWnluvVOn1URkXakvWwrFHlQhvsud6Lvc6Cm5RMQwRliXsBKom J8Q36D0GwbBR2Ck6pT/FdKSakTU7InBmdj6QUFqQ1zyyz8ZVSEPXZryQO60ZZhn3/EVB 1EC7EVPFQ9YRnEmr6xFgfGrn6+04QETx1ozWUPjN5sPit2zVAqpFv4B2TpZbeb776pfb hBSw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=TsLGBEmF; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y8-20020a056402270800b00531190128a0si16251713edd.511.2023.10.02.13.25.38; Mon, 02 Oct 2023 13:25:51 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=TsLGBEmF; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5978368CAAA; Mon, 2 Oct 2023 23:25:35 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D7C2768CAAA for ; Mon, 2 Oct 2023 23:25:27 +0300 (EEST) Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-77406094bc3so2420185a.1 for ; Mon, 02 Oct 2023 13:25:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696278326; x=1696883126; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=TFNshG8BbCdPWdHCFa8BenWDOHnck80H23DJ8DuwqLE=; b=TsLGBEmFelk5a5iR1mSBw6iZOCoIu2z03DxbirSU8ed17NgeFNUE7prYnpF3qHdRfa s37r95x8cuwXozPp0Moqaihe6HVqE48YlE1xBNoT89UrSSfMhTzQOhlvWM+kXlx2XmJJ TpxW/PVsq9nEQQVKiWYIJ32niuk44zO1XVf9VFlIuYdGWcXKvHJgAtk+C0QIgtpzBVxE Stz9ZUqeW+b9GGqSySQb7dHQA8xq+F0RU+yS4goIzQvDuvhiAMgLw2ch3ckEjb3yR8Ql oIwuNdcp/EY1j0+qTGQtPempcscTm8PyufbF3gx1ioE0CG1tElXf7omDfzne46VYkeWR s0xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696278326; x=1696883126; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TFNshG8BbCdPWdHCFa8BenWDOHnck80H23DJ8DuwqLE=; b=wUbnmS6Nju4LnTZO1M3fVeHTbmpXmJtEYk0c28jDWzxeXCHTHnkJOPSuPygXoq4bgu kN/CnWo2XBcrvO6HpULsd0pPUdZbE8B/KxMD+ycX2NBM67Q5RIzKo0Qqd6OMHb/1ZYaZ 2z7KeRNorNrXS4zsA77B86+GUV/WU1gPo1bONbWNER6ZyB0KmK59ANxpU78ihsdJCMXp kFbYyTatv9rOnl6zbjoZtbh1wPmaAHk897n75XW40xtZ4IPWaiYKanc27KWzA9amFzn0 qmsgmr3Dfr0pqLAhlDiWie5n2rjW0yIhqLIhmbNVp7Vno/umaGz2zY4kPYZbc6IZCOMD NaCg== X-Gm-Message-State: AOJu0Yyzc2YFtTw0P0Tj93jBSoZOjkK0fkbPeFy6223M0Zz5Tm8SJEJM ukTlU4JDlYxeLNcm9tOIQJsK/mcvXFradA== X-Received: by 2002:a05:620a:1a1f:b0:775:7921:732e with SMTP id bk31-20020a05620a1a1f00b007757921732emr13330799qkb.3.1696278326374; Mon, 02 Oct 2023 13:25:26 -0700 (PDT) Received: from gauss.local (c-68-56-149-176.hsd1.mi.comcast.net. [68.56.149.176]) by smtp.gmail.com with ESMTPSA id x18-20020a05620a099200b00767da10efb6sm9020145qkx.97.2023.10.02.13.25.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Oct 2023 13:25:25 -0700 (PDT) From: Leo Izen To: ffmpeg-devel@ffmpeg.org Date: Mon, 2 Oct 2023 16:25:23 -0400 Message-ID: <20231002202523.148560-1-leo.izen@gmail.com> X-Mailer: git-send-email 2.42.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avcodec/jpegxl_parser: fix various memory issues X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , Leo Izen Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: OPQ7eNGXsHr9 The spec caps the prefix alphabet size to 32768 (i.e. 1 << 15) so we need to check for that and reject alphabets that are too large. Additionally, there's no need to allocate buffers that are as large as the maximum alphabet size as these aren't stack-allocated, they're heap allocated and thus can be variable size. Added an overflow check as well, which fixes leaking the buffer, and capping the alphabet size fixes two potential overruns as well. Fixes: out of array access Fixes: 62089/clusterfuzz-testcase-minimized-ffmpeg_DEMUXER_fuzzer- 5437089094959104.fuzz Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Found-by: Hardik Shah of Vehere (Dawn Treaders team) Co-authored-by: Michael Niedermayer Signed-off-by: Leo Izen --- libavcodec/jpegxl_parser.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/libavcodec/jpegxl_parser.c b/libavcodec/jpegxl_parser.c index d25a1b6e1d..51af0f4ed1 100644 --- a/libavcodec/jpegxl_parser.c +++ b/libavcodec/jpegxl_parser.c @@ -46,6 +46,8 @@ #define JXL_FLAG_USE_LF_FRAME 32 #define JXL_FLAG_SKIP_ADAPTIVE_LF_SMOOTH 128 +#define MAX_PREFIX_ALPHABET_SIZE (1u << 15) + #define clog1p(x) (ff_log2(x) + !!(x)) #define unpack_signed(x) (((x) & 1 ? -(x)-1 : (x))/2) #define div_ceil(x, y) (((x) - 1) / (y) + 1) @@ -724,16 +726,17 @@ static int read_vlc_prefix(GetBitContext *gb, JXLEntropyDecoder *dec, JXLSymbolD if (ret < 0) goto end; - buf = av_calloc(1, 262148); // 32768 * 8 + 4 + buf = av_calloc(1, dist->alphabet_size * (2 * sizeof(int8_t) + sizeof(int16_t) + sizeof(uint32_t)) + + sizeof(uint32_t)); if (!buf) { ret = AVERROR(ENOMEM); goto end; } level2_lens = (int8_t *)buf; - level2_lens_s = (int8_t *)(buf + 32768); - level2_syms = (int16_t *)(buf + 65536); - level2_codecounts = (uint32_t *)(buf + 131072); + level2_lens_s = (int8_t *)(buf + dist->alphabet_size * sizeof(int8_t)); + level2_syms = (int16_t *)(buf + dist->alphabet_size * (2 * sizeof(int8_t))); + level2_codecounts = (uint32_t *)(buf + dist->alphabet_size * (2 * sizeof(int8_t) + sizeof(int16_t))); total_code = 0; for (int i = 0; i < dist->alphabet_size; i++) { @@ -742,6 +745,10 @@ static int read_vlc_prefix(GetBitContext *gb, JXLEntropyDecoder *dec, JXLSymbolD int extra = 3 + get_bits(gb, 2); if (repeat_count_prev) extra = 4 * (repeat_count_prev - 2) - repeat_count_prev + extra; + if (i + extra > dist->alphabet_size) { + ret = AVERROR_INVALIDDATA; + goto end; + } for (int j = 0; j < extra; j++) level2_lens[i + j] = prev; total_code += (32768 >> prev) * extra; @@ -772,8 +779,10 @@ static int read_vlc_prefix(GetBitContext *gb, JXLEntropyDecoder *dec, JXLSymbolD } } - if (total_code != 32768 && level2_codecounts[0] < dist->alphabet_size - 1) - return AVERROR_INVALIDDATA; + if (total_code != 32768 && level2_codecounts[0] < dist->alphabet_size - 1) { + ret = AVERROR_INVALIDDATA; + goto end; + } for (int i = 1; i < dist->alphabet_size + 1; i++) level2_codecounts[i] += level2_codecounts[i - 1]; @@ -848,6 +857,8 @@ static int read_distribution_bundle(GetBitContext *gb, JXLEntropyDecoder *dec, if (get_bits1(gb)) { int n = get_bits(gb, 4); dist->alphabet_size = 1 + (1 << n) + get_bitsz(gb, n); + if (dist->alphabet_size > MAX_PREFIX_ALPHABET_SIZE) + return AVERROR_INVALIDDATA; } else { dist->alphabet_size = 1; }