From patchwork Thu Aug 10 23:48:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pierre-Anthony Lemieux X-Patchwork-Id: 43200 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b1b:b0:130:ccc6:6c4b with SMTP id l27csp1006563pzh; Thu, 10 Aug 2023 16:49:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG9RUGNxH9lLHn0WbZtxqYAEWCxzRMbD/KlvAwp3PDwMTl1YCfACbv2zd8f9Bw+N7Z3O+Ik X-Received: by 2002:a05:6402:2806:b0:522:c226:34ea with SMTP id h6-20020a056402280600b00522c22634eamr4649033ede.7.1691711356815; Thu, 10 Aug 2023 16:49:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691711356; cv=none; d=google.com; s=arc-20160816; b=P6o2ZsFEEZgJYBul/hEqFBEsbkG/DyKcxF8xSvVrdeZcHCDcnuVQlvAqqCXP7xawbu sH/Rxx9ceZlr4htpPR4ZQ/heznVUWr04DjFjPNNjZOTm8DR44ukglYk8rbmXzLanyk0y 4XMptwoiZCLCul+mc/XpZhmHN76haNP0KGtS3MjF3OJ1D3pVgpWZzX1KrHrXlriaGFJh Y68P/wrtE8GsY1Wzb/6JmXKXK2JVtDXfl/IirMIwvjdikQx430jLXeHSxnxc77+mgBgw dV4nJVQDg9dPKmYHUsUz9u6v3PxI8AsEds1wvLoFlWjTiLKIRVkIInBlUisuW9YUGLqT rqyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=CR6ucGGU05FoEykgW9p47kbCmD5QepAEguIEqmbhG8I=; fh=4+6/4NzGOEepYtDkjx95X4pwwYTZ6KID5XoAxbSJl+M=; b=F8hc7kBuOnqV/NK4fk0EeepFBID0IkXrVR6XCUpOGlTsG64ACRjdI/twE9Wcp+UvOC ltG1hY9GByas+jTGTombRWwonRfvx91JcGwEp3PiXeUzA5CZuZL/mHmCyBSD8CQZnOQU oy7kT5csCxDykW2ZKkOECviK+/nTgFp2BGuvivpUWrL6nkN2GLaRAhIunF2ZSxkgXzQ4 TIPQSBsqHXxLmGq/2RZSwuZ6/c0gDSSeJjn/QLqRDnp8jQteLZgu1ScvBjJfxscylFNg 7+aDNvyYnUXRgfOF5DlefAvKlZdirU7j7z+Jt8othZemqqV/EQiIVnqDmOq6teOD2P8z 59Vg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z24-20020aa7cf98000000b005234f30c399si2339367edx.127.2023.08.10.16.49.16; Thu, 10 Aug 2023 16:49:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7B8EA68C8E1; Fri, 11 Aug 2023 02:49:12 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7A49668C789 for ; Fri, 11 Aug 2023 02:49:05 +0300 (EEST) Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2682e33509bso1001921a91.1 for ; Thu, 10 Aug 2023 16:49:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691711343; x=1692316143; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FGAgROnEX913BGP+cFkbGKCNJq4SCDfwWwmWPkbK6X8=; b=M1TReZwGtHnXtWDd6zsXZNtludit+a73RIOKGpf8oyEOOdpZvRbm/l/P+O6Y7Qz3JL PC/6SeE2G3qDewT+KFO5+XpCl+6akc0IqxdXcKkZa3lOTg0d0G8OvlbGZv1QhCMvIN3J ei3xgyn29aSGrzC8e33IzEQ74z3xZSGqVwLZgmZxMLmsp/QTRJB4rr2aRAhecTb6HjVl xRwtMuu8NcafcGlF2brBR9nVgYLgNhQx813JlpdnExz2CEdU2ZHa9w2kXOKYatR2cHSz mS0B3YT3iT7jGq/OzdwVPvX4v/uOb/QAmQQ6V1auvz29dR4sEDvtSDMIc1mURllPDLEk 1XKA== X-Gm-Message-State: AOJu0YxA1GC+Wucsqd4VKRQU5ooZbtB6pzWlqeCgS+hYIIzeRGXPUn7C 0vPbxc+5lLNnXxNETVzL1FVQaYbxCv4= X-Received: by 2002:a17:90b:1884:b0:263:311f:9bcc with SMTP id mn4-20020a17090b188400b00263311f9bccmr93614pjb.35.1691711342544; Thu, 10 Aug 2023 16:49:02 -0700 (PDT) Received: from localhost (76-14-89-2.sf-cable.astound.net. [76.14.89.2]) by smtp.gmail.com with ESMTPSA id p14-20020a17090adf8e00b0026971450601sm2041721pjv.7.2023.08.10.16.49.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Aug 2023 16:49:01 -0700 (PDT) Received: by localhost (sSMTP sendmail emulation); Thu, 10 Aug 2023 16:48:58 -0700 From: pal@sandflow.com To: ffmpeg-devel@ffmpeg.org Date: Thu, 10 Aug 2023 16:48:56 -0700 Message-Id: <20230810234856.2636-1-pal@sandflow.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v1] avcodec/jpeg2000htdec: check if block decoding will exceed internal precision X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Pierre-Anthony Lemieux Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: BKu0psh1iogd From: Pierre-Anthony Lemieux Intended to replace https://patchwork.ffmpeg.org/project/ffmpeg/patch/20230802000135.26482-3-michael@niedermayer.cc/ with a more accurate block decoding magnitude bound. --- libavcodec/jpeg2000htdec.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/libavcodec/jpeg2000htdec.c b/libavcodec/jpeg2000htdec.c index 2c4cea5dd9..713dba17e3 100644 --- a/libavcodec/jpeg2000htdec.c +++ b/libavcodec/jpeg2000htdec.c @@ -572,6 +572,14 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, const uint8_t *vlc_buf = Dcup + Pcup; + /* + * Bound on the recision needed to process the codeblock. The number of + * decoded bit planes is equal to at most cblk->zbp + 2 since S_blk = P if + * there are no placeholder passes or HT Sets and P = cblk->zbp. See Rec. + * ITU-T T.814, 7.6. + */ + int maxbp = cblk->zbp + 2; + /* convert to raster-scan */ const uint16_t is_border_x = width % 2; const uint16_t is_border_y = height % 2; @@ -590,6 +598,10 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, goto free; } + /* do we have enough precision, assuming a 32-bit decoding path */ + if (maxbp >= 32) + return AVERROR_INVALIDDATA; + sigma = sigma_n; mu = mu_n; @@ -676,6 +688,10 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, } U[J2K_Q1] = kappa[J2K_Q1] + u[J2K_Q1]; U[J2K_Q2] = kappa[J2K_Q2] + u[J2K_Q2]; + if (U[J2K_Q1] > maxbp || U[J2K_Q2] > maxbp) { + ret = AVERROR_INVALIDDATA; + goto free; + } for (int i = 0; i < 4; i++) { m[J2K_Q1][i] = sigma_n[4 * q1 + i] * U[J2K_Q1] - ((emb_pat_k[J2K_Q1] >> i) & 1); @@ -713,6 +729,10 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, } U[J2K_Q1] = kappa[J2K_Q1] + u[J2K_Q1]; + if (U[J2K_Q1] > maxbp) { + ret = AVERROR_INVALIDDATA; + goto free; + } for (int i = 0; i < 4; i++) m[J2K_Q1][i] = sigma_n[4 * q1 + i] * U[J2K_Q1] - ((emb_pat_k[J2K_Q1] >> i) & 1); @@ -842,6 +862,10 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, U[J2K_Q1] = kappa[J2K_Q1] + u[J2K_Q1]; U[J2K_Q2] = kappa[J2K_Q2] + u[J2K_Q2]; + if (U[J2K_Q1] > maxbp || U[J2K_Q2] > maxbp) { + ret = AVERROR_INVALIDDATA; + goto free; + } for (int i = 0; i < 4; i++) { m[J2K_Q1][i] = sigma_n[4 * q1 + i] * U[J2K_Q1] - ((emb_pat_k[J2K_Q1] >> i) & 1); @@ -910,6 +934,10 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, kappa[J2K_Q1] = FFMAX(1, gamma[J2K_Q1] * (max_e[J2K_Q1] - 1)); U[J2K_Q1] = kappa[J2K_Q1] + u[J2K_Q1]; + if (U[J2K_Q1] > maxbp) { + ret = AVERROR_INVALIDDATA; + goto free; + } for (int i = 0; i < 4; i++) m[J2K_Q1][i] = sigma_n[4 * q1 + i] * U[J2K_Q1] - ((emb_pat_k[J2K_Q1] >> i) & 1); @@ -1238,8 +1266,10 @@ ff_jpeg2000_decode_htj2k(const Jpeg2000DecoderContext *s, Jpeg2000CodingStyle *c } if ((ret = jpeg2000_decode_ht_cleanup_segment(s, cblk, t1, &mel_state, &mel, &vlc, &mag_sgn, Dcup, Lcup, Pcup, pLSB, width, - height, sample_buf, block_states)) < 0) + height, sample_buf, block_states)) < 0) { + av_log(s->avctx, AV_LOG_ERROR, "Bad HT cleanup segment\n"); goto free; + } if (cblk->npasses > 1) jpeg2000_decode_sigprop_segment(cblk, width, height, Dref, Lref,