From patchwork Sat Aug 12 20:31:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pierre-Anthony Lemieux X-Patchwork-Id: 43218 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6aa0:b0:130:ccc6:6c4b with SMTP id bi32csp529531pzb; Sat, 12 Aug 2023 13:31:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGs4Gd6xjiE9l16f3RjRSLZgZGyBH2gSumwIfIEcDE1xOGf7WetEhutNgQZDX9YFCeZO5Zx X-Received: by 2002:a17:906:23e2:b0:99c:331:4194 with SMTP id j2-20020a17090623e200b0099c03314194mr4637079ejg.45.1691872298987; Sat, 12 Aug 2023 13:31:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691872298; cv=none; d=google.com; s=arc-20160816; b=VeGVbd+wUK8JbbHlR6HUs8I64uJOVnZ5/lovotwPYLLJZf2s5Nr4BH9PUeUpXpK1yV ZoRCI03Ho5Ycwa5AaxDPJG7zLIQT3R8D/jqxCLOZijUpF7+2ABDxoh266IGNIKoFf2jZ w6fkEK10vwdPZyRTJv6m4PDqZQsdfoQeuy2GeOHcbmSaVhX3SJONZnmA7Djw9I3oIM5j w96t+a+8gXxTsCn87BvSOj3iLL112r18RaDCUj/ddOqQnW8+54+j0OZN2dD9EJewjSh/ zOJyeBnLgSE2QYa6vbjPdk+asiCEHVrBzpvem0CON+Dq++gRAftM6/T8xhFNVXrfA7Sd 4u2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=SJGYswBft96ItVeL4Gs7bzKxMsD2LGUvvzeUXJ8Fo2A=; fh=4+6/4NzGOEepYtDkjx95X4pwwYTZ6KID5XoAxbSJl+M=; b=mHjKavOCLB68O0mSd+bIEinW+QXE3qrC6rkoPa5qlYvCqVbnDdAzy130BZj4ynGKJY mAF7RxrgNQbTYEQKbH8T4NQ4IKCHcm+7RjS4vP0N6FLVOc6VkQz67ZvlxKLoAkzHgC// G8kg5uIQztjMBUORV6TQcw1wMoxQ75oDUEbI/0ld4M6Fn+IiyEGMmsu33+sBqscmNQAL +9cZ+2ZiN6PWSqFshbyldUR7AabgUU+5f0q3VJ9zSRfGXg7+o85zyS5+zT2D2EITKct5 p2EWzDfHIH9BCBIHh7oHIMClv9DEBs0/J3aQNHrI20Lc2zetTuJglY51sEIpd0OsmbC0 4/1Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id k19-20020a17090627d300b0098d7e44a637si5463322ejc.794.2023.08.12.13.31.36; Sat, 12 Aug 2023 13:31:38 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7C62868C5E7; Sat, 12 Aug 2023 23:31:32 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 942CA68C495 for ; Sat, 12 Aug 2023 23:31:25 +0300 (EEST) Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1bc83a96067so20025645ad.0 for ; Sat, 12 Aug 2023 13:31:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691872284; x=1692477084; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ejZs/r+lX+4jQ80/6W03YFrKctUzr9qvh94AWCWbo90=; b=VG1rC1MfetOhfmu5q2PJlx7z4IQ0yLTEXRAqMBVkOihTAx50giFqoH9gl7QaurZ917 0uWieJR6p46ISaNAcEMmi05DlpOVD/2H2SIk1uIrVO4KGeqb6TCLTAtjWYeiK4lDzrzE 4/QfNKSRp/OjeDJmaj3R3t66Ty2jLVwOBso89MFyCJKjesyvGrEq8sOcbsR0xsIRloqv kuyDaR3HVJe6j0cLy2U2X4AloDqqJ8b9FVC4Mz5NoIXyNu9r616W1K/2wL1aKuoNszo0 19shtXS6Isd0krewBPoBojfKCg4BWxt4raA3xWP+8rkPN+VlERPLbtMIXUrnA/LVIAjx EDdw== X-Gm-Message-State: AOJu0YzlBopaVS3BfcIzjoXNDZCyv3skKu69qhj4rj5CgroL/QYWr0CK 2cMQsCKLn6wrxKe4bJKhIgi3H2bHCh4= X-Received: by 2002:a17:902:ec92:b0:1b8:5a32:2345 with SMTP id x18-20020a170902ec9200b001b85a322345mr5352617plg.22.1691872283487; Sat, 12 Aug 2023 13:31:23 -0700 (PDT) Received: from localhost (76-14-89-2.sf-cable.astound.net. [76.14.89.2]) by smtp.gmail.com with ESMTPSA id jh3-20020a170903328300b001b866472e8dsm6299124plb.241.2023.08.12.13.31.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 12 Aug 2023 13:31:22 -0700 (PDT) Received: by localhost (sSMTP sendmail emulation); Sat, 12 Aug 2023 13:31:17 -0700 From: pal@sandflow.com To: ffmpeg-devel@ffmpeg.org Date: Sat, 12 Aug 2023 13:31:16 -0700 Message-Id: <20230812203116.31057-1-pal@sandflow.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2] avcodec/jpeg2000htdec: check if block decoding will exceed internal precision X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Pierre-Anthony Lemieux Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 38KHfBmB0OnJ From: Pierre-Anthony Lemieux Intended to replace https://patchwork.ffmpeg.org/project/ffmpeg/patch/20230802000135.26482-3-michael@niedermayer.cc/ with a more accurate block decoding magnitude bound. --- libavcodec/jpeg2000htdec.c | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/libavcodec/jpeg2000htdec.c b/libavcodec/jpeg2000htdec.c index 2c4cea5dd9..a7329206de 100644 --- a/libavcodec/jpeg2000htdec.c +++ b/libavcodec/jpeg2000htdec.c @@ -572,6 +572,14 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, const uint8_t *vlc_buf = Dcup + Pcup; + /* + * Bound on the recision needed to process the codeblock. The number of + * decoded bit planes is equal to at most cblk->zbp + 2 since S_blk = P if + * there are no placeholder passes or HT Sets and P = cblk->zbp. See Rec. + * ITU-T T.814, 7.6. + */ + int maxbp = cblk->zbp + 2; + /* convert to raster-scan */ const uint16_t is_border_x = width % 2; const uint16_t is_border_y = height % 2; @@ -590,6 +598,12 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, goto free; } + /* do we have enough precision, assuming a 32-bit decoding path */ + if (maxbp >= 32) { + return AVERROR_INVALIDDATA; + goto free; + } + sigma = sigma_n; mu = mu_n; @@ -676,6 +690,10 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, } U[J2K_Q1] = kappa[J2K_Q1] + u[J2K_Q1]; U[J2K_Q2] = kappa[J2K_Q2] + u[J2K_Q2]; + if (U[J2K_Q1] > maxbp || U[J2K_Q2] > maxbp) { + ret = AVERROR_INVALIDDATA; + goto free; + } for (int i = 0; i < 4; i++) { m[J2K_Q1][i] = sigma_n[4 * q1 + i] * U[J2K_Q1] - ((emb_pat_k[J2K_Q1] >> i) & 1); @@ -713,6 +731,10 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, } U[J2K_Q1] = kappa[J2K_Q1] + u[J2K_Q1]; + if (U[J2K_Q1] > maxbp) { + ret = AVERROR_INVALIDDATA; + goto free; + } for (int i = 0; i < 4; i++) m[J2K_Q1][i] = sigma_n[4 * q1 + i] * U[J2K_Q1] - ((emb_pat_k[J2K_Q1] >> i) & 1); @@ -842,6 +864,10 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, U[J2K_Q1] = kappa[J2K_Q1] + u[J2K_Q1]; U[J2K_Q2] = kappa[J2K_Q2] + u[J2K_Q2]; + if (U[J2K_Q1] > maxbp || U[J2K_Q2] > maxbp) { + ret = AVERROR_INVALIDDATA; + goto free; + } for (int i = 0; i < 4; i++) { m[J2K_Q1][i] = sigma_n[4 * q1 + i] * U[J2K_Q1] - ((emb_pat_k[J2K_Q1] >> i) & 1); @@ -910,6 +936,10 @@ static int jpeg2000_decode_ht_cleanup_segment(const Jpeg2000DecoderContext *s, kappa[J2K_Q1] = FFMAX(1, gamma[J2K_Q1] * (max_e[J2K_Q1] - 1)); U[J2K_Q1] = kappa[J2K_Q1] + u[J2K_Q1]; + if (U[J2K_Q1] > maxbp) { + ret = AVERROR_INVALIDDATA; + goto free; + } for (int i = 0; i < 4; i++) m[J2K_Q1][i] = sigma_n[4 * q1 + i] * U[J2K_Q1] - ((emb_pat_k[J2K_Q1] >> i) & 1); @@ -1238,8 +1268,10 @@ ff_jpeg2000_decode_htj2k(const Jpeg2000DecoderContext *s, Jpeg2000CodingStyle *c } if ((ret = jpeg2000_decode_ht_cleanup_segment(s, cblk, t1, &mel_state, &mel, &vlc, &mag_sgn, Dcup, Lcup, Pcup, pLSB, width, - height, sample_buf, block_states)) < 0) + height, sample_buf, block_states)) < 0) { + av_log(s->avctx, AV_LOG_ERROR, "Bad HT cleanup segment\n"); goto free; + } if (cblk->npasses > 1) jpeg2000_decode_sigprop_segment(cblk, width, height, Dref, Lref,