From patchwork Tue Apr 2 21:48:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank Plowman X-Patchwork-Id: 47739 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:9f96:b0:1a3:b6bb:3029 with SMTP id mm22csp41807pzb; Tue, 2 Apr 2024 14:49:04 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCW4oQ/07sbOCt5kTw4SSg+rms+91gNlMjFnWPyFJDl9oIJ2aFZmLC0jVitXfI2TDaXWfzpHnUI5cDXWSHPXYDiXtXc7qLVitIiIKA== X-Google-Smtp-Source: AGHT+IFYtTNXaFv99Zc/W4XLSntudLFvdxukAL5bH+a6/2KwxjBzYvSEtj1Qnj9cTLVFsWZ8mz4m X-Received: by 2002:a17:907:11c5:b0:a4e:68f3:b14f with SMTP id va5-20020a17090711c500b00a4e68f3b14fmr603674ejb.74.1712094544568; Tue, 02 Apr 2024 14:49:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712094544; cv=none; d=google.com; s=arc-20160816; b=uhlHzyWmPTTXvgERIi2Hl1+b4PEBX+KkYcXvmECzgm0HAP5mP78gDHGZQrhXwquk2/ BuQCtwikbrX3+3+JGEUekzQDBlmLB3rSLnGEl/3WUOqTRKumz9oMkMENIH+wuVplgnvg DfZ/Ethso+NHTdP/QukrP+pdrHnQsylLTE4RiRH6mMGunAJHDoJwVcbf/Xns2cq0wMDn 8e8EROHazD0Znwk5f9UV3F/6NyCEeB1v55U+tcpMwVX+zHItX6vZOkZCcrM9hwsxk0Zq ulOOdHX0pGXmyhgRo6dP2qUgAzh0dVDr6vmPkjZoyN1L9UFWd1L2WhsjbZSnUuuDzj6G 3IoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:message-id:date:from :dkim-signature:delivered-to; bh=UFv7Ip4HYB7KdgLFdHd4VAQ8Ux3cV0aobGkHTfAewL0=; fh=GABYHefZpsCOOVRn8a1IgmYuOvaCu1oFlwEOjvaxWTE=; b=ptjUyn8CtMKJPSIRVVbt9rD48TiCChEstxkFXYkLwKlFXlKPWqo/CW8NPTsTNrWza5 91FgdNOARDuNM1gfuYHBB65Lt8EAAcHlDWOiq7MxXMmRFaCZI8/feV00UGsvVzhAhiXQ fP/5ieyKk5jzkvsTdkun7bFK72q6aKQzQmxnUGnDmIFWmjdSQ68Iz8vWcbll1WDA4cxD d2xcmwdHd2GLFjRctEsMgird0f24QNlok8MHnloq/AoZSiV4MpJKybO6P0dnGqXy21/b xaQmLsULEFrV3B9wHaant09+AS7P/bNiSkClmGMM9w2dCHBmPE84MZYBXIwU7cYqWidc /caQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@frankplowman.com header.s=s1 header.b="s7q+VWw/"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r20-20020a170906549400b00a4a33a2d861si6187458ejo.752.2024.04.02.14.49.04; Tue, 02 Apr 2024 14:49:04 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@frankplowman.com header.s=s1 header.b="s7q+VWw/"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7A9A568CFFC; Wed, 3 Apr 2024 00:49:01 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from wrqvqshf.outbound-mail.sendgrid.net (wrqvqshf.outbound-mail.sendgrid.net [149.72.70.15]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 75AD368CD83 for ; Wed, 3 Apr 2024 00:48:54 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=frankplowman.com; h=from:subject:mime-version:to:cc:content-transfer-encoding: content-type:cc:content-type:from:subject:to; s=s1; bh=L0MQ2dPlV34/EvOvr29Z5eXHdEp8niQd/yDY09ravjE=; b=s7q+VWw/WSo9tqmbBdiirMi+s6JjUemi7OJL3mXFztA7iIGk7wozcUNfNxLoUzy8hN++ V45RdQ4pfbP0CRlBIjJ0n6apuOfhe+IbMv/S3WZD85176Xkh8Lj4usOp8wp2IRCmdGg7KT cPkEHijgGOEueo4aZDSxULejB1m2dV/yanxymO3RfqYqiVBxToIWhrQ6jT6QKH9agFvmWV Qr8O2btKIdEyZP53nw/fb6Y1kFW/p0naoXtUPxux1rg2dvi0GgIgElIcaQA7ihi5tpt3Ht /S5Ue6OFQr5yJ6vSpsyWWjGGMXQPn76/EnZE8v3PG61Y7aiXdSmsQiErmv5hqODA== Received: by filterdrecv-86675bc4b9-2clcd with SMTP id filterdrecv-86675bc4b9-2clcd-1-660C7D44-9 2024-04-02 21:48:52.592506206 +0000 UTC m=+1300989.153488583 Received: from localhost.localdomain (unknown) by geopod-ismtpd-3 (SG) with ESMTP id c8Mm-crfSdGId839eDTPaA Tue, 02 Apr 2024 21:48:52.280 +0000 (UTC) From: Frank Plowman Date: Tue, 02 Apr 2024 21:48:52 +0000 (UTC) Message-ID: <20240402214843.474910-1-post@frankplowman.com> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 X-SG-EID: u001.Z0KJCHpts8tvDq7PHgz5cpqJ+vJcSmdTtST/g91WT3qAbNxUpEMGIDQq91mXy3Yjvepp8fYsFUJzGsyhT1EwBe2kwsjy2Y9DttckmWRt2u002kw9YwUj6KEKw7Wplkh4Xyln4MiMLKg6u5RfgFogzekaWjgBICiC2TwkGawmn+Y2rmRqYjafDe1I7vDMMKPo1IdLApCh/YB1XzcfD9lK/j0ErnXQNHyZoH9au+KTg8pEMMWZFDAZAPznePi88KEl To: ffmpeg-devel@ffmpeg.org X-Entity-ID: u001.qzljkbu34TNIX4NwfTiKWA== Subject: [FFmpeg-devel] [PATCH] lavc/vvc: Only read split_cu_flag if a split is allowed X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Frank Plowman Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: hfUSS1FJPxuw Add a check to ensure some split is possible before reading the split_cu_flag. This is present in the spec, in VVCv3 section 7.3.11.4. Its omission could lead to infinite loops and ultimately crashing due to stack overflow. --- libavcodec/vvc/vvc_ctu.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/libavcodec/vvc/vvc_ctu.c b/libavcodec/vvc/vvc_ctu.c index 8ba12c8d9f..32d8bc8f5c 100644 --- a/libavcodec/vvc/vvc_ctu.c +++ b/libavcodec/vvc/vvc_ctu.c @@ -2095,6 +2095,7 @@ static int hls_coding_tree(VVCLocalContext *lc, const int ch_type = tree_type_curr == DUAL_TREE_CHROMA; int ret; VVCAllowedSplit allowed; + int split_cu_flag; if (pps->r->pps_cu_qp_delta_enabled_flag && qg_on_y && cb_sub_div <= sh->cu_qp_delta_subdiv) { lc->parse.is_cu_qp_delta_coded = 0; @@ -2109,7 +2110,11 @@ static int hls_coding_tree(VVCLocalContext *lc, can_split(lc, x0, y0, cb_width, cb_height, mtt_depth, depth_offset, part_idx, last_split_mode, tree_type_curr, mode_type_curr, &allowed); - if (ff_vvc_split_cu_flag(lc, x0, y0, cb_width, cb_height, ch_type, &allowed)) { + if (allowed.btv || allowed.bth || allowed.ttv || allowed.tth || allowed.qt) + split_cu_flag = ff_vvc_split_cu_flag(lc, x0, y0, cb_width, cb_height, ch_type, &allowed); + else + split_cu_flag = 0; + if (split_cu_flag) { VVCSplitMode split = ff_vvc_split_mode(lc, x0, y0, cb_width, cb_height, cqt_depth, mtt_depth, ch_type, &allowed); VVCModeType mode_type = mode_type_decode(lc, x0, y0, cb_width, cb_height, split, ch_type, mode_type_curr);