From patchwork Sun May 19 13:27:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nuo Mi X-Patchwork-Id: 49022 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp3927650pzb; Sun, 19 May 2024 06:50:09 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVFK8Rkw6/yLOK+MvAHSGeUDXQMOZDidTZO65m8t6kmIWLucmC00Y+hSnzJ9fkiBUfmIh0ugAp1dNILxlAsGHtHrXBm9Z3jjLBCHA== X-Google-Smtp-Source: AGHT+IHI0OAyeG4eR00hRXCcLsqpqZGj4QlEtHI69Z1rzrJGlXqOSrQIzwUWWfriRe3ztyw/BaMC X-Received: by 2002:a2e:744:0:b0:2e7:1b8:7b77 with SMTP id 38308e7fff4ca-2e701b87d03mr55471071fa.22.1716126608698; Sun, 19 May 2024 06:50:08 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 38308e7fff4ca-2e4d1949d86si68227821fa.603.2024.05.19.06.50.08; Sun, 19 May 2024 06:50:08 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@outlook.com header.s=selector1 header.b=cmJd9sGl; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AB6B068D222; Sun, 19 May 2024 16:40:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from SINPR02CU002.outbound.protection.outlook.com (mail-southeastasiaazolkn19011003.outbound.protection.outlook.com [52.103.65.3]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 74DD568D011 for ; Sun, 19 May 2024 16:39:57 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=c9VI9OyOykkUlrWN0njdh8yNAbJUC2hyUp50Stny8uxPSqPVxgOcp7wku5LAd5YtxPYJ8/zR80LYgpDcHm6VPiHuzhz5eLWOGSorsMDakKshrHFnO96jy+efoMhyTuF1IqQvbYJW8iE29SiUcN1YNBv1bS7ggDEKVXF23h0bHM7A2VXDje83Ifg/cyW7mWWMRMpVhlAsnB2F0PTqTyd/oqevL5+AUZ3FtroxFOlmZG6Liq5Rd9sLZzH+2mVWfxrAugEeUadL58TRaD/9W4PvCk5z2nWDyTEEpaNi5xDe1mxCdRhPw5oP+DA/Ksz/kBDJpPe7KyWACW+E2WaCw1LNeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DzL2llognoFT2O/u0P+aY+5BsRt8YhLqOEDAfRWNTGQ=; b=M7kq9f3GyIzdc2FnTx1+wJUsWzAg8D4cni6P0ziWul6YgIbExGjK91g13G+vnsmVeyO3jwqu5ZiDuVbgB87za8CfkJlGpDX/lXdr5zppASayycZfZkZcdwY466Y7lb7FLwVbwd1GVTpgDRl3SFrLWR0/gKT/CBJBFIHoJxViLBUua2Yjnzu5n67v/3uqe9Awaj30TrDbRzaDYmLHLlxNXu1nE1VrOujSILraraUc9dBugxZe5N1gIe/8tocVPIg8jiXT8prD5UU6Vw5Z1QXLWLT4nY4SPBcmtroIcK5IBITvbrUEkHyfB1VZxaZL6mu+gBBcPoOJDVmV1aBiaRdtfA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DzL2llognoFT2O/u0P+aY+5BsRt8YhLqOEDAfRWNTGQ=; b=cmJd9sGlDfqxNlJsbdxFQAuc8afE8iC08xUQODyiu2J4RasnZqoTfzGtSnkzEtmzzy/PimIwxDKzg4u7bUfgUxdGhym3e7uE8SgItFvecV6iIWiGqRYFqJY491sCxR0jPam1KujoTE4EtNsoBIGHUy/63hjJzcaQygrkJhES2DSALN+d1Wbak45r7VzuIjY5BmSZzvfVUvQi+QG9ecd1+bdjwdGORJ6fWAquYtvD/7j9afZ7/UA4T232e4aJIJbENsHxf2lKLFUlM004U5S8Oum0mEvry/SzrwnX2Q4peIo7jfFo3BIo0rhG6SHslx3QEndEbtdIGvLPH51F6yjyXA== Received: from TYSPR06MB6433.apcprd06.prod.outlook.com (2603:1096:400:47a::6) by TYUPR06MB6098.apcprd06.prod.outlook.com (2603:1096:400:358::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.34; Sun, 19 May 2024 13:39:41 +0000 Received: from TYSPR06MB6433.apcprd06.prod.outlook.com ([fe80::81f7:9125:583a:1cca]) by TYSPR06MB6433.apcprd06.prod.outlook.com ([fe80::81f7:9125:583a:1cca%4]) with mapi id 15.20.7587.030; Sun, 19 May 2024 13:39:41 +0000 From: Nuo Mi To: ffmpeg-devel@ffmpeg.org Date: Sun, 19 May 2024 21:27:43 +0800 Message-ID: X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240519132749.790832-1-nuomi2021@gmail.com> References: <20240519132749.790832-1-nuomi2021@gmail.com> X-TMN: [nOnqiX+Cs0Ma10W75O+qyHZZz9v9OX0N] X-ClientProxiedBy: TYCP301CA0017.JPNP301.PROD.OUTLOOK.COM (2603:1096:400:381::8) To TYSPR06MB6433.apcprd06.prod.outlook.com (2603:1096:400:47a::6) X-Microsoft-Original-Message-ID: <20240519132749.790832-12-nuomi2021@gmail.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 2 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: TYSPR06MB6433:EE_|TYUPR06MB6098:EE_ X-MS-Office365-Filtering-Correlation-Id: ffa269c9-c5d6-4eed-e8bb-08dc780922eb X-Microsoft-Antispam: BCL:0; ARA:14566002|461199019|440099019|3412199016|1710799017; X-Microsoft-Antispam-Message-Info: yW924WwlruaHOdaZx5GhFFE+aqW4eUYdQWLejHTbY893KqX3VqpVtRj+SZC54e0xmS3AJ1IUZbVleBWkhkFFB1Phu+Yq5CGI9/I1rw66dCzpvDhEVaLTbLQv+l+GztTw8zNnLwyjFicAtFQdkfwVqFL5yQV0v2+CdgUctjI6g07PKFr3U+6wLY6t9y0UwM5Wy5Lqtj8m1o4u3yOoMxI5twhhAvGk+aG3nsUxxrAZnG21pYkOn+gXYGLSNW2fJBGguK/O0KUJREfoPNBtON+rBl3MuB3zhQRLA4ll3YvXZEDs31cpWxOPg6bTs3nwcnHLx+pBTY8qFSJqdFcTY3O6c6VgNvT/WG7S+I7pmYGmmwrKxzr248bzTyYKoSBJEzdhdxEN9FRC7wF/J+eqS2ByxuFFDHMhorMyv1pa5FOOn5q+JNzme05EEDPKb6vM5VegXF+6NuVg+5hDws4tU2lWBwtf6Hng0NxKitvTwzA/s/WrbhS2n3nTtx3veTBLKE/tbaLYuoM7h5DHMqBwPweDj7I2uGCJKswaywVaJLMDwElA7Qk/suBhu+0MwXYgD3Pagiqmab/6uSwsswvSg0lELpyYbFUJyFTvFsFCGXZAg1Ljx1mwcJEP7SQCGud8YJke X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: avVNNR1JMEdiOlFWv19fcbsityES+ehEFluEym3Ut/zQllKOkiersI29B9NjTvmcIH6aIfAwvrUnJk5h6o77u9ycc1cPP1szPLJyQMJsFWzV3HUTN+tDF+RBtnD06va8Mrfx0tDuffa24JMnDEm001Ch5jF4XiOGuat1e9WsrobBB26o+GqeMxfJqWx1xWZ0VSiUCksjcYRbJTShGKv7+CFkW7JIn6Nt+LW6okYV5p4L/olj75fLaUmmsUuq9Zm3x+j0OPbl0rYvu4oksfMBTNR8IvbA+tF+O8LvZ9AWwUbd1EFhY0SNO9Su4kJvR6KRpldvRElZC5+EmFrWDdtQdTiAW/khKFLChX0n/qVqqAgClMONbBsLWyWJSTUSOVu1K5MviTdvJL3r1TEEEGIQJsXd0L6i2Tp+l8jeHJ6tv9+7yOteBe47qDoVlW0vRDrYygPAShPMtAQnkGFEIl+65jzeX/W4D7bioBTXRQNCw8fZSAlp/6sJtFjvysFKRkcH8usFixvAH7EjfzWNFVseXYxHPwgpN6FyBHYyUo6u7WYDUtBjKeQj/2r3cayBSdMXkaMAYiss0+vdbiOji7KoxG49KdZPgoaPy8eyZtTQUxRSJKYQxxyVV9/aNKeHc8n3GpgtWyl4j4NUv6zn5g0IGwYeCn+1k0qIwhEAuhjSZfXmOf9E4q4Gi5oq+gG6BMIvzNNueIsJq2gYbZx7r3zIJJ/vGAjPL8GKJur3Xw6FRu7AOIHGhrjAp5DRQeP9ydnahPwVwH9y/9pb8C+tTuQr4JQjwP4wLNyHxhNPrzck7X5wH+O0MFNOM7MA9KP+zeAPiWzLHnCZjYLRZ4EmAX34um/y0WSkGeV/Hm0cStWbqjezUGw2Bii1zgLJvAiN/aOxTcDF8sEkbym0xqXS/raX+D0L+rOesqEc4TqUAbfRs1CXBz26u3uhOUgVCI4tirVn+H+MyjWQJUNltQk+SUJtw7ypYoDfRUPhz+Ihkq6ZatqnqlKTCufN7v5iONV7DolNv1NZ+kSipQ2n5vj2aNuH76sGIP2IGHF1XsjTl8e/Iefoq0kYAjnWG63uq0blblOlm1WRIVb7RyQ5kTGALzdQ5rG+h7PpDcXkzZR2/u2yTN4OvAiex6LgoNw+wE7FGp9INB4/FWgEI365YgD07XuDaOb5djZ5cw3CzARAKCnnHuDizaJRLru2vK7nrB/Hng+z4NwAuDdB3e/uhLBHCwMb0SKnWmA7eE5tUU86HUfLGvMoB1DTXQbPkqtH3WwC4pZ1Mi2H+ISZdjpj/lEOyjxQjw== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: ffa269c9-c5d6-4eed-e8bb-08dc780922eb X-MS-Exchange-CrossTenant-AuthSource: TYSPR06MB6433.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 May 2024 13:39:41.7856 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYUPR06MB6098 Subject: [FFmpeg-devel] [PATCH 12/18] avcodec/vvcdec: inter, wait reference with a diffrent resolution X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Nuo Mi Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 9H+3j5q7phXC For RPR, the current frame may reference a frame with a different resolution. Therefore, we need to consider frame scaling when we wait for reference pixels. --- libavcodec/vvc/dec.c | 5 +++++ libavcodec/vvc/dec.h | 17 +++++++++++++++++ libavcodec/vvc/refs.c | 39 +++++++++++++++++++++++++++++++++++++++ libavcodec/vvc/thread.c | 10 +++++++--- 4 files changed, 68 insertions(+), 3 deletions(-) diff --git a/libavcodec/vvc/dec.c b/libavcodec/vvc/dec.c index 3325133efb..584c7b219f 100644 --- a/libavcodec/vvc/dec.c +++ b/libavcodec/vvc/dec.c @@ -573,6 +573,11 @@ static int ref_frame(VVCFrame *dst, const VVCFrame *src) dst->poc = src->poc; dst->ctb_count = src->ctb_count; + + dst->scaling_win = src->scaling_win; + dst->ref_width = src->ref_width; + dst->ref_height = src->ref_height; + dst->flags = src->flags; dst->sequence = src->sequence; diff --git a/libavcodec/vvc/dec.h b/libavcodec/vvc/dec.h index 6f14cc1860..1e0b76f283 100644 --- a/libavcodec/vvc/dec.h +++ b/libavcodec/vvc/dec.h @@ -46,6 +46,10 @@ typedef struct VVCRefPic { struct VVCFrame *ref; int poc; int is_lt; // is long term reference + + // for RPR + int is_scaled; ///< RprConstraintsActiveFlag + int scale[2]; ///< RefPicScale[] } VVCRefPic; typedef struct RefPicList { @@ -57,6 +61,13 @@ typedef struct RefPicListTab { RefPicList refPicList[2]; } RefPicListTab; +typedef struct VVCWindow { + int16_t left_offset; + int16_t right_offset; + int16_t top_offset; + int16_t bottom_offset; +} VVCWindow; + typedef struct VVCFrame { struct AVFrame *frame; @@ -71,6 +82,12 @@ typedef struct VVCFrame { int poc; + //for RPR + VVCWindow scaling_win; ///< pps_scaling_win_left_offset * SubWithC, pps_scaling_win_right_offset * SubWithC, + ///< pps_scaling_win_top_offset * SubHeigtC, pps_scaling_win_bottom_offset * SubHiehgtC + int ref_width; ///< CurrPicScalWinWidthL + int ref_height; ///< CurrPicScalWinHeightL + struct VVCFrame *collocated_ref; struct FrameProgress *progress; ///< RefStruct reference diff --git a/libavcodec/vvc/refs.c b/libavcodec/vvc/refs.c index 954db4a8c8..fb42963034 100644 --- a/libavcodec/vvc/refs.c +++ b/libavcodec/vvc/refs.c @@ -114,10 +114,12 @@ static FrameProgress *alloc_progress(void) static VVCFrame *alloc_frame(VVCContext *s, VVCFrameContext *fc) { + const VVCSPS *sps = fc->ps.sps; const VVCPPS *pps = fc->ps.pps; for (int i = 0; i < FF_ARRAY_ELEMS(fc->DPB); i++) { int ret; VVCFrame *frame = &fc->DPB[i]; + VVCWindow *win = &frame->scaling_win; if (frame->frame->buf[0]) continue; @@ -144,6 +146,13 @@ static VVCFrame *alloc_frame(VVCContext *s, VVCFrameContext *fc) for (int j = 0; j < frame->ctb_count; j++) frame->rpl_tab[j] = frame->rpl; + win->left_offset = pps->r->pps_scaling_win_left_offset << sps->hshift[CHROMA]; + win->right_offset = pps->r->pps_scaling_win_right_offset << sps->hshift[CHROMA]; + win->top_offset = pps->r->pps_scaling_win_top_offset << sps->vshift[CHROMA]; + win->bottom_offset = pps->r->pps_scaling_win_bottom_offset << sps->vshift[CHROMA]; + frame->ref_width = pps->r->pps_pic_width_in_luma_samples - win->left_offset - win->right_offset; + frame->ref_height = pps->r->pps_pic_height_in_luma_samples - win->bottom_offset - win->top_offset; + frame->progress = alloc_progress(); if (!frame->progress) goto fail; @@ -353,6 +362,24 @@ static VVCFrame *generate_missing_ref(VVCContext *s, VVCFrameContext *fc, int po return frame; } +#define CHECK_MAX(d) (frame->ref_##d * frame->sps->r->sps_pic_##d##_max_in_luma_samples >= ref->ref_##d * (frame->pps->r->pps_pic_##d##_in_luma_samples - max)) +#define CHECK_SAMPLES(d) (frame->pps->r->pps_pic_##d##_in_luma_samples == ref->pps->r->pps_pic_##d##_in_luma_samples) +static int check_candidate_ref(const VVCFrame *frame, const VVCRefPic *refp) +{ + const VVCFrame *ref = refp->ref; + + if (refp->is_scaled) { + const int max = FFMAX(8, frame->sps->min_cb_size_y); + return frame->ref_width * 2 >= ref->ref_width && + frame->ref_height * 2 >= ref->ref_height && + frame->ref_width <= ref->ref_width * 8 && + frame->ref_height <= ref->ref_height * 8 && + CHECK_MAX(width) && CHECK_MAX(height); + } + return CHECK_SAMPLES(width) && CHECK_SAMPLES(height); +} + +#define RPR_SCALE(f) (((ref->f << 14) + (fc->ref->f >> 1)) / fc->ref->f) /* add a reference with the given poc to the list and mark it as used in DPB */ static int add_candidate_ref(VVCContext *s, VVCFrameContext *fc, RefPicList *list, int poc, int ref_flag, uint8_t use_msb) @@ -372,6 +399,18 @@ static int add_candidate_ref(VVCContext *s, VVCFrameContext *fc, RefPicList *lis refp->poc = poc; refp->ref = ref; refp->is_lt = ref_flag & VVC_FRAME_FLAG_LONG_REF; + refp->is_scaled = ref->sps->r->sps_num_subpics_minus1 != fc->ref->sps->r->sps_num_subpics_minus1|| + memcmp(&ref->scaling_win, &fc->ref->scaling_win, sizeof(ref->scaling_win)) || + ref->pps->r->pps_pic_width_in_luma_samples != fc->ref->pps->r->pps_pic_width_in_luma_samples || + ref->pps->r->pps_pic_height_in_luma_samples != fc->ref->pps->r->pps_pic_height_in_luma_samples; + + if (!check_candidate_ref(fc->ref, refp)) + return AVERROR_INVALIDDATA; + + if (refp->is_scaled) { + refp->scale[0] = RPR_SCALE(ref_width); + refp->scale[1] = RPR_SCALE(ref_height); + } list->nb_refs++; mark_ref(ref, ref_flag); diff --git a/libavcodec/vvc/thread.c b/libavcodec/vvc/thread.c index 2654b40058..8777d380bf 100644 --- a/libavcodec/vvc/thread.c +++ b/libavcodec/vvc/thread.c @@ -293,10 +293,14 @@ static void schedule_inter(VVCContext *s, VVCFrameContext *fc, const SliceContex CTU *ctu = fc->tab.ctus + rs; for (int lx = 0; lx < 2; lx++) { for (int i = 0; i < sh->r->num_ref_idx_active[lx]; i++) { - const int y = ctu->max_y[lx][i]; - VVCFrame *ref = sc->rpl[lx].refs[i].ref; - if (ref && y >= 0) + int y = ctu->max_y[lx][i]; + VVCRefPic *refp = sc->rpl[lx].refs + i; + VVCFrame *ref = refp->ref; + if (ref && y >= 0) { + if (refp->is_scaled) + y = y * refp->scale[1] >> 14; add_progress_listener(ref, &t->listener[lx][i], t, s, VVC_PROGRESS_PIXEL, y + LUMA_EXTRA_AFTER); + } } } }