From patchwork Wed Sep 25 02:15:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Fei W" X-Patchwork-Id: 51812 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:9548:0:b0:48e:c0f8:d0de with SMTP id t8csp190414vqk; Tue, 24 Sep 2024 19:34:20 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCX4MFYSzcEFimuo+uZEKbg1Y7dRriAHfy8saeIE6heVtZbZpW/wk3fPqnHsmIQnRMSQvmX7SWxyc6vb1/7VpYsl@gmail.com X-Google-Smtp-Source: AGHT+IFChM0p92ahEPqGg4VByyitXH9lc9sMrz6GdOxuJvJuiRJRIlPJf+JSW7vPMeteICH/yRre X-Received: by 2002:a05:6512:39d0:b0:52c:d819:517e with SMTP id 2adb3069b0e04-53877530ba0mr553029e87.30.1727231660160; Tue, 24 Sep 2024 19:34:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1727231660; cv=none; d=google.com; s=arc-20240605; b=e4ZM0+u2NSDcnnS2Dbv0MRBDioLCVC6jTsdSVO9jtXdpqpF0tM0X6zm08142HFhMew wnbBkDY2RlPWZ9cFVH5+5I9orTTfNMvpmRBGIauzGN5EXFLhtOA/K4pAWUc4UO71ZaZW 8xEXtGI0bgfNF4Yao6mEBnfvADqWoVx9t9cU6tL9i5miHBI0089gHx2Cke88AhRqVGpL sy9+BMwJf5bkeExOReav27E/wmJmqHJUYQyKiw9FB97KMKyHCkkoDT9ccGtuL9hBlrUO iuCC10ZqmprmbjHiNpaxj8W6wmxAPIKYrZQmwuZXqM6CTV5E8NyPuaRWaY8IicXFvBX4 7lvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=fEk4Bu6WlgtBVQ17HJjWGT4fVjo2vcEUxchcSu9t7qo=; fh=i4ESP4ZRFDcfYfwXKWpOXjc2YhmIGuOsCfZUnwNO0gc=; b=lCMBLvjDZaveRqI0PG6ZkZvVG2ewQMNZOs0a3/x5h9jLgAs1v+gEyCH/BRwUm1nmcm R1BPhspUNWVdWz89BQbxuelbqRp28ddnyjCERQlFMClCVDI8JDLD4y4og5DM4ypuYcJo UlIjS7S4AJ3QryVsLcU6h0yv96qnEbdoDKo85nacwsiAqAJZ7c8rlTumll7UFH8grEAU rHfz8TA/0YLO+QkPG4nsyaCkFTpbRHqEspHdxBihsTasi0XaSZGLnYnN1LvijIGXwclL UqLNcZb/eAictRjCnIlqeivlXLj5oPXplrpniMtmFFCcwjIhrTrxGtttLazl1vnOuvGo bmNg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=HRv7xTwn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-537a85e1312si826227e87.97.2024.09.24.19.34.19; Tue, 24 Sep 2024 19:34:20 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=HRv7xTwn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 161DB68DC1A; Wed, 25 Sep 2024 05:15:33 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 74BE768DB18 for ; Wed, 25 Sep 2024 05:15:24 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727230531; x=1758766531; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Mwyqbi/9F2ctZVDuWhWfc6/vo3mf3vSRdU+IdE7wT48=; b=HRv7xTwnKCsWFhVynWt0joc1ziNrR7mW/QGDrhd7S9dtw+AOQijeWRnl mG9tBuBVxJV9ByVVoklW4l/ngBqUKAdf7pLeNsj5G0aY63S0JW8WlbHDq Ry8z+fR3+bDUNc3ej6VMReVeWs/k4OUglG5COmzprw7emRp5xJhgv+9Cp qWNM5E8+7Ker5vKffJYS5j+RRf2oUOfC3l9BP7naGASn2dX+EoSZYKo4E tSeX4VwKVcehwDbdcUiQmP3S1oXyylCBJJ433Ub4vhqS96A7AtVB+e5cE qTXNQQ0l5Kia/+qORO4xA0L+CEkgzdpwA1CTjNuR75fO4+lexB3x73Lci Q==; X-CSE-ConnectionGUID: XCPj3xcTSxSmQINRDuuxgw== X-CSE-MsgGUID: OsTk3sq3TRm7s+hU7izklg== X-IronPort-AV: E=McAfee;i="6700,10204,11205"; a="37634068" X-IronPort-AV: E=Sophos;i="6.10,256,1719903600"; d="scan'208";a="37634068" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Sep 2024 19:15:22 -0700 X-CSE-ConnectionGUID: 3cprAL2UR0+zHsXXRwKoOA== X-CSE-MsgGUID: 6Kl4g9oaTueSmr5XOClflg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,256,1719903600"; d="scan'208";a="71246400" Received: from feiwan1-desk3.sh.intel.com ([10.238.208.39]) by fmviesa007.fm.intel.com with ESMTP; 24 Sep 2024 19:13:16 -0700 From: fei.w.wang-at-intel.com@ffmpeg.org To: ffmpeg-devel@ffmpeg.org Date: Wed, 25 Sep 2024 10:15:32 +0800 Message-Id: <20240925021533.1742735-7-fei.w.wang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240925021533.1742735-1-fei.w.wang@intel.com> References: <20240925021533.1742735-1-fei.w.wang@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 7/8] lavc/vvc_dec: Add hardware decode API X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: fei.w.wang@intel.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: S6i10a+hGmXv From: Fei Wang Signed-off-by: Fei Wang --- libavcodec/vvc/dec.c | 90 ++++++++++++++++++++++++++++++++++++------- libavcodec/vvc/dec.h | 4 ++ libavcodec/vvc/refs.c | 6 +++ 3 files changed, 87 insertions(+), 13 deletions(-) diff --git a/libavcodec/vvc/dec.c b/libavcodec/vvc/dec.c index edf2607f50..19770de3fa 100644 --- a/libavcodec/vvc/dec.c +++ b/libavcodec/vvc/dec.c @@ -22,6 +22,8 @@ */ #include "libavcodec/codec_internal.h" #include "libavcodec/decode.h" +#include "libavcodec/hwaccel_internal.h" +#include "libavcodec/hwconfig.h" #include "libavcodec/profiles.h" #include "libavcodec/refstruct.h" #include "libavutil/cpu.h" @@ -610,6 +612,8 @@ static int ref_frame(VVCFrame *dst, const VVCFrame *src) ff_refstruct_replace(&dst->rpl_tab, src->rpl_tab); ff_refstruct_replace(&dst->rpl, src->rpl); + ff_refstruct_replace(&dst->hwaccel_picture_private, + src->hwaccel_picture_private); dst->nb_rpl_elems = src->nb_rpl_elems; dst->poc = src->poc; @@ -770,18 +774,50 @@ static int slice_start(SliceContext *sc, VVCContext *s, VVCFrameContext *fc, return 0; } -static void export_frame_params(VVCContext *s, const VVCFrameContext *fc) +static enum AVPixelFormat get_format(AVCodecContext *avctx, const VVCSPS *sps) +{ +#define HWACCEL_MAX 0 + + enum AVPixelFormat pix_fmts[HWACCEL_MAX + 2], *fmt = pix_fmts; + + switch (sps->pix_fmt) { + case AV_PIX_FMT_YUV420P: + break; + case AV_PIX_FMT_YUV420P10: + break; + } + + *fmt++ = sps->pix_fmt; + *fmt = AV_PIX_FMT_NONE; + + return ff_get_format(avctx, pix_fmts); +} + +static int export_frame_params(VVCContext *s, const VVCFrameContext *fc) { AVCodecContext *c = s->avctx; const VVCSPS *sps = fc->ps.sps; const VVCPPS *pps = fc->ps.pps; + int ret; - c->pix_fmt = sps->pix_fmt; - c->coded_width = pps->width; - c->coded_height = pps->height; - c->width = pps->width - ((pps->r->pps_conf_win_left_offset + pps->r->pps_conf_win_right_offset) << sps->hshift[CHROMA]); - c->height = pps->height - ((pps->r->pps_conf_win_top_offset + pps->r->pps_conf_win_bottom_offset) << sps->vshift[CHROMA]); + // Reset HW config if pix_fmt/w/h change. + if (s->pix_fmt != sps->pix_fmt || c->coded_width != pps->width || c->coded_height != pps->height) { + c->coded_width = pps->width; + c->coded_height = pps->height; + c->pix_fmt = get_format(c, sps); + ret = get_format(c, sps); + if (ret < 0) + return ret; + + c->pix_fmt = ret; + s->pix_fmt = sps->pix_fmt; + } + + c->width = pps->width - ((pps->r->pps_conf_win_left_offset + pps->r->pps_conf_win_right_offset) << sps->hshift[CHROMA]); + c->height = pps->height - ((pps->r->pps_conf_win_top_offset + pps->r->pps_conf_win_bottom_offset) << sps->vshift[CHROMA]); c->has_b_frames = sps->r->sps_dpb_params.dpb_max_num_reorder_pics[sps->r->sps_max_sublayers_minus1]; + + return 0; } static int frame_setup(VVCFrameContext *fc, VVCContext *s) @@ -794,8 +830,11 @@ static int frame_setup(VVCFrameContext *fc, VVCContext *s) if (ret < 0) return ret; - export_frame_params(s, fc); - return ret; + ret = export_frame_params(s, fc); + if (ret < 0) + return ret; + + return 0; } static int decode_slice(VVCContext *s, VVCFrameContext *fc, const H2645NAL *nal, const CodedBitstreamUnit *unit) @@ -824,6 +863,20 @@ static int decode_slice(VVCContext *s, VVCFrameContext *fc, const H2645NAL *nal, ret = slice_init_entry_points(sc, fc, nal, unit); if (ret < 0) return ret; + + if (s->avctx->hwaccel) { + if (is_first_slice) { + ret = FF_HW_CALL(s->avctx, start_frame, NULL, 0); + if (ret < 0) + return ret; + } + + ret = FF_HW_CALL(s->avctx, decode_slice, + nal->raw_data, nal->raw_size); + if (ret < 0) + return ret; + } + fc->nb_slices++; return 0; @@ -939,17 +992,26 @@ static int wait_delayed_frame(VVCContext *s, AVFrame *output, int *got_output) static int submit_frame(VVCContext *s, VVCFrameContext *fc, AVFrame *output, int *got_output) { - int ret = ff_vvc_frame_submit(s, fc); + int ret; - if (ret < 0) { - ff_vvc_report_frame_finished(fc->ref); - return ret; + if (s->avctx->hwaccel) { + if (ret = FF_HW_SIMPLE_CALL(s->avctx, end_frame) < 0) { + av_log(s->avctx, AV_LOG_ERROR, + "Hardware accelerator failed to decode picture\n"); + ff_vvc_unref_frame(fc, fc->ref, ~0); + return ret; + } + } else { + if (ret = ff_vvc_frame_submit(s, fc) < 0) { + ff_vvc_report_frame_finished(fc->ref); + return ret; + } } s->nb_frames++; s->nb_delayed++; - if (s->nb_delayed >= s->nb_fcs) { + if (s->nb_delayed >= s->nb_fcs || s->avctx->hwaccel) { if ((ret = wait_delayed_frame(s, output, got_output)) < 0) return ret; } @@ -1095,6 +1157,8 @@ static av_cold int vvc_decode_init(AVCodecContext *avctx) GDR_SET_RECOVERED(s); ff_thread_once(&init_static_once, init_default_scale_m); + s->pix_fmt = AV_PIX_FMT_NONE; + return 0; } diff --git a/libavcodec/vvc/dec.h b/libavcodec/vvc/dec.h index d27cf52ca2..776b38b20f 100644 --- a/libavcodec/vvc/dec.h +++ b/libavcodec/vvc/dec.h @@ -101,6 +101,8 @@ typedef struct VVCFrame { * A combination of VVC_FRAME_FLAG_* */ uint8_t flags; + + void *hwaccel_picture_private; ///< hardware accelerator private data } VVCFrame; typedef struct SliceContext { @@ -243,6 +245,8 @@ typedef struct VVCContext { uint64_t nb_frames; ///< processed frames int nb_delayed; ///< delayed frames + + enum AVPixelFormat pix_fmt; ///< pix format of current frame } VVCContext ; #endif /* AVCODEC_VVC_DEC_H */ diff --git a/libavcodec/vvc/refs.c b/libavcodec/vvc/refs.c index 3ed9d6768d..281410091e 100644 --- a/libavcodec/vvc/refs.c +++ b/libavcodec/vvc/refs.c @@ -26,6 +26,7 @@ #include "libavutil/thread.h" #include "libavcodec/refstruct.h" #include "libavcodec/thread.h" +#include "libavcodec/decode.h" #include "refs.h" @@ -59,6 +60,7 @@ void ff_vvc_unref_frame(VVCFrameContext *fc, VVCFrame *frame, int flags) ff_refstruct_unref(&frame->rpl_tab); frame->collocated_ref = NULL; + ff_refstruct_unref(&frame->hwaccel_picture_private); } } @@ -153,6 +155,10 @@ static VVCFrame *alloc_frame(VVCContext *s, VVCFrameContext *fc) if (!frame->progress) goto fail; + ret = ff_hwaccel_frame_priv_alloc(s->avctx, &frame->hwaccel_picture_private); + if (ret < 0) + goto fail; + return frame; fail: ff_vvc_unref_frame(fc, frame, ~0);