From patchwork Mon Sep 23 07:10:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Fei W" X-Patchwork-Id: 51731 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:d154:0:b0:48e:c0f8:d0de with SMTP id bt20csp2280049vqb; Mon, 23 Sep 2024 00:24:19 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXVFYxIfEIaaJRXptQzlBV+U+AtKJo9Gul2GtAfxCud0yg1fbFspCvPri9uW37kQga0XHO3STQYh/RzkPtnGuX+@gmail.com X-Google-Smtp-Source: AGHT+IHWi1VO+lS+4pXenh/D/wMyqJn/DLF0rXoLMOxfnf7psLi2XKEIbICw3YMTXtXhU9PufA3M X-Received: by 2002:a05:6512:2213:b0:535:6778:952b with SMTP id 2adb3069b0e04-536ac32f0bemr4873350e87.44.1727076259176; Mon, 23 Sep 2024 00:24:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1727076259; cv=none; d=google.com; s=arc-20240605; b=Ci93HGCr4TGVUaIfTrdojOftgM3NFmHRczGJTolyoCz37IBRcpchZnP0s9/5SyY/WX b8Rqx2VgqZlodMpGW+G6Pghyn3wJxRhNTpIuTjY98hf/OBCB29mIGH/l4pGvMkCZzwUi TaxvGyPccDNvvWpn/WjGPBUuzagyRLgv4jTGcPRyrj5heTEPVEqws1MliBDVnqoVFbbn 28bnKK2wM67GMmYYuRCAOJ6UnVmayVk8/I/iMoUiQfFLXKScuIvQFAK1qkK3seHMnuML eRa81WqP3VmOB+VHSxn44UKE80vko5p3onjAd7R/k7Za5nL3XROq3zVByfwEBFt5RCdy 5hbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=fcUYh2KtrhbFoDihbhshVn9DeiH8mFojFnI4irhgxCY=; fh=i4ESP4ZRFDcfYfwXKWpOXjc2YhmIGuOsCfZUnwNO0gc=; b=b0jqKSyEPHdKMLMqa2LzsBObDUpav9GT6Kz065q2Y3koqgI70Rajio9Tnp68SAXUJT HjAJbVcOpZhi7/B7jC2B2Lq5eePVq7sBp3im79NPbDzUjUmFea5X26IJYrtfSLLD9Nft d5xjI0XXocyjzm1q5lNBPDGPnAxbuZaaNOEApMwPEqRAg+wpWzw+zmx95U5t66ptlxdt WpWDxb8LNovMHSvxXKSz352St6Y8qL3Y8nF/HpcsZ3q9+3d86t4l93RvnBW6vYkxWsKX nBxNHNtsHiLAMVPFb8A7J+S/BC+1z93pxXhOiGkVTyN+DmySNYfSK0+i2JfCTxRUHib/ Q32A==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=Iw0AB+dA; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5c5ae20d8c7si3893131a12.608.2024.09.23.00.24.16; Mon, 23 Sep 2024 00:24:19 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=Iw0AB+dA; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5F19C68DB83; Mon, 23 Sep 2024 10:07:49 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 97DAD68CFC7 for ; Mon, 23 Sep 2024 10:07:42 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727075268; x=1758611268; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=Af7Y5lDvPAt3VEISWMGAgqd3YOOJvoKnK5+w2ChaIg8=; b=Iw0AB+dAjSfXB5cnLqPb1kYKyqCq7jvTIWMjkN02D9vCYKJl02o9ng77 VS9999yKuJc59tafmkCe/UTCd2cDZvyj9+dJl4BT7xkORDfwcopK8xDPm Yl2NRGcp5MGzTFBn1G63TMIFPUTgbgAc6r5LvUPmrt6Nv6XjFNMmuwTPh HagX1vBdhdwoBufTquWEuLgMyR1FaGXYJyxZ86oPi3BORWpNMPEYJPWfM JfPnG55YTKJtri5NSOCSQp8+tIhrPSOfKMDkV7NbOmHoyXunkVPwH9ZZf e3LMvt/F5KwL7IOLH76UjqHQR/Dwz9jWU/nQCBw0zmH7NdRKdZ/m0fbTx g==; X-CSE-ConnectionGUID: eEDNmwJUTuWEhUlaZU2Xhw== X-CSE-MsgGUID: IBU6t54cSnCxpYxxWvAddA== X-IronPort-AV: E=McAfee;i="6700,10204,11202"; a="43525025" X-IronPort-AV: E=Sophos;i="6.10,250,1719903600"; d="scan'208";a="43525025" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 00:07:40 -0700 X-CSE-ConnectionGUID: O7LGIztERYaAwelA7HCy4w== X-CSE-MsgGUID: MGvT/Yu0SUeQ/6Qbn4MQ8Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,250,1719903600"; d="scan'208";a="70560513" Received: from feiwan1-desk3.sh.intel.com ([10.238.208.39]) by fmviesa007.fm.intel.com with ESMTP; 23 Sep 2024 00:07:38 -0700 From: fei.w.wang-at-intel.com@ffmpeg.org To: ffmpeg-devel@ffmpeg.org Date: Mon, 23 Sep 2024 15:10:04 +0800 Message-Id: <20240923071008.1599696-1-fei.w.wang@intel.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/5] libavcodec/qsvenc: enable Alpha Encode for HEVC X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: fei.w.wang@intel.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: YLbsBGa7vajx From: Fei Wang This support alpha encode for HEVC introduced by Apple: https://developer.apple.com/videos/play/wwdc2019/506/ Currently, it only support RGBA video memory as input. RGB and alpha channel will be encoded in different layers with 4:2:0 color format. And set texture to shared to allow extract alpha channel internally: https://github.com/intel/libvpl/blob/5f6bd8a1e753c8f63a3fd8b36894d6968b808a6d/doc/spec/source/snippets/prg_encoding.c#L752 Example cmdline: ffmpeg.exe -v verbose -hwaccel qsv -hwaccel_output_format qsv -f rawvideo \ -pix_fmt bgra -s:v 1920x1080 -r:v 25 -i input.argb -vf \ 'format=bgra,hwupload=extra_hw_frames=120' -an -c:v hevc_qsv \ -alpha_encode 1 -y out.mp4 Signed-off-by: Fei Wang --- doc/encoders.texi | 4 ++ libavcodec/qsvenc.c | 79 +++++++++++++++++++++++++++++++++++++-- libavcodec/qsvenc.h | 9 ++++- libavcodec/qsvenc_hevc.c | 3 ++ libavutil/hwcontext_qsv.c | 4 +- 5 files changed, 93 insertions(+), 6 deletions(-) diff --git a/doc/encoders.texi b/doc/encoders.texi index 0749417db4..6094434ed4 100644 --- a/doc/encoders.texi +++ b/doc/encoders.texi @@ -4036,6 +4036,10 @@ skip_frame metadata indicates the number of missed frames before the current frame. @end table +@item @var{alpha_encode} +Encode Alpha and RGB into different layers introduced by Apple: +https://developer.apple.com/videos/play/wwdc2019/506/. Only support on Windows +with RGBA video memory as input. @end table @subsection MPEG2 Options diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c index 8200a14012..9af9932035 100644 --- a/libavcodec/qsvenc.c +++ b/libavcodec/qsvenc.c @@ -203,6 +203,9 @@ static void dump_video_param(AVCodecContext *avctx, QSVEncContext *q, #if QSV_HAVE_HE mfxExtHyperModeParam *exthypermodeparam = NULL; #endif +#if QSV_HAVE_AC + mfxExtAlphaChannelEncCtrl *extalphachannel = NULL; +#endif const char *tmp_str = NULL; @@ -220,6 +223,11 @@ static void dump_video_param(AVCodecContext *avctx, QSVEncContext *q, exthypermodeparam = (mfxExtHyperModeParam *)coding_opts[q->exthypermodeparam_idx]; #endif +#if QSV_HAVE_AC + if (q->extaplhachannel_idx > 0) + extalphachannel = (mfxExtAlphaChannelEncCtrl *)coding_opts[q->extaplhachannel_idx]; +#endif + av_log(avctx, AV_LOG_VERBOSE, "profile: %s; level: %"PRIu16"\n", print_profile(avctx->codec_id, info->CodecProfile), info->CodecLevel); @@ -400,6 +408,23 @@ static void dump_video_param(AVCodecContext *avctx, QSVEncContext *q, av_log(avctx, AV_LOG_VERBOSE, "\n"); } #endif + +#if QSV_HAVE_AC + if (extalphachannel) { + av_log(avctx, AV_LOG_VERBOSE, "AlphaChannel Encode: %s; ", print_threestate(extalphachannel->EnableAlphaChannelEncoding)); + + av_log(avctx, AV_LOG_VERBOSE, "Mode: "); + if (extalphachannel->AlphaChannelMode == MFX_ALPHA_MODE_PREMULTIPLIED) + av_log(avctx, AV_LOG_VERBOSE, "PREMULTIPLIED; "); + else if (extalphachannel->AlphaChannelMode == MFX_ALPHA_MODE_STRAIGHT) + av_log(avctx, AV_LOG_VERBOSE, "STRAIGHT; "); + else + av_log(avctx, AV_LOG_VERBOSE, "unknown; "); + av_log(avctx, AV_LOG_VERBOSE, "BitrateRatio: %d", extalphachannel->AlphaChannelBitrateRatio); + + av_log(avctx, AV_LOG_VERBOSE, "\n"); + } +#endif } static void dump_video_vp9_param(AVCodecContext *avctx, QSVEncContext *q, @@ -1150,7 +1175,8 @@ static int init_video_param(AVCodecContext *avctx, QSVEncContext *q) q->extco3.MaxFrameSizeP = q->max_frame_size_p; if (sw_format == AV_PIX_FMT_BGRA && (q->profile == MFX_PROFILE_HEVC_REXT || - q->profile == MFX_PROFILE_UNKNOWN)) + q->profile == MFX_PROFILE_UNKNOWN) && + !q->alpha_encode) q->extco3.TargetChromaFormatPlus1 = MFX_CHROMAFORMAT_YUV444 + 1; q->extco3.ScenarioInfo = q->scenario; @@ -1282,6 +1308,37 @@ static int init_video_param(AVCodecContext *avctx, QSVEncContext *q) } #endif +#if QSV_HAVE_AC + if (q->alpha_encode) { + if (QSV_RUNTIME_VERSION_ATLEAST(q->ver, 2, 13)) { + mfxIMPL impl; + MFXQueryIMPL(q->session, &impl); + + if (MFX_IMPL_VIA_MASK(impl) != MFX_IMPL_VIA_D3D11) { + av_log(avctx, AV_LOG_ERROR, "Alpha Channel Encode requires D3D11VA \n"); + return AVERROR_UNKNOWN; + } + + if (q->param.mfx.CodecId != MFX_CODEC_HEVC) { + av_log(avctx, AV_LOG_ERROR, "Not supported encoder for Alpha Channel Encode. " + "Supported: hevc_qsv \n"); + return AVERROR_UNKNOWN; + } + + q->extaplhachannelparam.Header.BufferId = MFX_EXTBUFF_ALPHA_CHANNEL_ENC_CTRL; + q->extaplhachannelparam.Header.BufferSz = sizeof(q->extaplhachannelparam); + q->extaplhachannelparam.EnableAlphaChannelEncoding = MFX_CODINGOPTION_ON; + q->extaplhachannelparam.AlphaChannelBitrateRatio = 25; + q->extaplhachannelparam.AlphaChannelMode = MFX_ALPHA_MODE_PREMULTIPLIED; + q->extparam_internal[q->nb_extparam_internal++] = (mfxExtBuffer *)&q->extaplhachannelparam; + } else { + av_log(avctx, AV_LOG_ERROR, + "This version of runtime doesn't support Alpha Channel Encode\n"); + return AVERROR_UNKNOWN; + } + } +#endif + if (!check_enc_param(avctx,q)) { av_log(avctx, AV_LOG_ERROR, "some encoding parameters are not supported by the QSV " @@ -1463,12 +1520,21 @@ static int qsv_retrieve_enc_params(AVCodecContext *avctx, QSVEncContext *q) }; #endif - mfxExtBuffer *ext_buffers[6 + QSV_HAVE_HE]; +#if QSV_HAVE_AC + mfxExtAlphaChannelEncCtrl alpha_encode_buf = { + .Header.BufferId = MFX_EXTBUFF_ALPHA_CHANNEL_ENC_CTRL, + .Header.BufferSz = sizeof(alpha_encode_buf), + }; +#endif + + mfxExtBuffer *ext_buffers[6 + QSV_HAVE_HE + QSV_HAVE_AC]; int need_pps = avctx->codec_id != AV_CODEC_ID_MPEG2VIDEO; int ret, ext_buf_num = 0, extradata_offset = 0; - q->co2_idx = q->co3_idx = q->exthevctiles_idx = q->exthypermodeparam_idx = -1; + q->co2_idx = q->co3_idx = q->exthevctiles_idx = q->exthypermodeparam_idx = + q->extaplhachannel_idx = -1; + ext_buffers[ext_buf_num++] = (mfxExtBuffer*)&extradata; ext_buffers[ext_buf_num++] = (mfxExtBuffer*)&co; @@ -1497,6 +1563,13 @@ static int qsv_retrieve_enc_params(AVCodecContext *avctx, QSVEncContext *q) } #endif +#if QSV_HAVE_AC + if (q->alpha_encode && QSV_RUNTIME_VERSION_ATLEAST(q->ver, 2, 13)) { + q->extaplhachannel_idx = ext_buf_num; + ext_buffers[ext_buf_num++] = (mfxExtBuffer*)&alpha_encode_buf; + } +#endif + q->param.ExtParam = ext_buffers; q->param.NumExtParam = ext_buf_num; diff --git a/libavcodec/qsvenc.h b/libavcodec/qsvenc.h index 4bc77f2f7c..559d40c919 100644 --- a/libavcodec/qsvenc.h +++ b/libavcodec/qsvenc.h @@ -44,11 +44,13 @@ #define QSV_HAVE_VCM 1 #define QSV_HAVE_MF 0 #define QSV_HAVE_HE QSV_VERSION_ATLEAST(2, 4) +#define QSV_HAVE_AC QSV_VERSION_ATLEAST(2, 13) #else #define QSV_HAVE_AVBR 0 #define QSV_HAVE_VCM 0 #define QSV_HAVE_MF !QSV_ONEVPL #define QSV_HAVE_HE 0 +#define QSV_HAVE_AC 0 #endif #define QSV_COMMON_OPTS \ @@ -188,10 +190,13 @@ typedef struct QSVEncContext { mfxFrameSurface1 **opaque_surfaces; AVBufferRef *opaque_alloc_buf; #endif +#if QSV_HAVE_AC + mfxExtAlphaChannelEncCtrl extaplhachannelparam; +#endif mfxExtVideoSignalInfo extvsi; - mfxExtBuffer *extparam_internal[5 + (QSV_HAVE_MF * 2) + (QSV_HAVE_EXT_AV1_PARAM * 2) + QSV_HAVE_HE]; + mfxExtBuffer *extparam_internal[5 + (QSV_HAVE_MF * 2) + (QSV_HAVE_EXT_AV1_PARAM * 2) + QSV_HAVE_HE + QSV_HAVE_AC]; int nb_extparam_internal; mfxExtBuffer **extparam_str; @@ -269,6 +274,7 @@ typedef struct QSVEncContext { int co2_idx; int co3_idx; + int extaplhachannel_idx; int exthevctiles_idx; int exthypermodeparam_idx; int vp9_idx; @@ -319,6 +325,7 @@ typedef struct QSVEncContext { int dual_gfx; AVDictionary *qsv_params; + int alpha_encode; } QSVEncContext; int ff_qsv_enc_init(AVCodecContext *avctx, QSVEncContext *q); diff --git a/libavcodec/qsvenc_hevc.c b/libavcodec/qsvenc_hevc.c index 2a397a2919..6ea9c4cdcb 100644 --- a/libavcodec/qsvenc_hevc.c +++ b/libavcodec/qsvenc_hevc.c @@ -365,6 +365,9 @@ static const AVOption options[] = { { "int_ref_qp_delta", "QP difference for the refresh MBs", OFFSET(qsv.int_ref_qp_delta), AV_OPT_TYPE_INT, { .i64 = INT16_MIN }, INT16_MIN, INT16_MAX, VE }, { "int_ref_cycle_dist", "Distance between the beginnings of the intra-refresh cycles in frames", OFFSET(qsv.int_ref_cycle_dist), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, INT16_MAX, VE }, +#if QSV_HAVE_AC + { "alpha_encode", "Encode with alpha channel", OFFSET(qsv.alpha_encode), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, VE}, +#endif { NULL }, }; diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c index 721c841c2a..8471b8bcdc 100644 --- a/libavutil/hwcontext_qsv.c +++ b/libavutil/hwcontext_qsv.c @@ -236,7 +236,7 @@ static uint32_t qsv_get_d3d11va_bind_flags(int mem_type) bind_flags = D3D11_BIND_DECODER; if ((MFX_MEMTYPE_FROM_VPPOUT & mem_type) || (MFX_MEMTYPE_VIDEO_MEMORY_PROCESSOR_TARGET & mem_type)) - bind_flags = D3D11_BIND_RENDER_TARGET; + bind_flags = D3D11_BIND_RENDER_TARGET | D3D11_BIND_SHADER_RESOURCE; return bind_flags; } @@ -590,7 +590,7 @@ static int qsv_init_child_ctx(AVHWFramesContext *ctx) if (child_device_ctx->type == AV_HWDEVICE_TYPE_D3D11VA) { AVD3D11VAFramesContext *child_frames_hwctx = child_frames_ctx->hwctx; if (hwctx->frame_type == 0) - hwctx->frame_type = MFX_MEMTYPE_VIDEO_MEMORY_PROCESSOR_TARGET; + hwctx->frame_type = MFX_MEMTYPE_VIDEO_MEMORY_PROCESSOR_TARGET | MFX_MEMTYPE_SHARED_RESOURCE; if (hwctx->frame_type & MFX_MEMTYPE_SHARED_RESOURCE) child_frames_hwctx->MiscFlags = D3D11_RESOURCE_MISC_SHARED; child_frames_hwctx->BindFlags = qsv_get_d3d11va_bind_flags(hwctx->frame_type); From patchwork Mon Sep 23 07:10:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Fei W" X-Patchwork-Id: 51729 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:d154:0:b0:48e:c0f8:d0de with SMTP id bt20csp2276391vqb; Mon, 23 Sep 2024 00:14:21 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCV3zW2DX8hAAzlyPCsfjPPExH+2MMf0mJWPAGhH68uIHuFwennuAVQQtWhF8vt5yQqBaVkZKBYVpHSDeFUbch8x@gmail.com X-Google-Smtp-Source: AGHT+IEZ74fKe2UbbYvsTiw3c64jBhTUG8vlsKk2UP6ofOz1urB1AW6dXJmCQ6dLNraRwz5VmZop X-Received: by 2002:a05:6512:ea8:b0:533:4722:ebbe with SMTP id 2adb3069b0e04-536ac2f2b4cmr5144861e87.26.1727075660676; Mon, 23 Sep 2024 00:14:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1727075660; cv=none; d=google.com; s=arc-20240605; b=LfP75ot4tk3PQbOJPHL4exMfaLaPOChLScdaUB0SNBC5xRekpDqVq2u6SYF6AJXS6l NQuVEdYF7LaDDXsjr9irpZUFdznP6Xob6WER4cOk+nnshb4n9SOZT48hn5u89mps4MCl zYgFV7VzdcgtYQL7j+BR8HsoXWN+IcrMroCk5EfrGNxCA6FBjj8XJxFjY41FVR7pbUP5 Jn1r+Nk1HyQRQW5oznl2M8luG0+peq5W415D0PfFAA8GwiuSr4unh8BOljAE7FcW+FgG uXUkE+YEy9CVQBqJW9K/7gVTqPukDFFhCGcKTV60Ii0EisEm9/ZaJzUBmp1OcqPYqf/c pKnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=J59RnuDrt861gI5Ufr6LE01MoLUt/+we2SCGj6SFElI=; fh=i4ESP4ZRFDcfYfwXKWpOXjc2YhmIGuOsCfZUnwNO0gc=; b=ZDh792xWHnGPFmFQalO2/FcYpae3XdgxAvDysEnUMKCnA5Y0w40q/Hspgl1ZBTkysB AZjhLmdJkWZ+ou20WHOK9zJEHpxIKTFk57c17tCgRkIxnq3mK6E7+GH4bBRSkiDpzW/t 5crzRl9BMVo8/nT58PxcPAaOQbMBDwHTodZOa4Rj3B4CmbgxxJQUBEGcK3DQ6jnPzaWo Ukm8m3lrnWSBBHQ2Gy4X/i2Cy+4SwKns1KbKax7uHUG+sj3+J4GTu/XHQ7aA6mLtjpei pEl40MmI8x6lfcWpybjAnEiL5WmbZ+jXeCZhjWVSlzXv+q3RRYljRN210xWIEyuoMxxb q4nQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=XZYUmm4j; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5c42bc9079fsi13586476a12.437.2024.09.23.00.14.19; Mon, 23 Sep 2024 00:14:20 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=XZYUmm4j; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B7BBE68DB63; Mon, 23 Sep 2024 10:07:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1011D68DB73 for ; Mon, 23 Sep 2024 10:07:43 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727075269; x=1758611269; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Dsrxwu1yCkSfRQiXnKYLqAtotUlRhnQh5YOupTFJA/Y=; b=XZYUmm4jAt3kVXRMcFG527aUoqVC+/hxLpx19BYAgaNpLq1An/VBx3zK Ww/Zs07jK/3KQAj5bdPCySlPnIF/Zoj0Uis3txxKRZJlXhI6r3X3dJ5EJ zXocWdd/GvMF1kIEsya7QvxcZ8BAHxsY8Kal8Ko836EX83yxOqIVtXRx0 6fYiv4jAFQFUc8d4CaQ74HZMEAYoO3MFHe0La677HeB1vAox+L6QQcH1F aP1qdVIvhZCIab4IfuRkTskifm1blWue5u+4wfwseG6nSXcrf9tK2ZI6C 23EZ/K4azCIyow11QffTW1YqUhgacAnBg9/AK9tazLBj2nIj3TDmR35qG A==; X-CSE-ConnectionGUID: JE5cp3sZTYSzq0g1bVGN7A== X-CSE-MsgGUID: peAdGMffTs6EYsovf+0oeg== X-IronPort-AV: E=McAfee;i="6700,10204,11202"; a="43525026" X-IronPort-AV: E=Sophos;i="6.10,250,1719903600"; d="scan'208";a="43525026" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 00:07:41 -0700 X-CSE-ConnectionGUID: HdvwPKlkTDKpZZEcBPn4HQ== X-CSE-MsgGUID: /JukKuStQs+o6L5o7dSj0w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,250,1719903600"; d="scan'208";a="70560522" Received: from feiwan1-desk3.sh.intel.com ([10.238.208.39]) by fmviesa007.fm.intel.com with ESMTP; 23 Sep 2024 00:07:40 -0700 From: fei.w.wang-at-intel.com@ffmpeg.org To: ffmpeg-devel@ffmpeg.org Date: Mon, 23 Sep 2024 15:10:05 +0800 Message-Id: <20240923071008.1599696-2-fei.w.wang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240923071008.1599696-1-fei.w.wang@intel.com> References: <20240923071008.1599696-1-fei.w.wang@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/5] libavcodec/qsvenc: enable Screen Content Tool Encode for AV1 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: fei.w.wang@intel.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: FXWMjGPZfO+M From: Fei Wang Screen Content Tool provides Intra Block Copy and Palette Mode when encoding. Signed-off-by: Fei Wang --- doc/encoders.texi | 6 ++++++ libavcodec/qsvenc.c | 43 +++++++++++++++++++++++++++++++++++++++++ libavcodec/qsvenc.h | 9 ++++++++- libavcodec/qsvenc_av1.c | 4 ++++ 4 files changed, 61 insertions(+), 1 deletion(-) diff --git a/doc/encoders.texi b/doc/encoders.texi index 6094434ed4..85be976a46 100644 --- a/doc/encoders.texi +++ b/doc/encoders.texi @@ -4121,6 +4121,12 @@ than zero, then for I frames the value set by max_frame_size is ignored. @item @var{max_frame_size_p} Maximum encoded frame size for P frames in bytes. If this value is set as larger than zero, then for P frames the value set by max_frame_size is ignored. + +@item @var{palette_mode} +Use palette mode in Screen Content Tool. + +@item @var{intrabc} +Use intra block copy in Screen Content Tool. @end table @section snow diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c index 9af9932035..dfef15fa5a 100644 --- a/libavcodec/qsvenc.c +++ b/libavcodec/qsvenc.c @@ -519,6 +519,9 @@ static void dump_video_av1_param(AVCodecContext *avctx, QSVEncContext *q, mfxExtAV1BitstreamParam *av1_bs_param = (mfxExtAV1BitstreamParam *)coding_opts[1]; mfxExtCodingOption2 *co2 = (mfxExtCodingOption2*)coding_opts[2]; mfxExtCodingOption3 *co3 = (mfxExtCodingOption3*)coding_opts[3]; +#if QSV_HAVE_EXT_AV1_SCC + mfxExtAV1ScreenContentTools *scc = (mfxExtAV1ScreenContentTools*)coding_opts[4]; +#endif av_log(avctx, AV_LOG_VERBOSE, "profile: %s; level: %"PRIu16"\n", print_profile(avctx->codec_id, info->CodecProfile), info->CodecLevel); @@ -591,6 +594,14 @@ static void dump_video_av1_param(AVCodecContext *avctx, QSVEncContext *q, print_threestate(av1_bs_param->WriteIVFHeaders)); av_log(avctx, AV_LOG_VERBOSE, "LowDelayBRC: %s\n", print_threestate(co3->LowDelayBRC)); av_log(avctx, AV_LOG_VERBOSE, "MaxFrameSize: %d;\n", co2->MaxFrameSize); + +#if QSV_HAVE_EXT_AV1_SCC + if (scc) { + av_log(avctx, AV_LOG_VERBOSE, + "Palette: %s; IntraBlockCopy: %s\n", + print_threestate(scc->Palette), print_threestate(scc->IntraBlockCopy)); + } +#endif } #endif @@ -1339,6 +1350,28 @@ static int init_video_param(AVCodecContext *avctx, QSVEncContext *q) } #endif +#if QSV_HAVE_EXT_AV1_SCC + if (q->palette_mode || q->intrabc) { + if (QSV_RUNTIME_VERSION_ATLEAST(q->ver, 2, 13)) { + if (q->param.mfx.CodecId != MFX_CODEC_AV1) { + av_log(avctx, AV_LOG_ERROR, "Not supported encoder for Screen Content Tool Encode. " + "Supported: av1_qsv \n"); + return AVERROR_UNKNOWN; + } + + q->extsccparam.Header.BufferId = MFX_EXTBUFF_AV1_SCREEN_CONTENT_TOOLS; + q->extsccparam.Header.BufferSz = sizeof(q->extsccparam); + q->extsccparam.Palette = q->palette_mode ? MFX_CODINGOPTION_ON : MFX_CODINGOPTION_OFF; + q->extsccparam.IntraBlockCopy = q->intrabc ? MFX_CODINGOPTION_ON : MFX_CODINGOPTION_OFF; + q->extparam_internal[q->nb_extparam_internal++] = (mfxExtBuffer *)&q->extsccparam; + } else { + av_log(avctx, AV_LOG_ERROR, + "This version of runtime doesn't support Screen Content Tool Encode\n"); + return AVERROR_UNKNOWN; + } + } +#endif + if (!check_enc_param(avctx,q)) { av_log(avctx, AV_LOG_ERROR, "some encoding parameters are not supported by the QSV " @@ -1446,11 +1479,21 @@ static int qsv_retrieve_enc_av1_params(AVCodecContext *avctx, QSVEncContext *q) .Header.BufferSz = sizeof(co3), }; +#if QSV_HAVE_EXT_AV1_SCC + mfxExtAV1ScreenContentTools scc_buf = { + .Header.BufferId = MFX_EXTBUFF_AV1_SCREEN_CONTENT_TOOLS, + .Header.BufferSz = sizeof(scc_buf), + }; +#endif + mfxExtBuffer *ext_buffers[] = { (mfxExtBuffer*)&av1_extend_tile_buf, (mfxExtBuffer*)&av1_bs_param, (mfxExtBuffer*)&co2, (mfxExtBuffer*)&co3, +#if QSV_HAVE_EXT_AV1_SCC + (mfxExtBuffer*)&scc_buf, +#endif }; if (!QSV_RUNTIME_VERSION_ATLEAST(q->ver, 2, 5)) { diff --git a/libavcodec/qsvenc.h b/libavcodec/qsvenc.h index 559d40c919..1e1474d37c 100644 --- a/libavcodec/qsvenc.h +++ b/libavcodec/qsvenc.h @@ -38,6 +38,7 @@ #define QSV_HAVE_EXT_VP9_TILES QSV_VERSION_ATLEAST(1, 29) #define QSV_HAVE_EXT_AV1_PARAM QSV_VERSION_ATLEAST(2, 5) +#define QSV_HAVE_EXT_AV1_SCC QSV_VERSION_ATLEAST(2, 13) #if defined(_WIN32) || defined(__CYGWIN__) #define QSV_HAVE_AVBR 1 @@ -193,10 +194,14 @@ typedef struct QSVEncContext { #if QSV_HAVE_AC mfxExtAlphaChannelEncCtrl extaplhachannelparam; #endif +#if QSV_HAVE_EXT_AV1_SCC + mfxExtAV1ScreenContentTools extsccparam; +#endif mfxExtVideoSignalInfo extvsi; - mfxExtBuffer *extparam_internal[5 + (QSV_HAVE_MF * 2) + (QSV_HAVE_EXT_AV1_PARAM * 2) + QSV_HAVE_HE + QSV_HAVE_AC]; + mfxExtBuffer *extparam_internal[5 + (QSV_HAVE_MF * 2) + (QSV_HAVE_EXT_AV1_PARAM * 2) + + QSV_HAVE_HE + QSV_HAVE_AC + QSV_HAVE_EXT_AV1_SCC]; int nb_extparam_internal; mfxExtBuffer **extparam_str; @@ -326,6 +331,8 @@ typedef struct QSVEncContext { AVDictionary *qsv_params; int alpha_encode; + int palette_mode; + int intrabc; } QSVEncContext; int ff_qsv_enc_init(AVCodecContext *avctx, QSVEncContext *q); diff --git a/libavcodec/qsvenc_av1.c b/libavcodec/qsvenc_av1.c index a86b409bed..4f035f3d83 100644 --- a/libavcodec/qsvenc_av1.c +++ b/libavcodec/qsvenc_av1.c @@ -189,6 +189,10 @@ static const AVOption options[] = { { "tile_cols", "Number of columns for tiled encoding", OFFSET(qsv.tile_cols), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, UINT16_MAX, VE }, { "tile_rows", "Number of rows for tiled encoding", OFFSET(qsv.tile_rows), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, UINT16_MAX, VE }, { "look_ahead_depth", "Depth of look ahead in number frames, available when extbrc option is enabled", OFFSET(qsv.look_ahead_depth), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 100, VE }, +#if QSV_HAVE_EXT_AV1_SCC + { "palette_mode", "Enable palette mode of Screen Content Tool for encoding", OFFSET(qsv.palette_mode), AV_OPT_TYPE_BOOL, { .i64 = 0}, 0, 1, VE}, + { "intrabc", "Enable intra block copy of Screen Content Tool for encoding", OFFSET(qsv.intrabc), AV_OPT_TYPE_BOOL, { .i64 = 0}, 0, 1, VE}, +#endif { NULL }, }; From patchwork Mon Sep 23 07:10:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Fei W" X-Patchwork-Id: 51732 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:d154:0:b0:48e:c0f8:d0de with SMTP id bt20csp2302577vqb; Mon, 23 Sep 2024 01:24:18 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUzvDAAShciywlOayz7ykpR2MsBkDoZ2ox06ZIdbEOBQBdofdQwK8QPSj9kx2H0pfkAm1bnIX52M7ngRMuMUWxp@gmail.com X-Google-Smtp-Source: AGHT+IEn68ZOSmc3t0xbVFAVp4NoKxy82zYDtnHqYOSkQiV2OU60FozEepgnBtoNNrNpPi1zAYE5 X-Received: by 2002:a05:651c:1543:b0:2f7:c7f3:1d2e with SMTP id 38308e7fff4ca-2f7cb31c4e8mr59350581fa.19.1727079857767; Mon, 23 Sep 2024 01:24:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1727079857; cv=none; d=google.com; s=arc-20240605; b=EdiTu8ya7fnk7nHJiaifG9SCTeuhFA44OWhckgWyFOzFP/GJ3OFlw7/bvyvsYZc9IX tHwuHbpdL/MBM8bcrZJSbEe9KdgfpQaOQivrQCP9WqIkJRm8s7WoUZBItLJX5bc4H/4f fAmbxrVE09LhoStXV75O+jdgE45V9jtDdCROkCaAL70EDf6zQKSxXLTOcDJ/xGeROpWr X+lAFD0IyMluigh8CW5m/2h6XRjdSduK46+YjyyW4BJi4Ricz8ilml/m7bbLeoOorufS 4UicDTB+WJzIWF37XSddcGaFqI06X5KI4H3kG3zNYrznKoQd2gDfgwoxvVwFCtQi0++c a07A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=H9QqfF6zaUeISKOYqdAO9E1dCwr7qG4qdu6pScm5V1A=; fh=i4ESP4ZRFDcfYfwXKWpOXjc2YhmIGuOsCfZUnwNO0gc=; b=ZaDj26z1xmsMCBJsrxineEeo6rz81Vrd6Koth3yYDUFAb7gcXviVy1ZWuUkEvin4Sq Ai8BFrNX5ulVvTp/Z3RdUvB7qFELbpxSFkr2lbwUMIppoxUGWZGKdPeuxtHMauY22t7N BH0beYpiBsBUYVWqttPL+C5WtJeS1yXFnyadXZyjlhQtNm6t5U1S2+lkDHM86XhP11qz fECDbPERQuUJUJ5iMlo2LbPfdy4IPxxd2zWcXv1ian3Twl3Z33Lu8oOFlxMuvMuMv99Y VPrfJIWEJoNmmTchPmhIN29goxzMPeDvmLvQjc2Cn43MVhqELE32Y9EitSVYUBfXK3E8 T16w==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=T+gLOYoQ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 38308e7fff4ca-2f79d2c80c9si61881961fa.47.2024.09.23.01.24.17; Mon, 23 Sep 2024 01:24:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=T+gLOYoQ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3188768DBA0; Mon, 23 Sep 2024 10:07:53 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 353B168D7CA for ; Mon, 23 Sep 2024 10:07:45 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727075270; x=1758611270; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Yz9I+msC6VMnyYmvZCJIS6xBqcac2oRJKCNrGb0kGOM=; b=T+gLOYoQCUKa1Ug9Bz18zCJTJsXXUsdLjGvDoJgvuSqQilcJFPKjKBh8 p+YRJdhlJdt/yohKwReh1JIU+y096rTvKiEarIR+go+B2xy2AYBtz3f0t TiDsNBdwWVnSq6EPZJgW3qx8xDrKL+iGBqAyquLce1k22tuG0vEIhCWWS A4xjeNj1W3ES0dTgGzlGjvd6J46nfQuWzq+7bLiinIrDi01saVmf4Kwo5 2MFhfC7swddupt7QthpifHGEDFTMUVcKbXC0YM2sJqzVWjrULJUYCuDoh JpvDK7UHuj0lwxC2OkwfyL8zxLolxfoe2Ak6YE7+Nxi/mCXCKCBojCYjG Q==; X-CSE-ConnectionGUID: VzucUHEeRmeUAHwxCbXNuQ== X-CSE-MsgGUID: D6zzZpN8RtyGT7KUPVid2Q== X-IronPort-AV: E=McAfee;i="6700,10204,11202"; a="43525029" X-IronPort-AV: E=Sophos;i="6.10,250,1719903600"; d="scan'208";a="43525029" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 00:07:42 -0700 X-CSE-ConnectionGUID: GCha8a2vQhCmzZRSQ0/4lg== X-CSE-MsgGUID: XxC/l5A6TaKiEtu4n9TMog== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,250,1719903600"; d="scan'208";a="70560529" Received: from feiwan1-desk3.sh.intel.com ([10.238.208.39]) by fmviesa007.fm.intel.com with ESMTP; 23 Sep 2024 00:07:41 -0700 From: fei.w.wang-at-intel.com@ffmpeg.org To: ffmpeg-devel@ffmpeg.org Date: Mon, 23 Sep 2024 15:10:06 +0800 Message-Id: <20240923071008.1599696-3-fei.w.wang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240923071008.1599696-1-fei.w.wang@intel.com> References: <20240923071008.1599696-1-fei.w.wang@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/5] lavc/qsvenc: Support calculate encoded frame quality(MSE/PSNR) when encoding X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: fei.w.wang@intel.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: O8Y4AQW2ym6H From: Fei Wang Once the '-mse' option enabled, MSE/PSNR of each frame will be shown in VERBOSE debug level log. Signed-off-by: Fei Wang --- doc/encoders.texi | 4 + libavcodec/qsvenc.c | 162 +++++++++++++++++++++++++++++++++++---- libavcodec/qsvenc.h | 12 +++ libavcodec/qsvenc_av1.c | 3 + libavcodec/qsvenc_h264.c | 3 + libavcodec/qsvenc_hevc.c | 3 + 6 files changed, 173 insertions(+), 14 deletions(-) diff --git a/doc/encoders.texi b/doc/encoders.texi index 85be976a46..7d1373e0e0 100644 --- a/doc/encoders.texi +++ b/doc/encoders.texi @@ -3624,6 +3624,10 @@ ffmpeg -i input.mp4 -c:v h264_qsv -qsv_params "CodingOption1=1:CodingOption2=2" @end example This option allows fine-grained control over various encoder-specific settings provided by the QSV encoder. + +@item @var{mse} +Supported in h264_qsv, hevc_qsv, and av1_qsv on Windows. Output encoded +frame's quality(MSE/PSNR) information in VERBOSE log. @end table @subsection H264 options diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c index dfef15fa5a..cd89656f74 100644 --- a/libavcodec/qsvenc.c +++ b/libavcodec/qsvenc.c @@ -95,6 +95,9 @@ typedef struct QSVPacket { AVPacket pkt; mfxSyncPoint *sync; mfxBitstream *bs; + int bs_buf_num; + int frameinfo_buf_idx; + int mse_buf_idx; } QSVPacket; static const char *print_profile(enum AVCodecID codec_id, mfxU16 profile) @@ -206,6 +209,9 @@ static void dump_video_param(AVCodecContext *avctx, QSVEncContext *q, #if QSV_HAVE_AC mfxExtAlphaChannelEncCtrl *extalphachannel = NULL; #endif +#if QSV_HAVE_MSE + mfxExtQualityInfoMode *extmse = NULL; +#endif const char *tmp_str = NULL; @@ -228,6 +234,11 @@ static void dump_video_param(AVCodecContext *avctx, QSVEncContext *q, extalphachannel = (mfxExtAlphaChannelEncCtrl *)coding_opts[q->extaplhachannel_idx]; #endif +#if QSV_HAVE_MSE + if (q->extmse_idx > 0) + extmse = (mfxExtQualityInfoMode *)coding_opts[q->extmse_idx]; +#endif + av_log(avctx, AV_LOG_VERBOSE, "profile: %s; level: %"PRIu16"\n", print_profile(avctx->codec_id, info->CodecProfile), info->CodecLevel); @@ -425,6 +436,22 @@ static void dump_video_param(AVCodecContext *avctx, QSVEncContext *q, av_log(avctx, AV_LOG_VERBOSE, "\n"); } #endif + +#if QSV_HAVE_MSE + if (extmse) { + av_log(avctx, AV_LOG_VERBOSE, "MSE: "); + + if (extmse->QualityInfoMode == MFX_QUALITY_INFO_LEVEL_FRAME) + av_log(avctx, AV_LOG_VERBOSE, "ON"); + else if (extmse->QualityInfoMode == MFX_QUALITY_INFO_DISABLE) + av_log(avctx, AV_LOG_VERBOSE, "OFF"); + else + av_log(avctx, AV_LOG_VERBOSE, "unknown"); + + av_log(avctx, AV_LOG_VERBOSE, "\n"); + } +#endif + } static void dump_video_vp9_param(AVCodecContext *avctx, QSVEncContext *q, @@ -522,6 +549,9 @@ static void dump_video_av1_param(AVCodecContext *avctx, QSVEncContext *q, #if QSV_HAVE_EXT_AV1_SCC mfxExtAV1ScreenContentTools *scc = (mfxExtAV1ScreenContentTools*)coding_opts[4]; #endif +#if QSV_HAVE_MSE + mfxExtQualityInfoMode *mse = (mfxExtQualityInfoMode*)coding_opts[5]; +#endif av_log(avctx, AV_LOG_VERBOSE, "profile: %s; level: %"PRIu16"\n", print_profile(avctx->codec_id, info->CodecProfile), info->CodecLevel); @@ -602,6 +632,21 @@ static void dump_video_av1_param(AVCodecContext *avctx, QSVEncContext *q, print_threestate(scc->Palette), print_threestate(scc->IntraBlockCopy)); } #endif + +#if QSV_HAVE_MSE + if (mse) { + av_log(avctx, AV_LOG_VERBOSE, "MSE: "); + + if (mse->QualityInfoMode == MFX_QUALITY_INFO_LEVEL_FRAME) + av_log(avctx, AV_LOG_VERBOSE, "ON"); + else if (mse->QualityInfoMode == MFX_QUALITY_INFO_DISABLE) + av_log(avctx, AV_LOG_VERBOSE, "OFF"); + else + av_log(avctx, AV_LOG_VERBOSE, "unknown"); + + av_log(avctx, AV_LOG_VERBOSE, "\n"); + } +#endif } #endif @@ -1372,6 +1417,21 @@ static int init_video_param(AVCodecContext *avctx, QSVEncContext *q) } #endif +#if QSV_HAVE_MSE + if (q->mse) { + if (QSV_RUNTIME_VERSION_ATLEAST(q->ver, 2, 13)) { + q->extmseparam.Header.BufferId = MFX_EXTBUFF_ENCODED_QUALITY_INFO_MODE; + q->extmseparam.Header.BufferSz = sizeof(q->extmseparam); + q->extmseparam.QualityInfoMode = MFX_QUALITY_INFO_LEVEL_FRAME; + q->extparam_internal[q->nb_extparam_internal++] = (mfxExtBuffer *)&q->extmseparam; + } else { + av_log(avctx, AV_LOG_ERROR, + "This version of runtime doesn't support Mean Squared Error\n"); + return AVERROR_UNKNOWN; + } + } +#endif + if (!check_enc_param(avctx,q)) { av_log(avctx, AV_LOG_ERROR, "some encoding parameters are not supported by the QSV " @@ -1486,6 +1546,12 @@ static int qsv_retrieve_enc_av1_params(AVCodecContext *avctx, QSVEncContext *q) }; #endif +#if QSV_HAVE_MSE + mfxExtQualityInfoMode mse_buf = { + .Header.BufferId = MFX_EXTBUFF_ENCODED_QUALITY_INFO_MODE, + .Header.BufferSz = sizeof(mse_buf), + }; +#endif mfxExtBuffer *ext_buffers[] = { (mfxExtBuffer*)&av1_extend_tile_buf, (mfxExtBuffer*)&av1_bs_param, @@ -1493,6 +1559,9 @@ static int qsv_retrieve_enc_av1_params(AVCodecContext *avctx, QSVEncContext *q) (mfxExtBuffer*)&co3, #if QSV_HAVE_EXT_AV1_SCC (mfxExtBuffer*)&scc_buf, +#endif +#if QSV_HAVE_MSE + (mfxExtBuffer*)&mse_buf, #endif }; @@ -1570,7 +1639,14 @@ static int qsv_retrieve_enc_params(AVCodecContext *avctx, QSVEncContext *q) }; #endif - mfxExtBuffer *ext_buffers[6 + QSV_HAVE_HE + QSV_HAVE_AC]; +#if QSV_HAVE_MSE + mfxExtQualityInfoMode mse_buf = { + .Header.BufferId = MFX_EXTBUFF_ENCODED_QUALITY_INFO_MODE, + .Header.BufferSz = sizeof(mse_buf), + }; +#endif + + mfxExtBuffer *ext_buffers[6 + QSV_HAVE_HE + QSV_HAVE_AC + QSV_HAVE_MSE]; int need_pps = avctx->codec_id != AV_CODEC_ID_MPEG2VIDEO; int ret, ext_buf_num = 0, extradata_offset = 0; @@ -1613,6 +1689,13 @@ static int qsv_retrieve_enc_params(AVCodecContext *avctx, QSVEncContext *q) } #endif +#if QSV_HAVE_MSE + if (q->mse && QSV_RUNTIME_VERSION_ATLEAST(q->ver, 2, 13)) { + q->extmse_idx = ext_buf_num; + ext_buffers[ext_buf_num++] = (mfxExtBuffer*)&mse_buf; + } +#endif + q->param.ExtParam = ext_buffers; q->param.NumExtParam = ext_buf_num; @@ -2228,7 +2311,8 @@ static int submit_frame(QSVEncContext *q, const AVFrame *frame, return ret; } } - qf->surface.Data.TimeStamp = av_rescale_q(frame->pts, q->avctx->time_base, (AVRational){1, 90000}); + qf->surface.Data.TimeStamp = av_rescale_q(frame->pts, q->avctx->time_base, (AVRational){1, 90000}); + qf->surface.Data.FrameOrder = MFX_FRAMEORDER_UNKNOWN; *new_frame = qf; @@ -2578,11 +2662,15 @@ static int encode_frame(AVCodecContext *avctx, QSVEncContext *q, { QSVPacket pkt = { { 0 } }; mfxExtAVCEncodedFrameInfo *enc_info = NULL; +#if QSV_HAVE_MSE + mfxExtQualityInfoOutput *mse = NULL; +#endif mfxExtBuffer **enc_buf = NULL; mfxFrameSurface1 *surf = NULL; QSVFrame *qsv_frame = NULL; mfxEncodeCtrl* enc_ctrl = NULL; + mfxExtBuffer *bs_buf[2]; int ret; if (frame) { @@ -2622,13 +2710,30 @@ static int encode_frame(AVCodecContext *avctx, QSVEncContext *q, enc_info->Header.BufferId = MFX_EXTBUFF_ENCODED_FRAME_INFO; enc_info->Header.BufferSz = sizeof (*enc_info); - pkt.bs->NumExtParam = 1; - enc_buf = av_mallocz(sizeof(mfxExtBuffer *)); - if (!enc_buf) + pkt.frameinfo_buf_idx = pkt.bs_buf_num; + bs_buf[pkt.bs_buf_num++] = (mfxExtBuffer *)enc_info; + } + +#if QSV_HAVE_MSE + if (q->mse) { + mse = av_mallocz(sizeof(*mse)); + if (!mse) goto nomem; - enc_buf[0] = (mfxExtBuffer *)enc_info; + mse->Header.BufferId = MFX_EXTBUFF_ENCODED_QUALITY_INFO_OUTPUT; + mse->Header.BufferSz = sizeof(*mse); + pkt.mse_buf_idx = pkt.bs_buf_num; + bs_buf[pkt.bs_buf_num++] = (mfxExtBuffer *)mse; + } +#endif + + if (pkt.bs_buf_num) { + enc_buf = av_mallocz(sizeof(mfxExtBuffer *) * pkt.bs_buf_num); + if (!enc_buf) + goto nomem; + memcpy(enc_buf, bs_buf, pkt.bs_buf_num * sizeof(mfxExtBuffer *)); pkt.bs->ExtParam = enc_buf; + pkt.bs->NumExtParam = pkt.bs_buf_num; } if (q->set_encode_ctrl_cb && enc_ctrl) { @@ -2683,8 +2788,13 @@ free: av_freep(&pkt.bs); if (avctx->codec_id == AV_CODEC_ID_H264) { av_freep(&enc_info); - av_freep(&enc_buf); } +#if QSV_HAVE_MSE + if (q->mse) + av_freep(&mse); +#endif + if (pkt.bs_buf_num) + av_freep(&enc_buf); } return ret; @@ -2770,7 +2880,7 @@ int ff_qsv_encode(AVCodecContext *avctx, QSVEncContext *q, (!frame && av_fifo_can_read(q->async_fifo))) { QSVPacket qpkt; mfxExtAVCEncodedFrameInfo *enc_info; - mfxExtBuffer **enc_buf; + mfxExtBuffer *enc_buf; enum AVPictureType pict_type; av_fifo_read(q->async_fifo, &qpkt, 1); @@ -2803,13 +2913,29 @@ int ff_qsv_encode(AVCodecContext *avctx, QSVEncContext *q, } if (avctx->codec_id == AV_CODEC_ID_H264) { - enc_buf = qpkt.bs->ExtParam; - enc_info = (mfxExtAVCEncodedFrameInfo *)(*enc_buf); + enc_buf = qpkt.bs->ExtParam[qpkt.frameinfo_buf_idx]; + enc_info = (mfxExtAVCEncodedFrameInfo *)enc_buf; ff_side_data_set_encoder_stats(&qpkt.pkt, enc_info->QP * FF_QP2LAMBDA, NULL, 0, pict_type); av_freep(&enc_info); - av_freep(&enc_buf); } +#if QSV_HAVE_MSE + if (q->mse) { + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(avctx->pix_fmt); + mfxExtQualityInfoOutput *mse; + enc_buf = qpkt.bs->ExtParam[qpkt.mse_buf_idx]; + mse = (mfxExtQualityInfoOutput *)enc_buf; + av_log(avctx, AV_LOG_VERBOSE, "Frame Display order:%d, MSE Y/U/V: %0.2f/%0.2f/%0.2f, " + "PSNR Y/U/V: %0.2f/%0.2f/%0.2f\n", + mse->FrameOrder, mse->MSE[0] / 256.0, mse->MSE[1] / 256.0, mse->MSE[2] / 256.0, + 10.0 * log10(pow(((1 << desc->comp[0].depth) -1), 2) / (mse->MSE[0] / 256.0)), + 10.0 * log10(pow(((1 << desc->comp[1].depth) -1), 2) / (mse->MSE[1] / 256.0)), + 10.0 * log10(pow(((1 << desc->comp[2].depth) -1), 2) / (mse->MSE[2] / 256.0))); + av_freep(&mse); + } +#endif + if (qpkt.bs_buf_num) + av_freep(&qpkt.bs->ExtParam); av_freep(&qpkt.bs); av_freep(&qpkt.sync); @@ -2847,11 +2973,19 @@ int ff_qsv_enc_close(AVCodecContext *avctx, QSVEncContext *q) QSVPacket pkt; while (av_fifo_read(q->async_fifo, &pkt, 1) >= 0) { if (avctx->codec_id == AV_CODEC_ID_H264) { - mfxExtBuffer **enc_buf = pkt.bs->ExtParam; - mfxExtAVCEncodedFrameInfo *enc_info = (mfxExtAVCEncodedFrameInfo *)(*enc_buf); + mfxExtBuffer *enc_buf = pkt.bs->ExtParam[pkt.frameinfo_buf_idx]; + mfxExtAVCEncodedFrameInfo *enc_info = (mfxExtAVCEncodedFrameInfo *)enc_buf; av_freep(&enc_info); - av_freep(&enc_buf); } +#if QSV_HAVE_MSE + if (q->mse) { + mfxExtBuffer *enc_buf = pkt.bs->ExtParam[pkt.mse_buf_idx]; + mfxExtQualityInfoOutput *mse = (mfxExtQualityInfoOutput *)enc_buf; + av_freep(&mse); + } +#endif + if (pkt.bs_buf_num) + av_freep(&pkt.bs->ExtParam); av_freep(&pkt.sync); av_freep(&pkt.bs); av_packet_unref(&pkt.pkt); diff --git a/libavcodec/qsvenc.h b/libavcodec/qsvenc.h index 1e1474d37c..5aa0aae790 100644 --- a/libavcodec/qsvenc.h +++ b/libavcodec/qsvenc.h @@ -46,12 +46,14 @@ #define QSV_HAVE_MF 0 #define QSV_HAVE_HE QSV_VERSION_ATLEAST(2, 4) #define QSV_HAVE_AC QSV_VERSION_ATLEAST(2, 13) +#define QSV_HAVE_MSE QSV_VERSION_ATLEAST(2, 13) #else #define QSV_HAVE_AVBR 0 #define QSV_HAVE_VCM 0 #define QSV_HAVE_MF !QSV_ONEVPL #define QSV_HAVE_HE 0 #define QSV_HAVE_AC 0 +#define QSV_HAVE_MSE 0 #endif #define QSV_COMMON_OPTS \ @@ -151,6 +153,11 @@ { "brc_only", "skip_frame metadata indicates the number of missed frames before the current frame", \ 0, AV_OPT_TYPE_CONST, { .i64 = MFX_SKIPFRAME_BRC_ONLY }, .flags = VE, .unit = "skip_frame" }, +#if QSV_HAVE_MSE +#define QSV_MSE_OPTIONS \ +{ "mse", "Enable output MSE(Mean Squared Error) of each frame", OFFSET(qsv.mse), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, +#endif + extern const AVCodecHWConfigInternal *const ff_qsv_enc_hw_configs[]; typedef int SetEncodeCtrlCB (AVCodecContext *avctx, @@ -197,6 +204,9 @@ typedef struct QSVEncContext { #if QSV_HAVE_EXT_AV1_SCC mfxExtAV1ScreenContentTools extsccparam; #endif +#if QSV_HAVE_MSE + mfxExtQualityInfoMode extmseparam; +#endif mfxExtVideoSignalInfo extvsi; @@ -282,6 +292,7 @@ typedef struct QSVEncContext { int extaplhachannel_idx; int exthevctiles_idx; int exthypermodeparam_idx; + int extmse_idx; int vp9_idx; int max_qp_i; @@ -333,6 +344,7 @@ typedef struct QSVEncContext { int alpha_encode; int palette_mode; int intrabc; + int mse; } QSVEncContext; int ff_qsv_enc_init(AVCodecContext *avctx, QSVEncContext *q); diff --git a/libavcodec/qsvenc_av1.c b/libavcodec/qsvenc_av1.c index 4f035f3d83..ee0617714a 100644 --- a/libavcodec/qsvenc_av1.c +++ b/libavcodec/qsvenc_av1.c @@ -183,6 +183,9 @@ static const AVOption options[] = { QSV_OPTION_EXTBRC QSV_OPTION_LOW_DELAY_BRC QSV_OPTION_MAX_FRAME_SIZE +#if QSV_HAVE_MSE + QSV_MSE_OPTIONS +#endif { "profile", NULL, OFFSET(qsv.profile), AV_OPT_TYPE_INT, { .i64 = MFX_PROFILE_UNKNOWN }, 0, INT_MAX, VE, .unit = "profile" }, { "unknown" , NULL, 0, AV_OPT_TYPE_CONST, { .i64 = MFX_PROFILE_UNKNOWN }, INT_MIN, INT_MAX, VE, .unit = "profile" }, { "main" , NULL, 0, AV_OPT_TYPE_CONST, { .i64 = MFX_PROFILE_AV1_MAIN }, INT_MIN, INT_MAX, VE, .unit = "profile" }, diff --git a/libavcodec/qsvenc_h264.c b/libavcodec/qsvenc_h264.c index 304d1e7dcb..0fc0c852ac 100644 --- a/libavcodec/qsvenc_h264.c +++ b/libavcodec/qsvenc_h264.c @@ -120,6 +120,9 @@ static const AVOption options[] = { #if QSV_HAVE_HE QSV_HE_OPTIONS #endif +#if QSV_HAVE_MSE + QSV_MSE_OPTIONS +#endif { "cavlc", "Enable CAVLC", OFFSET(qsv.cavlc), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, #if QSV_HAVE_VCM diff --git a/libavcodec/qsvenc_hevc.c b/libavcodec/qsvenc_hevc.c index 6ea9c4cdcb..8197e0958e 100644 --- a/libavcodec/qsvenc_hevc.c +++ b/libavcodec/qsvenc_hevc.c @@ -323,6 +323,9 @@ static const AVOption options[] = { #if QSV_HAVE_HE QSV_HE_OPTIONS #endif +#if QSV_HAVE_MSE + QSV_MSE_OPTIONS +#endif { "idr_interval", "Distance (in I-frames) between IDR frames", OFFSET(qsv.idr_interval), AV_OPT_TYPE_INT, { .i64 = 0 }, -1, INT_MAX, VE, .unit = "idr_interval" }, { "begin_only", "Output an IDR-frame only at the beginning of the stream", 0, AV_OPT_TYPE_CONST, { .i64 = -1 }, 0, 0, VE, .unit = "idr_interval" }, From patchwork Mon Sep 23 07:10:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Fei W" X-Patchwork-Id: 51728 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:d154:0:b0:48e:c0f8:d0de with SMTP id bt20csp2274068vqb; Mon, 23 Sep 2024 00:08:24 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXp0+rOrJ6H+ZdbmrkAzi6U6vrojatBnyHs5ikXdQAMIOtHijovb6NLmpTmA4Ve3gejzFO2gpexa/OcTVwz6fr+@gmail.com X-Google-Smtp-Source: AGHT+IFHqm9vEETHN7vMQcjVZjMroaP6okWsLkX7jd+Ke0JsPShMyFr+zwT0zU2Y5wv5f4QlFMDE X-Received: by 2002:a05:6402:520d:b0:5c4:58e3:a8a7 with SMTP id 4fb4d7f45d1cf-5c4637c7fb1mr13130096a12.9.1727075304085; Mon, 23 Sep 2024 00:08:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1727075304; cv=none; d=google.com; s=arc-20240605; b=WYMZr+YxvZ11f/hRrxHPE8FBMOf73LCVjVjeijRV4iBiBUlZ0M9uPHlyBETCh4tGa0 MY5ClhVGBFK8j7A0pglS+p1Tt14/vAcNRx16vjpnsY7jVhZcQ/Wfali9zSKklAyFBOzM /zs8hjtnnphj97MhuIHlSahH3azUEx+hHBeqfum8nPnHVSJetClYAhUgZ6A+MdkY+SmR 3rK+555M61lHcA0ObW5cHsjNzIkB0TsMUBfYSqn6elx1715KEQQPXU1y8BonLfhgjlUt Dy0RjRFZexC0VStI+Tr8Gp7ovB4KlYZjSWtgRxCvw7y4kJx8iVT4Zz7DQiIa3C9fMUNF ob2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=3AEJmlQI1K5FbUFIGBNWfjJM1rJ3NXQObUeZzk/PPZA=; fh=i4ESP4ZRFDcfYfwXKWpOXjc2YhmIGuOsCfZUnwNO0gc=; b=FE5cDNe5ii5tMSDyaJPtIcWvWz62uQPNjyMFG9BsYvBItj3LpBE/Nm/I+Y7yhHYjuQ GcaSo7BbwQAT8sKsRmo5OJRztIjM/89u+xEwGFYWmyVwJas6uvK3kXkEz6Nch1o/jhwz rVVitac9vq01679fvVZuzvPyrC7ppNfKqZpvanGoxLq4dhvcD1cP/pgoc3jNz5EmXlxZ YmQobttGL5wkBAFKNHQLdVoxf2OtoSuY1PVL+kykkWtDjSK05mjYfV9NbiyKw1SvUHVk ZS3soALi8Ir+2LBmFvBMrwdxDgPd+NBD2ZlrIat06N99BUu41Q2LlidzC9j260cdZGzL J19w==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=n1sUpjke; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5c42bb5cf7bsi13035684a12.184.2024.09.23.00.08.23; Mon, 23 Sep 2024 00:08:24 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=n1sUpjke; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9B21E68DBE9; Mon, 23 Sep 2024 10:07:55 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8E3DC68DBBB for ; Mon, 23 Sep 2024 10:07:48 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727075274; x=1758611274; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RMQXWRa+kvpEJ1E9Ctkcr0WJUW2X96sZxbrvnVyvZXY=; b=n1sUpjkesNupKKhpv/DUOyD6xl81DPkxCDQtiL9rInQGZgiZEcVOv448 OKDpqk4hACGnMliSRlOWrPKPJw9O3laX/WjxrHldyTdRwXwjAfIc7Iv4Q T3Kx0ynt4KEwT7/qZDWzNv55+/faYZTW9krCOSOWQUOh9miR7uRVxniB3 5qsJcEEWxq+FcuGB13xScdoxc1dRv32UWMJQzQNoEp3EW7tMZylfIEM+I OdZBjmOjRn6RApkbne44mjbz/PJEVp0Zn4UiXzCDZU0s8TKljPxp2KJzI 1lXDq6SITQqRbRyNvjTY7O38wF1Yw78ImmuiI9FOEvTi+Tp/VvufSaLvH w==; X-CSE-ConnectionGUID: s9KJzC9VQ7eSUS8wR4VuJg== X-CSE-MsgGUID: NDJoS8YMQGOnXWZtbUFCOQ== X-IronPort-AV: E=McAfee;i="6700,10204,11202"; a="43525030" X-IronPort-AV: E=Sophos;i="6.10,250,1719903600"; d="scan'208";a="43525030" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 00:07:43 -0700 X-CSE-ConnectionGUID: DQg8TwLYRw27bd1Mlztj8w== X-CSE-MsgGUID: X6FVPDgKR3al0MATEozZQg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,250,1719903600"; d="scan'208";a="70560534" Received: from feiwan1-desk3.sh.intel.com ([10.238.208.39]) by fmviesa007.fm.intel.com with ESMTP; 23 Sep 2024 00:07:42 -0700 From: fei.w.wang-at-intel.com@ffmpeg.org To: ffmpeg-devel@ffmpeg.org Date: Mon, 23 Sep 2024 15:10:07 +0800 Message-Id: <20240923071008.1599696-4-fei.w.wang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240923071008.1599696-1-fei.w.wang@intel.com> References: <20240923071008.1599696-1-fei.w.wang@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 4/5] lavc/qsvenc: Add sliding window bitrate control for CBR X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: fei.w.wang@intel.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ZvWjdEGw3e1v From: Fei Wang Sliding window bitrate control will provide a more stable bitrate when use CBR bitrate control mode. Signed-off-by: Fei Wang --- doc/encoders.texi | 9 +++++++++ libavcodec/qsvenc.c | 25 +++++++++++++++++++++++++ libavcodec/qsvenc.h | 11 +++++++++++ libavcodec/qsvenc_av1.c | 3 +++ libavcodec/qsvenc_h264.c | 3 +++ libavcodec/qsvenc_hevc.c | 3 +++ 6 files changed, 54 insertions(+) diff --git a/doc/encoders.texi b/doc/encoders.texi index 7d1373e0e0..b9f7f80e74 100644 --- a/doc/encoders.texi +++ b/doc/encoders.texi @@ -3628,6 +3628,15 @@ This option allows fine-grained control over various encoder-specific settings p @item @var{mse} Supported in h264_qsv, hevc_qsv, and av1_qsv on Windows. Output encoded frame's quality(MSE/PSNR) information in VERBOSE log. + +@item @var{sw_size} +Number of frames used for sliding window(Only available for CBR bitrate +control mode on Windows). Range from 30 to 60 for AVC and HEVC. 30 to 120 for AV1. + +@item @var{sw_max_bitrate_factor} +Factor between bitrate and max bitrate for frames in sliding window. +Range from 1.1 to 2.0. + @end table @subsection H264 options diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c index cd89656f74..872a4d2cdf 100644 --- a/libavcodec/qsvenc.c +++ b/libavcodec/qsvenc.c @@ -398,6 +398,11 @@ static void dump_video_param(AVCodecContext *avctx, QSVEncContext *q, av_log(avctx, AV_LOG_VERBOSE, "MaxFrameSizeI: %d; ", co3->MaxFrameSizeI); av_log(avctx, AV_LOG_VERBOSE, "MaxFrameSizeP: %d\n", co3->MaxFrameSizeP); av_log(avctx, AV_LOG_VERBOSE, "ScenarioInfo: %"PRId16"\n", co3->ScenarioInfo); +#if QSV_HAVE_SW + av_log(avctx, AV_LOG_VERBOSE, + "WinBRCSize: %"PRIu16"; WinBRCMaxAvgKbps: %"PRIu16"\n", + co3->WinBRCSize, co3->WinBRCMaxAvgKbps); +#endif } if (exthevctiles) { @@ -647,6 +652,12 @@ static void dump_video_av1_param(AVCodecContext *avctx, QSVEncContext *q, av_log(avctx, AV_LOG_VERBOSE, "\n"); } #endif + +#if QSV_HAVE_SW + av_log(avctx, AV_LOG_VERBOSE, + "WinBRCSize: %"PRIu16"; WinBRCMaxAvgKbps: %"PRIu16"\n", + co3->WinBRCSize, co3->WinBRCMaxAvgKbps); +#endif } #endif @@ -1250,6 +1261,20 @@ static int init_video_param(AVCodecContext *avctx, QSVEncContext *q) q->extco3.TransformSkip = MFX_CODINGOPTION_UNKNOWN; q->extco3.GPB = q->gpb ? MFX_CODINGOPTION_ON : MFX_CODINGOPTION_OFF; } + +#if QSV_HAVE_SW + if ((avctx->codec_id == AV_CODEC_ID_H264 || avctx->codec_id == AV_CODEC_ID_HEVC || + avctx->codec_id == AV_CODEC_ID_AV1) && q->sw_size) { + if (QSV_RUNTIME_VERSION_ATLEAST(q->ver, 2, 13)) { + q->extco3.WinBRCSize = q->sw_size; + q->extco3.WinBRCMaxAvgKbps = (int)(q->sw_max_bitrate_factor * q->param.mfx.TargetKbps); + } else { + av_log(avctx, AV_LOG_ERROR, + "This version of runtime doesn't support sliding windows bitrate control\n"); + return AVERROR_UNKNOWN; + } + } +#endif q->extparam_internal[q->nb_extparam_internal++] = (mfxExtBuffer *)&q->extco3; } diff --git a/libavcodec/qsvenc.h b/libavcodec/qsvenc.h index 5aa0aae790..64c8cabbc7 100644 --- a/libavcodec/qsvenc.h +++ b/libavcodec/qsvenc.h @@ -24,6 +24,7 @@ #define AVCODEC_QSVENC_H #include +#include #include #include "libavutil/common.h" @@ -47,6 +48,7 @@ #define QSV_HAVE_HE QSV_VERSION_ATLEAST(2, 4) #define QSV_HAVE_AC QSV_VERSION_ATLEAST(2, 13) #define QSV_HAVE_MSE QSV_VERSION_ATLEAST(2, 13) +#define QSV_HAVE_SW QSV_VERSION_ATLEAST(2, 13) #else #define QSV_HAVE_AVBR 0 #define QSV_HAVE_VCM 0 @@ -54,6 +56,7 @@ #define QSV_HAVE_HE 0 #define QSV_HAVE_AC 0 #define QSV_HAVE_MSE 0 +#define QSV_HAVE_SW 0 #endif #define QSV_COMMON_OPTS \ @@ -158,6 +161,12 @@ { "mse", "Enable output MSE(Mean Squared Error) of each frame", OFFSET(qsv.mse), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, #endif +#if QSV_HAVE_SW +#define QSV_SW_OPTIONS \ +{ "sw_size", "Number of frames used for sliding window(Only available for CBR bitrate control mode)", OFFSET(qsv.sw_size), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, VE }, \ +{ "sw_max_bitrate_factor", "Factor between bitrate and max bitrate for frames in sliding window(Only available when sw_size is set)", OFFSET(qsv.sw_max_bitrate_factor), AV_OPT_TYPE_FLOAT, { .i64 = 0 }, 0, FLT_MAX, VE }, +#endif + extern const AVCodecHWConfigInternal *const ff_qsv_enc_hw_configs[]; typedef int SetEncodeCtrlCB (AVCodecContext *avctx, @@ -345,6 +354,8 @@ typedef struct QSVEncContext { int palette_mode; int intrabc; int mse; + int sw_size; + float sw_max_bitrate_factor; } QSVEncContext; int ff_qsv_enc_init(AVCodecContext *avctx, QSVEncContext *q); diff --git a/libavcodec/qsvenc_av1.c b/libavcodec/qsvenc_av1.c index ee0617714a..f685214daa 100644 --- a/libavcodec/qsvenc_av1.c +++ b/libavcodec/qsvenc_av1.c @@ -185,6 +185,9 @@ static const AVOption options[] = { QSV_OPTION_MAX_FRAME_SIZE #if QSV_HAVE_MSE QSV_MSE_OPTIONS +#endif +#if QSV_HAVE_SW + QSV_SW_OPTIONS #endif { "profile", NULL, OFFSET(qsv.profile), AV_OPT_TYPE_INT, { .i64 = MFX_PROFILE_UNKNOWN }, 0, INT_MAX, VE, .unit = "profile" }, { "unknown" , NULL, 0, AV_OPT_TYPE_CONST, { .i64 = MFX_PROFILE_UNKNOWN }, INT_MIN, INT_MAX, VE, .unit = "profile" }, diff --git a/libavcodec/qsvenc_h264.c b/libavcodec/qsvenc_h264.c index 0fc0c852ac..d05371350f 100644 --- a/libavcodec/qsvenc_h264.c +++ b/libavcodec/qsvenc_h264.c @@ -123,6 +123,9 @@ static const AVOption options[] = { #if QSV_HAVE_MSE QSV_MSE_OPTIONS #endif +#if QSV_HAVE_SW + QSV_SW_OPTIONS +#endif { "cavlc", "Enable CAVLC", OFFSET(qsv.cavlc), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, #if QSV_HAVE_VCM diff --git a/libavcodec/qsvenc_hevc.c b/libavcodec/qsvenc_hevc.c index 8197e0958e..8d08991979 100644 --- a/libavcodec/qsvenc_hevc.c +++ b/libavcodec/qsvenc_hevc.c @@ -326,6 +326,9 @@ static const AVOption options[] = { #if QSV_HAVE_MSE QSV_MSE_OPTIONS #endif +#if QSV_HAVE_SW + QSV_SW_OPTIONS +#endif { "idr_interval", "Distance (in I-frames) between IDR frames", OFFSET(qsv.idr_interval), AV_OPT_TYPE_INT, { .i64 = 0 }, -1, INT_MAX, VE, .unit = "idr_interval" }, { "begin_only", "Output an IDR-frame only at the beginning of the stream", 0, AV_OPT_TYPE_CONST, { .i64 = -1 }, 0, 0, VE, .unit = "idr_interval" }, From patchwork Mon Sep 23 07:10:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Fei W" X-Patchwork-Id: 51730 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:d154:0:b0:48e:c0f8:d0de with SMTP id bt20csp2280037vqb; Mon, 23 Sep 2024 00:24:17 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCV9IlQfCuvy/Rez9/tw+aLMRip5lWN4QV2/5kUEHp11iqOSVSaeqBBmdt5Ixps4jRabVOCHMH6CkeU9HbPZkUVu@gmail.com X-Google-Smtp-Source: AGHT+IGls2aWpH6jvOAMJ9q/CxL/aeCzIWbZTcmRovnuw3wVk6MyprVxSXcdrjha8iPudX1ERoOq X-Received: by 2002:a05:6512:1107:b0:52c:dd3d:85af with SMTP id 2adb3069b0e04-536ad17d3ecmr4334985e87.25.1727076257553; Mon, 23 Sep 2024 00:24:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1727076257; cv=none; d=google.com; s=arc-20240605; b=jU5Om0MTp/s78IZKD/u80HqT8o8eYBIZ58Px74CaWCSE3mElsIS64Fyy7z1drOanzE pOZ7OOxkkyECRSK8tMIrvq2ABcy3FZa3zY1MIxdTlppi7NAfanBcKhc6mNqTqY2OXux5 T4n1tSLkeq7PRVezZL5IRselKEmf337J7nAe2/WUii9QrI77+Q3baOK6rSsNsS1+Pj/q 9iu0xZQJflepcGdi3/A5ATVQnpps8+6BLc3zLDBIlGv9qml+RJY79p03hAcli7Bhfmft sCdWbLdUdb1zWCaq51Mw49WSzc9C7cQzG7h6cvPfDJmqWUoXSYpF9zmWOluKtpu0yI8h Aw/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=oNY1kfI1EtB/y1UFyWpy9KavAUcxfWaTk9bU5qrLoI4=; fh=i4ESP4ZRFDcfYfwXKWpOXjc2YhmIGuOsCfZUnwNO0gc=; b=Oo9l0MYTBtFG+VWTBWjsWKJz/jbcb3eLDl8KybVg/PugSsz95b0K3QqeBdFkDcjazu NdNFxEHmvxOepSpeljsHQBidVJobSUS0RHrcm4qjzFoCfksBqkRNpX+hinniQwaD3sQF 3BltBhaGX1HjfqnOEY4BThfRlMAXOXMlgFl3jm3CboNEkUyMxKG6cf7U1VltYHQjF2jV 7JI7fKpuPgtcID0SlFVRVzWDBAh41tvqVJdrIXSA/X687r0zldsmznCwo+GVLLVqONgA MEFsfFyHTrXT8LKTdQFlPqT6qeBzmjJKgA+uHuPhs/8LUA6e5g04VkvIScgbj4nvd9B1 wPpQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b="QLrFo6n/"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-5368708e207si6723856e87.260.2024.09.23.00.24.16; Mon, 23 Sep 2024 00:24:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b="QLrFo6n/"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 295CA68DC23; Mon, 23 Sep 2024 10:07:58 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9871A68DBA6 for ; Mon, 23 Sep 2024 10:07:49 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727075275; x=1758611275; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dAvC29ggl0ImKZsCXXVvu9fnIF/SHJLzqpJS++hzp1I=; b=QLrFo6n/ZzWB0c9bnCx14axn83s6ir7IUVJ4AuIkH+7w1ogq6smLYvx6 UStkxPqbY/Ad1ZNf6npQVJ5NN9gOh0JAf2ya+l3+vAIp030E4A9j/BswL 83pTOkpPwqo63m6U1o0HqL8sMTL7+2gMph35Bkehu35A3Zh1bYIRlD7Op w7pg6btJ6R7oAjKUvdvBTCDvoEv4qJIMoVKayHnhI/XA3IGWpl1ga7onj ldQDiT3UgFX5xR8arNR72CH4uLu0i329y9KilJJd2LCRizgjFjU6hSW94 Qw0VSZT1iNpDCMKo14QKmfZ2cXBY/ErQrHxT7udLrk7Wx3NH5WmV2qhPb w==; X-CSE-ConnectionGUID: rSirPutES3mNtPu8CjqlpQ== X-CSE-MsgGUID: aCU2kd8NQp2P6/MKBiAiGQ== X-IronPort-AV: E=McAfee;i="6700,10204,11202"; a="43525032" X-IronPort-AV: E=Sophos;i="6.10,250,1719903600"; d="scan'208";a="43525032" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 00:07:44 -0700 X-CSE-ConnectionGUID: aNcDfMyySHuXp08Okw8GEw== X-CSE-MsgGUID: cwULXlu3TP2urupN/JTlTQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,250,1719903600"; d="scan'208";a="70560541" Received: from feiwan1-desk3.sh.intel.com ([10.238.208.39]) by fmviesa007.fm.intel.com with ESMTP; 23 Sep 2024 00:07:43 -0700 From: fei.w.wang-at-intel.com@ffmpeg.org To: ffmpeg-devel@ffmpeg.org Date: Mon, 23 Sep 2024 15:10:08 +0800 Message-Id: <20240923071008.1599696-5-fei.w.wang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240923071008.1599696-1-fei.w.wang@intel.com> References: <20240923071008.1599696-1-fei.w.wang@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 5/5] Changelog: Add new entries for QSV encoder and bump lavc version X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: fei.w.wang@intel.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: EfzTasjmPU0a From: Fei Wang Signed-off-by: Fei Wang --- Changelog | 4 ++++ libavcodec/version.h | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/Changelog b/Changelog index 49a16da7ca..3f9bfcdd4e 100644 --- a/Changelog +++ b/Changelog @@ -26,6 +26,10 @@ version : - stream specifiers in fftools can now match by stream disposition - LCEVC enhancement data exporting in H.26x and MP4/ISOBMFF - LCEVC filter +- Alpha encode for hevc_qsv encoder +- Screen Content Tool for av1_qsv encoder +- Encode frame quality capture for {h264, hevc, av1}_qsv encoder +- Sliding windows bitrate control for {h264, hevc, av1}_qsv encoder version 7.0: diff --git a/libavcodec/version.h b/libavcodec/version.h index 2618016a83..65bc52fb24 100644 --- a/libavcodec/version.h +++ b/libavcodec/version.h @@ -29,7 +29,7 @@ #include "version_major.h" -#define LIBAVCODEC_VERSION_MINOR 17 +#define LIBAVCODEC_VERSION_MINOR 18 #define LIBAVCODEC_VERSION_MICRO 100 #define LIBAVCODEC_VERSION_INT AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \