From patchwork Wed Jun 8 05:02:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chen, Wenbin" X-Patchwork-Id: 36092 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp7303096pzj; Tue, 7 Jun 2022 22:03:21 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwOKUzX5yqcJolgjNwGqUCBkZYEvcOrMf2/AHubxhik4aYSBY6vRdYCSAEsIoGo1v4K2x9I X-Received: by 2002:a05:6402:2405:b0:431:6ef1:e2a with SMTP id t5-20020a056402240500b004316ef10e2amr13254537eda.26.1654664601063; Tue, 07 Jun 2022 22:03:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1654664601; cv=none; d=google.com; s=arc-20160816; b=Xu4o3Z/rqsjHaJYvc6FDmg99us88YJ/hbTcEUsJQ1B2rNXwo8YOBmX9+G/S+3sdESa twWHppV7rD5rZFiQQfXKU8fkrdIHzlMKdLmPuFMP8NF0qb1Qz8MA44z8zvhvZoZp99tK rCfUTcEZpyLqfK3rdkYKOwI90GFSqfCVMLRYn0weaWGBwGNVy58dkSb/BJlkWLYMFA4G rDFeMIfvo0nk2VVFk9KwZDaZ1haxs7yXnMpzWKJgU5Xly9Qt8EMN6hqB143Lk+s+OhAK XJcz0U8w9QwoAXe4U3HsKjJPM+b870rC4FLgd+bcjFTLe5QPAY8gsPjZhCBCxAMz/o3u SaNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=1h7vrndqPA46hI/bUa/jfrlts+iv5IqIRgTbMoWZJu8=; b=NPoKW+fz+PUlojocOOMBb5xdDfPMcf3nHgT+vcDPnA2OYd2tTauL/TSWrN5BhB6VaA 5P/4rdmEhcVDJki9urklNLKKKye99LoHDYXIUmCjqaT1GRV39snvzJG2niMrCfkt09p+ ds2JuLMy+HEtdm2yuoPxZqjqeGKdf2rB+vJ0luLbYwnD4+xg3sbVJliM5kLsnbRvHxk8 dK8z+AU4vBesdoH71k+B+YFCGFw3hsUvk7A4C1cNjcYlNlGOrkz13eSutBxq0D4Buakm mCYKh3RO3AMEiJ9q/DsgySi2/labmSsRkhi+n5GGiG4ypzWLSOek3LBgE43z3cfKDWl9 irvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b="O9SXqY/H"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id h23-20020a170906261700b006f3c442e22fsi19399730ejc.361.2022.06.07.22.03.19; Tue, 07 Jun 2022 22:03:21 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b="O9SXqY/H"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A5AD968B2A8; Wed, 8 Jun 2022 08:03:16 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7881C68B0DB for ; Wed, 8 Jun 2022 08:03:08 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1654664594; x=1686200594; h=from:to:subject:date:message-id:mime-version: content-transfer-encoding; bh=1Eh6AyMXPcsPPOzD5DppWobA3BGnqW1pAzU+5IrXAYE=; b=O9SXqY/HDrjdAEyZsKYzC4z+o4s/Go9kR9J1tFBe73QLRQn3YdfaZ/1m n+DzZom0VomGu0sFQUtK/aLGpzrnTlxZiJYSLZRQYlI6cKoVzHpmEMwNK MzJnUsxiFSnmkGugn4JBIVVy2o4A8RsqvI1TBTik1kRmrl4W49Nnu1TRv cgiElY7K2JBIMeglZrOn6BlcNg8qSDx8EsqDK5UPWLUbaZJI/+T2pgWG+ GE/KnGyqAz3OwMudQ5CC6zMDUuZIE8ubZ4JRcKbrvcmOIaG05t64p9vXK N9JGYwyVGb0yTmLee03dAbVJ5jqNHo2Zy0+dc7XlLtic7ByRe/VpBnEaJ Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10371"; a="277579748" X-IronPort-AV: E=Sophos;i="5.91,285,1647327600"; d="scan'208";a="277579748" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jun 2022 22:03:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,285,1647327600"; d="scan'208";a="670349391" Received: from wenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.4]) by FMSMGA003.fm.intel.com with ESMTP; 07 Jun 2022 22:03:04 -0700 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Wed, 8 Jun 2022 13:02:48 +0800 Message-Id: <20220608050248.3669348-1-wenbin.chen@intel.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4] libavcodec/qsvenc: add ROI support to qsv encoder X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: wmukRcTwNYg1 Use The mfxEncoderCtrl parameter to enable ROI. Get side data "AVRegionOfInterest" and use it to configure "mfxExtEncoderROI" which is the MediaSDK's ROI configuration. Signed-off-by: Wenbin Chen --- libavcodec/qsv_internal.h | 4 ++ libavcodec/qsvenc.c | 85 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 89 insertions(+) diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h index e2aecdcbd6..8131acdae9 100644 --- a/libavcodec/qsv_internal.h +++ b/libavcodec/qsv_internal.h @@ -51,6 +51,9 @@ #define ASYNC_DEPTH_DEFAULT 4 // internal parallelism #define QSV_MAX_ENC_PAYLOAD 2 // # of mfxEncodeCtrl payloads supported +#define QSV_MAX_ENC_EXTPARAM 2 + +#define QSV_MAX_ROI_NUM 256 #define QSV_MAX_FRAME_EXT_PARAMS 4 @@ -83,6 +86,7 @@ typedef struct QSVFrame { int num_ext_params; mfxPayload *payloads[QSV_MAX_ENC_PAYLOAD]; ///< used for enc_ctrl.Payload + mfxExtBuffer *extparam[QSV_MAX_ENC_EXTPARAM]; ///< used for enc_ctrl.ExtParam int queued; int used; diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c index 03e9e5523d..902bada55b 100644 --- a/libavcodec/qsvenc.c +++ b/libavcodec/qsvenc.c @@ -1390,15 +1390,29 @@ static void free_encoder_ctrl_payloads(mfxEncodeCtrl* enc_ctrl) } } +static void free_encoder_ctrl_extparam(mfxEncodeCtrl* enc_ctrl) +{ + if (enc_ctrl) { + int i; + for (i = 0; i < enc_ctrl->NumExtParam && i < QSV_MAX_ENC_EXTPARAM; i++) { + if (enc_ctrl->ExtParam[i]) + av_freep(&(enc_ctrl->ExtParam[i])); + } + enc_ctrl->NumExtParam = 0; + } +} + static void clear_unused_frames(QSVEncContext *q) { QSVFrame *cur = q->work_frames; while (cur) { if (cur->used && !cur->surface.Data.Locked) { free_encoder_ctrl_payloads(&cur->enc_ctrl); + free_encoder_ctrl_extparam(&cur->enc_ctrl); //do not reuse enc_ctrl from previous frame memset(&cur->enc_ctrl, 0, sizeof(cur->enc_ctrl)); cur->enc_ctrl.Payload = cur->payloads; + cur->enc_ctrl.ExtParam = cur->extparam; if (cur->frame->format == AV_PIX_FMT_QSV) { av_frame_unref(cur->frame); } @@ -1436,6 +1450,7 @@ static int get_free_frame(QSVEncContext *q, QSVFrame **f) return AVERROR(ENOMEM); } frame->enc_ctrl.Payload = frame->payloads; + frame->enc_ctrl.ExtParam = frame->extparam; *last = frame; *f = frame; @@ -1537,6 +1552,67 @@ static void print_interlace_msg(AVCodecContext *avctx, QSVEncContext *q) } } +static int set_roi_encode_ctrl(AVCodecContext *avctx, const AVFrame *frame, + mfxEncodeCtrl *enc_ctrl) +{ + AVFrameSideData *sd = NULL; + int mb_size; + + if (avctx->codec_id == AV_CODEC_ID_H264) + mb_size = 16; + else if (avctx->codec_id == AV_CODEC_ID_H265) + mb_size = 32; + else + return 0; + + if (frame) + sd = av_frame_get_side_data(frame, AV_FRAME_DATA_REGIONS_OF_INTEREST); + + if (sd) { + mfxExtEncoderROI *enc_roi = NULL; + AVRegionOfInterest *roi; + uint32_t roi_size; + int nb_roi, i; + + roi = (AVRegionOfInterest *)sd->data; + roi_size = roi->self_size; + if (!roi_size || sd->size % roi_size) { + av_log(avctx, AV_LOG_ERROR, "Invalid ROI Data.\n"); + return AVERROR(EINVAL); + } + nb_roi = sd->size / roi_size; + if (nb_roi > QSV_MAX_ROI_NUM) { + av_log(avctx, AV_LOG_WARNING, "More ROIs set than " + "supported by driver (%d > %d).\n", + nb_roi, QSV_MAX_ROI_NUM); + nb_roi = QSV_MAX_ROI_NUM; + } + + enc_roi = av_mallocz(sizeof(*enc_roi)); + if (!enc_roi) + return AVERROR(ENOMEM); + enc_roi->Header.BufferId = MFX_EXTBUFF_ENCODER_ROI; + enc_roi->Header.BufferSz = sizeof(*enc_roi); + enc_roi->NumROI = nb_roi; + enc_roi->ROIMode = MFX_ROI_MODE_QP_DELTA; + for (i = 0; i < nb_roi; i++) { + roi = (AVRegionOfInterest *)(sd->data + roi_size * i); + enc_roi->ROI[i].Top = FFALIGN(roi->top, mb_size); + enc_roi->ROI[i].Bottom = FFALIGN(roi->bottom, mb_size); + enc_roi->ROI[i].Left = FFALIGN(roi->left, mb_size); + enc_roi->ROI[i].Right = FFALIGN(roi->right, mb_size); + enc_roi->ROI[i].DeltaQP = + roi->qoffset.num * 51 / roi->qoffset.den; + av_log(avctx, AV_LOG_DEBUG, "ROI: (%d,%d)-(%d,%d) -> %+d.\n", + roi->top, roi->left, roi->bottom, roi->right, + enc_roi->ROI[i].DeltaQP); + } + enc_ctrl->ExtParam[enc_ctrl->NumExtParam] = (mfxExtBuffer *)enc_roi; + enc_ctrl->NumExtParam++; + } + return 0; +} + static int encode_frame(AVCodecContext *avctx, QSVEncContext *q, const AVFrame *frame) { @@ -1599,6 +1675,14 @@ static int encode_frame(AVCodecContext *avctx, QSVEncContext *q, q->set_encode_ctrl_cb(avctx, frame, &qsv_frame->enc_ctrl); } + if ((avctx->codec_id == AV_CODEC_ID_H264 || + avctx->codec_id == AV_CODEC_ID_H265) && + enc_ctrl && QSV_RUNTIME_VERSION_ATLEAST(q->ver, 1, 8)) { + ret = set_roi_encode_ctrl(avctx, frame, enc_ctrl); + if (ret < 0) + goto free; + } + pkt.sync = av_mallocz(sizeof(*pkt.sync)); if (!pkt.sync) goto nomem; @@ -1721,6 +1805,7 @@ int ff_qsv_enc_close(AVCodecContext *avctx, QSVEncContext *q) while (cur) { q->work_frames = cur->next; av_frame_free(&cur->frame); + free_encoder_ctrl_extparam(&cur->enc_ctrl); free_encoder_ctrl_payloads(&cur->enc_ctrl); av_freep(&cur); cur = q->work_frames;