From patchwork Mon Jun 27 07:24:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chen, Wenbin" X-Patchwork-Id: 36482 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp1855837pzh; Mon, 27 Jun 2022 00:32:12 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tHKkt0iwLngsi04m3w6hVg0UFKUuScTg8SE7KxJkdDwmCJ3zssOPFWlznmicp8ElUI9bYi X-Received: by 2002:a17:906:1018:b0:718:dd3f:f28c with SMTP id 24-20020a170906101800b00718dd3ff28cmr11753545ejm.55.1656315132419; Mon, 27 Jun 2022 00:32:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656315132; cv=none; d=google.com; s=arc-20160816; b=AdTwl9cWHemCOOlYjs+Ywa4AJn1+pKyc5ToWaTQuCqj0PXJVwa9CI6PGcFbdiSfrGY RMbu5+1zZ69mcenc7HRWXP1XlQ/k4feQ6sylgsaEpZWHH76aGGVdc+FPQRQ2f/M3hpqP WH4ptNl950f1j0c+YRAdCm4sUAj57/1wlckLKbyDr9+XqtWlQw2NPGQ0mW1DuHzauEVo pMyfmKi+8+ouyG590OXEdOFyBd6EhzUneFCvaUIblL+tRg5k3S7FtCLcV5cyRVqgv55M i20RGrs7CRd4lIM2PLLf7NC7rni6MBAn4LttAoAtqdKPqayVpzgMHMgmdqhyVyr+3WMH XKGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=jPkOAsG0kDNxob6cQxVTufavCjGLE7fsDqmoJtZSSZk=; b=K/dxq1BeNTI0d1Cu9K8Un0A360UAVuBT7Y9dm+CLeQQ8C2VUS2wyC/k90XsgrWEPMb 454v3silgD8nXEjIVzuv3cGVIAcLPJnzr/mCYHqDvnpwFcET75eO5ZaLI/k+wfR0W7iK uQ0d9YbslQOAoAcBcnwtc1Jk9k/sMT2B66jwyTDtuAbP0ZS4jTNknnfTC1s6avizjSYl /bOtkI8HxLRvQSJygggrARab0TF1NGevxE1BXdpoS9BjHYQAzXU77aQPNiG61VeC/afg mcLscNIA8HyePr3/+eQ0T2sE4QGqbQOm3DllYmOO/abgz3bL9RFLWpX8B1Ii5KBbNuKY 2JgA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=kXLxCrIo; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b11-20020a056402084b00b00435a2bfe481si13336742edz.597.2022.06.27.00.32.07; Mon, 27 Jun 2022 00:32:12 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=kXLxCrIo; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9663E68B8B8; Mon, 27 Jun 2022 10:32:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D7FEB68B875 for ; Mon, 27 Jun 2022 10:31:57 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1656315123; x=1687851123; h=from:to:subject:date:message-id:mime-version: content-transfer-encoding; bh=KrkAfFMO4REQ1lv4BJsuwaD8FDrkj6KjWMzeO7w4G38=; b=kXLxCrIoBR3cLYsllgGZ/CvqUhRuJiofo3OgisDl+u80Bd8LsZR/kZUH JbQgXVNmHEwaVi2kVQD85aKJxi6Z6iGj4jmKqnuXvL0pUUJjaBkc9mNTj WjnHj8fS+1pcyudiX5u2ufDMEexZwt6BFrATsxCfXQ/tMTo9aNrPn83Mz WIeHOtY0ESI3u+kZMBdj3Bq/dPSRRshJVus3W6qAKmIJZQeeb7o8FUJuh LsXOxM9pB2YeLEgaptiTD4GdcwZUHk8IueRBObVf7IBJla1Q3Uc22piKY JPbdHKtOEdo0nlyjVlv9PdV+CfzQCkEc0GAUeeIuW+6OhLHDmtwpzv8Ov A==; X-IronPort-AV: E=McAfee;i="6400,9594,10390"; a="261798729" X-IronPort-AV: E=Sophos;i="5.92,225,1650956400"; d="scan'208";a="261798729" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jun 2022 00:28:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,225,1650956400"; d="scan'208";a="646312218" Received: from wenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.4]) by fmsmga008.fm.intel.com with ESMTP; 27 Jun 2022 00:28:47 -0700 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Mon, 27 Jun 2022 15:24:19 +0800 Message-Id: <20220627072419.589256-1-wenbin.chen@intel.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] libavcodec/qsvenc: Add support to qsv to encode external surface. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 3OnGaDVMXjc/ Qsv encoder only encode the frame that are pre-allocated, so qsv encoder cannot encode the frame mapped from other components. In fact, MediaSDK can encode frame that are dynamically created. I add the support to qsv to encode external frame. Now the following command line works: ffmpeg -v verbose -init_hw_device vaapi=va -init_hw_device qsv=qs@va \ -hwaccel qsv -hwaccel_device qs -c:v h264_qsv -i input.264 -vf \ "hwmap=derive_device=vaapi,format=vaapi,hwmap=derive_device=vulkan, \ format=vulkan,scale_vulkan=w=1920:h=1080,hwmap=derive_device=vaapi, \ format=vaapi,hwmap=derive_device=qsv:reverse=1:extra_hw_frames=16, \ format=qsv" -c:v h264_qsv output.264 Signed-off-by: Wenbin Chen Signed-off-by: Tong Wu --- libavcodec/qsv_internal.h | 1 + libavcodec/qsvenc.c | 19 ++++++++-- libavutil/hwcontext_qsv.c | 79 ++++++++++++++++++++++++++++++++++----- 3 files changed, 86 insertions(+), 13 deletions(-) diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h index 8131acdae9..2ee523ec3d 100644 --- a/libavcodec/qsv_internal.h +++ b/libavcodec/qsv_internal.h @@ -90,6 +90,7 @@ typedef struct QSVFrame { int queued; int used; + int external_frame; struct QSVFrame *next; } QSVFrame; diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c index 2382c2f5f7..e269ece5d9 100644 --- a/libavcodec/qsvenc.c +++ b/libavcodec/qsvenc.c @@ -1421,6 +1421,10 @@ static void clear_unused_frames(QSVEncContext *q) memset(&cur->enc_ctrl, 0, sizeof(cur->enc_ctrl)); cur->enc_ctrl.Payload = cur->payloads; cur->enc_ctrl.ExtParam = cur->extparam; + if (cur->external_frame) { + av_freep(&cur->surface.Data.MemId); + cur->external_frame = 0; + } if (cur->frame->format == AV_PIX_FMT_QSV) { av_frame_unref(cur->frame); } @@ -1486,10 +1490,17 @@ static int submit_frame(QSVEncContext *q, const AVFrame *frame, if (q->frames_ctx.mids) { ret = ff_qsv_find_surface_idx(&q->frames_ctx, qf); - if (ret < 0) - return ret; - - qf->surface.Data.MemId = &q->frames_ctx.mids[ret]; + if (ret >= 0) + qf->surface.Data.MemId = &q->frames_ctx.mids[ret]; + } + if (!q->frames_ctx.mids || ret < 0) { + QSVMid *mid = NULL; + mid = (QSVMid *)av_mallocz(sizeof(*mid)); + if (!mid) + return AVERROR(ENOMEM); + mid->handle_pair = (mfxHDLPair *)qf->surface.Data.MemId; + qf->surface.Data.MemId = mid; + qf->external_frame = 1; } } else { /* make a copy if the input is not padded as libmfx requires */ diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c index 56dffa1f25..565c79a4e0 100644 --- a/libavutil/hwcontext_qsv.c +++ b/libavutil/hwcontext_qsv.c @@ -1337,11 +1337,24 @@ static int qsv_frames_derive_to(AVHWFramesContext *dst_ctx, return 0; } +static void qsv_umap_external_surface(AVHWFramesContext *dst_fc, + HWMapDescriptor *hwmap) +{ + mfxFrameSurface1 *new_sur = (mfxFrameSurface1 *)hwmap->priv; + mfxHDLPair *hdlpair = (mfxHDLPair *)new_sur->Data.MemId; + if (hwmap->source->format == AV_PIX_FMT_VAAPI) + av_freep(&hdlpair->first); + av_freep(&new_sur->Data.MemId); + av_freep(&new_sur); +} + static int qsv_map_to(AVHWFramesContext *dst_ctx, AVFrame *dst, const AVFrame *src, int flags) { AVQSVFramesContext *hwctx = dst_ctx->hwctx; int i, err, index = -1; + mfxFrameSurface1 *new_sur = NULL; + mfxHDLPair *new_hdlpair = NULL; for (i = 0; i < hwctx->nb_surfaces && index < 0; i++) { switch(src->format) { @@ -1379,22 +1392,70 @@ static int qsv_map_to(AVHWFramesContext *dst_ctx, #endif } } + /* If the surface is not in the pool, create a new surface */ if (index < 0) { - av_log(dst_ctx, AV_LOG_ERROR, "Trying to map from a surface which " - "is not in the mapped frames context.\n"); - return AVERROR(EINVAL); + new_sur = (mfxFrameSurface1 *)av_mallocz(sizeof(*new_sur)); + if (!new_sur) { + err = AVERROR(ENOMEM); + goto fail; + } + err = qsv_init_surface(dst_ctx, new_sur); + if (err < 0) + goto fail; + + new_hdlpair = (mfxHDLPair *)av_mallocz(sizeof(*new_hdlpair)); + if (!new_hdlpair) { + err = AVERROR(ENOMEM); + goto fail; + } + switch (src->format) { +#if CONFIG_VAAPI + case AV_PIX_FMT_VAAPI: + new_hdlpair->first = (VASurfaceID *)av_mallocz(sizeof(VASurfaceID)); + if (!new_hdlpair->first) { + err = AVERROR(ENOMEM); + goto fail; + } + *(VASurfaceID*)new_hdlpair->first = (VASurfaceID)(uintptr_t)src->data[3]; + break; +#endif + case AV_PIX_FMT_D3D11: + new_hdlpair->first = src->data[0]; + new_hdlpair->second = src->data[1]; + break; + case AV_PIX_FMT_DXVA2_VLD: + new_hdlpair->first = src->data[3]; + break; + default: + return AVERROR(ENOSYS); + } + av_log(dst_ctx, AV_LOG_DEBUG, "Trying to map from a surface which " + "is not in the mapped frames context, so create a new surface\n"); + new_sur->Data.MemId = new_hdlpair; + err = ff_hwframe_map_create(dst->hw_frames_ctx, dst, src, + &qsv_umap_external_surface, + (void*)new_sur); + if (err) + goto fail; + } else { + err = ff_hwframe_map_create(dst->hw_frames_ctx, + dst, src, NULL, NULL); + if (err) + return err; } - err = ff_hwframe_map_create(dst->hw_frames_ctx, - dst, src, NULL, NULL); - if (err) - return err; - dst->width = src->width; dst->height = src->height; - dst->data[3] = (uint8_t*)&hwctx->surfaces[index]; + dst->data[3] = (uint8_t *)((index < 0) ? new_sur : &hwctx->surfaces[index]); return 0; + +fail: + av_freep(&new_sur); + if (src->format == AV_PIX_FMT_VAAPI && new_hdlpair) + av_freep(&new_hdlpair->first); + av_freep(&new_hdlpair); + return err; } static int qsv_frames_get_constraints(AVHWDeviceContext *ctx,