From patchwork Tue Jun 7 09:22:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wu, Tong1" X-Patchwork-Id: 36086 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp6754571pzj; Tue, 7 Jun 2022 02:23:52 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy33FvXVaLvLOi+9srsD00wPiCZL7DHxo8z7nVEvbZb3V6efjHoKvvyPkxrWv1vly56/86v X-Received: by 2002:a05:6402:3899:b0:42d:d79e:acef with SMTP id fd25-20020a056402389900b0042dd79eacefmr31982307edb.12.1654593831827; Tue, 07 Jun 2022 02:23:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1654593831; cv=none; d=google.com; s=arc-20160816; b=d7GSdeKTQVmKT/sJmdWFMpqE/I25KPMVqixoPLTXqHXxeaLHo1JldZm9schcJNyrBi IejcWz5M0otXghiE4l8o1ObPTvNIaoh5VWohFs2GUxSppJebYdkYzms4S61zHeeSIP8f Dn2icLXg+hOmjJe/WtXPyeoaRTmVdgjG5b8mlQ9C6Miw5zbWp9tb9f/1nacXIvT5Eo5+ oettrdbGScGupsPe54mI67FLLf/oF0jNdY6YC7yhB7xTypql4GaY6UDyi+osnVzl6snT lSltOoPRS5KyXGhs7eID1XQ01fCAKWGcc7b7cidyRY1g4bJswyH8Uk0MjYtyCsDqTjVJ TEPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=Csx+4NfC2YoW//gVhx97sZ9TCzGrl51gzkEdKI+xNyM=; b=bVKairCK4xl2GW77Xi7wOt8rmqC3xW5wtnxJP5Sez+AgaZmu1cR9vZueCMAYBPlMuF fz8fTRm4uP4ymmND5LnjY4xffhCBkgg25+bCuQfJF/ETyl18Vi2ETgvg6oBA5ISQ9Ie5 OCEUyPcX9mVZD8flwy7z2uOSWCyozyC91+xXia+V0biBQXFJdUfmDtCeUcST4A+cwreL EQbyGg/ZRQSoz8OePNUUT0uaudrDp/J9VnTF2vYuIXZcGHEYh3twXFUEcb8NsmBQpHqf LngicjKnU8B8OMg4cw0Gz0ZUGmgyWBkyqUNVwTvOU1upSLx8eQ9oMXdZBuC3brFT93YP mJMA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=ChaQ3C2j; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id sh39-20020a1709076ea700b006f4adc54180si14509449ejc.611.2022.06.07.02.23.51; Tue, 07 Jun 2022 02:23:51 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=ChaQ3C2j; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DB67D68B635; Tue, 7 Jun 2022 12:23:47 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D15C468B31F for ; Tue, 7 Jun 2022 12:23:40 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1654593826; x=1686129826; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=Qx0e2Fjda2K9RIvvOCrqhinaq5JB0vCjzSk2fpzaU90=; b=ChaQ3C2jaufbqf1zVmEferrshiE0uHvcOrAImwdXaQZPygGaayVh/4ao lfoQvUMUzMuZ23MhcyO2J0IPC6k/NEjZcGJGwTG2rTpQEr8Xk3PMjNBQr k+ur03g3R20hIS7TWYhYqdKUO2zMvpopcb4qXqu8a5PY7OdO9ZMdQgfwD KajnDsnn78yJ5+onjYsmrntP0WnrF/ZR1S3L86q0qzxEK24I6W8ENlQnG ERBx7hBgp/KiX/8ULi6ZaglljN5Ht+p5W4RiPhi4rtXjO2o6xZQT6PBfS 5/EhZ1rz/y9jnoaCQ3ElM/VZSsWUngxtkfEVXc0fdKRx0Q63k2MnDB++m Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10370"; a="302076436" X-IronPort-AV: E=Sophos;i="5.91,283,1647327600"; d="scan'208";a="302076436" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jun 2022 02:23:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,283,1647327600"; d="scan'208";a="826276980" Received: from desktop-qn7n0nf.sh.intel.com (HELO localhost.localdomain) ([10.239.160.39]) by fmsmga006.fm.intel.com with ESMTP; 07 Jun 2022 02:23:36 -0700 From: Tong Wu To: ffmpeg-devel@ffmpeg.org Date: Tue, 7 Jun 2022 17:22:16 +0800 Message-Id: <20220607092216.405-1-tong1.wu@intel.com> X-Mailer: git-send-email 2.35.1.windows.2 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avcodec/qsvenc: make QSV encoder encode VAAPI and D3D11 frames directly X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Tong Wu , Wenbin Chen Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: QcXu851w3Qgd QSV encoder is able to encode frames with VAAPI or D3D11 pixel format directly. This patch adds support for qsv encoder to accept VAAPI and D3D11 pixel formats as input. Signed-off-by: Wenbin Chen Signed-off-by: Tong Wu --- libavcodec/qsvenc.c | 59 ++++++++++++++++++++++++++++++++++----- libavcodec/qsvenc_h264.c | 2 ++ libavcodec/qsvenc_hevc.c | 2 ++ libavcodec/qsvenc_jpeg.c | 2 ++ libavcodec/qsvenc_mpeg2.c | 2 ++ libavcodec/qsvenc_vp9.c | 2 ++ 6 files changed, 62 insertions(+), 7 deletions(-) diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c index 2b3b06767d..132d9ba93b 100644 --- a/libavcodec/qsvenc.c +++ b/libavcodec/qsvenc.c @@ -524,7 +524,9 @@ static int check_enc_param(AVCodecContext *avctx, QSVEncContext *q) static int init_video_param_jpeg(AVCodecContext *avctx, QSVEncContext *q) { - enum AVPixelFormat sw_format = avctx->pix_fmt == AV_PIX_FMT_QSV ? + enum AVPixelFormat sw_format = avctx->pix_fmt == AV_PIX_FMT_QSV || + avctx->pix_fmt == AV_PIX_FMT_VAAPI || + avctx->pix_fmt == AV_PIX_FMT_D3D11 ? avctx->sw_pix_fmt : avctx->pix_fmt; const AVPixFmtDescriptor *desc; int ret; @@ -591,7 +593,9 @@ static int init_video_param_jpeg(AVCodecContext *avctx, QSVEncContext *q) static int init_video_param(AVCodecContext *avctx, QSVEncContext *q) { - enum AVPixelFormat sw_format = avctx->pix_fmt == AV_PIX_FMT_QSV ? + enum AVPixelFormat sw_format = avctx->pix_fmt == AV_PIX_FMT_QSV || + avctx->pix_fmt == AV_PIX_FMT_VAAPI || + avctx->pix_fmt == AV_PIX_FMT_D3D11 ? avctx->sw_pix_fmt : avctx->pix_fmt; const AVPixFmtDescriptor *desc; float quant; @@ -1247,7 +1251,31 @@ int ff_qsv_enc_init(AVCodecContext *avctx, QSVEncContext *q) if (avctx->hw_frames_ctx) { AVHWFramesContext *frames_ctx = (AVHWFramesContext*)avctx->hw_frames_ctx->data; - AVQSVFramesContext *frames_hwctx = frames_ctx->hwctx; + AVQSVFramesContext *frames_hwctx = NULL; + + if (frames_ctx->format == AV_PIX_FMT_VAAPI || frames_ctx->format == AV_PIX_FMT_D3D11) { + AVBufferRef *derive_device_ref = NULL; + AVBufferRef *derive_frames_ref = NULL; + ret = av_hwdevice_ctx_create_derived(&derive_device_ref, + AV_HWDEVICE_TYPE_QSV, frames_ctx->device_ref, 0); + if (ret < 0) { + av_log(avctx, AV_LOG_ERROR, "Failed to derive QSV device context: %d.\n", ret); + return ret; + } + ret = av_hwframe_ctx_create_derived(&derive_frames_ref, + AV_PIX_FMT_QSV, derive_device_ref, avctx->hw_frames_ctx, 0); + if (ret < 0) { + av_log(avctx, AV_LOG_ERROR, "Failed to derive QSV frames context: %d.\n", ret); + av_buffer_unref(&derive_device_ref); + return ret; + } + av_buffer_unref(&avctx->hw_device_ctx); + avctx->hw_device_ctx = derive_device_ref; + av_buffer_unref(&avctx->hw_frames_ctx); + avctx->hw_frames_ctx = derive_frames_ref; + frames_ctx = (AVHWFramesContext*)avctx->hw_frames_ctx->data; + } + frames_hwctx = frames_ctx->hwctx; if (!iopattern) { if (frames_hwctx->frame_type & MFX_MEMTYPE_OPAQUE_FRAME) @@ -1437,10 +1465,25 @@ static int submit_frame(QSVEncContext *q, const AVFrame *frame, if (ret < 0) return ret; - if (frame->format == AV_PIX_FMT_QSV) { - ret = av_frame_ref(qf->frame, frame); - if (ret < 0) - return ret; + if (frame->format == AV_PIX_FMT_QSV || frame->format == AV_PIX_FMT_VAAPI || frame->format == AV_PIX_FMT_D3D11) { + if (frame->format == AV_PIX_FMT_QSV) { + ret = av_frame_ref(qf->frame, frame); + if (ret < 0) + return ret; + } else { + qf->frame->format = AV_PIX_FMT_QSV; + qf->frame->hw_frames_ctx = av_buffer_ref(q->avctx->hw_frames_ctx); + if (!qf->frame->hw_frames_ctx) + return AVERROR(ENOMEM); + ret = av_hwframe_map(qf->frame, frame, 0); + if (ret < 0) { + av_log(q->avctx, AV_LOG_ERROR, "Failed to map to QSV frames\n"); + return ret; + } + ret = av_frame_copy_props(qf->frame, frame); + if (ret < 0) + return ret; + } qf->surface = *(mfxFrameSurface1*)qf->frame->data[3]; @@ -1735,6 +1778,8 @@ int ff_qsv_enc_close(AVCodecContext *avctx, QSVEncContext *q) const AVCodecHWConfigInternal *const ff_qsv_enc_hw_configs[] = { HW_CONFIG_ENCODER_FRAMES(QSV, QSV), + HW_CONFIG_ENCODER_FRAMES(VAAPI,VAAPI), + HW_CONFIG_ENCODER_FRAMES(D3D11,D3D11VA), HW_CONFIG_ENCODER_DEVICE(NV12, QSV), HW_CONFIG_ENCODER_DEVICE(P010, QSV), NULL, diff --git a/libavcodec/qsvenc_h264.c b/libavcodec/qsvenc_h264.c index cf77ea575b..93ba8d8ded 100644 --- a/libavcodec/qsvenc_h264.c +++ b/libavcodec/qsvenc_h264.c @@ -196,6 +196,8 @@ const FFCodec ff_h264_qsv_encoder = { .p.pix_fmts = (const enum AVPixelFormat[]){ AV_PIX_FMT_NV12, AV_PIX_FMT_P010, AV_PIX_FMT_QSV, + AV_PIX_FMT_VAAPI, + AV_PIX_FMT_D3D11, AV_PIX_FMT_NONE }, .p.priv_class = &class, .defaults = qsv_enc_defaults, diff --git a/libavcodec/qsvenc_hevc.c b/libavcodec/qsvenc_hevc.c index a6bf39c148..63b6ad9150 100644 --- a/libavcodec/qsvenc_hevc.c +++ b/libavcodec/qsvenc_hevc.c @@ -309,6 +309,8 @@ const FFCodec ff_hevc_qsv_encoder = { AV_PIX_FMT_YUYV422, AV_PIX_FMT_Y210, AV_PIX_FMT_QSV, + AV_PIX_FMT_VAAPI, + AV_PIX_FMT_D3D11, AV_PIX_FMT_BGRA, AV_PIX_FMT_X2RGB10, AV_PIX_FMT_NONE }, diff --git a/libavcodec/qsvenc_jpeg.c b/libavcodec/qsvenc_jpeg.c index 825eb8dc06..5b7611bb85 100644 --- a/libavcodec/qsvenc_jpeg.c +++ b/libavcodec/qsvenc_jpeg.c @@ -91,6 +91,8 @@ const FFCodec ff_mjpeg_qsv_encoder = { .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HYBRID, .p.pix_fmts = (const enum AVPixelFormat[]){ AV_PIX_FMT_NV12, AV_PIX_FMT_QSV, + AV_PIX_FMT_VAAPI, + AV_PIX_FMT_D3D11, AV_PIX_FMT_NONE }, .p.priv_class = &class, .defaults = qsv_enc_defaults, diff --git a/libavcodec/qsvenc_mpeg2.c b/libavcodec/qsvenc_mpeg2.c index 5cb12a2582..cba4001ee1 100644 --- a/libavcodec/qsvenc_mpeg2.c +++ b/libavcodec/qsvenc_mpeg2.c @@ -105,6 +105,8 @@ const FFCodec ff_mpeg2_qsv_encoder = { .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HYBRID, .p.pix_fmts = (const enum AVPixelFormat[]){ AV_PIX_FMT_NV12, AV_PIX_FMT_QSV, + AV_PIX_FMT_VAAPI, + AV_PIX_FMT_D3D11, AV_PIX_FMT_NONE }, .p.priv_class = &class, .defaults = qsv_enc_defaults, diff --git a/libavcodec/qsvenc_vp9.c b/libavcodec/qsvenc_vp9.c index 4b2a6ce77f..2825b98a4a 100644 --- a/libavcodec/qsvenc_vp9.c +++ b/libavcodec/qsvenc_vp9.c @@ -115,6 +115,8 @@ const FFCodec ff_vp9_qsv_encoder = { .p.pix_fmts = (const enum AVPixelFormat[]){ AV_PIX_FMT_NV12, AV_PIX_FMT_P010, AV_PIX_FMT_QSV, + AV_PIX_FMT_VAAPI, + AV_PIX_FMT_D3D11, AV_PIX_FMT_NONE }, .p.priv_class = &class, .defaults = qsv_enc_defaults,