From patchwork Tue Aug 16 10:10:03 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nablet Developer X-Patchwork-Id: 189 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.140.67 with SMTP id o64csp2033142vsd; Tue, 16 Aug 2016 03:11:56 -0700 (PDT) X-Received: by 10.194.223.40 with SMTP id qr8mr37977510wjc.16.1471342316812; Tue, 16 Aug 2016 03:11:56 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b73si19954643wmi.47.2016.08.16.03.11.56; Tue, 16 Aug 2016 03:11:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 56599689B07; Tue, 16 Aug 2016 13:11:42 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mout.kundenserver.de (mout.kundenserver.de [217.72.192.74]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0462F689AFA for ; Tue, 16 Aug 2016 13:11:27 +0300 (EEST) Received: from localhost.localdomain ([91.216.211.197]) by mrelayeu.kundenserver.de (mreue103) with ESMTPSA (Nemesis) id 0LmNdq-1aziIA0Yra-00ZwRl; Tue, 16 Aug 2016 12:11:24 +0200 From: Nablet Developer To: ffmpeg-devel@ffmpeg.org Date: Tue, 16 Aug 2016 17:10:03 +0700 Message-Id: <1471342207-11982-3-git-send-email-sdk@nablet.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1471342207-11982-1-git-send-email-sdk@nablet.com> References: <1471342207-11982-1-git-send-email-sdk@nablet.com> X-Provags-ID: V03:K0:NPF/o9ZDjylW3o3+cAoNcPBl0VdNYsSCS2N3/khl0B8mYvRMrCT 2PIm9iMAZ8GPUK20tAZO5t2MwKngRp+dUkLhs6cAd5Yo7Yjc7ZgjgbZny9ILD3YLnCSeiHB kSKBtSpO4Q4oiUswc0CQYs1sSbA8TL2jJpWwHcmH/Ktqb7fh8dJz/2djP6RAwPSV3Dg9qRY wtzKuoytqCKsQJ28LBC1A== X-UI-Out-Filterresults: notjunk:1; V01:K0:lG1mlmICIvg=:0ZZkUDxR6FsexeTKiNLpx5 bfDhouKG501cDDhA6N7ik6S7Qhu5k167B+lM5LTjSZbHIiDmBG3YvNS4eomnxNwEs+5f8RQ9I giFTAuTYnhjUvEtCO+JVbzsSzqtm2+AlediafBSRihYTnzlFaxGmCA4SpcSy+6Dvx1frTJz6+ uYDBGkaOELgv/D5uJeI0spt6os3PRBrOGlGMgGk+sZYxKGCRJ5G1fNW4RcmSquNmFY5WLR5lU 6I5hSBLuIVdrwBMp5tE9biwVbRCdYKwPSy0JSaArA4qUjwPcoiUhOi5jRdlZr46qlj7Kkhy9J /d/9a80dqHhFergt6RwmOqi/wYSk0LpH/lQGXBtp8alGFaDOmnEgmdOGe9t/PsgPbE2eo+l6x tjf2kzB/39hcs5Xsj/DoeAy9S0XQ7kdgPeQEKuN1S2e7tpOY+LaOd24e79rpIEdPXpkETnnBF 2hbLddVp8eQumE4jUTp2P6pB7rUzp2egzDDa+lbml/a6t9i5DEGoXKdqFlgh8gXLAr3mRrmV9 G4Z6NVgGQdH2KPfMRIm6tZieOIQju0jMFlz0FQe5lJbJ+4Ghhi76dLoWiOf//u79zgVvBhm5h oJ9r945iyfkbnbh8eoR18A/TJ7+VZOD+kJbZjin+Y/KapOeSbkM/ZHqhmYyyK5wv/sqr2hk+5 khdiBQYtb2DEA+yTmu8I9jcteD3jJcHwsXMSbbFgmbhTG+h8/pRV8DKUgnzIfNtfcyBiFQnrg kAqYD38aSXdNvfIp Subject: [FFmpeg-devel] [PATCH 2/6] lavf/vpp: Enable vpp filter, an Intel GPU accelerated scaler. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: ChaoX A Liu MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: ChaoX A Liu Signed-off-by: ChaoX A Liu --- configure | 3 + libavcodec/qsv.c | 2 +- libavcodec/qsv_internal.h | 2 +- libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_vpp.c | 863 ++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 870 insertions(+), 2 deletions(-) create mode 100644 libavfilter/vf_vpp.c diff --git a/configure b/configure index 9b92426..7eb7ffd 100755 --- a/configure +++ b/configure @@ -2086,6 +2086,7 @@ CONFIG_EXTRA=" qsv qsvdec qsvenc + qsvvpp rangecoder riffdec riffenc @@ -3076,6 +3077,8 @@ tinterlace_pad_test_deps="tinterlace_filter" uspp_filter_deps="gpl avcodec" vidstabdetect_filter_deps="libvidstab" vidstabtransform_filter_deps="libvidstab" +vpp_filter_deps="libmfx" +vpp_filter_select="qsv" zmq_filter_deps="libzmq" zoompan_filter_deps="swscale" zscale_filter_deps="libzimg" diff --git a/libavcodec/qsv.c b/libavcodec/qsv.c index b505e14..c180ca8 100644 --- a/libavcodec/qsv.c +++ b/libavcodec/qsv.c @@ -168,7 +168,7 @@ static int ff_qsv_set_display_handle(AVCodecContext *avctx, QSVSession *qs) * @param avctx ffmpeg metadata for this codec context * @param session the MSDK session used */ -int ff_qsv_init_internal_session(AVCodecContext *avctx, QSVSession *qs) +int ff_qsv_init_internal_session(void *avctx, QSVSession *qs) { mfxIMPL impl = MFX_IMPL_AUTO_ANY; mfxVersion ver = { { QSV_VERSION_MINOR, QSV_VERSION_MAJOR } }; diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h index 59d1336..e43728b 100644 --- a/libavcodec/qsv_internal.h +++ b/libavcodec/qsv_internal.h @@ -80,7 +80,7 @@ int ff_qsv_error(int mfx_err); int ff_qsv_codec_id_to_mfx(enum AVCodecID codec_id); -int ff_qsv_init_internal_session(AVCodecContext *avctx, QSVSession *qs); +int ff_qsv_init_internal_session(void *avctx, QSVSession *qs); int ff_qsv_load_plugins(mfxSession session, const char *load_plugins); diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 0d94f84..a15bf3c 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -286,6 +286,7 @@ OBJS-$(CONFIG_VFLIP_FILTER) += vf_vflip.o OBJS-$(CONFIG_VIDSTABDETECT_FILTER) += vidstabutils.o vf_vidstabdetect.o OBJS-$(CONFIG_VIDSTABTRANSFORM_FILTER) += vidstabutils.o vf_vidstabtransform.o OBJS-$(CONFIG_VIGNETTE_FILTER) += vf_vignette.o +OBJS-$(CONFIG_VPP_FILTER) += vf_vpp.o OBJS-$(CONFIG_VSTACK_FILTER) += vf_stack.o framesync.o OBJS-$(CONFIG_W3FDIF_FILTER) += vf_w3fdif.o OBJS-$(CONFIG_WAVEFORM_FILTER) += vf_waveform.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index feed4f8..ce209cc 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -301,6 +301,7 @@ void avfilter_register_all(void) REGISTER_FILTER(VIDSTABDETECT, vidstabdetect, vf); REGISTER_FILTER(VIDSTABTRANSFORM, vidstabtransform, vf); REGISTER_FILTER(VIGNETTE, vignette, vf); + REGISTER_FILTER(VPP, vpp, vf); REGISTER_FILTER(VSTACK, vstack, vf); REGISTER_FILTER(W3FDIF, w3fdif, vf); REGISTER_FILTER(WAVEFORM, waveform, vf); diff --git a/libavfilter/vf_vpp.c b/libavfilter/vf_vpp.c new file mode 100644 index 0000000..0cfeafc --- /dev/null +++ b/libavfilter/vf_vpp.c @@ -0,0 +1,863 @@ +/* + * Intel MediaSDK Quick Sync Video VPP filter + * + * copyright (c) 2015 Sven Dueking + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with Libav; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "internal.h" +#include +#include "libavutil/parseutils.h" +#include "libavutil/timestamp.h" +#include "libavcodec/qsv.h" + +/** + * ToDo : + * + * - double check surface pointers for different fourccs + * - handle empty extbuffers + * - cropping + * - use AV_PIX_FMT_QSV to pass surfaces to encoder + * - deinterlace check settings etc. + * - allocate number of surfaces depending modules and number of b frames + */ + +#define VPP_ZERO_MEMORY(VAR) { memset(&VAR, 0, sizeof(VAR)); } +#define VPP_ALIGN16(value) (((value + 15) >> 4) << 4) // round up to a multiple of 16 +#define VPP_ALIGN32(value) (((value + 31) >> 5) << 5) // round up to a multiple of 32 +#define VPP_CHECK_POINTER(P, ...) {if (!(P)) {return __VA_ARGS__;}} + +#define OFFSET(x) offsetof(VPPContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_FILTERING_PARAM + +static const AVOption vpp_options[] = { + { "deinterlace", "deinterlace mode: 0=off, 1=bob, 2=advanced", OFFSET(deinterlace), AV_OPT_TYPE_INT, {.i64=0}, 0, MFX_DEINTERLACING_ADVANCED, .flags = FLAGS }, + { "denoise", "denoise level [0, 100]", OFFSET(denoise), AV_OPT_TYPE_INT, {.i64=0}, 0, 100, .flags = FLAGS }, + { "detail", "detail enhancement level [0, 100]", OFFSET(detail), AV_OPT_TYPE_INT, {.i64=0}, 0, 100, .flags = FLAGS }, + { "w", "Output video width", OFFSET(out_width), AV_OPT_TYPE_INT, {.i64=0}, 0, 4096, .flags = FLAGS }, + { "width", "Output video width", OFFSET(out_width), AV_OPT_TYPE_INT, {.i64=0}, 0, 4096, .flags = FLAGS }, + { "h", "Output video height", OFFSET(out_height), AV_OPT_TYPE_INT, {.i64=0}, 0, 2304, .flags = FLAGS }, + { "height", "Output video height : ", OFFSET(out_height), AV_OPT_TYPE_INT, {.i64=0}, 0, 2304, .flags = FLAGS }, + { "dpic", "dest pic struct: 0=tff, 1=progressive [default], 2=bff", OFFSET(dpic), AV_OPT_TYPE_INT, {.i64 = 1 }, 0, 2, .flags = FLAGS }, + { "framerate", "output framerate", OFFSET(framerate), AV_OPT_TYPE_RATIONAL, { .dbl = 0.0 },0, DBL_MAX, .flags = FLAGS }, + { "async_depth", "Maximum processing parallelism [default = 4]", OFFSET(async_depth), AV_OPT_TYPE_INT, { .i64 = ASYNC_DEPTH_DEFAULT }, 0, INT_MAX, .flags = FLAGS }, + { "max_b_frames","Maximum number of b frames [default = 3]", OFFSET(max_b_frames), AV_OPT_TYPE_INT, { .i64 = 3 }, 0, INT_MAX, .flags = FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(vpp); + +static int option_id_to_mfx_pic_struct(int id) +{ + switch (id) { + case 0: + return MFX_PICSTRUCT_FIELD_TFF; + case 1: + return MFX_PICSTRUCT_PROGRESSIVE; + case 2: + return MFX_PICSTRUCT_FIELD_BFF; + default: + return MFX_PICSTRUCT_UNKNOWN; + } + return MFX_PICSTRUCT_UNKNOWN; +} + +static int get_chroma_fourcc(unsigned int fourcc) +{ + switch (fourcc) { + case MFX_FOURCC_YUY2: + return MFX_CHROMAFORMAT_YUV422; + case MFX_FOURCC_RGB4: + return MFX_CHROMAFORMAT_YUV444; + default: + return MFX_CHROMAFORMAT_YUV420; + } + return MFX_CHROMAFORMAT_YUV420; +} + +static int avframe_id_to_mfx_pic_struct(AVFrame * pic) +{ + if (!pic->interlaced_frame) + return MFX_PICSTRUCT_PROGRESSIVE; + + if (pic->top_field_first == 1) + return MFX_PICSTRUCT_FIELD_TFF; + + return MFX_PICSTRUCT_FIELD_BFF; +} + +static int avpix_fmt_to_mfx_fourcc(int format) +{ + switch(format){ + case AV_PIX_FMT_YUV420P: + return MFX_FOURCC_YV12; + case AV_PIX_FMT_NV12: + return MFX_FOURCC_NV12; + case AV_PIX_FMT_YUYV422: + return MFX_FOURCC_YUY2; + case AV_PIX_FMT_RGB32: + return MFX_FOURCC_RGB4; + } + + return MFX_FOURCC_NV12; +} + +static void vidmem_init_surface(VPPContext *vpp) +{ + int i; + + av_log(vpp->ctx, AV_LOG_INFO, "vpp: vidmem_init_surface: "); + + /* + * Input surfaces are allocated by decoder. + * We needn't allocate them. + */ + vpp->num_surfaces_in = 0; + vpp->in_surface = NULL; + vpp->in_response = NULL; + + /* + * We should care about next stage vpp or encoder's input surfaces. + */ + av_log(vpp->ctx, AV_LOG_INFO, "in.num = %d, out.num = %d, ", + vpp->req[0].NumFrameSuggested, vpp->req[1].NumFrameSuggested); + if (vpp->enc_ctx) { + vpp->req[1].NumFrameSuggested += vpp->enc_ctx->req.NumFrameSuggested; + av_log(vpp->ctx, AV_LOG_INFO, "enc_ctx.num=%d\n", vpp->enc_ctx->req.NumFrameSuggested); + } else { + av_log(vpp->ctx, AV_LOG_INFO, "enc_ctx.num=%d\n", 0); + } + + vpp->req[0].NumFrameSuggested = FFMAX(vpp->req[0].NumFrameSuggested, 1); + + vpp->num_surfaces_out = FFMAX(vpp->req[1].NumFrameSuggested, 1); + vpp->out_response = av_mallocz(sizeof(*vpp->out_response)); + VPP_CHECK_POINTER(vpp->out_response); + vpp->out_surface = av_mallocz(sizeof(*vpp->out_surface) * vpp->num_surfaces_out); + VPP_CHECK_POINTER(vpp->out_surface); + + vpp->pFrameAllocator->Alloc( vpp->pFrameAllocator->pthis, &(vpp->req[1]), vpp->out_response); + for (i = 0; i < vpp->num_surfaces_out; i++) { + memcpy(&vpp->out_surface[i].Info, &vpp->pVppParam->vpp.Out, sizeof(vpp->out_surface[i].Info)); + vpp->out_surface[i].Data.MemId = vpp->out_response->mids[i]; + } +} + +static void vidmem_free_surface(AVFilterContext *ctx) +{ + VPPContext *vpp= ctx->priv; + + av_log(vpp->ctx, AV_LOG_DEBUG, "vpp: vidmem_free_surface\n"); + + if (vpp->out_surface) + av_freep(&vpp->out_surface); + + if (vpp->out_response) { + vpp->pFrameAllocator->Free(vpp->pFrameAllocator->pthis, vpp->out_response); + av_freep(&vpp->out_response); + } + + vpp->num_surfaces_in = 0; + vpp->num_surfaces_out = 0; +} + +static void sysmem_init_surface(VPPContext *vpp) +{ + int i, j; + av_log(vpp->ctx, AV_LOG_INFO, "vpp: sysmem_init_surface: "); + av_log(vpp->ctx, AV_LOG_INFO, "in.num = %d, out.num = %d\n", + vpp->req[0].NumFrameSuggested, vpp->req[1].NumFrameSuggested); + + vpp->num_surfaces_in = FFMAX(vpp->req[0].NumFrameSuggested, vpp->async_depth + vpp->max_b_frames + 1); + vpp->in_surface = av_mallocz(sizeof(*vpp->in_surface) * vpp->num_surfaces_in); + VPP_CHECK_POINTER(vpp->in_surface); + for (j = 0; j < vpp->num_surfaces_in; j++) + memcpy(&vpp->in_surface[j].Info, &vpp->pVppParam->vpp.In, sizeof(vpp->in_surface[j].Info)); + + vpp->num_surfaces_out = FFMAX(vpp->req[1].NumFrameSuggested, 1); + vpp->out_surface = av_mallocz(sizeof(*vpp->out_surface) * vpp->num_surfaces_out); + VPP_CHECK_POINTER(vpp->out_surface); + for (i = 0; i < vpp->num_surfaces_out; i++) + memcpy(&vpp->out_surface[i].Info, &vpp->pVppParam->vpp.Out, sizeof(vpp->out_surface[i].Info)); +} + +static void sysmem_free_surface(AVFilterContext *ctx) +{ + VPPContext *vpp= ctx->priv; + + av_log(vpp->ctx, AV_LOG_DEBUG, "vpp: sysmem_free_surface\n"); + if (vpp->in_surface) + av_freep(&vpp->in_surface); + + if (vpp->out_surface) + av_freep(&vpp->out_surface); + + vpp->num_surfaces_in = 0; + vpp->num_surfaces_out = 0; +} + +static int get_free_surface_index_in(AVFilterContext *ctx, mfxFrameSurface1 *surface_pool, int pool_size) +{ + if (surface_pool) { + for (mfxU16 i = 0; i < pool_size; i++) { + if (!surface_pool[i].Data.Locked) + return i; + } + } + + av_log(ctx, AV_LOG_ERROR, "Error getting a free surface, pool size is %d\n", pool_size); + return MFX_ERR_NOT_FOUND; +} + +static int get_free_surface_index_out(AVFilterContext *ctx, mfxFrameSurface1 *surface_pool, int pool_size) +{ + int i; + if (surface_pool) + for (i = 0; i < pool_size; i++) + if (!surface_pool[i].Data.Locked) + return i; + + av_log(ctx, AV_LOG_ERROR, "Error getting a free surface, pool size is %d\n", pool_size); + return MFX_ERR_NOT_FOUND; +} + +static int sysmem_map_frame_to_surface(VPPContext *vpp, AVFrame *frame, mfxFrameSurface1 *surface) +{ + surface->Info.PicStruct = avframe_id_to_mfx_pic_struct(frame); + + switch (frame->format) { + case AV_PIX_FMT_NV12: + surface->Data.Y = frame->data[0]; + surface->Data.VU = frame->data[1]; + break; + + case AV_PIX_FMT_YUV420P: + surface->Data.Y = frame->data[0]; + surface->Data.U = frame->data[1]; + surface->Data.V = frame->data[2]; + break; + + case AV_PIX_FMT_YUYV422: + surface->Data.Y = frame->data[0]; + surface->Data.U = frame->data[0] + 1; + surface->Data.V = frame->data[0] + 3; + break; + + case AV_PIX_FMT_RGB32: + surface->Data.B = frame->data[0]; + surface->Data.G = frame->data[0] + 1; + surface->Data.R = frame->data[0] + 2; + surface->Data.A = frame->data[0] + 3; + break; + + default: + return MFX_ERR_UNSUPPORTED; + } + surface->Data.Pitch = frame->linesize[0]; + + return 0; +} + +static int vidmem_map_frame_to_surface(VPPContext *vpp, AVFrame *frame, mfxFrameSurface1 *surface) +{ + int i; + mfxFrameData data; + + vpp->pFrameAllocator->Lock(vpp->pFrameAllocator->pthis, surface->Data.MemId, &data); + switch (frame->format) { + case AV_PIX_FMT_NV12: + for (i = 0; iheight; i++) + memcpy(data.Y + data.Pitch*i, frame->data[0] + frame->linesize[0]*i, frame->linesize[0]); + for (i = 0; iheight/2; i++) + memcpy(data.UV + data.Pitch*i, frame->data[1] + frame->linesize[1]*i, frame->linesize[1]); + break; + + case AV_PIX_FMT_YUV420P: + for (i = 0; iheight; i++) + memcpy(data.Y + data.Pitch*i, frame->data[0] + frame->linesize[0]*i, frame->linesize[0]); + for (i = 0; iheight/2; i++) + memcpy(data.U + data.Pitch*i, frame->data[1] + frame->linesize[1]*i, frame->linesize[1]); + for (i = 0; iheight/2; i++) + memcpy(data.V + data.Pitch*i, frame->data[2] + frame->linesize[2]*i, frame->linesize[2]); + break; + + case AV_PIX_FMT_YUYV422: + for (i = 0; iheight; i++) + memcpy(data.Y + data.Pitch*i, frame->data[0] + frame->linesize[0]*i, frame->linesize[0]); + break; + + case AV_PIX_FMT_RGB32: + for (i = 0; iheight; i++) + memcpy(data.B + data.Pitch*i, frame->data[0] + frame->linesize[0]*i, frame->linesize[0]); + break; + + default: + return MFX_ERR_UNSUPPORTED; + } + vpp->pFrameAllocator->Unlock(vpp->pFrameAllocator->pthis, surface->Data.MemId, &data); + + return 0; +} + +static int sysmem_input_get_surface( AVFilterLink *inlink, AVFrame* picref, mfxFrameSurface1 **surface ) +{ + AVFilterContext *ctx = inlink->dst; + VPPContext *vpp = ctx->priv; + int surf_idx = 0; + + surf_idx = get_free_surface_index_in(ctx, vpp->in_surface, vpp->num_surfaces_in); + if ( MFX_ERR_NOT_FOUND == surf_idx ) + return MFX_ERR_NOT_FOUND; + + *surface = &vpp->in_surface[surf_idx]; + sysmem_map_frame_to_surface(vpp, picref, *surface); + (*surface)->Data.TimeStamp = av_rescale_q(picref->pts, inlink->time_base, (AVRational){1,90000}); + + return 0; +} + +static int vidmem_input_get_surface( AVFilterLink *inlink, AVFrame* picref, mfxFrameSurface1 **surface ) +{ + if (picref->data[3]) { + *surface = (mfxFrameSurface1*)picref->data[3]; + } else { + return MFX_ERR_NOT_FOUND; + } + + (*surface)->Data.TimeStamp = av_rescale_q(picref->pts, inlink->time_base, (AVRational){1,90000}); + + return 0; +} + +static int sysmem_output_get_surface( AVFilterLink *inlink, mfxFrameSurface1 **surface ) +{ + AVFilterContext *ctx = inlink->dst; + VPPContext *vpp = ctx->priv; + int out_idx = 0; + + if (vpp->out_surface) { + for (out_idx = vpp->sysmem_cur_out_idx; out_idx < vpp->num_surfaces_out; out_idx++) + if (!vpp->out_surface[out_idx].Data.Locked) + break; + } else { + return MFX_ERR_NOT_INITIALIZED; + } + + if ( vpp->num_surfaces_out == out_idx ) + return MFX_ERR_NOT_FOUND; + + *surface = &vpp->out_surface[out_idx]; + + vpp->sysmem_cur_out_idx = out_idx + 1; + if (vpp->sysmem_cur_out_idx >= vpp->num_surfaces_out) + vpp->sysmem_cur_out_idx = 0; + + return 0; +} + +static int vidmem_output_get_surface( AVFilterLink *inlink, mfxFrameSurface1 **surface ) +{ + AVFilterContext *ctx = inlink->dst; + VPPContext *vpp = ctx->priv; + int out_idx = 0; + + out_idx = get_free_surface_index_out(ctx, vpp->out_surface, vpp->num_surfaces_out); + + if (MFX_ERR_NOT_FOUND == out_idx) + return out_idx; + + *surface = &vpp->out_surface[out_idx]; + + return 0; +} + +static int init_vpp_param( VPPContext *vpp, int format, int input_w, int input_h, int frame_rate_num, int frame_rate_den, int pic_struct ) +{ + // input data + vpp->pVppParam->vpp.In.FourCC = avpix_fmt_to_mfx_fourcc(format); + vpp->pVppParam->vpp.In.ChromaFormat = get_chroma_fourcc(vpp->pVppParam->vpp.In.FourCC); + vpp->pVppParam->vpp.In.CropX = 0; + vpp->pVppParam->vpp.In.CropY = 0; + vpp->pVppParam->vpp.In.CropW = input_w; + vpp->pVppParam->vpp.In.CropH = input_h; + vpp->pVppParam->vpp.In.PicStruct = pic_struct; + vpp->pVppParam->vpp.In.FrameRateExtN = frame_rate_num; + vpp->pVppParam->vpp.In.FrameRateExtD = frame_rate_den; + vpp->pVppParam->vpp.In.BitDepthLuma = 8; + vpp->pVppParam->vpp.In.BitDepthChroma = 8; + + // width must be a multiple of 16 + // height must be a multiple of 16 in case of frame picture and a multiple of 32 in case of field picture + vpp->pVppParam->vpp.In.Width = VPP_ALIGN16(vpp->pVppParam->vpp.In.CropW); + vpp->pVppParam->vpp.In.Height = + (MFX_PICSTRUCT_PROGRESSIVE == vpp->pVppParam->vpp.In.PicStruct) ? + VPP_ALIGN16(vpp->pVppParam->vpp.In.CropH) : + VPP_ALIGN32(vpp->pVppParam->vpp.In.CropH); + + // output data + vpp->pVppParam->vpp.Out.FourCC = MFX_FOURCC_NV12; + vpp->pVppParam->vpp.Out.ChromaFormat = MFX_CHROMAFORMAT_YUV420; + vpp->pVppParam->vpp.Out.CropX = 0; + vpp->pVppParam->vpp.Out.CropY = 0; + vpp->pVppParam->vpp.Out.CropW = vpp->out_width; + vpp->pVppParam->vpp.Out.CropH = vpp->out_height; + vpp->pVppParam->vpp.Out.PicStruct = option_id_to_mfx_pic_struct(vpp->dpic); + vpp->pVppParam->vpp.Out.FrameRateExtN = vpp->framerate.num; + vpp->pVppParam->vpp.Out.FrameRateExtD = vpp->framerate.den; + vpp->pVppParam->vpp.Out.BitDepthLuma = 8; + vpp->pVppParam->vpp.Out.BitDepthChroma = 8; + + if ((vpp->pVppParam->vpp.In.FrameRateExtN / vpp->pVppParam->vpp.In.FrameRateExtD) != + (vpp->pVppParam->vpp.Out.FrameRateExtN / vpp->pVppParam->vpp.Out.FrameRateExtD)) { + vpp->use_frc = 1; + } else { + vpp->use_frc = 0; + } + + // width must be a multiple of 16 + // height must be a multiple of 16 in case of frame picture and a multiple of 32 in case of field picture + vpp->pVppParam->vpp.Out.Width = VPP_ALIGN16(vpp->pVppParam->vpp.Out.CropW); + vpp->pVppParam->vpp.Out.Height = + (MFX_PICSTRUCT_PROGRESSIVE == vpp->pVppParam->vpp.Out.PicStruct) ? + VPP_ALIGN16(vpp->pVppParam->vpp.Out.CropH) : + VPP_ALIGN32(vpp->pVppParam->vpp.Out.CropH); + + if (vpp->pFrameAllocator) { + vpp->pVppParam->IOPattern = MFX_IOPATTERN_IN_VIDEO_MEMORY | MFX_IOPATTERN_OUT_VIDEO_MEMORY; + } else { + vpp->pVppParam->IOPattern = MFX_IOPATTERN_IN_SYSTEM_MEMORY | MFX_IOPATTERN_OUT_SYSTEM_MEMORY; + } + + av_log(vpp->ctx, AV_LOG_INFO, "VPP: In %dx%d %4.2f fps\t Out %dx%d %4.2f fps\n", + (vpp->pVppParam->vpp.In.Width), + (vpp->pVppParam->vpp.In.Height), + vpp->pVppParam->vpp.In.FrameRateExtN / (float)vpp->pVppParam->vpp.In.FrameRateExtD, + (vpp->pVppParam->vpp.Out.Width), + (vpp->pVppParam->vpp.Out.Height), + vpp->pVppParam->vpp.Out.FrameRateExtN / (float)vpp->pVppParam->vpp.Out.FrameRateExtD); + + if (vpp->use_frc == 1) + av_log(vpp->ctx, AV_LOG_INFO, "VPP: Framerate conversion enabled\n"); + + return 0; +} + +static int initial_vpp( VPPContext *vpp ) +{ + int ret = 0; + vpp->pVppParam->NumExtParam = 0; + vpp->frame_number = 0; + vpp->pVppParam->ExtParam = (mfxExtBuffer**)vpp->pExtBuf; + + if (vpp->deinterlace) { + av_log(vpp->ctx, AV_LOG_DEBUG, "Deinterlace enabled\n"); + memset(&vpp->deinterlace_conf, 0, sizeof(mfxExtVPPDeinterlacing)); + vpp->deinterlace_conf.Header.BufferId = MFX_EXTBUFF_VPP_DEINTERLACING; + vpp->deinterlace_conf.Header.BufferSz = sizeof(mfxExtVPPDeinterlacing); + vpp->deinterlace_conf.Mode = vpp->deinterlace == 1 ? MFX_DEINTERLACING_BOB : MFX_DEINTERLACING_ADVANCED; + + vpp->pVppParam->ExtParam[vpp->pVppParam->NumExtParam++] = (mfxExtBuffer*)&(vpp->deinterlace_conf); + } + + if (vpp->use_frc) { + av_log(vpp->ctx, AV_LOG_DEBUG, "Framerate conversion enabled\n"); + memset(&vpp->frc_conf, 0, sizeof(mfxExtVPPFrameRateConversion)); + vpp->frc_conf.Header.BufferId = MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION; + vpp->frc_conf.Header.BufferSz = sizeof(mfxExtVPPFrameRateConversion); + vpp->frc_conf.Algorithm = MFX_FRCALGM_DISTRIBUTED_TIMESTAMP; // make optional + + vpp->pVppParam->ExtParam[vpp->pVppParam->NumExtParam++] = (mfxExtBuffer*)&(vpp->frc_conf); + } + + if (vpp->denoise) { + av_log(vpp->ctx, AV_LOG_DEBUG, "Denoise enabled\n"); + memset(&vpp->denoise_conf, 0, sizeof(mfxExtVPPDenoise)); + vpp->denoise_conf.Header.BufferId = MFX_EXTBUFF_VPP_DENOISE; + vpp->denoise_conf.Header.BufferSz = sizeof(mfxExtVPPDenoise); + vpp->denoise_conf.DenoiseFactor = vpp->denoise; + + vpp->pVppParam->ExtParam[vpp->pVppParam->NumExtParam++] = (mfxExtBuffer*)&(vpp->denoise_conf); + } + + if (vpp->detail) { + av_log(vpp->ctx, AV_LOG_DEBUG, "Detail enabled\n"); + memset(&vpp->detail_conf, 0, sizeof(mfxExtVPPDetail)); + vpp->detail_conf.Header.BufferId = MFX_EXTBUFF_VPP_DETAIL; + vpp->detail_conf.Header.BufferSz = sizeof(mfxExtVPPDetail); + vpp->detail_conf.DetailFactor = vpp->detail; + + vpp->pVppParam->ExtParam[vpp->pVppParam->NumExtParam++] = (mfxExtBuffer*)&(vpp->detail_conf); + } + + memset(&vpp->req, 0, sizeof(mfxFrameAllocRequest) * 2); + ret = MFXVideoVPP_QueryIOSurf(vpp->session, vpp->pVppParam, &vpp->req[0]); + if (ret < 0) { + av_log(vpp->ctx, AV_LOG_ERROR, "Error querying the VPP IO surface\n"); + return ff_qsv_error(ret); + } + + if (vpp->pFrameAllocator) { + vidmem_init_surface(vpp); + } else { + sysmem_init_surface(vpp); + } + + ret = MFXVideoVPP_Init(vpp->session, vpp->pVppParam); + + if (MFX_WRN_PARTIAL_ACCELERATION == ret) { + av_log(vpp->ctx, AV_LOG_WARNING, "VPP will work with partial HW acceleration\n"); + } else if (ret < 0) { + av_log(vpp->ctx, AV_LOG_ERROR, "Error initializing the VPP %d\n",ret); + return ff_qsv_error(ret); + } + + vpp->vpp_ready = 1; + return 0; +} + +static int config_vpp(AVFilterLink *inlink, AVFrame * pic) +{ + AVFilterContext *ctx = inlink->dst; + VPPContext *vpp= ctx->priv; + mfxVideoParam mfxParamsVideo; + int ret; + + av_log(vpp->ctx, AV_LOG_INFO, "vpp configuration and call mfxVideoVPP_Init\n"); + if (!vpp->session) { + ret = ff_qsv_init_internal_session(ctx, &vpp->internal_qs); + if (ret < 0) + return ret; + + vpp->session = vpp->internal_qs.session; + } + + av_log(ctx, AV_LOG_INFO, "vpp initializing with session = %p\n", vpp->session); + VPP_ZERO_MEMORY(mfxParamsVideo); + vpp->pVppParam = &mfxParamsVideo; + + init_vpp_param(vpp, inlink->format, inlink->w, inlink->h, + inlink->frame_rate.num, inlink->frame_rate.den, + avframe_id_to_mfx_pic_struct(pic) ); + + return initial_vpp( vpp ); +} + +static void deconf_vpp(AVFilterContext *ctx) +{ + VPPContext *vpp = ctx->priv; + + MFXVideoVPP_Close(vpp->session); + + if (vpp->pFrameAllocator) { + vidmem_free_surface(ctx); + } else { + sysmem_free_surface(ctx); + } + + ff_qsv_close_internal_session(&vpp->internal_qs); + vpp->vpp_ready = 0; +} + +/* + * Real filter func. + * Push frame into mSDK and pop out filtered frames. + */ +static int filter_frame(AVFilterLink *inlink, AVFrame *picref) +{ + int ret = 0; + int filter_frame_ret = 0; + AVFilterContext *ctx = inlink->dst; + VPPContext *vpp = ctx->priv; + mfxSyncPoint sync = NULL; + mfxFrameSurface1 *pInSurface = NULL; + mfxFrameSurface1 *pOutSurface = NULL; + AVFilterLink *outlink = inlink->dst->outputs[0]; + AVFrame *out = NULL; + + /* + * we re-config local params when getting 1st main frame. + */ + if (!vpp->vpp_ready) { + ret = config_vpp(inlink, picref); + if (ret < 0) { + av_frame_free(&picref); + return ret; + } + vpp->vpp_ready = 1; + } + av_log(ctx, AV_LOG_DEBUG, "Filtering frame from %s, count %"PRIi64", ts %s\n", + inlink->src->name, inlink->frame_count, av_ts2str(picref->pts)); + + do { + /*get input surface*/ + if (vpp->pFrameAllocator) { + vidmem_input_get_surface( inlink, picref, &pInSurface ); + vidmem_output_get_surface( inlink, &pOutSurface ); + } else { + sysmem_input_get_surface( inlink, picref, &pInSurface ); + sysmem_output_get_surface( inlink, &pOutSurface ); + } + + if (!pInSurface || !pOutSurface) { + av_log(ctx, AV_LOG_ERROR, "no free input or output surface\n"); + ret = MFX_ERR_MEMORY_ALLOC; + break; + } + + /* + * get an AVFrame for output. + * @NOTE: frame buffer is aligned with 128x64 to compat with GPU-copy. + */ + out = ff_get_video_buffer(outlink, FFALIGN(vpp->out_width, 128), FFALIGN(vpp->out_height, 64)); + if (!out) { + ret = MFX_ERR_MEMORY_ALLOC; + break; + } + av_frame_copy_props(out, picref); + out->width = vpp->out_width; + out->height = vpp->out_height; + + /*map avframe->data into outsurface*/ + if (!vpp->pFrameAllocator) { + pOutSurface->Data.Y = out->data[0]; + pOutSurface->Data.VU = out->data[1]; + pOutSurface->Data.Pitch = out->linesize[0]; + } + + do { + ret = MFXVideoVPP_RunFrameVPPAsync(vpp->session, pInSurface, pOutSurface, NULL, &sync); + if (ret == MFX_WRN_DEVICE_BUSY) { + av_usleep(500); + continue; + } + break; + } while (1); + + if (ret < 0 && ret != MFX_ERR_MORE_SURFACE) { + av_frame_free(&out); + /*Ignore more_data error*/ + if (ret == MFX_ERR_MORE_DATA) + ret = 0; + break; + } + + if (ret == MFX_WRN_INCOMPATIBLE_VIDEO_PARAM) + av_log(ctx, AV_LOG_WARNING, + "EncodeFrameAsync returned 'incompatible param' code\n"); + + MFXVideoCORE_SyncOperation(vpp->session, sync, 60000); + + out->interlaced_frame = !(vpp->dpic == 1 || vpp->dpic == 3); + out->top_field_first = vpp->dpic == 1; + if (pOutSurface->Data.TimeStamp == MFX_TIMESTAMP_UNKNOWN) { + out->pts = AV_NOPTS_VALUE; + } else { + out->pts = av_rescale_q(pOutSurface->Data.TimeStamp, (AVRational){1,90000}, outlink->time_base); + } + + /*For video mem, we use AVFrame->data[3] to transfer surface*/ + if (vpp->pFrameAllocator) + out->data[3] = (void*) pOutSurface; + + filter_frame_ret = ff_filter_frame(inlink->dst->outputs[0], out); + if (filter_frame_ret < 0) + break; + + vpp->frame_number++; + } while (ret == MFX_ERR_MORE_SURFACE); + av_frame_free(&picref); + + return ret < 0 ? ff_qsv_error(ret) : filter_frame_ret; +} + +/* + * Configure each inputs. + */ +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + VPPContext *vpp = ctx->priv; + + if (!vpp->framerate.den || !vpp->framerate.num) + vpp->framerate = inlink->frame_rate; + + /* + * if out_w is not set(<=0), we calc it based on out_h; + * if out_h is not set(<=0), we calc it based on out_w; + * if both are not set, we set out_rect = in_rect. + */ + if (vpp->out_width <= 0) + vpp->out_width = av_rescale(vpp->out_height, inlink->w, inlink->h); + if (vpp->out_height <= 0) + vpp->out_height = av_rescale(vpp->out_width, inlink->h, inlink->w); + if (vpp->out_height <= 0 || vpp->out_width <= 0) { + vpp->out_width = inlink->w; + vpp->out_height = inlink->h; + } + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + VPPContext *vpp = ctx->priv; + + outlink->w = vpp->out_width; + outlink->h = vpp->out_height; + outlink->frame_rate = vpp->framerate; + outlink->time_base = av_inv_q(vpp->framerate); + outlink->format = AV_PIX_FMT_NV12; + + return 0; +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *main_fmts, *out_fmts; + + static const enum AVPixelFormat main_in_fmts[] = { + AV_PIX_FMT_YUV420P, + AV_PIX_FMT_NV12, + AV_PIX_FMT_YUYV422, + AV_PIX_FMT_RGB32, + AV_PIX_FMT_QSV, + AV_PIX_FMT_NONE + }; + static const enum AVPixelFormat out_pix_fmts[] = { + AV_PIX_FMT_NV12, + AV_PIX_FMT_QSV, + AV_PIX_FMT_NONE + }; + + main_fmts = ff_make_format_list(main_in_fmts); + out_fmts = ff_make_format_list(out_pix_fmts); + + ff_formats_ref(main_fmts, &ctx->inputs[0]->out_formats); + ff_formats_ref(out_fmts, &ctx->outputs[0]->in_formats); + + return 0; +} + +static av_cold int vpp_init(AVFilterContext *ctx) +{ + VPPContext *vpp = ctx->priv; + + vpp->frame_number = 0; + vpp->pFrameAllocator = NULL; + vpp->vpp_ready = 0; + vpp->ctx = ctx; + vpp->sysmem_cur_out_idx = 0; + + return 0; +} + +static av_cold void vpp_uninit(AVFilterContext *ctx) +{ + deconf_vpp(ctx); +} + +static int vpp_cmd_size(AVFilterContext *ctx, const char *arg) +{ + VPPContext *vpp = ctx->priv; + int w,h,ret; + + ret = av_parse_video_size(&w, &h, arg); + if (ret) + return ret; + + if (w != vpp->out_width || h != vpp->out_height) { + if (vpp->vpp_ready) + deconf_vpp(ctx); + vpp->out_width = w; + vpp->out_height = h; + } + + return ret; +} + +static int vpp_process_cmd(AVFilterContext *ctx, const char *cmd, const char *arg, char *res, int res_len, int flags) +{ + int ret = 0, i; + +#undef NELEMS +#define NELEMS(x) (sizeof(x)/sizeof((x)[0])) + static const struct{ + const char *short_name; + const char *long_name; + const char *desc; + int (*func)(AVFilterContext*, const char *); + int need_arg; + const char *arg_desc; + }cmdlist[] = { + {"h", "help", "Show this help.", NULL, 0, NULL}, + {"s", "size", "Output resolution", vpp_cmd_size, 1, "wxh"}, + }; + + for (i = 0; i < NELEMS(cmdlist); i++) { + if (!av_strcasecmp(cmd, cmdlist[i].long_name) + || !av_strcasecmp(cmd, cmdlist[i].short_name)) + break; + } + + if ((i > NELEMS(cmdlist)) || (i <= 0) || (cmdlist[i].need_arg && !arg)) { + for (i = 0; i < NELEMS(cmdlist); i++) + av_log(ctx, AV_LOG_INFO, "%2s|%-12s %12s\t%s\n", cmdlist[i].short_name, + cmdlist[i].long_name, cmdlist[i].desc, cmdlist[i].arg_desc); + + return AVERROR(EINVAL); + } + + if (cmdlist[i].func) + ret = cmdlist[i].func(ctx, arg); + av_log(ctx, AV_LOG_DEBUG, "Dealing with cmd: %s, args: %s, ret: %d.\n", cmd, arg, ret); + + return ret; +} + +static const AVFilterPad vpp_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input, + .filter_frame = filter_frame, + }, + { NULL } +}; + +static const AVFilterPad vpp_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, + { NULL } +}; + +AVFilter ff_vf_vpp = { + .name = "vpp", + .description = NULL_IF_CONFIG_SMALL("Quick Sync Video VPP."), + .priv_size = sizeof(VPPContext), + .query_formats = query_formats, + .init = vpp_init, + .uninit = vpp_uninit, + .inputs = vpp_inputs, + .outputs = vpp_outputs, + .priv_class = &vpp_class, + .process_command = vpp_process_cmd, +};