From patchwork Fri Jul 31 14:57:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zong, Wei" X-Patchwork-Id: 21395 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 54E7A44B984 for ; Fri, 31 Jul 2020 10:07:25 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 29A8D68B6E8; Fri, 31 Jul 2020 10:07:25 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 85F8468B641 for ; Fri, 31 Jul 2020 10:07:18 +0300 (EEST) IronPort-SDR: BST92MH+9JugjiMCzAZStEYtejE635tu/mdHbYAb0bfXtrdmjImVz2tpLsHJSqbaItmvbJ5nKI 9GjvGO/9Pv8w== X-IronPort-AV: E=McAfee;i="6000,8403,9698"; a="139723608" X-IronPort-AV: E=Sophos;i="5.75,417,1589266800"; d="scan'208";a="139723608" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 Jul 2020 00:07:16 -0700 IronPort-SDR: HugosoY+jCY68GBVjSIHHuqCv//IVaMznyeM/VkB32eXg3/D8DxZB1H2oYwzR+r3GpR+xhSKBt xNO1v1YQilHA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,417,1589266800"; d="scan'208";a="321327210" Received: from zongwei-im360.bj.intel.com ([10.238.153.107]) by orsmga008.jf.intel.com with ESMTP; 31 Jul 2020 00:07:13 -0700 From: zongwave To: ffmpeg-devel@ffmpeg.org Date: Fri, 31 Jul 2020 22:57:10 +0800 Message-Id: <20200731145710.114479-1-wei.zong@intel.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH] avfilter/vf_xcam: add xcam video filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zong Wei , Yinhang Liu MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Yinhang Liu xcam filter is a wrapper of libxcam project which supports 360 video stitching, automotive surround view stitching, digital video stabilization (DVS), wavelet noise reduction, 3D noise reduction, etc. libxcam library optimized algorithms by AVX, GLES and Vulkan depends on hardware configuration to get good performance. Here are demos of xcam filter * video stitching: https://github.com/intel/libxcam/wiki#libxcam-stitch-processing-flow https://www.youtube.com/watch?v=J2Cr09KfIQ0 https://www.youtube.com/watch?v=z5MvzhRKcE8 * digital video stabilization: https://www.youtube.com/watch?v=f5JFhZPa4Xc libxcam project is hosted at: https://github.com/intel/libxcam Build instruction is at: https://github.com/intel/libxcam/wiki/Build To enable xcam video filter, configure FFmpeg with option --enable-libxcam Here are features provided by xcam video filter: - stitch CPU|GLES|Vulkan stitching - stitchcl OpenCL stitching - fisheye Fisheye calibration - 3dnr 3D denoising - waveletnr Wavelet denoising - dvs Digital video stabilization - defog Fog removal Use 'ffmpeg -h filter=xcam' to get the common parameters, use 'params=help=1' to get the private parameters. Detailed test cases at: https://github.com/intel/libxcam/wiki/Tests#1-ffmpeg-xcam Signed-off-by: Yinhang Liu Signed-off-by: Zong Wei Reviewed-by: Zong Wei --- Changelog | 1 + configure | 5 + doc/filters.texi | 79 ++++++++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_xcam.c | 326 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 413 insertions(+) create mode 100644 libavfilter/vf_xcam.c diff --git a/Changelog b/Changelog index 6f648bff2b..1d5872e216 100644 --- a/Changelog +++ b/Changelog @@ -10,6 +10,7 @@ version : - ADPCM IMA Ubisoft APM encoder - Rayman 2 APM muxer - AV1 encoding support SVT-AV1 +- xcam filter version 4.3: diff --git a/configure b/configure index 169f23e17f..7664a077b3 100755 --- a/configure +++ b/configure @@ -290,6 +290,7 @@ External library support: --enable-libx265 enable HEVC encoding via x265 [no] --enable-libxavs enable AVS encoding via xavs [no] --enable-libxavs2 enable AVS2 encoding via xavs2 [no] + --enable-libxcam enable image processing via xcam [no] --enable-libxcb enable X11 grabbing using XCB [autodetect] --enable-libxcb-shm enable X11 grabbing shm communication [autodetect] --enable-libxcb-xfixes enable X11 grabbing mouse rendering [autodetect] @@ -1817,6 +1818,7 @@ EXTERNAL_LIBRARY_LIST=" libvpx libwavpack libwebp + libxcam libxml2 libzimg libzmq @@ -3639,6 +3641,7 @@ scale_vaapi_filter_deps="vaapi" scale_vulkan_filter_deps="vulkan libglslang" vpp_qsv_filter_deps="libmfx" vpp_qsv_filter_select="qsvvpp" +xcam_filter_deps="libxcam" xfade_opencl_filter_deps="opencl" yadif_cuda_filter_deps="ffnvcodec" yadif_cuda_filter_deps_any="cuda_nvcc cuda_llvm" @@ -6451,6 +6454,8 @@ enabled libx265 && require_pkg_config libx265 x265 x265.h x265_api_get require_cpp_condition libx265 x265.h "X265_BUILD >= 70" enabled libxavs && require libxavs "stdint.h xavs.h" xavs_encoder_encode "-lxavs $pthreads_extralibs $libm_extralibs" enabled libxavs2 && require_pkg_config libxavs2 "xavs2 >= 1.3.0" "stdint.h xavs2.h" xavs2_api_get +enabled libxcam && { check_pkg_config libxcam "libxcam >= 1.4.0" "capi/xcam_handle.h" xcam_create_handle || + die "ERROR: libXCam must be installed and version must be >= 1.4.0"; } enabled libxvid && require libxvid xvid.h xvid_global -lxvidcore enabled libzimg && require_pkg_config libzimg "zimg >= 2.7.0" zimg.h zimg_get_api_version enabled libzmq && require_pkg_config libzmq "libzmq >= 4.2.1" zmq.h zmq_ctx_new diff --git a/doc/filters.texi b/doc/filters.texi index 561aa98a9d..ab6f231025 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -20399,6 +20399,85 @@ Set the scaling dimension: @code{2} for @code{2xBR}, @code{3} for Default is @code{3}. @end table +@section xcam +Image processing supported through libXCam. + +libXCam supports automotive surround view stitching, 360 video stitching, +digital video stabilization, noise reduction and so on. For more information +about libxcam see @url{https://github.com/intel/libxcam}. + +Please refer to @url{https://github.com/intel/libxcam/wiki/Build} to build +libxcam. + +To enable compilation of @var{xcam} video filter you need to configure FFmpeg +with @code{--enable-libxcam}. + +@subsection Options + +@table @option + +@item inputs +The number of inputs. Default is @code{1}. @b{stitch} and @b{stitchcl} +handlers support dynamic inputs, @b{fisheye}, @b{3dnr}, @b{waveletnr}, +@b{dvs} and @b{defog} handlers support one input. + +@item w +Output video width. Default is @code{0}. +If the value is 0, the corresponding input width is used for the output. + +@item h +Output video height. Default is @code{0}. +If the value is 0, the corresponding input height is used for the output. + +@item fmt +Pixel format. Default is @code{nv12}. + +@table @samp +@item auto +Negotiate pixel format automatically, selects the input pixel format as the +processing pixel format. +@item nv12 +NV12 pixel format. All handlers support NV12 pixel format. +@item yuv420 +YUV420 pixel format. Currently, only @b{CPU} stitching supports YUV420 +pixel format. +@end table + +@item name +Handler name. Default is @code{stitch}. + +@table @samp +@item stitch +CPU|GLES|Vulkan stitching, supports automotive surround view stitching and +360 video stitching. +@item stitchcl +OpenCL stitching, supports automotive surround view stitching and 360 video +stitching. +@item fisheye +Fisheye calibration +@item 3dnr +3D denoising +@item waveletnr +Wavelet denoising +@item dvs +Digital video stabilization +@item defog +Fog removal +@end table + +@item params +Private parameters for each handler. Currently, only @b{stitch} and +@b{stitchcl} handlers have private parameters. + +@end table + +@subsection Examples +Use @code{ffmpeg -h filter=xcam} to get the common parameters, @b{stitch} +and @b{stitchcl} handlers have private parameters, use @code{params=help=1} +to get the private parameters. + +For more detailed examples see @url{https://github.com/intel/libxcam/wiki/Tests#1-ffmpeg-xcam}. + @section xfade Apply cross fade from one input video stream to another input video stream. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 0dc74f8b70..4f9acd7770 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -456,6 +456,7 @@ OBJS-$(CONFIG_W3FDIF_FILTER) += vf_w3fdif.o OBJS-$(CONFIG_WAVEFORM_FILTER) += vf_waveform.o OBJS-$(CONFIG_WEAVE_FILTER) += vf_weave.o OBJS-$(CONFIG_XBR_FILTER) += vf_xbr.o +OBJS-$(CONFIG_XCAM_FILTER) += vf_xcam.o OBJS-$(CONFIG_XFADE_FILTER) += vf_xfade.o OBJS-$(CONFIG_XFADE_OPENCL_FILTER) += vf_xfade_opencl.o opencl.o opencl/xfade.o OBJS-$(CONFIG_XMEDIAN_FILTER) += vf_xmedian.o framesync.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 3f70153986..728c11b303 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -435,6 +435,7 @@ extern AVFilter ff_vf_w3fdif; extern AVFilter ff_vf_waveform; extern AVFilter ff_vf_weave; extern AVFilter ff_vf_xbr; +extern AVFilter ff_vf_xcam; extern AVFilter ff_vf_xfade; extern AVFilter ff_vf_xfade_opencl; extern AVFilter ff_vf_xmedian; diff --git a/libavfilter/vf_xcam.c b/libavfilter/vf_xcam.c new file mode 100644 index 0000000000..d3384f6f74 --- /dev/null +++ b/libavfilter/vf_xcam.c @@ -0,0 +1,326 @@ +/* + * Copyright (c) 2020 Intel Corporation. + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * libXCam wrapper functions + */ + +#include +#include "libavutil/avstring.h" +#include "libavutil/opt.h" +#include "framesync.h" +#include "internal.h" + +typedef struct XCamVideoFilterBuf { + XCamVideoBuffer buf; + AVFrame *frame; +} XCamVideoFilterBuf; + +typedef struct XCAMContext { + const AVClass *class; + + int nb_inputs; + int w; + int h; + char *fmt; + char *name; + char *params; + + XCamHandle *handle; + uint32_t v4l2_fmt; + + XCamVideoFilterBuf **inbufs; + XCamVideoFilterBuf *outbuf; + + FFFrameSync fs; +} XCAMContext; + +static uint8_t *xcambuf_map(XCamVideoBuffer *buf) +{ + XCamVideoFilterBuf *avfilter_buf = (XCamVideoFilterBuf *)(buf); + return avfilter_buf->frame->data[0]; +} + +static void xcambuf_unmap(XCamVideoBuffer *buf) +{ + return; +} + +static uint32_t avfmt_to_v4l2fmt(AVFilterContext *ctx, int avfmt) +{ + uint32_t v4l2fmt = 0; + if (avfmt == AV_PIX_FMT_YUV420P) + v4l2fmt = V4L2_PIX_FMT_YUV420; + else if (avfmt == AV_PIX_FMT_NV12) + v4l2fmt = V4L2_PIX_FMT_NV12; + else + av_log(ctx, AV_LOG_ERROR, "unsupported pixel format %d\n", avfmt); + + return v4l2fmt; +} + +static int set_parameters(AVFilterContext *ctx, const AVFilterLink *inlink, const AVFilterLink *outlink) +{ + XCAMContext *s = inlink->dst->priv; + + char params[XCAM_MAX_PARAMS_LENGTH] = { 0 }; + snprintf(params, XCAM_MAX_PARAMS_LENGTH - 1, "inw=%d inh=%d outw=%d outh=%d fmt=%d %s", + inlink->w, inlink->h, outlink->w, outlink->h, s->v4l2_fmt, s->params); + + if (xcam_handle_set_parameters(s->handle, params) != XCAM_RETURN_NO_ERROR) { + av_log(ctx, AV_LOG_ERROR, "xcam handler set parameters failed\n"); + return AVERROR(EINVAL); + } + + return 0; +} + +static int init_xcambuf_info(XCAMContext *s, XCamVideoBuffer *buf, AVFrame *frame) +{ + XCamReturn ret = xcam_video_buffer_info_reset( + &buf->info, s->v4l2_fmt, frame->width, frame->height, frame->linesize[0], frame->height, 0); + if (ret != XCAM_RETURN_NO_ERROR) + return AVERROR(EINVAL); + + for (int i = 0; frame->linesize[i]; i++) { + buf->info.offsets[i] = frame->data[i] - frame->data[0]; + buf->info.strides[i] = frame->linesize[i]; + } + + return 0; +} + +static int xcam_execute(FFFrameSync *fs) +{ + AVFilterContext *ctx = fs->parent; + XCAMContext *s = fs->opaque; + AVFilterLink *outlink; + AVFrame *outframe, *frame; + + XCamVideoFilterBuf **inbufs = s->inbufs; + XCamVideoFilterBuf *outbuf = s->outbuf; + + for (int i = 0; i < ctx->nb_inputs; i++) { + int error = ff_framesync_get_frame(&s->fs, i, &frame, 0); + if (error < 0) + return error; + if (init_xcambuf_info(s, &inbufs[i]->buf, frame) != 0) + return AVERROR(EINVAL); + inbufs[i]->frame = frame; + } + + outlink = ctx->outputs[0]; + if (!(outframe = ff_get_video_buffer(outlink, outlink->w, outlink->h))) { + av_frame_free(&frame); + return AVERROR(ENOMEM); + } + outframe->pts = av_rescale_q(s->fs.pts, s->fs.time_base, outlink->time_base); + + if (init_xcambuf_info(s, &outbuf->buf, outframe) != 0) + return AVERROR(EINVAL); + outbuf->frame = outframe; + + if (xcam_handle_execute(s->handle, (XCamVideoBuffer **)inbufs, (XCamVideoBuffer **)&outbuf) != XCAM_RETURN_NO_ERROR) { + av_log(ctx, AV_LOG_ERROR, "execute xcam handler failed\n"); + return AVERROR(EINVAL); + } + + return ff_filter_frame(outlink, outframe); +} + +static int xcam_query_formats(AVFilterContext *ctx) +{ + XCAMContext *s = ctx->priv; + AVFilterFormats *formats = NULL; + + static const enum AVPixelFormat nv12_fmts[] = {AV_PIX_FMT_NV12, AV_PIX_FMT_NONE}; + static const enum AVPixelFormat yuv420_fmts[] = {AV_PIX_FMT_YUV420P, AV_PIX_FMT_NONE}; + static const enum AVPixelFormat auto_fmts[] = {AV_PIX_FMT_NV12, AV_PIX_FMT_YUV420P, AV_PIX_FMT_NONE}; + + const enum AVPixelFormat *pix_fmts = NULL; + if (!av_strcasecmp(s->fmt, "nv12")) + pix_fmts = nv12_fmts; + else if (!av_strcasecmp(s->fmt, "yuv420")) + pix_fmts = yuv420_fmts; + else + pix_fmts = auto_fmts; + + if (!(formats = ff_make_format_list(pix_fmts))) + return AVERROR(ENOMEM); + + return ff_set_common_formats(ctx, formats); +} + +static int xcam_config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + XCAMContext *s = ctx->priv; + AVFilterLink *inlink = ctx->inputs[0]; + int ret = 0; + + s->v4l2_fmt = avfmt_to_v4l2fmt(ctx, inlink->format); + if (s->v4l2_fmt == 0) + return AVERROR(EINVAL); + + if (s->w && s->h) { + outlink->w = s->w; + outlink->h = s->h; + } else { + outlink->w = inlink->w; + outlink->h = inlink->h; + } + + set_parameters(ctx, inlink, outlink); + if (xcam_handle_init(s->handle) != XCAM_RETURN_NO_ERROR) { + av_log(ctx, AV_LOG_ERROR, "init xcam handler failed\n"); + return AVERROR(EINVAL); + } + + if ((ret = ff_framesync_init(&s->fs, ctx, ctx->nb_inputs)) < 0) + return ret; + s->fs.opaque = s; + s->fs.on_event = xcam_execute; + for (int i = 0; i < ctx->nb_inputs; i++) { + FFFrameSyncIn *in = &s->fs.in[i]; + in->time_base = ctx->inputs[i]->time_base; + in->sync = 1; + in->before = EXT_STOP; + in->after = EXT_STOP; + } + ret = ff_framesync_configure(&s->fs); + outlink->time_base = s->fs.time_base; + + return ret; +} + +static av_cold int xcam_init(AVFilterContext *ctx) +{ + XCAMContext *s = ctx->priv; + int ret = 0; + + s->handle = xcam_create_handle(s->name); + if (!s->handle) { + av_log(ctx, AV_LOG_ERROR, "create xcam handler failed\n"); + return AVERROR(EINVAL); + } + + s->inbufs = av_mallocz_array(s->nb_inputs + 1, sizeof(XCamVideoFilterBuf *)); + for (int i = 0; i < s->nb_inputs; i++) { + s->inbufs[i] = av_mallocz_array(1, sizeof(XCamVideoFilterBuf)); + if (!s->inbufs[i]) + return AVERROR(ENOMEM); + s->inbufs[i]->buf.map = xcambuf_map; + s->inbufs[i]->buf.unmap = xcambuf_unmap; + } + + s->outbuf = av_mallocz_array(1, sizeof(XCamVideoFilterBuf)); + s->outbuf->buf.map = xcambuf_map; + s->outbuf->buf.unmap = xcambuf_unmap; + + for (int i = 0; i < s->nb_inputs; i++) { + AVFilterPad pad = { .type = AVMEDIA_TYPE_VIDEO }; + pad.name = av_asprintf("input%d", i); + if (!pad.name) + return AVERROR(ENOMEM); + + if ((ret = ff_insert_inpad(ctx, i, &pad)) < 0) { + av_freep(&pad.name); + return ret; + } + } + + return 0; +} + +static av_cold void xcam_uninit(AVFilterContext *ctx) +{ + XCAMContext *s = ctx->priv; + + ff_framesync_uninit(&s->fs); + + for (int i = 0; i < s->nb_inputs; i++) { + if (s->inbufs && s->inbufs[i]) + av_freep(&s->inbufs[i]); + if (ctx->input_pads) + av_freep(&ctx->input_pads[i].name); + } + av_freep(&s->inbufs); + av_freep(&s->outbuf); + + xcam_destroy_handle(s->handle); + s->handle = NULL; +} + +static int xcam_activate(AVFilterContext *ctx) +{ + XCAMContext *s = ctx->priv; + return ff_framesync_activate(&s->fs); +} + +#define OFFSET(x) offsetof(XCAMContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_FILTERING_PARAM +#define CONST_STRING(name, help, unit) \ + { name, help, 0, AV_OPT_TYPE_CONST, { .str=name }, 0, 0, FLAGS, unit } + +static const AVOption xcam_options[] = { + { "inputs", "number of inputs", OFFSET(nb_inputs), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, XCAM_MAX_INPUTS_NUM, FLAGS }, + { "w", "output width", OFFSET(w), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, + { "h", "output height", OFFSET(h), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, + { "fmt", "pixel format", OFFSET(fmt), AV_OPT_TYPE_STRING, { .str = "nv12" }, 0, 0, FLAGS, "fmt" }, + CONST_STRING("auto", "automatic pixel format negotiation", "fmt"), + CONST_STRING("nv12", "NV12 pixel format, all handlers support NV12 pixel format", "fmt"), + CONST_STRING("yuv420", "YUV420 pixel format, only CPU stitching supports YUV420 pixel format", "fmt"), + { "name", "handler name", OFFSET(name), AV_OPT_TYPE_STRING, { .str = "stitch" }, 0, 0, FLAGS, "name" }, + CONST_STRING("stitch", "CPU|GLES|Vulkan stitching", "name"), + CONST_STRING("stitchcl", "OpenCL stitching", "name"), + CONST_STRING("fisheye", "fisheye calibration", "name"), + CONST_STRING("3dnr", "3d denoising", "name"), + CONST_STRING("waveletnr", "wavelet denoising", "name"), + CONST_STRING("dvs", "digital video stabilizer", "name"), + CONST_STRING("defog", "fog removal", "name"), + { "params", "private parameters for each handle, usage: params=help=1 field0=value0 field1=value1 ...", + OFFSET(params), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(xcam); + +static const AVFilterPad xcam_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = xcam_config_output + }, + { NULL } +}; + +AVFilter ff_vf_xcam = { + .name = "xcam", + .description = NULL_IF_CONFIG_SMALL("Apply image processing using libXCam"), + .priv_size = sizeof(XCAMContext), + .priv_class = &xcam_class, + .init = xcam_init, + .query_formats = xcam_query_formats, + .outputs = xcam_outputs, + .activate = xcam_activate, + .uninit = xcam_uninit, + .flags = AVFILTER_FLAG_DYNAMIC_INPUTS +};