From patchwork Mon Jun 14 02:08:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuewei Meng <928826483@qq.com> X-Patchwork-Id: 28263 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5e:c91a:0:0:0:0:0 with SMTP id z26csp2464894iol; Sun, 13 Jun 2021 19:08:25 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxQ39ne01oRic54LeTsxxyzeC8ikrUmx0cxjqf62k2MaZjnomd+kLT5tbpwY47HLSdbI5Gh X-Received: by 2002:a17:906:3402:: with SMTP id c2mr4393676ejb.213.1623636505020; Sun, 13 Jun 2021 19:08:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1623636505; cv=none; d=google.com; s=arc-20160816; b=YO5Y7DBZS3g6CwmYbeOaZznBRhwiQmwUj7Pnd+KXf/MzKCTEGcnMZNZ7LSzjHvJ+UL Wj32A2kNsOu4dq87wl94jVdjbnBqcK6a9ueu8H+bWoxKd6kjviErkcP6Z4AdGj7WDl2M ln60C8AyDL+kJJ7xrnegbXaaqKIR+Bu9cC1Ph0ad5zcJXV2MaACiSOcrJtSu8+gqd73F 6kVXh0DsXbZHnVw/i+5QuRYHcY4BC4Da9hfdozGiZCbtB9vjZkBD0H3+aVID3GfY15Dz O1v4+LNknsV2yRyfN+QGwnQxxnAcWkNfSdIlSSKiGle+pnBbikpzG6Ib/2Q20VVi3HoW aMYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:date:to:from:message-id:dkim-signature :delivered-to; bh=VBZf+nm3h5t/1pGx580bD8OVDDCnx8qjMYsfeP3oC+k=; b=B7SOx/o+TGAhm3RpIRcGOBz1xtdVfP0kv/XP0K9CVu/Gh5IDNkr/iHBzbxDmaSOaho bnegUENf29MRAiKa0CQsJAvgGijgKr12I5IKU8NV3E440bIoBM4sPVGoSmqwtONg2E2G PtDFmtjcGIN5zxuCbOKkDKNj+nSYCq6mDXU0iNal0BgokfKgCivz4yVBwlmPEZHEcwOY jtE9o4dkj3Wx8gi9WOr/f3ZbLjhPuBxVrgi8yN61hM0I0fpJbX0xMv6G902AJfEABniE fL2cYqG+BoICHX1euwAJs0kRlKpwUGiPFiafzvnNC7ENPYoNCQ8kTR/n6lateuKgEd1t CNqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@qq.com header.s=s201512 header.b=QY0dTbRy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=qq.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 89si11167115eda.437.2021.06.13.19.08.24; Sun, 13 Jun 2021 19:08:25 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@qq.com header.s=s201512 header.b=QY0dTbRy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=qq.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E8C35688335; Mon, 14 Jun 2021 05:08:19 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-191.mail.qq.com (out203-205-221-191.mail.qq.com [203.205.221.191]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 12861680998 for ; Mon, 14 Jun 2021 05:08:11 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s201512; t=1623636487; bh=MfclJPwGQNT/1Czorkzp9SU9myQ0s9/VeGdGQdTg3jg=; h=From:To:Cc:Subject:Date; b=QY0dTbRyg/BE0bW7Yq+eQvt2BGdq/MqXQOwCPalg2P+q9epiY3XeEG9RzBn+VxqMp MDoMyIocQx4UFrZiNKZsUJ3kF5SjSLcOWQJgu9GJLQnMUakiQqLOLXT8o4p2+0Qb5H 5a14zIgnKx6Au+5dx8CBoWPagMJvs3XXx2jm7Olk= Received: from meng.localdomain ([162.105.23.172]) by newxmesmtplogicsvrszc6.qq.com (NewEsmtp) with SMTP id 2060B6E6; Mon, 14 Jun 2021 10:08:06 +0800 X-QQ-mid: xmsmtpt1623636486tb1a4pqlu Message-ID: X-QQ-XMAILINFO: OX+uablwd2Jr1ayQ7V6eEHj0mjkqBj8nXvVnUDhKwtVQNGrLh8HZyOgYtit1el Xb7PHHODGkpG/rswnJ2bV+PZL1+v0+Bnn+MqJhV2Ar35DUgKQLUPV6OOXuYEsUH0HRIdta+Cjs/F q9mr31dA57fwyLsykmorN0B87OyLPtcw86qdKZK7MmMjNnDzga/Ya0vlVyynxZB9JMxhQbTqZxhm ieaZ0vnc9l72yJ9GEw8MWRYUEvNT0+7kB4AVdyJJQXYGH6DaszvMMbCrpVxGlvOg1qRy37U8GhBI Gt6cfA5M69o5GwzBzo9gj6WJVg/G3m3D22WhltP0U8QHaSjf/tlFtEcTTr3jAO04doEyQbzzeWNQ uprxrZ76qjc3vWLumU/P1qpI8qO6+pmgJgYYPD1lSrQwtYpX2IQ/d4PAlDgwDS1TS9L9BVVV76Cj Vf5c9XtHH61UUiB5o9JxIt+Vwsu7Fzuea3G09n0UQp1b6gl6E7eOpA7W3dM8vnwufZpTRqhx8oGH CKM34AalEWUIPjj6Mpey/v0sRe11lsgaR8qPwNzw4kC5pYWLAwyYEsSdzTuBtIgRb8JvDPxIzR2Y f2v6Z8n2ZQ8s5EbS4gVEzdmBVA1RfergUJ9ewWOjGuO+kmGG8PHLHuyDH52gpix/mY3kceTSTIMs 0niJH5DRpIO0uOnk8VUax9FutUT+DJODBtTB7rwGZHhb0hApynkOLhBZpQDcvRcxF5GAfPPGE3tY 9SPelFDVeT+Dxhe8lEMqWK5ZsFmCd7scSsm01rxxLLNxcUrAc8f/8F0236IPW19MjCqVdETECvPO qZz7HUeYGt2gjug7oirIYxBLEjEsBoDjM= From: Xuewei Meng <928826483@qq.com> To: ffmpeg-devel@ffmpeg.org Date: Mon, 14 Jun 2021 10:08:01 +0800 X-OQ-MSGID: <1623636481-70-1-git-send-email-928826483@qq.com> X-Mailer: git-send-email 1.9.1 Subject: [FFmpeg-devel] [PATCH] avfilter/vf_guided: support single input X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Xuewei Meng MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: LkJ5eTQSwzH+ From: Xuewei Meng Support single input for guided filter by adding guidance mode. If the guidance mode is off, single input is required. And edge-preserving smoothing is conducted. If the mode is on, two inputs are needed. The second input serves as the guidance. For this mode, more tasks are supported, such as detail enhancement, dehazing and so on. Signed-off-by: Xuewei Meng --- doc/filters.texi | 12 ++-- libavfilter/vf_guided.c | 168 ++++++++++++++++++++++++++++++++---------------- 2 files changed, 119 insertions(+), 61 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index 78faf76..5c362c0 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -12975,8 +12975,6 @@ greyedge=difford=1:minknorm=0:sigma=2 @section guided Apply guided filter for edge-preserving smoothing, dehazing and so on. -This filter requires two inputs of same resolution and pixel format. -The second input serves as the reference. The filter accepts the following options: @table @option @@ -12997,6 +12995,12 @@ Set subsampling ratio for @code{fast} mode. Range is 2 to 64. Default is 4. No subsampling occurs in @code{basic} mode. +@item guidance +Set guidance mode. Can be @code{off} or @code{on}. Default is @code{off}. +If @code{off}, single input is required. +If @code{on}, two inputs of the same resolution and pixel format are required. +The second input serves as the guidance. + @item planes Set planes to filter. Default is first only. @end table @@ -13009,7 +13013,7 @@ This filter supports the all above options as @ref{commands}. @item Edge-preserving smoothing with guided filter: @example -ffmpeg -i in.png -i in.png -filter_complex guided out.png +ffmpeg -i in.png -vf guided out.png @end example @item @@ -13017,7 +13021,7 @@ Dehazing, structure-transferring filtering, detail enhancement with guided filte For the generation of guidance image, refer to paper "Guided Image Filtering". See: @url{http://kaiminghe.com/publications/pami12guidedfilter.pdf}. @example -ffmpeg -i in.png -i guidance.png -filter_complex guided out.png +ffmpeg -i in.png -i guidance.png -filter_complex guided=guidance=on out.png @end example @end itemize diff --git a/libavfilter/vf_guided.c b/libavfilter/vf_guided.c index ea537e4..739d615 100644 --- a/libavfilter/vf_guided.c +++ b/libavfilter/vf_guided.c @@ -22,6 +22,7 @@ #include "libavutil/opt.h" #include "libavutil/pixdesc.h" #include "avfilter.h" +#include "filters.h" #include "formats.h" #include "framesync.h" #include "internal.h" @@ -33,6 +34,12 @@ enum FilterModes { NB_MODES, }; +enum GuidanceModes { + OFF, + ON, + NB_GUIDANCE_MODES, +}; + typedef struct GuidedContext { const AVClass *class; FFFrameSync fs; @@ -41,7 +48,7 @@ typedef struct GuidedContext { float eps; int mode; int sub; - + int guidance; int planes; int width; @@ -59,13 +66,16 @@ typedef struct GuidedContext { #define FLAGS AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_RUNTIME_PARAM static const AVOption guided_options[] = { - { "radius", "set the box radius", OFFSET(radius), AV_OPT_TYPE_INT, {.i64 = 3 }, 1, 20, FLAGS }, - { "eps", "set the regularization parameter (with square)", OFFSET(eps), AV_OPT_TYPE_FLOAT, {.dbl = 0.01 }, 0.0, 1, FLAGS }, - { "mode", "set filtering mode (0: basic mode; 1: fast mode)", OFFSET(mode), AV_OPT_TYPE_INT, {.i64 = BASIC}, BASIC, NB_MODES - 1, FLAGS, "mode" }, - { "basic", "basic guided filter", 0, AV_OPT_TYPE_CONST, {.i64 = BASIC}, 0, 0, FLAGS, "mode" }, - { "fast", "fast guided filter", 0, AV_OPT_TYPE_CONST, {.i64 = FAST }, 0, 0, FLAGS, "mode" }, - { "sub", "subsampling ratio for fast mode", OFFSET(sub), AV_OPT_TYPE_INT, {.i64 = 4 }, 2, 64, FLAGS }, - { "planes", "set planes to filter", OFFSET(planes), AV_OPT_TYPE_INT, {.i64=1 }, 0, 0xF, FLAGS }, + { "radius", "set the box radius", OFFSET(radius), AV_OPT_TYPE_INT, {.i64 = 3 }, 1, 20, FLAGS }, + { "eps", "set the regularization parameter (with square)", OFFSET(eps), AV_OPT_TYPE_FLOAT, {.dbl = 0.01 }, 0.0, 1, FLAGS }, + { "mode", "set filtering mode (0: basic mode; 1: fast mode)", OFFSET(mode), AV_OPT_TYPE_INT, {.i64 = BASIC}, BASIC, NB_MODES - 1, FLAGS, "mode" }, + { "basic", "basic guided filter", 0, AV_OPT_TYPE_CONST, {.i64 = BASIC}, 0, 0, FLAGS, "mode" }, + { "fast", "fast guided filter", 0, AV_OPT_TYPE_CONST, {.i64 = FAST }, 0, 0, FLAGS, "mode" }, + { "sub", "subsampling ratio for fast mode", OFFSET(sub), AV_OPT_TYPE_INT, {.i64 = 4 }, 2, 64, FLAGS }, + { "guidance", "set guidance mode (0: off mode; 1: on mode)", OFFSET(guidance), AV_OPT_TYPE_INT, {.i64 = OFF }, OFF, NB_GUIDANCE_MODES - 1, FLAGS, "guidance" }, + { "off", "only one input is enabled", 0, AV_OPT_TYPE_CONST, {.i64 = OFF }, 0, 0, FLAGS, "guidance" }, + { "on", "two inputs are required", 0, AV_OPT_TYPE_CONST, {.i64 = ON }, 0, 0, FLAGS, "guidance" }, + { "planes", "set planes to filter", OFFSET(planes), AV_OPT_TYPE_INT, {.i64 = 1 }, 0, 0xF, FLAGS }, { NULL } }; @@ -149,16 +159,6 @@ static int config_input(AVFilterLink *inlink) GuidedContext *s = ctx->priv; const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(inlink->format); - if (ctx->inputs[0]->w != ctx->inputs[1]->w || - ctx->inputs[0]->h != ctx->inputs[1]->h) { - av_log(ctx, AV_LOG_ERROR, "Width and height of input videos must be same.\n"); - return AVERROR(EINVAL); - } - if (ctx->inputs[0]->format != ctx->inputs[1]->format) { - av_log(ctx, AV_LOG_ERROR, "Inputs must be of same pixel format.\n"); - return AVERROR(EINVAL); - } - if (s->mode == BASIC) { s->sub = 1; } @@ -230,7 +230,7 @@ static int guided_##name(AVFilterContext *ctx, GuidedContext *s, meanB = av_calloc(w * h, sizeof(float)); \ \ if (!I || !II || !P || !IP || !meanI || !meanII || !meanP || \ - !meanIP || !A || !B || !meanA || !meanB){ \ + !meanIP || !A || !B || !meanA || !meanB) { \ ret = AVERROR(ENOMEM); \ goto end; \ } \ @@ -304,47 +304,54 @@ end: GUIDED(uint8_t, byte) GUIDED(uint16_t, word) -static int process_frame(FFFrameSync *fs) +static int filter_frame(AVFilterContext *ctx, AVFrame **out, AVFrame *in, AVFrame *ref) { - AVFilterContext *ctx = fs->parent; - GuidedContext *s = fs->opaque; + GuidedContext *s = ctx->priv; AVFilterLink *outlink = ctx->outputs[0]; - AVFrame *out_frame = NULL, *main_frame = NULL, *ref_frame = NULL; - int ret; - - ret = ff_framesync_dualinput_get(fs, &main_frame, &ref_frame); - if (ret < 0) - return ret; - - out_frame = ff_get_video_buffer(outlink, outlink->w, outlink->h); - if (!out_frame) { - av_frame_free(&main_frame); + *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!*out) return AVERROR(ENOMEM); - } - av_frame_copy_props(out_frame, main_frame); + av_frame_copy_props(*out, in); for (int plane = 0; plane < s->nb_planes; plane++) { if (!(s->planes & (1 << plane))) { - av_image_copy_plane(out_frame->data[plane], out_frame->linesize[plane], - main_frame->data[plane], main_frame->linesize[plane], + av_image_copy_plane((*out)->data[plane], (*out)->linesize[plane], + in->data[plane], in->linesize[plane], s->planewidth[plane] * ((s->depth + 7) / 8), s->planeheight[plane]); continue; } if (s->depth <= 8) - guided_byte(ctx, s, main_frame->data[plane], ref_frame->data[plane], out_frame->data[plane], s->radius, s->eps, + guided_byte(ctx, s, in->data[plane], ref->data[plane], (*out)->data[plane], s->radius, s->eps, s->planewidth[plane], s->planeheight[plane], - main_frame->linesize[plane], ref_frame->linesize[plane], out_frame->linesize[plane], (1 << s->depth) - 1.f); + in->linesize[plane], ref->linesize[plane], (*out)->linesize[plane], (1 << s->depth) - 1.f); else - guided_word(ctx, s, main_frame->data[plane], ref_frame->data[plane], out_frame->data[plane], s->radius, s->eps, + guided_word(ctx, s, in->data[plane], ref->data[plane], (*out)->data[plane], s->radius, s->eps, s->planewidth[plane], s->planeheight[plane], - main_frame->linesize[plane] / 2, ref_frame->linesize[plane] / 2, out_frame->linesize[plane] / 2, (1 << s->depth) - 1.f); + in->linesize[plane] / 2, ref->linesize[plane] / 2, (*out)->linesize[plane] / 2, (1 << s->depth) - 1.f); + } + + return 0; +} + +static int process_frame(FFFrameSync *fs) +{ + AVFilterContext *ctx = fs->parent; + AVFilterLink *outlink = ctx->outputs[0]; + AVFrame *out_frame = NULL, *main_frame = NULL, *ref_frame = NULL; + int ret; + ret = ff_framesync_dualinput_get(fs, &main_frame, &ref_frame); + if (ret < 0) + return ret; + + ret = filter_frame(ctx, &out_frame, main_frame, ref_frame); + if(ret < 0) { + return ret; } av_frame_free(&main_frame); return ff_filter_frame(outlink, out_frame); } - static int config_output(AVFilterLink *outlink) { AVFilterContext *ctx = outlink->src; @@ -354,12 +361,27 @@ static int config_output(AVFilterLink *outlink) FFFrameSyncIn *in; int ret; + if(s->guidance == ON) { + if (ctx->inputs[0]->w != ctx->inputs[1]->w || + ctx->inputs[0]->h != ctx->inputs[1]->h) { + av_log(ctx, AV_LOG_ERROR, "Width and height of input videos must be same.\n"); + return AVERROR(EINVAL); + } + if (ctx->inputs[0]->format != ctx->inputs[1]->format) { + av_log(ctx, AV_LOG_ERROR, "Inputs must be of same pixel format.\n"); + return AVERROR(EINVAL); + } + } outlink->w = mainlink->w; outlink->h = mainlink->h; outlink->time_base = mainlink->time_base; outlink->sample_aspect_ratio = mainlink->sample_aspect_ratio; outlink->frame_rate = mainlink->frame_rate; + + if (s->guidance == OFF) + return 0; + if ((ret = ff_framesync_init(&s->fs, ctx, 2)) < 0) return ret; @@ -383,22 +405,66 @@ static int config_output(AVFilterLink *outlink) static int activate(AVFilterContext *ctx) { GuidedContext *s = ctx->priv; - return ff_framesync_activate(&s->fs); + AVFrame *frame = NULL; + AVFrame *out = NULL; + int ret, status; + int64_t pts; + if(s->guidance) + return ff_framesync_activate(&s->fs); + + FF_FILTER_FORWARD_STATUS_BACK(ctx->outputs[0], ctx->inputs[0]); + + if ((ret = ff_inlink_consume_frame(ctx->inputs[0], &frame)) > 0) { + ret = filter_frame(ctx, &out, frame, frame); + av_frame_free(&frame); + if (ret < 0) + return ret; + ret = ff_filter_frame(ctx->outputs[0], out); + } + if (ret < 0) + return ret; + if (ff_inlink_acknowledge_status(ctx->inputs[0], &status, &pts)) { + ff_outlink_set_status(ctx->outputs[0], status, pts); + return 0; + } + if (ff_outlink_frame_wanted(ctx->outputs[0])) + ff_inlink_request_frame(ctx->inputs[0]); + return 0; } static av_cold int init(AVFilterContext *ctx) { + GuidedContext *s = ctx->priv; + AVFilterPad pad = { 0 }; + int ret; + + pad.type = AVMEDIA_TYPE_VIDEO; + pad.name = "source"; + pad.config_props = config_input; + + if ((ret = ff_insert_inpad(ctx, 0, &pad)) < 0) + return ret; + + if (s->guidance == ON) { + pad.type = AVMEDIA_TYPE_VIDEO; + pad.name = "guidance"; + pad.config_props = NULL; + + if ((ret = ff_insert_inpad(ctx, 1, &pad)) < 0) + return ret; + } + return 0; } static av_cold void uninit(AVFilterContext *ctx) { GuidedContext *s = ctx->priv; - ff_framesync_uninit(&s->fs); + if(s->guidance == ON) + ff_framesync_uninit(&s->fs); return; } - static int process_command(AVFilterContext *ctx, const char *cmd, const char *arg, @@ -414,18 +480,6 @@ static int process_command(AVFilterContext *ctx, return 0; } -static const AVFilterPad guided_inputs[] = { - { - .name = "main", - .type = AVMEDIA_TYPE_VIDEO, - },{ - .name = "reference", - .type = AVMEDIA_TYPE_VIDEO, - .config_props = config_input, - }, - { NULL } -}; - static const AVFilterPad guided_outputs[] = { { .name = "default", @@ -444,7 +498,7 @@ const AVFilter ff_vf_guided = { .priv_size = sizeof(GuidedContext), .priv_class = &guided_class, .activate = activate, - .inputs = guided_inputs, + .inputs = NULL, .outputs = guided_outputs, .flags = AVFILTER_FLAG_SUPPORT_TIMELINE_GENERIC | AVFILTER_FLAG_SLICE_THREADS, .process_command = process_command,