| Message ID | CAPYw7P4d0wSChMyCu7TujYDFibXCQXm38uJE=HdOkthALQ1OHQ@mail.gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [FFmpeg-devel] avfilter: add backgroundkey video filter | expand |
| Context | Check | Description |
|---|---|---|
| yinshiyou/configure_loongarch64 | warning | Failed to apply patch |
| andriy/make_x86 | success | Make finished |
| andriy/make_fate_x86 | success | Make fate finished |
On 10/30/22, Paul B Mahol <onemda@gmail.com> wrote: > Patch attached. > Improved patch attached.
Paul B Mahol: > +static int filter_frame(AVFilterLink *link, AVFrame *frame) > +{ > + AVFilterContext *avctx = link->dst; > + BackgroundkeyContext *s = avctx->priv; > + int64_t sum = 0; > + int ret; > + > + if (!s->background) { > + s->background = av_frame_clone(frame); > + if (!s->background) > + return AVERROR(ENOMEM); > + av_frame_make_writable(s->background); You are never writing to the background frame, so there is no point in making it writable; what you actually want to achieve here is making frame writable again and to achieve this you should make frame, not background writable (and of course you should check said call). (Actually, you never > + } > + > + if (ret = ff_filter_execute(avctx, s->do_slice, frame, NULL, > + FFMIN(frame->height, s->nb_threads))) > + return ret; > + > + for (int n = 0; n < s->nb_threads; n++) > + sum += s->sums[n]; > + if (s->max_sum * s->threshold < sum) { > + av_frame_free(&s->background); > + s->background = av_frame_clone(frame); > + if (!s->background) > + return AVERROR(ENOMEM); > + av_frame_make_writable(s->background); Given that you never write to background, there is no need to make it writable. This time, there is also no need to make frame writable (the next filter in the chain may not need a writable frame anyway), so just remove this. And the av_frame_free+av_frame_clone can become an av_frame_unref+av_frame_ref (this will necessitate modifying the check above to not only check for to existence of s->background). > + } > + > + return ff_filter_frame(avctx->outputs[0], frame); > +}
On 11/2/22, Andreas Rheinhardt <andreas.rheinhardt@outlook.com> wrote: > Paul B Mahol: >> +static int filter_frame(AVFilterLink *link, AVFrame *frame) >> +{ >> + AVFilterContext *avctx = link->dst; >> + BackgroundkeyContext *s = avctx->priv; >> + int64_t sum = 0; >> + int ret; >> + >> + if (!s->background) { >> + s->background = av_frame_clone(frame); >> + if (!s->background) >> + return AVERROR(ENOMEM); >> + av_frame_make_writable(s->background); > > You are never writing to the background frame, so there is no point in > making it writable; what you actually want to achieve here is making > frame writable again and to achieve this you should make frame, not > background writable (and of course you should check said call). > This is invalid, input pad receives always writable frames as there is flag. > (Actually, you never > >> + } >> + >> + if (ret = ff_filter_execute(avctx, s->do_slice, frame, NULL, >> + FFMIN(frame->height, s->nb_threads))) >> + return ret; >> + >> + for (int n = 0; n < s->nb_threads; n++) >> + sum += s->sums[n]; >> + if (s->max_sum * s->threshold < sum) { >> + av_frame_free(&s->background); >> + s->background = av_frame_clone(frame); >> + if (!s->background) >> + return AVERROR(ENOMEM); >> + av_frame_make_writable(s->background); > > Given that you never write to background, there is no need to make it > writable. This time, there is also no need to make frame writable (the > next filter in the chain may not need a writable frame anyway), so just > remove this. > And the av_frame_free+av_frame_clone can become an > av_frame_unref+av_frame_ref (this will necessitate modifying the check > above to not only check for to existence of s->background). > >> + } >> + >> + return ff_filter_frame(avctx->outputs[0], frame); >> +} > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >
Paul B Mahol: > On 11/2/22, Andreas Rheinhardt <andreas.rheinhardt@outlook.com> wrote: >> Paul B Mahol: >>> +static int filter_frame(AVFilterLink *link, AVFrame *frame) >>> +{ >>> + AVFilterContext *avctx = link->dst; >>> + BackgroundkeyContext *s = avctx->priv; >>> + int64_t sum = 0; >>> + int ret; >>> + >>> + if (!s->background) { >>> + s->background = av_frame_clone(frame); >>> + if (!s->background) >>> + return AVERROR(ENOMEM); >>> + av_frame_make_writable(s->background); >> >> You are never writing to the background frame, so there is no point in >> making it writable; what you actually want to achieve here is making >> frame writable again and to achieve this you should make frame, not >> background writable (and of course you should check said call). >> > > This is invalid, input pad receives always writable frames as there is flag. > But in case this branch here is executed, the av_frame_clone() makes frame non-writable, so it needs to be made writable again. >> (Actually, you never >> >>> + } >>> + >>> + if (ret = ff_filter_execute(avctx, s->do_slice, frame, NULL, >>> + FFMIN(frame->height, s->nb_threads))) >>> + return ret; >>> + >>> + for (int n = 0; n < s->nb_threads; n++) >>> + sum += s->sums[n]; >>> + if (s->max_sum * s->threshold < sum) { >>> + av_frame_free(&s->background); >>> + s->background = av_frame_clone(frame); >>> + if (!s->background) >>> + return AVERROR(ENOMEM); >>> + av_frame_make_writable(s->background); >> >> Given that you never write to background, there is no need to make it >> writable. This time, there is also no need to make frame writable (the >> next filter in the chain may not need a writable frame anyway), so just >> remove this. >> And the av_frame_free+av_frame_clone can become an >> av_frame_unref+av_frame_ref (this will necessitate modifying the check >> above to not only check for to existence of s->background). >> >>> + } >>> + >>> + return ff_filter_frame(avctx->outputs[0], frame); >>> +} >> >> _______________________________________________ >> ffmpeg-devel mailing list >> ffmpeg-devel@ffmpeg.org >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel >> >> To unsubscribe, visit link above, or email >> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >> > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
On 11/3/22, Andreas Rheinhardt <andreas.rheinhardt@outlook.com> wrote: > Paul B Mahol: >> On 11/2/22, Andreas Rheinhardt <andreas.rheinhardt@outlook.com> wrote: >>> Paul B Mahol: >>>> +static int filter_frame(AVFilterLink *link, AVFrame *frame) >>>> +{ >>>> + AVFilterContext *avctx = link->dst; >>>> + BackgroundkeyContext *s = avctx->priv; >>>> + int64_t sum = 0; >>>> + int ret; >>>> + >>>> + if (!s->background) { >>>> + s->background = av_frame_clone(frame); >>>> + if (!s->background) >>>> + return AVERROR(ENOMEM); >>>> + av_frame_make_writable(s->background); >>> >>> You are never writing to the background frame, so there is no point in >>> making it writable; what you actually want to achieve here is making >>> frame writable again and to achieve this you should make frame, not >>> background writable (and of course you should check said call). >>> >> >> This is invalid, input pad receives always writable frames as there is >> flag. >> > > But in case this branch here is executed, the av_frame_clone() makes > frame non-writable, so it needs to be made writable again. Than will use copy frame. > >>> (Actually, you never >>> >>>> + } >>>> + >>>> + if (ret = ff_filter_execute(avctx, s->do_slice, frame, NULL, >>>> + FFMIN(frame->height, s->nb_threads))) >>>> + return ret; >>>> + >>>> + for (int n = 0; n < s->nb_threads; n++) >>>> + sum += s->sums[n]; >>>> + if (s->max_sum * s->threshold < sum) { >>>> + av_frame_free(&s->background); >>>> + s->background = av_frame_clone(frame); >>>> + if (!s->background) >>>> + return AVERROR(ENOMEM); >>>> + av_frame_make_writable(s->background); >>> >>> Given that you never write to background, there is no need to make it >>> writable. This time, there is also no need to make frame writable (the >>> next filter in the chain may not need a writable frame anyway), so just >>> remove this. >>> And the av_frame_free+av_frame_clone can become an >>> av_frame_unref+av_frame_ref (this will necessitate modifying the check >>> above to not only check for to existence of s->background). >>> >>>> + } >>>> + >>>> + return ff_filter_frame(avctx->outputs[0], frame); >>>> +} >>> >>> _______________________________________________ >>> ffmpeg-devel mailing list >>> ffmpeg-devel@ffmpeg.org >>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel >>> >>> To unsubscribe, visit link above, or email >>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >>> >> _______________________________________________ >> ffmpeg-devel mailing list >> ffmpeg-devel@ffmpeg.org >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel >> >> To unsubscribe, visit link above, or email >> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >
On 11/3/22, Paul B Mahol <onemda@gmail.com> wrote: > On 11/3/22, Andreas Rheinhardt <andreas.rheinhardt@outlook.com> wrote: >> Paul B Mahol: >>> On 11/2/22, Andreas Rheinhardt <andreas.rheinhardt@outlook.com> wrote: >>>> Paul B Mahol: >>>>> +static int filter_frame(AVFilterLink *link, AVFrame *frame) >>>>> +{ >>>>> + AVFilterContext *avctx = link->dst; >>>>> + BackgroundkeyContext *s = avctx->priv; >>>>> + int64_t sum = 0; >>>>> + int ret; >>>>> + >>>>> + if (!s->background) { >>>>> + s->background = av_frame_clone(frame); >>>>> + if (!s->background) >>>>> + return AVERROR(ENOMEM); >>>>> + av_frame_make_writable(s->background); >>>> >>>> You are never writing to the background frame, so there is no point in >>>> making it writable; what you actually want to achieve here is making >>>> frame writable again and to achieve this you should make frame, not >>>> background writable (and of course you should check said call). >>>> >>> >>> This is invalid, input pad receives always writable frames as there is >>> flag. >>> >> >> But in case this branch here is executed, the av_frame_clone() makes >> frame non-writable, so it needs to be made writable again. > > Than will use copy frame. > >> >>>> (Actually, you never >>>> >>>>> + } >>>>> + >>>>> + if (ret = ff_filter_execute(avctx, s->do_slice, frame, NULL, >>>>> + FFMIN(frame->height, s->nb_threads))) >>>>> + return ret; >>>>> + >>>>> + for (int n = 0; n < s->nb_threads; n++) >>>>> + sum += s->sums[n]; >>>>> + if (s->max_sum * s->threshold < sum) { >>>>> + av_frame_free(&s->background); >>>>> + s->background = av_frame_clone(frame); >>>>> + if (!s->background) >>>>> + return AVERROR(ENOMEM); >>>>> + av_frame_make_writable(s->background); >>>> >>>> Given that you never write to background, there is no need to make it >>>> writable. This time, there is also no need to make frame writable (the >>>> next filter in the chain may not need a writable frame anyway), so just >>>> remove this. >>>> And the av_frame_free+av_frame_clone can become an >>>> av_frame_unref+av_frame_ref (this will necessitate modifying the check >>>> above to not only check for to existence of s->background). >>>> >>>>> + } >>>>> + >>>>> + return ff_filter_frame(avctx->outputs[0], frame); >>>>> +} >>>> >>>> _______________________________________________ >>>> ffmpeg-devel mailing list >>>> ffmpeg-devel@ffmpeg.org >>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel >>>> >>>> To unsubscribe, visit link above, or email >>>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >>>> >>> _______________________________________________ >>> ffmpeg-devel mailing list >>> ffmpeg-devel@ffmpeg.org >>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel >>> >>> To unsubscribe, visit link above, or email >>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >> >> _______________________________________________ >> ffmpeg-devel mailing list >> ffmpeg-devel@ffmpeg.org >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel >> >> To unsubscribe, visit link above, or email >> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >> > Will apply witjh av_copy_frame() solution.
On 11/2/22, Paul B Mahol <onemda@gmail.com> wrote: > On 10/30/22, Paul B Mahol <onemda@gmail.com> wrote: >> Patch attached. >> > > Improved patch attached. > Another improvement.
From 72aebb5d79d60d267def506092e13b0d35c4df3d Mon Sep 17 00:00:00 2001 From: Paul B Mahol <onemda@gmail.com> Date: Fri, 28 Oct 2022 22:02:29 +0200 Subject: [PATCH] avfilter: add backgroundkey video filter Signed-off-by: Paul B Mahol <onemda@gmail.com> --- libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_backgroundkey.c | 218 +++++++++++++++++++++++++++++++++ 3 files changed, 220 insertions(+) create mode 100644 libavfilter/vf_backgroundkey.c diff --git a/libavfilter/Makefile b/libavfilter/Makefile index ff2a06c262..ace0e60ba1 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -191,6 +191,7 @@ OBJS-$(CONFIG_AVGBLUR_FILTER) += vf_avgblur.o OBJS-$(CONFIG_AVGBLUR_OPENCL_FILTER) += vf_avgblur_opencl.o opencl.o \ opencl/avgblur.o boxblur.o OBJS-$(CONFIG_AVGBLUR_VULKAN_FILTER) += vf_avgblur_vulkan.o vulkan.o vulkan_filter.o +OBJS-$(CONFIG_BACKGROUNDKEY_FILTER) += vf_backgroundkey.o OBJS-$(CONFIG_BBOX_FILTER) += bbox.o vf_bbox.o OBJS-$(CONFIG_BENCH_FILTER) += f_bench.o OBJS-$(CONFIG_BILATERAL_FILTER) += vf_bilateral.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 119de40b25..e0598e9986 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -177,6 +177,7 @@ extern const AVFilter ff_vf_atadenoise; extern const AVFilter ff_vf_avgblur; extern const AVFilter ff_vf_avgblur_opencl; extern const AVFilter ff_vf_avgblur_vulkan; +extern const AVFilter ff_vf_backgroundkey; extern const AVFilter ff_vf_bbox; extern const AVFilter ff_vf_bench; extern const AVFilter ff_vf_bilateral; diff --git a/libavfilter/vf_backgroundkey.c b/libavfilter/vf_backgroundkey.c new file mode 100644 index 0000000000..7502ee79b2 --- /dev/null +++ b/libavfilter/vf_backgroundkey.c @@ -0,0 +1,218 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/opt.h" +#include "libavutil/imgutils.h" +#include "libavutil/intreadwrite.h" +#include "avfilter.h" +#include "formats.h" +#include "internal.h" +#include "video.h" + +typedef struct BackgroundkeyContext { + const AVClass *class; + + float similarity; + float blend; + int max; + + int hsub_log2; + int vsub_log2; + + AVFrame *background; + + int (*do_slice)(AVFilterContext *avctx, void *arg, + int jobnr, int nb_jobs); +} BackgroundkeyContext; + +static int do_backgroundkey_slice(AVFilterContext *avctx, void *arg, int jobnr, int nb_jobs) +{ + BackgroundkeyContext *s = avctx->priv; + AVFrame *frame = arg; + const int slice_start = (frame->height * jobnr) / nb_jobs; + const int slice_end = (frame->height * (jobnr + 1)) / nb_jobs; + const int min_diff = s->max * s->similarity; + const int hsub = s->hsub_log2; + const int vsub = s->vsub_log2; + const float blend = s->blend; + + for (int y = slice_start; y < slice_end; y++) { + const uint8_t *srcy = frame->data[0] + frame->linesize[0] * y; + const uint8_t *srcu = frame->data[1] + frame->linesize[1] * (y >> vsub); + const uint8_t *srcv = frame->data[2] + frame->linesize[2] * (y >> vsub); + const uint8_t *bsrcy = s->background->data[0] + s->background->linesize[0] * y; + const uint8_t *bsrcu = s->background->data[1] + s->background->linesize[1] * (y >> vsub); + const uint8_t *bsrcv = s->background->data[2] + s->background->linesize[2] * (y >> vsub); + uint8_t *dst = frame->data[3] + frame->linesize[3] * y; + for (int x = 0; x < frame->width; x++) { + const int xx = x >> hsub; + const int diff = FFABS(srcy[x] - bsrcy[x]) + + FFABS(srcu[xx] - bsrcu[xx]) + + FFABS(srcv[xx] - bsrcv[xx]); + int A; + + if (diff > min_diff) { + A = 255; + } else if (blend > 0.f) { + A = FFMIN(255, (min_diff - diff) * blend * 255); + } else { + A = 0; + } + + dst[x] = A; + } + } + + return 0; +} + +static int do_backgroundkey16_slice(AVFilterContext *avctx, void *arg, int jobnr, int nb_jobs) +{ + BackgroundkeyContext *s = avctx->priv; + AVFrame *frame = arg; + const int slice_start = (frame->height * jobnr) / nb_jobs; + const int slice_end = (frame->height * (jobnr + 1)) / nb_jobs; + const int min_diff = s->max * s->similarity; + const int hsub = s->hsub_log2; + const int vsub = s->vsub_log2; + const float blend = s->blend; + const int max = s->max; + + for (int y = slice_start; y < slice_end; y++) { + const uint16_t *srcy = (const uint16_t *)(frame->data[0] + frame->linesize[0] * y); + const uint16_t *srcu = (const uint16_t *)(frame->data[1] + frame->linesize[1] * (y >> vsub)); + const uint16_t *srcv = (const uint16_t *)(frame->data[2] + frame->linesize[2] * (y >> vsub)); + const uint16_t *bsrcy = (const uint16_t *)(s->background->data[0] + s->background->linesize[0] * y); + const uint16_t *bsrcu = (const uint16_t *)(s->background->data[1] + s->background->linesize[1] * (y >> vsub)); + const uint16_t *bsrcv = (const uint16_t *)(s->background->data[2] + s->background->linesize[2] * (y >> vsub)); + uint16_t *dst = (uint16_t *)(frame->data[3] + frame->linesize[3] * y); + for (int x = 0; x < frame->width; x++) { + const int xx = x >> hsub; + const int diff = FFABS(srcy[x] - bsrcy[x]) + + FFABS(srcu[xx] - bsrcu[xx]) + + FFABS(srcv[xx] - bsrcv[xx]); + int A; + + if (diff > min_diff) { + A = max; + } else if (blend > 0.f) { + A = FFMIN(max, (min_diff - diff) * blend * max); + } else { + A = 0; + } + + dst[x] = A; + } + } + + return 0; +} + +static int filter_frame(AVFilterLink *link, AVFrame *frame) +{ + AVFilterContext *avctx = link->dst; + BackgroundkeyContext *s = avctx->priv; + int ret; + + if (!s->background) { + s->background = av_frame_clone(frame); + if (!s->background) + return AVERROR(ENOMEM); + av_frame_make_writable(s->background); + } + + if (ret = ff_filter_execute(avctx, s->do_slice, frame, NULL, + FFMIN(frame->height, ff_filter_get_nb_threads(avctx)))) + return ret; + + return ff_filter_frame(avctx->outputs[0], frame); +} + +static av_cold int config_output(AVFilterLink *outlink) +{ + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(outlink->format); + AVFilterContext *avctx = outlink->src; + BackgroundkeyContext *s = avctx->priv; + int depth; + + depth = desc->comp[0].depth; + s->do_slice = depth <= 8 ? do_backgroundkey_slice : do_backgroundkey16_slice; + s->max = (1 << depth) - 1; + s->hsub_log2 = desc->log2_chroma_w; + s->vsub_log2 = desc->log2_chroma_h; + + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + BackgroundkeyContext *s = ctx->priv; + + av_freep(&s->background); +} + +static const AVFilterPad backgroundkey_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .flags = AVFILTERPAD_FLAG_NEEDS_WRITABLE, + .filter_frame = filter_frame, + }, +}; + +static const AVFilterPad backgroundkey_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +#define OFFSET(x) offsetof(BackgroundkeyContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_RUNTIME_PARAM + +static const AVOption backgroundkey_options[] = { + { "similarity", "set the similarity", OFFSET(similarity), AV_OPT_TYPE_FLOAT, { .dbl = 0.02 }, 0.0, 1.0, FLAGS }, + { "blend", "set the blend value", OFFSET(blend), AV_OPT_TYPE_FLOAT, { .dbl = 0.0 }, 0.0, 1.0, FLAGS }, + { NULL } +}; + +static const enum AVPixelFormat backgroundkey_fmts[] = { + AV_PIX_FMT_YUVA420P, + AV_PIX_FMT_YUVA422P, + AV_PIX_FMT_YUVA444P, + AV_PIX_FMT_YUVA420P9, AV_PIX_FMT_YUVA422P9, AV_PIX_FMT_YUVA444P9, + AV_PIX_FMT_YUVA420P10, AV_PIX_FMT_YUVA422P10, AV_PIX_FMT_YUVA444P10, + AV_PIX_FMT_YUVA422P12, AV_PIX_FMT_YUVA444P12, + AV_PIX_FMT_YUVA420P16, AV_PIX_FMT_YUVA422P16, AV_PIX_FMT_YUVA444P16, + AV_PIX_FMT_NONE +}; + +AVFILTER_DEFINE_CLASS(backgroundkey); + +const AVFilter ff_vf_backgroundkey = { + .name = "backgroundkey", + .description = NULL_IF_CONFIG_SMALL("Turns a background into transparency."), + .priv_size = sizeof(BackgroundkeyContext), + .priv_class = &backgroundkey_class, + .uninit = uninit, + FILTER_INPUTS(backgroundkey_inputs), + FILTER_OUTPUTS(backgroundkey_outputs), + FILTER_PIXFMTS_ARRAY(backgroundkey_fmts), + .flags = AVFILTER_FLAG_SUPPORT_TIMELINE_GENERIC | AVFILTER_FLAG_SLICE_THREADS, +}; -- 2.37.2