From patchwork Sun Oct 30 14:34:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul B Mahol X-Patchwork-Id: 39049 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8e01:b0:a3:a294:714c with SMTP id y1csp410626pzj; Sun, 30 Oct 2022 07:35:11 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6XOqhUpnOTX8ZYx0FOJydNEJi15+SzYntdPjvX1+lXDul9vAWik2yYx1ViK7YHgDtPKcKI X-Received: by 2002:aa7:c792:0:b0:453:98b7:213c with SMTP id n18-20020aa7c792000000b0045398b7213cmr8935307eds.159.1667140510735; Sun, 30 Oct 2022 07:35:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667140510; cv=none; d=google.com; s=arc-20160816; b=jbIVfNFUWLWwa3SB89UkCvKCmQC5S6Phb5Sffh57lJyQPxR0hAL4EWE/DkZ/kL/UYu nSQgxFuy0iUt891yebKWHgm4LrVrOq9uT8z057+ArAHjZlF+DjqhbIjbg3mK9rxFhuwZ iKGTkS1J7Hzll86e+DrSiP/RIvx5ttqG/Wi6+9lQ8q/qVqDhkEy7zjEcDt2+aldSi83l tj0D79Rrgkrzz2mQ8adis4oAwimu38iYHV/T1qMPYaBpeXXsz1UbldB65AHp+Z0BQito T/59/ilTGkM2t55TAcqc/IeIPOZKOyFPibfS5dUakj/5smI8ZDpb6+gs0CiBzUP9R2sZ VhzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject:to :message-id:date:from:mime-version:dkim-signature:delivered-to; bh=t2d9S0/FF2qc3kGupk5NFLhX2nVySV48U4pS6t+7Jwo=; b=uzoTVmXrGWZZ4vbU4roV3+G/jQvwPKGewIa+buC4uh4nE+3Snv0zjL60NbjrDpCkM7 CmmXCtJ79VNMDnmDskX9lnJFElszYFpnSROW9ryFtiCfBSBbUS8KJ05yI1xvAYj8UuZ1 JQCrWTns4JYFn96kw7ukShlinfXmnYOGaWssbdtcMX8I9J5mRdmKbpfzK8QY2fDlmejY AyaAJ0QXk3pCSN7oyHNNYcW5ZEnkPQhkZrbw6JZn2nv6djNJLNVj63AvALDOPa+kNLYH r779wiJeByyNfd9j2MktYm0T9ERTPWiazVBIma4KkAxHTfmylo7uqy6V3zg/8fCpwxEo 85JA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b="Qivr/dMT"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 17-20020a170906059100b0079ca59305a6si4440933ejn.354.2022.10.30.07.35.09; Sun, 30 Oct 2022 07:35:10 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b="Qivr/dMT"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 711F068BD1D; Sun, 30 Oct 2022 16:35:05 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ua1-f41.google.com (mail-ua1-f41.google.com [209.85.222.41]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BF92568B8C8 for ; Sun, 30 Oct 2022 16:34:58 +0200 (EET) Received: by mail-ua1-f41.google.com with SMTP id ca15so4125038uab.0 for ; Sun, 30 Oct 2022 07:34:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=dItnY/obuMogiy3VzsN5ulytvdLEtQJ4cyiBqoxK+r0=; b=Qivr/dMTo5j2AxD4RxtG1Oco7xjEaq7hi7Do7s9yW7L3/aKisQopZc5zeR6YVsKGcp DRMf0GOeCY0hqfDM8EMVZqASq2QvgZnhHBUjIw8EQ/3hjTfcrHjHU6Phj4ko0pvvutKY gQ4PUNGAw27ARqymrE/qzkyhEPDUcp/hQAvzEn8ajm0YWofgUUA7fxZPglFnliF1SA0f JbjuOrG/gAgcuuEN1fY11hz0aQU4gwYPR6q42NkXluwOfYxowIb8x4L8O6v7Ch5BNHx7 r1L/0UG5vu/ec6aCiDRexFHEWlTZlQKrPx/mIxDIhh7thMCyfR2x7wxjXgh/qJeCTNhd 9f4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=dItnY/obuMogiy3VzsN5ulytvdLEtQJ4cyiBqoxK+r0=; b=Xa3IgXj6GzCRhPgZ7V+DQoKDmphCttGYBOg1WcKActIpdhkFPeuRRP3QnRSRkZM18F pw8juALNuNCUvopYM+BclaaxcZ9vw2EvVVXWOQkjNtvd/3L1eDAqxauRn4RjAjVzJ+3K Ejty5GozX798yo7oLaAYHkU6tG1lkPsr9CnHoWyPhP+UX9VbZehO1Ge7TdL+VQTqQJM9 Ous02ynUcev3PfGdRaYRJPPO9oOpqAYG4PYqSMCKlRxhRdDsdlhGHrPYzT3W5P3oSfCd yf4y497OY7UoXZ5irxABYhHtJ5gBsSylhSG2RIE4D3mpVuj5X9IjFZYh8TBVRzzG5raB XV8g== X-Gm-Message-State: ACrzQf3EsL0VONR787tpiTiiJKbj9qFstCTc7Pm3Z/CZqm12ptZ8htKU UXj/dTLX7sSfZfOPhM1qpkCIfXAq404JrzHmDlk//wAi X-Received: by 2002:ab0:29d8:0:b0:40a:c9:9ff7 with SMTP id i24-20020ab029d8000000b0040a00c99ff7mr1888286uaq.41.1667140497036; Sun, 30 Oct 2022 07:34:57 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a05:612c:612:b0:314:ac6a:1eb7 with HTTP; Sun, 30 Oct 2022 07:34:55 -0700 (PDT) From: Paul B Mahol Date: Sun, 30 Oct 2022 15:34:55 +0100 Message-ID: To: FFmpeg development discussions and patches Subject: [FFmpeg-devel] [PATCH] avfilter: add backgroundkey video filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: e/uCkyov0Q7w Patch attached. From 72aebb5d79d60d267def506092e13b0d35c4df3d Mon Sep 17 00:00:00 2001 From: Paul B Mahol Date: Fri, 28 Oct 2022 22:02:29 +0200 Subject: [PATCH] avfilter: add backgroundkey video filter Signed-off-by: Paul B Mahol --- libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_backgroundkey.c | 218 +++++++++++++++++++++++++++++++++ 3 files changed, 220 insertions(+) create mode 100644 libavfilter/vf_backgroundkey.c diff --git a/libavfilter/Makefile b/libavfilter/Makefile index ff2a06c262..ace0e60ba1 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -191,6 +191,7 @@ OBJS-$(CONFIG_AVGBLUR_FILTER) += vf_avgblur.o OBJS-$(CONFIG_AVGBLUR_OPENCL_FILTER) += vf_avgblur_opencl.o opencl.o \ opencl/avgblur.o boxblur.o OBJS-$(CONFIG_AVGBLUR_VULKAN_FILTER) += vf_avgblur_vulkan.o vulkan.o vulkan_filter.o +OBJS-$(CONFIG_BACKGROUNDKEY_FILTER) += vf_backgroundkey.o OBJS-$(CONFIG_BBOX_FILTER) += bbox.o vf_bbox.o OBJS-$(CONFIG_BENCH_FILTER) += f_bench.o OBJS-$(CONFIG_BILATERAL_FILTER) += vf_bilateral.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 119de40b25..e0598e9986 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -177,6 +177,7 @@ extern const AVFilter ff_vf_atadenoise; extern const AVFilter ff_vf_avgblur; extern const AVFilter ff_vf_avgblur_opencl; extern const AVFilter ff_vf_avgblur_vulkan; +extern const AVFilter ff_vf_backgroundkey; extern const AVFilter ff_vf_bbox; extern const AVFilter ff_vf_bench; extern const AVFilter ff_vf_bilateral; diff --git a/libavfilter/vf_backgroundkey.c b/libavfilter/vf_backgroundkey.c new file mode 100644 index 0000000000..7502ee79b2 --- /dev/null +++ b/libavfilter/vf_backgroundkey.c @@ -0,0 +1,218 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/opt.h" +#include "libavutil/imgutils.h" +#include "libavutil/intreadwrite.h" +#include "avfilter.h" +#include "formats.h" +#include "internal.h" +#include "video.h" + +typedef struct BackgroundkeyContext { + const AVClass *class; + + float similarity; + float blend; + int max; + + int hsub_log2; + int vsub_log2; + + AVFrame *background; + + int (*do_slice)(AVFilterContext *avctx, void *arg, + int jobnr, int nb_jobs); +} BackgroundkeyContext; + +static int do_backgroundkey_slice(AVFilterContext *avctx, void *arg, int jobnr, int nb_jobs) +{ + BackgroundkeyContext *s = avctx->priv; + AVFrame *frame = arg; + const int slice_start = (frame->height * jobnr) / nb_jobs; + const int slice_end = (frame->height * (jobnr + 1)) / nb_jobs; + const int min_diff = s->max * s->similarity; + const int hsub = s->hsub_log2; + const int vsub = s->vsub_log2; + const float blend = s->blend; + + for (int y = slice_start; y < slice_end; y++) { + const uint8_t *srcy = frame->data[0] + frame->linesize[0] * y; + const uint8_t *srcu = frame->data[1] + frame->linesize[1] * (y >> vsub); + const uint8_t *srcv = frame->data[2] + frame->linesize[2] * (y >> vsub); + const uint8_t *bsrcy = s->background->data[0] + s->background->linesize[0] * y; + const uint8_t *bsrcu = s->background->data[1] + s->background->linesize[1] * (y >> vsub); + const uint8_t *bsrcv = s->background->data[2] + s->background->linesize[2] * (y >> vsub); + uint8_t *dst = frame->data[3] + frame->linesize[3] * y; + for (int x = 0; x < frame->width; x++) { + const int xx = x >> hsub; + const int diff = FFABS(srcy[x] - bsrcy[x]) + + FFABS(srcu[xx] - bsrcu[xx]) + + FFABS(srcv[xx] - bsrcv[xx]); + int A; + + if (diff > min_diff) { + A = 255; + } else if (blend > 0.f) { + A = FFMIN(255, (min_diff - diff) * blend * 255); + } else { + A = 0; + } + + dst[x] = A; + } + } + + return 0; +} + +static int do_backgroundkey16_slice(AVFilterContext *avctx, void *arg, int jobnr, int nb_jobs) +{ + BackgroundkeyContext *s = avctx->priv; + AVFrame *frame = arg; + const int slice_start = (frame->height * jobnr) / nb_jobs; + const int slice_end = (frame->height * (jobnr + 1)) / nb_jobs; + const int min_diff = s->max * s->similarity; + const int hsub = s->hsub_log2; + const int vsub = s->vsub_log2; + const float blend = s->blend; + const int max = s->max; + + for (int y = slice_start; y < slice_end; y++) { + const uint16_t *srcy = (const uint16_t *)(frame->data[0] + frame->linesize[0] * y); + const uint16_t *srcu = (const uint16_t *)(frame->data[1] + frame->linesize[1] * (y >> vsub)); + const uint16_t *srcv = (const uint16_t *)(frame->data[2] + frame->linesize[2] * (y >> vsub)); + const uint16_t *bsrcy = (const uint16_t *)(s->background->data[0] + s->background->linesize[0] * y); + const uint16_t *bsrcu = (const uint16_t *)(s->background->data[1] + s->background->linesize[1] * (y >> vsub)); + const uint16_t *bsrcv = (const uint16_t *)(s->background->data[2] + s->background->linesize[2] * (y >> vsub)); + uint16_t *dst = (uint16_t *)(frame->data[3] + frame->linesize[3] * y); + for (int x = 0; x < frame->width; x++) { + const int xx = x >> hsub; + const int diff = FFABS(srcy[x] - bsrcy[x]) + + FFABS(srcu[xx] - bsrcu[xx]) + + FFABS(srcv[xx] - bsrcv[xx]); + int A; + + if (diff > min_diff) { + A = max; + } else if (blend > 0.f) { + A = FFMIN(max, (min_diff - diff) * blend * max); + } else { + A = 0; + } + + dst[x] = A; + } + } + + return 0; +} + +static int filter_frame(AVFilterLink *link, AVFrame *frame) +{ + AVFilterContext *avctx = link->dst; + BackgroundkeyContext *s = avctx->priv; + int ret; + + if (!s->background) { + s->background = av_frame_clone(frame); + if (!s->background) + return AVERROR(ENOMEM); + av_frame_make_writable(s->background); + } + + if (ret = ff_filter_execute(avctx, s->do_slice, frame, NULL, + FFMIN(frame->height, ff_filter_get_nb_threads(avctx)))) + return ret; + + return ff_filter_frame(avctx->outputs[0], frame); +} + +static av_cold int config_output(AVFilterLink *outlink) +{ + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(outlink->format); + AVFilterContext *avctx = outlink->src; + BackgroundkeyContext *s = avctx->priv; + int depth; + + depth = desc->comp[0].depth; + s->do_slice = depth <= 8 ? do_backgroundkey_slice : do_backgroundkey16_slice; + s->max = (1 << depth) - 1; + s->hsub_log2 = desc->log2_chroma_w; + s->vsub_log2 = desc->log2_chroma_h; + + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + BackgroundkeyContext *s = ctx->priv; + + av_freep(&s->background); +} + +static const AVFilterPad backgroundkey_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .flags = AVFILTERPAD_FLAG_NEEDS_WRITABLE, + .filter_frame = filter_frame, + }, +}; + +static const AVFilterPad backgroundkey_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +#define OFFSET(x) offsetof(BackgroundkeyContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_RUNTIME_PARAM + +static const AVOption backgroundkey_options[] = { + { "similarity", "set the similarity", OFFSET(similarity), AV_OPT_TYPE_FLOAT, { .dbl = 0.02 }, 0.0, 1.0, FLAGS }, + { "blend", "set the blend value", OFFSET(blend), AV_OPT_TYPE_FLOAT, { .dbl = 0.0 }, 0.0, 1.0, FLAGS }, + { NULL } +}; + +static const enum AVPixelFormat backgroundkey_fmts[] = { + AV_PIX_FMT_YUVA420P, + AV_PIX_FMT_YUVA422P, + AV_PIX_FMT_YUVA444P, + AV_PIX_FMT_YUVA420P9, AV_PIX_FMT_YUVA422P9, AV_PIX_FMT_YUVA444P9, + AV_PIX_FMT_YUVA420P10, AV_PIX_FMT_YUVA422P10, AV_PIX_FMT_YUVA444P10, + AV_PIX_FMT_YUVA422P12, AV_PIX_FMT_YUVA444P12, + AV_PIX_FMT_YUVA420P16, AV_PIX_FMT_YUVA422P16, AV_PIX_FMT_YUVA444P16, + AV_PIX_FMT_NONE +}; + +AVFILTER_DEFINE_CLASS(backgroundkey); + +const AVFilter ff_vf_backgroundkey = { + .name = "backgroundkey", + .description = NULL_IF_CONFIG_SMALL("Turns a background into transparency."), + .priv_size = sizeof(BackgroundkeyContext), + .priv_class = &backgroundkey_class, + .uninit = uninit, + FILTER_INPUTS(backgroundkey_inputs), + FILTER_OUTPUTS(backgroundkey_outputs), + FILTER_PIXFMTS_ARRAY(backgroundkey_fmts), + .flags = AVFILTER_FLAG_SUPPORT_TIMELINE_GENERIC | AVFILTER_FLAG_SLICE_THREADS, +}; -- 2.37.2