From patchwork Fri Apr 29 14:05:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thilo Borgmann X-Patchwork-Id: 35508 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:a885:b0:7f:4be2:bd17 with SMTP id ca5csp314942pzb; Fri, 29 Apr 2022 07:05:46 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwhPeuxIFC/j7YeRorNjpqf3ZcGH/QSnKh2BKaR9iO54na7geuYgbaxCVUnkGdabjbOKEej X-Received: by 2002:a17:907:1b26:b0:6ef:eaca:d2d8 with SMTP id mp38-20020a1709071b2600b006efeacad2d8mr36541713ejc.604.1651241145734; Fri, 29 Apr 2022 07:05:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1651241145; cv=none; d=google.com; s=arc-20160816; b=pTAOnm7TIGnBbifmRzEUMDTVsqs31vQqaSoHFhL3YTLYWcIJIdgOnPzcg6TXTqA0YE 5hzP8lcUSFmc+DiR+EeikGQmQA2Oybv08GDAFk24upyymLhtJeUAPkdobVg15LnlAB2L ZqkqZnhh7WFO4EBk12H/SMP8W/IA6kDCMlrYQJUQhpWb++8/EjhKPiVRWfjdEmYhUKZC wKOIJ+PuxWHj1ZhH12LaKa8ZQAsnUBUqHom72W4V6CBFWd2r37bIRFHUy6LTfyx7Tz5e whRZO3WS5Cnjf4UQJF7OHLrRiXgqffuKEKsQNSmTqmuyV/GDDBK+ydmaNbdBLvGkAFIS DE7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject:from:to :content-language:mime-version:date:message-id:dkim-signature :delivered-to; bh=2jbKVjzQeclCjHNqgG7G9j7T7cE0M7D0dKLJy+M0Jfk=; b=Pm8mFLmrHxAu6m80yiOL0jWAe6PFSAgJJaALoo02kaKtrkPj3rADlCR5SYLMIgrfoF Ou73okmv3IQLWX/FMzTWsq6wS5YC1nR9vLXgWDT6dvoCisjj4G0jaXohhsiLAqGxrVpH E1W9rM/zeXVnt/myFwt1kK7AiI+iSdJ9+hDHzo6ssKJvKAna6OfmhR4gbFgIW77ia8EM aULfSprCNtSl0jSrdOw7VjlpxDrTRfR3aEPILwtu24/ZDDd2pBdgr1UcTPQiDdnTikth L1hgzS5VIm5AKd7R7knCWWZrZGgV7neq30MBqwftZ+Re++/B8igDEG89dyH2eSXQvgyS kuOA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@mail.de header.s=mailde202009 header.b=OZZOzq9i; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=mail.de Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t3-20020a50ab43000000b00425d808e448si6188993edc.304.2022.04.29.07.05.44; Fri, 29 Apr 2022 07:05:45 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@mail.de header.s=mailde202009 header.b=OZZOzq9i; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=mail.de Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 06CBB68B2BC; Fri, 29 Apr 2022 17:05:40 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from shout02.mail.de (shout02.mail.de [62.201.172.25]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 56B62689CFC for ; Fri, 29 Apr 2022 17:05:33 +0300 (EEST) Received: from postfix01.mail.de (postfix01.bt.mail.de [10.0.121.125]) by shout02.mail.de (Postfix) with ESMTP id BCFE6A0C82 for ; Fri, 29 Apr 2022 16:05:32 +0200 (CEST) Received: from smtp02.mail.de (smtp02.bt.mail.de [10.0.121.212]) by postfix01.mail.de (Postfix) with ESMTP id 36028801B2 for ; Fri, 29 Apr 2022 16:05:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mail.de; s=mailde202009; t=1651241132; bh=0MiMepJEWcIImhAyrNq+lVTkVSjsWJ2AS0wpwrTgYD0=; h=Message-ID:Date:To:From:Subject:From:To:CC:Subject:Reply-To; b=OZZOzq9iKu29RZEP0E4CN2TIgE/1kzVTezj1lnZUYxazmYAV/6XjpzRvRVmZrrFNz BYoY05QDmRGcA0tKagetpTGBYzN0uqL36DC0IN4+Dx2qljKKE+sK7EIOnQpaJQZ3ew W3IToQituBfw8LOlWJgKnxEEa4NjD7vzj1V41oVV3yFx/fLtH4w9h0SOHZKTfoAncS f8qxHHcfiiL0KVBZ+5uUv6h0WyTn6wdZ5dmJTOT0Dn/ecLgOaEB20oi8XrOIQjeqvu RutTIW548ZX1OlZpQoiie2QVvOgtpoydc+IrH0FjDXZF25Vs5/3ZSFtXxWUvZ6faCk LtTIWsH4/YHog== Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by smtp02.mail.de (Postfix) with ESMTPSA id C1CD1A05D7 for ; Fri, 29 Apr 2022 16:05:30 +0200 (CEST) Message-ID: Date: Fri, 29 Apr 2022 16:05:30 +0200 MIME-Version: 1.0 Content-Language: en-US To: FFmpeg development discussions and patches From: Thilo Borgmann X-purgate: clean X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate-type: clean X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate: clean X-purgate-size: 18888 X-purgate-ID: 154282::1651241131-0000737C-5060E584/0/0 Subject: [FFmpeg-devel] [PATCH] avfilter: Add blockdetect filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: HRC+1ZRJMiGg Hi, $subject based on http://www.eurasip.org/Proceedings/Eusipco/Eusipco2005/defevent/papers/cr1042.pdf -Thilo From e69095b17619454e671e401ec5c01f130c308a76 Mon Sep 17 00:00:00 2001 From: Thilo Borgmann Date: Fri, 29 Apr 2022 15:58:11 +0200 Subject: [PATCH] avfilter: Add blockdetect filter --- doc/filters.texi | 29 ++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_blockdetect.c | 294 +++++++++++++++++++++++++++++++++++ 4 files changed, 325 insertions(+) create mode 100644 libavfilter/vf_blockdetect.c diff --git a/doc/filters.texi b/doc/filters.texi index 52c40833eb..e550eb8824 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -7997,6 +7997,35 @@ tblend=all_mode=grainextract @subsection Commands This filter supports same @ref{commands} as options. +@anchor{blockdetect} +@section blockdetect + +Determines blockiness of frames without altering the input frames. + +Based on Remco Muijs and Ihor Kirenko: "A no-reference blocking artifact measure for adaptive video processing." 2005 13th European signal processing conference. + +The filter accepts the following options: + +@table @option +@item period_min +@item period_max +Set minimum and maximum values for determining pixel grids (periods). +Default values are [3,24]. + +@item planes +Set planes to filter. Default is first only. +@end table + +@subsection Examples + +@itemize +@item +Determine blockiness for the first plane and search for periods within [8,32]: +@example +blockdetect=period_min=8:period_max=32:planes=1 +@end example +@end itemize + @anchor{blurdetect} @section blurdetect diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 1db097b464..35a430878f 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -195,6 +195,7 @@ OBJS-$(CONFIG_BLACKDETECT_FILTER) += vf_blackdetect.o OBJS-$(CONFIG_BLACKFRAME_FILTER) += vf_blackframe.o OBJS-$(CONFIG_BLEND_FILTER) += vf_blend.o framesync.o OBJS-$(CONFIG_BLEND_VULKAN_FILTER) += vf_blend_vulkan.o framesync.o vulkan.o vulkan_filter.o +OBJS-$(CONFIG_BLOCKDETECT_FILTER) += vf_blockdetect.o OBJS-$(CONFIG_BLURDETECT_FILTER) += vf_blurdetect.o edge_common.o OBJS-$(CONFIG_BM3D_FILTER) += vf_bm3d.o framesync.o OBJS-$(CONFIG_BOXBLUR_FILTER) += vf_boxblur.o boxblur.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 2ad523fd0f..30936392ce 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -183,6 +183,7 @@ extern const AVFilter ff_vf_blackdetect; extern const AVFilter ff_vf_blackframe; extern const AVFilter ff_vf_blend; extern const AVFilter ff_vf_blend_vulkan; +extern const AVFilter ff_vf_blockdetect; extern const AVFilter ff_vf_blurdetect; extern const AVFilter ff_vf_bm3d; extern const AVFilter ff_vf_boxblur; diff --git a/libavfilter/vf_blockdetect.c b/libavfilter/vf_blockdetect.c new file mode 100644 index 0000000000..f15dd19646 --- /dev/null +++ b/libavfilter/vf_blockdetect.c @@ -0,0 +1,294 @@ +/* + * Copyright (c) 2021 Thilo Borgmann + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * No-reference blockdetect filter + * + * Implementing: + * Remco Muijs and Ihor Kirenko: "A no-reference blocking artifact measure for adaptive video processing." 2005 13th European signal processing conference. IEEE, 2005. + * http://www.eurasip.org/Proceedings/Eusipco/Eusipco2005/defevent/papers/cr1042.pdf + * + * @author Thilo Borgmann + */ + +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "internal.h" + +typedef struct BLKContext { + const AVClass *class; + + int hsub, vsub; + int nb_planes; + + int period_min; // minimum period to search for + int period_max; // maximum period to search for + int planes; // number of planes to filter + + double block_total; + uint64_t nb_frames; + + float *gradients; +} BLKContext; + +#define OFFSET(x) offsetof(BLKContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM +static const AVOption blockdetect_options[] = { + { "period_min", "Minimum period to search for", OFFSET(period_min), AV_OPT_TYPE_INT, {.i64=3}, 2, 32, FLAGS}, + { "period_max", "Maximum period to search for", OFFSET(period_max), AV_OPT_TYPE_INT, {.i64=24}, 2, 64, FLAGS}, + { "planes", "set planes to filter", OFFSET(planes), AV_OPT_TYPE_INT, {.i64=1}, 0, 15, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(blockdetect); + +static av_cold int blockdetect_init(AVFilterContext *ctx) +{ + return 0; +} + +static int blockdetect_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + BLKContext *s = ctx->priv; + const int bufsize = inlink->w * inlink->h; + const AVPixFmtDescriptor *pix_desc; + + pix_desc = av_pix_fmt_desc_get(inlink->format); + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + s->nb_planes = av_pix_fmt_count_planes(inlink->format); + + s->gradients = av_calloc(bufsize, sizeof(*s->gradients)); + + if (!s->gradients) + return AVERROR(ENOMEM); + + return 0; +} + +static float calculate_blockiness(BLKContext *s, int w, int h, + float *grad, int grad_linesize, + uint8_t* src, int src_linesize) +{ + float block = 0.0f; + float nonblock = 0.0f; + int block_count = 0; + int nonblock_count = 0; + float ret = 0; + + // Calculate BS in horizontal and vertical directions according to (1)(2)(3). + // Also try to find integer pixel periods (grids) even for scaled images. + // In case of fractional periods, FFMAX of current and neighbor pixels + // can help improve the correlation with MQS. + // Skip linear correction term (4)(5), as it appears only valid for their own test samples. + + // horizontal blockiness (fixed width) + for (int j = 1; j < h; j++) { + for (int i = 3; i < w - 4; i++) { + float temp = 0.0f; + grad[j * grad_linesize + i] = + abs(src[j * src_linesize + i + 0] - src[j * src_linesize + i + 1]); + temp += abs(src[j * src_linesize + i + 1] - src[j * src_linesize + i + 2]); + temp += abs(src[j * src_linesize + i + 2] - src[j * src_linesize + i + 3]); + temp += abs(src[j * src_linesize + i + 3] - src[j * src_linesize + i + 4]); + temp += abs(src[j * src_linesize + i - 0] - src[j * src_linesize + i - 1]); + temp += abs(src[j * src_linesize + i - 1] - src[j * src_linesize + i - 2]); + temp += abs(src[j * src_linesize + i - 2] - src[j * src_linesize + i - 3]); + temp = FFMAX(1, temp); + grad[j * grad_linesize + i] /= temp; + + // use first row to store acculated results + grad[i] += grad[j * grad_linesize + i]; + } + } + + // find horizontal period + for (int period = s->period_min; period < s->period_max + 1; period++) { + float temp; + block = 0; + nonblock = 0; + block_count = 0; + nonblock_count = 0; + for (int i = 3; i < w - 4; i++) { + if ((i % period) == (period - 1)) { + block += FFMAX(FFMAX(grad[i + 0], grad[i + 1]), grad[i - 1]); + block_count++; + } else { + nonblock += grad[i]; + nonblock_count++; + } + } + temp = (block / block_count) / (nonblock / nonblock_count); + ret = FFMAX(ret, temp); + } + + // vertical blockiness (fixed height) + block_count = 0; + for (int j = 3; j < h - 4; j++) { + for (int i = 1; i < w; i++) { + float temp = 0.0f; + grad[j * grad_linesize + i] = + abs(src[(j + 0) * src_linesize + i] - src[(j + 1) * src_linesize + i]); + temp += abs(src[(j + 1) * src_linesize + i] - src[(j + 2) * src_linesize + i]); + temp += abs(src[(j + 2) * src_linesize + i] - src[(j + 3) * src_linesize + i]); + temp += abs(src[(j + 3) * src_linesize + i] - src[(j + 4) * src_linesize + i]); + temp += abs(src[(j - 0) * src_linesize + i] - src[(j - 1) * src_linesize + i]); + temp += abs(src[(j - 1) * src_linesize + i] - src[(j - 2) * src_linesize + i]); + temp += abs(src[(j - 2) * src_linesize + i] - src[(j - 3) * src_linesize + i]); + temp = FFMAX(1, temp); + grad[j * grad_linesize + i] /= temp; + + // use first column to store accumulated results + grad[j * grad_linesize] += grad[j * grad_linesize + i]; + } + } + + // find vertical period + for (int period = s->period_min; period < s->period_max + 1; period++) { + float temp; + block = 0; + nonblock = 0; + block_count = 0; + nonblock_count = 0; + for (int j = 3; j < h - 4; j++) { + if ((j % period) == (period - 1)) { + block += FFMAX(FFMAX(grad[(j + 0) * grad_linesize], + grad[(j + 1) * grad_linesize]), + grad[(j - 1) * grad_linesize]); + block_count++; + } else { + nonblock += grad[j * grad_linesize]; + nonblock_count++; + } + } + temp = (block / block_count) / (nonblock / nonblock_count); + ret = FFMAX(ret, temp); + } + + // return highest value of horz||vert + return ret; +} + +static void set_meta(AVDictionary **metadata, const char *key, float d) +{ + char value[128]; + snprintf(value, sizeof(value), "%f", d); + av_dict_set(metadata, key, value, 0); +} + +static int blockdetect_filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx = inlink->dst; + BLKContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + + const int inw = inlink->w; + const int inh = inlink->h; + + float *gradients = s->gradients; + + float block = 0.0f; + int nplanes = 0; + AVDictionary **metadata; + metadata = &in->metadata; + + for (int plane = 0; plane < s->nb_planes; plane++) { + int hsub = plane == 1 || plane == 2 ? s->hsub : 0; + int vsub = plane == 1 || plane == 2 ? s->vsub : 0; + int w = AV_CEIL_RSHIFT(inw, hsub); + int h = AV_CEIL_RSHIFT(inh, vsub); + + if (!((1 << plane) & s->planes)) + continue; + + nplanes++; + + block += calculate_blockiness(s, w, h, gradients, w, in->data[plane], in->linesize[plane]); + } + + if (nplanes) + block /= nplanes; + + s->block_total += block; + + // write stats + av_log(ctx, AV_LOG_VERBOSE, "block: %.7f\n", block); + + set_meta(metadata, "lavfi.block", block); + + s->nb_frames = inlink->frame_count_in; + + return ff_filter_frame(outlink, in); +} + +static av_cold void blockdetect_uninit(AVFilterContext *ctx) +{ + BLKContext *s = ctx->priv; + + if (s->nb_frames > 0) { + av_log(ctx, AV_LOG_INFO, "block mean: %.7f\n", + s->block_total / s->nb_frames); + } + + av_freep(&s->gradients); +} + +static const enum AVPixelFormat pix_fmts[] = { + AV_PIX_FMT_GRAY8, + AV_PIX_FMT_GBRP, AV_PIX_FMT_GBRAP, + AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV420P, + AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUV440P, + AV_PIX_FMT_YUV411P, AV_PIX_FMT_YUV410P, + AV_PIX_FMT_YUVJ440P, AV_PIX_FMT_YUVJ411P, AV_PIX_FMT_YUVJ420P, + AV_PIX_FMT_YUVJ422P, AV_PIX_FMT_YUVJ444P, + AV_PIX_FMT_YUVA444P, AV_PIX_FMT_YUVA422P, AV_PIX_FMT_YUVA420P, + AV_PIX_FMT_NONE +}; + +static const AVFilterPad blockdetect_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = blockdetect_config_input, + .filter_frame = blockdetect_filter_frame, + }, +}; + +static const AVFilterPad blockdetect_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + }, +}; + +const AVFilter ff_vf_blockdetect = { + .name = "blockdetect", + .description = NULL_IF_CONFIG_SMALL("blockdetect filter."), + .priv_size = sizeof(BLKContext), + .init = blockdetect_init, + .uninit = blockdetect_uninit, + FILTER_PIXFMTS_ARRAY(pix_fmts), + FILTER_INPUTS(blockdetect_inputs), + FILTER_OUTPUTS(blockdetect_outputs), + .priv_class = &blockdetect_class, + .flags = AVFILTER_FLAG_METADATA_ONLY, +};