| Message ID | 00ca6f6f-fc18-8b6b-1ff0-f24db7f3714b@mail.de |
|---|---|
| State | New |
| Headers | show |
| Series | [FFmpeg-devel,1/2] lafi/vf_edgedetect: Move some common functions into seperate file | expand |
| Context | Check | Description |
|---|---|---|
| andriy/configurex86 | warning | Failed to apply patch |
| andriy/configureppc | warning | Failed to apply patch |
On Tue, 30 Nov 2021, Thilo Borgmann wrote: > Hi, > > $subject Can you name this blurdetect to be more in line with existing similar filters? (blackdetect, freezedetect) Thanks, Marton
Am 30.11.21 um 17:11 schrieb Marton Balint: > > > On Tue, 30 Nov 2021, Thilo Borgmann wrote: > >> Hi, >> >> $subject > > Can you name this blurdetect to be more in line with existing similar filters? (blackdetect, freezedetect) Totally will in the next version. Thanks, Thilo
Hi,
v3 updated to current HEAD.
Named blurdetect filter now.
Minor fixes on allocation and removed -f option.
-Thilo
From 1c81cf6f0586a3cc670e6d8354f35e5bdbe2a282 Mon Sep 17 00:00:00 2001
From: Thilo Borgmann <thilo.borgmann@mail.de>
Date: Mon, 4 Apr 2022 12:21:14 +0200
Subject: [PATCH v3 2/2] lavfi: Add blurdetect filter
---
doc/filters.texi | 49 +++++
libavfilter/Makefile | 1 +
libavfilter/allfilters.c | 1 +
libavfilter/vf_blurdetect.c | 368 ++++++++++++++++++++++++++++++++++++
4 files changed, 419 insertions(+)
create mode 100644 libavfilter/vf_blurdetect.c
diff --git a/doc/filters.texi b/doc/filters.texi
index 636c80dbff..f15fb719d5 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -7990,6 +7990,55 @@ tblend=all_mode=grainextract
@subsection Commands
This filter supports same @ref{commands} as options.
+@anchor{blurdetect}
+@section blurdetect
+
+Determines blurriness of frames without altering the input frames.
+
+Based on Marziliano, Pina, et al. "A no-reference perceptual blur metric."
+Allows for a block-based abbreviation.
+
+The filter accepts the following options:
+
+@table @option
+@item low
+@item high
+Set low and high threshold values used by the Canny thresholding
+algorithm.
+
+The high threshold selects the "strong" edge pixels, which are then
+connected through 8-connectivity with the "weak" edge pixels selected
+by the low threshold.
+
+@var{low} and @var{high} threshold values must be chosen in the range
+[0,1], and @var{low} should be lesser or equal to @var{high}.
+
+Default value for @var{low} is @code{20/255}, and default value for @var{high}
+is @code{50/255}.
+
+@item radius
+Define the radius to search around an edge pixel for local maxima.
+
+@item block_pct
+Determine blurriness only for the most significant blocks, given in percentage.
+
+@item block_width
+Determine blurriness for blocks of width @var{block_width}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_height}.
+
+@item block_height
+Determine blurriness for blocks of height @var{block_height}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_width}.
+@end table
+
+@subsection Examples
+
+@itemize
+@item
+Determine blur for 80% of most significant 32x32 blocks:
+@example
+blurdetect=block_width=32:block_height=32:block_pct=80
+@end example
+@end itemize
+
@section bm3d
Denoise frames using Block-Matching 3D algorithm.
diff --git a/libavfilter/Makefile b/libavfilter/Makefile
index 006e59b2bd..6332a6f799 100644
--- a/libavfilter/Makefile
+++ b/libavfilter/Makefile
@@ -195,6 +195,7 @@ OBJS-$(CONFIG_BLACKDETECT_FILTER) += vf_blackdetect.o
OBJS-$(CONFIG_BLACKFRAME_FILTER) += vf_blackframe.o
OBJS-$(CONFIG_BLEND_FILTER) += vf_blend.o framesync.o
OBJS-$(CONFIG_BLEND_VULKAN_FILTER) += vf_blend_vulkan.o framesync.o vulkan.o vulkan_filter.o
+OBJS-$(CONFIG_BLURDETECT_FILTER) += vf_blurdetect.o edge_common.o
OBJS-$(CONFIG_BM3D_FILTER) += vf_bm3d.o framesync.o
OBJS-$(CONFIG_BOXBLUR_FILTER) += vf_boxblur.o boxblur.o
OBJS-$(CONFIG_BOXBLUR_OPENCL_FILTER) += vf_avgblur_opencl.o opencl.o \
diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
index 9fbaaacf47..2667d153ad 100644
--- a/libavfilter/allfilters.c
+++ b/libavfilter/allfilters.c
@@ -183,6 +183,7 @@ extern const AVFilter ff_vf_blackdetect;
extern const AVFilter ff_vf_blackframe;
extern const AVFilter ff_vf_blend;
extern const AVFilter ff_vf_blend_vulkan;
+extern const AVFilter ff_vf_blurdetect;
extern const AVFilter ff_vf_bm3d;
extern const AVFilter ff_vf_boxblur;
extern const AVFilter ff_vf_boxblur_opencl;
diff --git a/libavfilter/vf_blurdetect.c b/libavfilter/vf_blurdetect.c
new file mode 100644
index 0000000000..050cec9529
--- /dev/null
+++ b/libavfilter/vf_blurdetect.c
@@ -0,0 +1,368 @@
+/*
+ * Copyright (c) 2021 Thilo Borgmann <thilo.borgmann _at_ mail.de>
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * No-reference blurdetect filter
+ *
+ * Implementing:
+ * Marziliano, Pina, et al. "A no-reference perceptual blur metric." Proceedings.
+ * International conference on image processing. Vol. 3. IEEE, 2002.
+ * https://infoscience.epfl.ch/record/111802/files/14%20A%20no-reference%20perceptual%20blur%20metric.pdf
+ *
+ * @author Thilo Borgmann <thilo.borgmann _at_ mail.de>
+ */
+
+#include "libavutil/avassert.h"
+#include "libavutil/imgutils.h"
+#include "libavutil/opt.h"
+#include "libavutil/pixelutils.h"
+#include "libavutil/motion_vector.h"
+#include "avfilter.h"
+#include "formats.h"
+#include "internal.h"
+#include "video.h"
+#include "edge_common.h"
+
+static int comp(const void *a,const void *b)
+{
+ float x = *(float*)a;
+ float y = *(float*)b;
+ if (x > y) return 1;
+ if (x < y) return -1;
+ return 0;
+}
+
+typedef struct BLRContext {
+ const AVClass *class;
+
+ float low, high;
+ uint8_t low_u8, high_u8;
+ int radius; // radius during local maxima detection
+ int block_pct; // percentage of "sharpest" blocks in the image to use for bluriness calculation
+ int block_width; // width for block abbreviation
+ int block_height; // height for block abbreviation
+
+ double blur_total;
+ uint64_t nb_frames;
+
+ float *blks;
+ uint8_t *filterbuf;
+ uint8_t *tmpbuf;
+ uint16_t *gradients;
+ char *directions;
+} BLRContext;
+
+#define OFFSET(x) offsetof(BLRContext, x)
+#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM
+static const AVOption blr_options[] = {
+ { "high", "set high threshold", OFFSET(high), AV_OPT_TYPE_FLOAT, {.dbl=30/255.}, 0, 1, FLAGS },
+ { "low", "set low threshold", OFFSET(low), AV_OPT_TYPE_FLOAT, {.dbl=15/255.}, 0, 1, FLAGS },
+ { "radius", "search radius for maxima detection", OFFSET(radius), AV_OPT_TYPE_INT, {.i64=50}, 1, 100, FLAGS },
+ { "block_pct", "block pooling threshold when calculating blurriness", OFFSET(block_pct), AV_OPT_TYPE_INT, {.i64=80}, 1, 100, FLAGS },
+ { "block_width", "block size for block-based abbreviation of blurriness", OFFSET(block_width), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS },
+ { "block_height", "block size for block-based abbreviation of blurriness", OFFSET(block_height), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS },
+ { NULL }
+};
+
+AVFILTER_DEFINE_CLASS(blr);
+
+static av_cold int blr_init(AVFilterContext *ctx)
+{
+ BLRContext *blr = ctx->priv;
+
+ blr->low_u8 = blr->low * 255. + .5;
+ blr->high_u8 = blr->high * 255. + .5;
+
+ return 0;
+}
+
+static int blr_config_input(AVFilterLink *inlink)
+{
+ AVFilterContext *ctx = inlink->dst;
+ BLRContext *blr = ctx->priv;
+ const int bufsize = inlink->w * inlink->h;
+
+ if (blr->block_width < 1 || blr->block_height < 1) {
+ blr->block_width = inlink->w;
+ blr->block_height = inlink->h;
+ }
+
+ blr->tmpbuf = av_malloc(bufsize);
+ blr->filterbuf = av_malloc(bufsize);
+ blr->gradients = av_calloc(bufsize, sizeof(*blr->gradients));
+ blr->directions = av_malloc(bufsize);
+ blr->blks = av_calloc((inlink->w / blr->block_width) * (inlink->h / blr->block_height),
+ sizeof(*blr->blks));
+
+ if (!blr->tmpbuf || !blr->filterbuf || !blr->gradients ||
+ !blr->directions || !blr->blks)
+ return AVERROR(ENOMEM);
+
+ return 0;
+}
+
+// edge width is defined as the distance between surrounding maxima of the edge pixel
+static float edge_width(BLRContext *blr, int i, int j, int8_t dir, int w, int h,
+ int edge, const uint8_t *src, int src_linesize)
+{
+ float width = 0;
+ int dX, dY;
+ int sign;
+ int tmp;
+ int p1;
+ int p2;
+ int k, x, y;
+ int edge1;
+ int edge2;
+ float luma1 = 0.0; // average luma difference per edge pixel
+ float luma2 = 0.0;
+ int radius = blr->radius;
+
+ switch(dir)
+ {
+ case DIRECTION_HORIZONTAL: dX = 1; dY = 0; break;
+ case DIRECTION_VERTICAL: dX = 0; dY = 1; break;
+ case DIRECTION_45UP: dX = 1; dY = -1; break;
+ case DIRECTION_45DOWN: dX = 1; dY = 1; break;
+ }
+ if (dir == DIRECTION_HORIZONTAL) return 0;
+
+ // determines if search in direction dX/dY is looking for a maximum or minimum
+ sign = src[j * src_linesize + i] > src[(j - dY) * src_linesize + i - dX] ? 1 : -1;
+
+ // search in -(dX/dY) direction
+ for (k = 0; k < radius; k++)
+ {
+ x = i - k*dX;
+ y = j - k*dY;
+ p1 = y * src_linesize + x;
+ x -= dX;
+ y -= dY;
+ p2 = y * src_linesize + x;
+ if (x < 0 || x >= w || y < 0 || y >= h)
+ return 0;
+
+ tmp = (src[p1] - src[p2]) * sign;
+
+ if (tmp <= 0) // local maximum found
+ break;
+
+ luma1 += tmp;
+ }
+ if (k > 0) luma1 /= k;
+ edge1 = k;
+ width += k;
+
+ // search in +(dX/dY) direction
+ for (k = 0; k < radius; k++)
+ {
+ x = i + k * dX;
+ y = j + k * dY;
+ p1 = y * src_linesize + x;
+ x += dX;
+ y += dY;
+ p2 = y * src_linesize + x;
+ if (x < 0 || x >= w || y < 0 || y >= h)
+ return 0;
+
+ tmp = (src[p1] - src[p2]) * sign;
+
+ if (tmp >= 0) // local maximum found
+ break;
+
+ luma2 -= tmp;
+ }
+ if (k > 0) luma2 /= k;
+ edge2 = k;
+ width += k;
+
+ // for 45 degree directions approximate edge width in pixel units: 0.7 ~= sqrt(2)/2
+ if (dir == DIRECTION_45UP || dir == DIRECTION_45DOWN)
+ width *= 0.7;
+
+ return width;
+}
+
+static float calculate_blur(BLRContext *blr, int w, int h,
+ uint8_t* dir, int dir_linesize,
+ uint8_t* dst, int dst_linesize,
+ uint8_t* src, int src_linesize)
+{
+ float total_width = 0.0;
+ int block_count;
+ float block_total_width;
+
+ int i, j;
+ int blkcnt = 0;
+
+ float *blks = blr->blks;
+ int block_pct = blr->block_pct;
+ float block_pool_threshold = block_pct / 100.0;
+
+ int block_width = blr->block_width;
+ int block_height = blr->block_height;
+ int brows = h / block_height;
+ int bcols = w / block_width;
+
+ for (int blkj = 0; blkj < brows; blkj++) {
+ for (int blki = 0; blki < bcols; blki++) {
+ block_total_width = 0;
+ block_count = 0;
+ for (int inj = 0; inj < block_height; inj++) {
+ for (int ini = 0; ini < block_width; ini++) {
+ i = blki * block_width + ini;
+ j = blkj * block_height + inj;
+
+ if (dst[j * dst_linesize + i] > 0) {
+ float width = edge_width(blr, i, j, dir[j*dir_linesize+i],
+ w, h, dst[j*dst_linesize+i],
+ src, src_linesize);
+ if (width > 0.001) { // throw away zeros
+ block_count++;
+ block_total_width += width;
+ }
+ }
+ }
+ }
+ // if not enough edge pixels in a block, consider it smooth
+ if (block_total_width >= 2) {
+ blks[blkcnt] = block_total_width / block_count;
+ blkcnt++;
+ }
+ }
+ }
+
+ // simple block pooling by sorting and keeping the sharper blocks
+ qsort(blks, blkcnt, sizeof(*blks), comp);
+ blkcnt = ceil(blkcnt * block_pool_threshold);
+ for (int i = 0; i < blkcnt; i++) {
+ total_width += blks[i];
+ }
+
+ return total_width / blkcnt;
+}
+
+static void set_meta(AVDictionary **metadata, const char *key, float d)
+{
+ char value[128];
+ snprintf(value, sizeof(value), "%f", d);
+ av_dict_set(metadata, key, value, 0);
+}
+
+static int blr_filter_frame(AVFilterLink *inlink, AVFrame *in)
+{
+ AVFilterContext *ctx = inlink->dst;
+ BLRContext *blr = ctx->priv;
+
+ const int w = inlink->w;
+ const int h = inlink->h;
+
+ uint8_t *tmpbuf = blr->tmpbuf;
+ uint8_t *filterbuf = blr->filterbuf;
+ uint16_t *gradients = blr->gradients;
+ int8_t *directions = blr->directions;
+
+ float blur;
+ AVDictionary **metadata;
+ metadata = &in->metadata;
+
+ // gaussian filter to reduce noise
+ ff_gaussian_blur(w, h,
+ filterbuf, w,
+ in->data[0], in->linesize[0]);
+
+ // compute the 16-bits gradients and directions for the next step
+ ff_sobel(w, h, gradients, w, directions, w, filterbuf, w);
+
+ // non_maximum_suppression() will actually keep & clip what's necessary and
+ // ignore the rest, so we need a clean output buffer
+ memset(tmpbuf, 0, w * h);
+ ff_non_maximum_suppression(w, h, tmpbuf, w, directions, w, gradients, w);
+
+
+ // keep high values, or low values surrounded by high values
+ ff_double_threshold(blr->low_u8, blr->high_u8, w, h,
+ tmpbuf, w, tmpbuf, w);
+
+
+ blur = calculate_blur(blr, w, h, directions, w,
+ tmpbuf, w, filterbuf, w);
+
+ blr->blur_total += blur;
+
+ // write stats
+ av_log(ctx, AV_LOG_VERBOSE, "blur: %.7f\n", blur);
+
+ set_meta(metadata, "lavfi.blur", blur);
+
+ blr->nb_frames++;
+
+ return ff_filter_frame(ctx->outputs[0], in);
+}
+
+static av_cold void blr_uninit(AVFilterContext *ctx)
+{
+ BLRContext *blr = ctx->priv;
+
+ if (blr->nb_frames > 0) {
+ int nb_frames = blr->nb_frames;
+ av_log(ctx, AV_LOG_INFO, "blur mean: %.7f\n",
+ blr->blur_total / nb_frames);
+ }
+
+ av_freep(&blr->tmpbuf);
+ av_freep(&blr->filterbuf);
+ av_freep(&blr->gradients);
+ av_freep(&blr->directions);
+ av_freep(&blr->blks);
+}
+
+static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_YUV420P, AV_PIX_FMT_GRAY8, AV_PIX_FMT_NONE };
+
+static const AVFilterPad blr_inputs[] = {
+ {
+ .name = "default",
+ .type = AVMEDIA_TYPE_VIDEO,
+ .config_props = blr_config_input,
+ .filter_frame = blr_filter_frame,
+ },
+};
+
+static const AVFilterPad blr_outputs[] = {
+ {
+ .name = "default",
+ .type = AVMEDIA_TYPE_VIDEO,
+ },
+};
+
+AVFilter ff_vf_blurdetect = {
+ .name = "blurdetect",
+ .description = NULL_IF_CONFIG_SMALL("Blurdetect filter."),
+ .priv_size = sizeof(BLRContext),
+ .init = blr_init,
+ .uninit = blr_uninit,
+ FILTER_PIXFMTS_ARRAY(pix_fmts),
+ FILTER_INPUTS(blr_inputs),
+ FILTER_OUTPUTS(blr_outputs),
+ .priv_class = &blr_class,
+ .flags = AVFILTER_FLAG_METADATA_ONLY,
+};
+
On Mon, Apr 4, 2022 at 12:25 PM Thilo Borgmann <thilo.borgmann@mail.de> wrote: > Hi, > > v3 updated to current HEAD. > > Named blurdetect filter now. > Minor fixes on allocation and removed -f option. > Please make this per plane filtering, with default to measure only first plane. > -Thilo_______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >
Hi, >> v3 updated to current HEAD. >> >> Named blurdetect filter now. >> Minor fixes on allocation and removed -f option. >> > > Please make this per plane filtering, with default to measure only first > plane. done in v4. (Will add Changelog, version.h and fate test once the filter itself looks ok) Thanks, Thilo From 6411603fe86d4fb6a781aec2ff48e48be12d269e Mon Sep 17 00:00:00 2001 From: Thilo Borgmann <thilo.borgmann@mail.de> Date: Tue, 5 Apr 2022 10:54:19 +0200 Subject: [PATCH v4 2/2] lavfi: Add blurdetect filter --- doc/filters.texi | 52 +++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_blurdetect.c | 394 ++++++++++++++++++++++++++++++++++++ 4 files changed, 448 insertions(+) create mode 100644 libavfilter/vf_blurdetect.c diff --git a/doc/filters.texi b/doc/filters.texi index 636c80dbff..c6d4537804 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -7990,6 +7990,58 @@ tblend=all_mode=grainextract @subsection Commands This filter supports same @ref{commands} as options. +@anchor{blurdetect} +@section blurdetect + +Determines blurriness of frames without altering the input frames. + +Based on Marziliano, Pina, et al. "A no-reference perceptual blur metric." +Allows for a block-based abbreviation. + +The filter accepts the following options: + +@table @option +@item low +@item high +Set low and high threshold values used by the Canny thresholding +algorithm. + +The high threshold selects the "strong" edge pixels, which are then +connected through 8-connectivity with the "weak" edge pixels selected +by the low threshold. + +@var{low} and @var{high} threshold values must be chosen in the range +[0,1], and @var{low} should be lesser or equal to @var{high}. + +Default value for @var{low} is @code{20/255}, and default value for @var{high} +is @code{50/255}. + +@item radius +Define the radius to search around an edge pixel for local maxima. + +@item block_pct +Determine blurriness only for the most significant blocks, given in percentage. + +@item block_width +Determine blurriness for blocks of width @var{block_width}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_height}. + +@item block_height +Determine blurriness for blocks of height @var{block_height}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_width}. + +@item planes +Set planes to filter. Default is first only. +@end table + +@subsection Examples + +@itemize +@item +Determine blur for 80% of most significant 32x32 blocks: +@example +blurdetect=block_width=32:block_height=32:block_pct=80 +@end example +@end itemize + @section bm3d Denoise frames using Block-Matching 3D algorithm. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 006e59b2bd..6332a6f799 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -195,6 +195,7 @@ OBJS-$(CONFIG_BLACKDETECT_FILTER) += vf_blackdetect.o OBJS-$(CONFIG_BLACKFRAME_FILTER) += vf_blackframe.o OBJS-$(CONFIG_BLEND_FILTER) += vf_blend.o framesync.o OBJS-$(CONFIG_BLEND_VULKAN_FILTER) += vf_blend_vulkan.o framesync.o vulkan.o vulkan_filter.o +OBJS-$(CONFIG_BLURDETECT_FILTER) += vf_blurdetect.o edge_common.o OBJS-$(CONFIG_BM3D_FILTER) += vf_bm3d.o framesync.o OBJS-$(CONFIG_BOXBLUR_FILTER) += vf_boxblur.o boxblur.o OBJS-$(CONFIG_BOXBLUR_OPENCL_FILTER) += vf_avgblur_opencl.o opencl.o \ diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 9fbaaacf47..2667d153ad 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -183,6 +183,7 @@ extern const AVFilter ff_vf_blackdetect; extern const AVFilter ff_vf_blackframe; extern const AVFilter ff_vf_blend; extern const AVFilter ff_vf_blend_vulkan; +extern const AVFilter ff_vf_blurdetect; extern const AVFilter ff_vf_bm3d; extern const AVFilter ff_vf_boxblur; extern const AVFilter ff_vf_boxblur_opencl; diff --git a/libavfilter/vf_blurdetect.c b/libavfilter/vf_blurdetect.c new file mode 100644 index 0000000000..71e46fa962 --- /dev/null +++ b/libavfilter/vf_blurdetect.c @@ -0,0 +1,394 @@ +/* + * Copyright (c) 2021 Thilo Borgmann <thilo.borgmann _at_ mail.de> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * No-reference blurdetect filter + * + * Implementing: + * Marziliano, Pina, et al. "A no-reference perceptual blur metric." Proceedings. + * International conference on image processing. Vol. 3. IEEE, 2002. + * https://infoscience.epfl.ch/record/111802/files/14%20A%20no-reference%20perceptual%20blur%20metric.pdf + * + * @author Thilo Borgmann <thilo.borgmann _at_ mail.de> + */ + +#include "libavutil/avassert.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "libavutil/pixelutils.h" +#include "libavutil/motion_vector.h" +#include "avfilter.h" +#include "formats.h" +#include "internal.h" +#include "video.h" +#include "edge_common.h" + +static int comp(const void *a,const void *b) +{ + float x = *(float*)a; + float y = *(float*)b; + if (x > y) return 1; + if (x < y) return -1; + return 0; +} + +typedef struct BLRContext { + const AVClass *class; + + int hsub, vsub; + int nb_planes; + + float low, high; + uint8_t low_u8, high_u8; + int radius; // radius during local maxima detection + int block_pct; // percentage of "sharpest" blocks in the image to use for bluriness calculation + int block_width; // width for block abbreviation + int block_height; // height for block abbreviation + int planes; // number of planes to filter + + double blur_total; + uint64_t nb_frames; + + float *blks; + uint8_t *filterbuf; + uint8_t *tmpbuf; + uint16_t *gradients; + char *directions; +} BLRContext; + +#define OFFSET(x) offsetof(BLRContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM +static const AVOption blr_options[] = { + { "high", "set high threshold", OFFSET(high), AV_OPT_TYPE_FLOAT, {.dbl=30/255.}, 0, 1, FLAGS }, + { "low", "set low threshold", OFFSET(low), AV_OPT_TYPE_FLOAT, {.dbl=15/255.}, 0, 1, FLAGS }, + { "radius", "search radius for maxima detection", OFFSET(radius), AV_OPT_TYPE_INT, {.i64=50}, 1, 100, FLAGS }, + { "block_pct", "block pooling threshold when calculating blurriness", OFFSET(block_pct), AV_OPT_TYPE_INT, {.i64=80}, 1, 100, FLAGS }, + { "block_width", "block size for block-based abbreviation of blurriness", OFFSET(block_width), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS }, + { "block_height", "block size for block-based abbreviation of blurriness", OFFSET(block_height), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS }, + { "planes", "set planes to filter", OFFSET(planes), AV_OPT_TYPE_INT, {.i64=1}, 0, 15, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(blr); + +static av_cold int blr_init(AVFilterContext *ctx) +{ + BLRContext *blr = ctx->priv; + + blr->low_u8 = blr->low * 255. + .5; + blr->high_u8 = blr->high * 255. + .5; + + return 0; +} + +static int blr_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + BLRContext *blr = ctx->priv; + const int bufsize = inlink->w * inlink->h; + const AVPixFmtDescriptor *pix_desc; + + pix_desc = av_pix_fmt_desc_get(inlink->format); + blr->hsub = pix_desc->log2_chroma_w; + blr->vsub = pix_desc->log2_chroma_h; + blr->nb_planes = av_pix_fmt_count_planes(inlink->format); + + if (blr->block_width < 1 || blr->block_height < 1) { + blr->block_width = inlink->w; + blr->block_height = inlink->h; + } + + blr->tmpbuf = av_malloc(bufsize); + blr->filterbuf = av_malloc(bufsize); + blr->gradients = av_calloc(bufsize, sizeof(*blr->gradients)); + blr->directions = av_malloc(bufsize); + blr->blks = av_calloc((inlink->w / blr->block_width) * (inlink->h / blr->block_height), + sizeof(*blr->blks)); + + if (!blr->tmpbuf || !blr->filterbuf || !blr->gradients || + !blr->directions || !blr->blks) + return AVERROR(ENOMEM); + + return 0; +} + +// edge width is defined as the distance between surrounding maxima of the edge pixel +static float edge_width(BLRContext *blr, int i, int j, int8_t dir, int w, int h, + int edge, const uint8_t *src, int src_linesize) +{ + float width = 0; + int dX, dY; + int sign; + int tmp; + int p1; + int p2; + int k, x, y; + int edge1; + int edge2; + float luma1 = 0.0; // average luma difference per edge pixel + float luma2 = 0.0; + int radius = blr->radius; + + switch(dir) + { + case DIRECTION_HORIZONTAL: dX = 1; dY = 0; break; + case DIRECTION_VERTICAL: dX = 0; dY = 1; break; + case DIRECTION_45UP: dX = 1; dY = -1; break; + case DIRECTION_45DOWN: dX = 1; dY = 1; break; + } + if (dir == DIRECTION_HORIZONTAL) return 0; + + // determines if search in direction dX/dY is looking for a maximum or minimum + sign = src[j * src_linesize + i] > src[(j - dY) * src_linesize + i - dX] ? 1 : -1; + + // search in -(dX/dY) direction + for (k = 0; k < radius; k++) + { + x = i - k*dX; + y = j - k*dY; + p1 = y * src_linesize + x; + x -= dX; + y -= dY; + p2 = y * src_linesize + x; + if (x < 0 || x >= w || y < 0 || y >= h) + return 0; + + tmp = (src[p1] - src[p2]) * sign; + + if (tmp <= 0) // local maximum found + break; + + luma1 += tmp; + } + if (k > 0) luma1 /= k; + edge1 = k; + width += k; + + // search in +(dX/dY) direction + for (k = 0; k < radius; k++) + { + x = i + k * dX; + y = j + k * dY; + p1 = y * src_linesize + x; + x += dX; + y += dY; + p2 = y * src_linesize + x; + if (x < 0 || x >= w || y < 0 || y >= h) + return 0; + + tmp = (src[p1] - src[p2]) * sign; + + if (tmp >= 0) // local maximum found + break; + + luma2 -= tmp; + } + if (k > 0) luma2 /= k; + edge2 = k; + width += k; + + // for 45 degree directions approximate edge width in pixel units: 0.7 ~= sqrt(2)/2 + if (dir == DIRECTION_45UP || dir == DIRECTION_45DOWN) + width *= 0.7; + + return width; +} + +static float calculate_blur(BLRContext *blr, int w, int h, int hsub, int vsub, + uint8_t* dir, int dir_linesize, + uint8_t* dst, int dst_linesize, + uint8_t* src, int src_linesize) +{ + float total_width = 0.0; + int block_count; + double block_total_width; + + int i, j; + int blkcnt = 0; + + float *blks = blr->blks; + float block_pool_threshold = blr->block_pct / 100.0; + + int block_width = AV_CEIL_RSHIFT(blr->block_width, hsub); + int block_height = AV_CEIL_RSHIFT(blr->block_height, vsub); + int brows = h / block_height; + int bcols = w / block_width; + + for (int blkj = 0; blkj < brows; blkj++) { + for (int blki = 0; blki < bcols; blki++) { + block_total_width = 0.0; + block_count = 0; + for (int inj = 0; inj < block_height; inj++) { + for (int ini = 0; ini < block_width; ini++) { + i = blki * block_width + ini; + j = blkj * block_height + inj; + + if (dst[j * dst_linesize + i] > 0) { + float width = edge_width(blr, i, j, dir[j*dir_linesize+i], + w, h, dst[j*dst_linesize+i], + src, src_linesize); + if (width > 0.001) { // throw away zeros + block_count++; + block_total_width += width; + } + } + } + } + // if not enough edge pixels in a block, consider it smooth + if (block_total_width >= 2) { + blks[blkcnt] = block_total_width / block_count; + blkcnt++; + } + } + } + + // simple block pooling by sorting and keeping the sharper blocks + qsort(blks, blkcnt, sizeof(*blks), comp); + blkcnt = ceil(blkcnt * block_pool_threshold); + for (int i = 0; i < blkcnt; i++) { + total_width += blks[i]; + } + + return total_width / blkcnt; +} + +static void set_meta(AVDictionary **metadata, const char *key, float d) +{ + char value[128]; + snprintf(value, sizeof(value), "%f", d); + av_dict_set(metadata, key, value, 0); +} + +static int blr_filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx = inlink->dst; + BLRContext *blr = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + + const int inw = inlink->w; + const int inh = inlink->h; + + uint8_t *tmpbuf = blr->tmpbuf; + uint8_t *filterbuf = blr->filterbuf; + uint16_t *gradients = blr->gradients; + int8_t *directions = blr->directions; + + float blur = 0.0f; + int nplanes = 0; + AVDictionary **metadata; + metadata = &in->metadata; + + for (int plane = 0; plane < blr->nb_planes; plane++) { + int hsub = plane == 1 || plane == 2 ? blr->hsub : 0; + int vsub = plane == 1 || plane == 2 ? blr->vsub : 0; + int w = AV_CEIL_RSHIFT(inw, hsub); + int h = AV_CEIL_RSHIFT(inh, vsub); + + if (!((1 << plane) & blr->planes)) + continue; + + nplanes++; + + // gaussian filter to reduce noise + ff_gaussian_blur(w, h, + filterbuf, w, + in->data[plane], in->linesize[plane]); + + // compute the 16-bits gradients and directions for the next step + ff_sobel(w, h, gradients, w, directions, w, filterbuf, w); + + // non_maximum_suppression() will actually keep & clip what's necessary and + // ignore the rest, so we need a clean output buffer + memset(tmpbuf, 0, inw * inh); + ff_non_maximum_suppression(w, h, tmpbuf, w, directions, w, gradients, w); + + + // keep high values, or low values surrounded by high values + ff_double_threshold(blr->low_u8, blr->high_u8, w, h, + tmpbuf, w, tmpbuf, w); + + blur += calculate_blur(blr, w, h, hsub, vsub, directions, w, + tmpbuf, w, filterbuf, w); + } + + if (nplanes) + blur /= nplanes; + + blr->blur_total += blur; + + // write stats + av_log(ctx, AV_LOG_VERBOSE, "blur: %.7f\n", blur); + + set_meta(metadata, "lavfi.blur", blur); + + blr->nb_frames++; + + return ff_filter_frame(outlink, in); +} + +static av_cold void blr_uninit(AVFilterContext *ctx) +{ + BLRContext *blr = ctx->priv; + + if (blr->nb_frames > 0) { + int nb_frames = blr->nb_frames; + av_log(ctx, AV_LOG_INFO, "blur mean: %.7f\n", + blr->blur_total / nb_frames); + } + + av_freep(&blr->tmpbuf); + av_freep(&blr->filterbuf); + av_freep(&blr->gradients); + av_freep(&blr->directions); + av_freep(&blr->blks); +} + +static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_YUV420P, AV_PIX_FMT_GRAY8, AV_PIX_FMT_NONE }; + +static const AVFilterPad blr_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = blr_config_input, + .filter_frame = blr_filter_frame, + }, +}; + +static const AVFilterPad blr_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + }, +}; + +AVFilter ff_vf_blurdetect = { + .name = "blurdetect", + .description = NULL_IF_CONFIG_SMALL("Blurdetect filter."), + .priv_size = sizeof(BLRContext), + .init = blr_init, + .uninit = blr_uninit, + FILTER_PIXFMTS_ARRAY(pix_fmts), + FILTER_INPUTS(blr_inputs), + FILTER_OUTPUTS(blr_outputs), + .priv_class = &blr_class, + .flags = AVFILTER_FLAG_METADATA_ONLY, +}; +
Am 05.04.22 um 10:56 schrieb Thilo Borgmann: > Hi, > >>> v3 updated to current HEAD. >>> >>> Named blurdetect filter now. >>> Minor fixes on allocation and removed -f option. >>> >> >> Please make this per plane filtering, with default to measure only first >> plane. > > done in v4. > > (Will add Changelog, version.h and fate test once the filter itself looks ok) Ping. -Thilo
Hi, >>>> v3 updated to current HEAD. >>>> >>>> Named blurdetect filter now. >>>> Minor fixes on allocation and removed -f option. >>>> >>> >>> Please make this per plane filtering, with default to measure only first >>> plane. >> >> done in v4. >> >> (Will add Changelog, version.h and fate test once the filter itself looks ok) > > Ping. v5 according to IRC comments. -Thilo From be0bed07d36d957caf328f91a6cd0402c27fb6b4 Mon Sep 17 00:00:00 2001 From: Thilo Borgmann <thilo.borgmann@mail.de> Date: Sat, 23 Apr 2022 15:29:36 +0200 Subject: [PATCH v5 2/2] lavfi: Add blurdetect filter --- doc/filters.texi | 52 +++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_blurdetect.c | 395 ++++++++++++++++++++++++++++++++++++ 4 files changed, 449 insertions(+) create mode 100644 libavfilter/vf_blurdetect.c diff --git a/doc/filters.texi b/doc/filters.texi index 636c80dbff..c6d4537804 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -7990,6 +7990,58 @@ tblend=all_mode=grainextract @subsection Commands This filter supports same @ref{commands} as options. +@anchor{blurdetect} +@section blurdetect + +Determines blurriness of frames without altering the input frames. + +Based on Marziliano, Pina, et al. "A no-reference perceptual blur metric." +Allows for a block-based abbreviation. + +The filter accepts the following options: + +@table @option +@item low +@item high +Set low and high threshold values used by the Canny thresholding +algorithm. + +The high threshold selects the "strong" edge pixels, which are then +connected through 8-connectivity with the "weak" edge pixels selected +by the low threshold. + +@var{low} and @var{high} threshold values must be chosen in the range +[0,1], and @var{low} should be lesser or equal to @var{high}. + +Default value for @var{low} is @code{20/255}, and default value for @var{high} +is @code{50/255}. + +@item radius +Define the radius to search around an edge pixel for local maxima. + +@item block_pct +Determine blurriness only for the most significant blocks, given in percentage. + +@item block_width +Determine blurriness for blocks of width @var{block_width}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_height}. + +@item block_height +Determine blurriness for blocks of height @var{block_height}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_width}. + +@item planes +Set planes to filter. Default is first only. +@end table + +@subsection Examples + +@itemize +@item +Determine blur for 80% of most significant 32x32 blocks: +@example +blurdetect=block_width=32:block_height=32:block_pct=80 +@end example +@end itemize + @section bm3d Denoise frames using Block-Matching 3D algorithm. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 006e59b2bd..6332a6f799 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -195,6 +195,7 @@ OBJS-$(CONFIG_BLACKDETECT_FILTER) += vf_blackdetect.o OBJS-$(CONFIG_BLACKFRAME_FILTER) += vf_blackframe.o OBJS-$(CONFIG_BLEND_FILTER) += vf_blend.o framesync.o OBJS-$(CONFIG_BLEND_VULKAN_FILTER) += vf_blend_vulkan.o framesync.o vulkan.o vulkan_filter.o +OBJS-$(CONFIG_BLURDETECT_FILTER) += vf_blurdetect.o edge_common.o OBJS-$(CONFIG_BM3D_FILTER) += vf_bm3d.o framesync.o OBJS-$(CONFIG_BOXBLUR_FILTER) += vf_boxblur.o boxblur.o OBJS-$(CONFIG_BOXBLUR_OPENCL_FILTER) += vf_avgblur_opencl.o opencl.o \ diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 9fbaaacf47..2667d153ad 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -183,6 +183,7 @@ extern const AVFilter ff_vf_blackdetect; extern const AVFilter ff_vf_blackframe; extern const AVFilter ff_vf_blend; extern const AVFilter ff_vf_blend_vulkan; +extern const AVFilter ff_vf_blurdetect; extern const AVFilter ff_vf_bm3d; extern const AVFilter ff_vf_boxblur; extern const AVFilter ff_vf_boxblur_opencl; diff --git a/libavfilter/vf_blurdetect.c b/libavfilter/vf_blurdetect.c new file mode 100644 index 0000000000..c74f45ccb8 --- /dev/null +++ b/libavfilter/vf_blurdetect.c @@ -0,0 +1,395 @@ +/* + * Copyright (c) 2021 Thilo Borgmann <thilo.borgmann _at_ mail.de> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * No-reference blurdetect filter + * + * Implementing: + * Marziliano, Pina, et al. "A no-reference perceptual blur metric." Proceedings. + * International conference on image processing. Vol. 3. IEEE, 2002. + * https://infoscience.epfl.ch/record/111802/files/14%20A%20no-reference%20perceptual%20blur%20metric.pdf + * + * @author Thilo Borgmann <thilo.borgmann _at_ mail.de> + */ + +#include "libavutil/avassert.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "libavutil/pixelutils.h" +#include "libavutil/motion_vector.h" +#include "libavutil/qsort.h" +#include "avfilter.h" +#include "formats.h" +#include "internal.h" +#include "video.h" +#include "edge_common.h" + +static int comp(const float *a,const float *b) +{ + return FFDIFFSIGN(*a, *b); +} + +typedef struct BLRContext { + const AVClass *class; + + int hsub, vsub; + int nb_planes; + + float low, high; + uint8_t low_u8, high_u8; + int radius; // radius during local maxima detection + int block_pct; // percentage of "sharpest" blocks in the image to use for bluriness calculation + int block_width; // width for block abbreviation + int block_height; // height for block abbreviation + int planes; // number of planes to filter + + double blur_total; + uint64_t nb_frames; + + float *blks; + uint8_t *filterbuf; + uint8_t *tmpbuf; + uint16_t *gradients; + char *directions; +} BLRContext; + +#define OFFSET(x) offsetof(BLRContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM +static const AVOption blurdetect_options[] = { + { "high", "set high threshold", OFFSET(high), AV_OPT_TYPE_FLOAT, {.dbl=30/255.}, 0, 1, FLAGS }, + { "low", "set low threshold", OFFSET(low), AV_OPT_TYPE_FLOAT, {.dbl=15/255.}, 0, 1, FLAGS }, + { "radius", "search radius for maxima detection", OFFSET(radius), AV_OPT_TYPE_INT, {.i64=50}, 1, 100, FLAGS }, + { "block_pct", "block pooling threshold when calculating blurriness", OFFSET(block_pct), AV_OPT_TYPE_INT, {.i64=80}, 1, 100, FLAGS }, + { "block_width", "block size for block-based abbreviation of blurriness", OFFSET(block_width), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS }, + { "block_height", "block size for block-based abbreviation of blurriness", OFFSET(block_height), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS }, + { "planes", "set planes to filter", OFFSET(planes), AV_OPT_TYPE_INT, {.i64=1}, 0, 15, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(blurdetect); + +static av_cold int blurdetect_init(AVFilterContext *ctx) +{ + BLRContext *s = ctx->priv; + + s->low_u8 = s->low * 255. + .5; + s->high_u8 = s->high * 255. + .5; + + return 0; +} + +static int blurdetect_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + BLRContext *s = ctx->priv; + const int bufsize = inlink->w * inlink->h; + const AVPixFmtDescriptor *pix_desc; + + pix_desc = av_pix_fmt_desc_get(inlink->format); + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + s->nb_planes = av_pix_fmt_count_planes(inlink->format); + + if (s->block_width < 1 || s->block_height < 1) { + s->block_width = inlink->w; + s->block_height = inlink->h; + } + + s->tmpbuf = av_malloc(bufsize); + s->filterbuf = av_malloc(bufsize); + s->gradients = av_calloc(bufsize, sizeof(*s->gradients)); + s->directions = av_malloc(bufsize); + s->blks = av_calloc((inlink->w / s->block_width) * (inlink->h / s->block_height), + sizeof(*s->blks)); + + if (!s->tmpbuf || !s->filterbuf || !s->gradients || !s->directions || !s->blks) + return AVERROR(ENOMEM); + + return 0; +} + +// edge width is defined as the distance between surrounding maxima of the edge pixel +static float edge_width(BLRContext *blr, int i, int j, int8_t dir, int w, int h, + int edge, const uint8_t *src, int src_linesize) +{ + float width = 0; + int dX, dY; + int sign; + int tmp; + int p1; + int p2; + int k, x, y; + int edge1; + int edge2; + float luma1 = 0.0; // average luma difference per edge pixel + float luma2 = 0.0; + int radius = blr->radius; + + switch(dir) { + case DIRECTION_HORIZONTAL: dX = 1; dY = 0; break; + case DIRECTION_VERTICAL: dX = 0; dY = 1; break; + case DIRECTION_45UP: dX = 1; dY = -1; break; + case DIRECTION_45DOWN: dX = 1; dY = 1; break; + } + if (dir == DIRECTION_HORIZONTAL) return 0; + + // determines if search in direction dX/dY is looking for a maximum or minimum + sign = src[j * src_linesize + i] > src[(j - dY) * src_linesize + i - dX] ? 1 : -1; + + // search in -(dX/dY) direction + for (k = 0; k < radius; k++) { + x = i - k*dX; + y = j - k*dY; + p1 = y * src_linesize + x; + x -= dX; + y -= dY; + p2 = y * src_linesize + x; + if (x < 0 || x >= w || y < 0 || y >= h) + return 0; + + tmp = (src[p1] - src[p2]) * sign; + + if (tmp <= 0) // local maximum found + break; + + luma1 += tmp; + } + if (k > 0) luma1 /= k; + edge1 = k; + width += k; + + // search in +(dX/dY) direction + for (k = 0; k < radius; k++) { + x = i + k * dX; + y = j + k * dY; + p1 = y * src_linesize + x; + x += dX; + y += dY; + p2 = y * src_linesize + x; + if (x < 0 || x >= w || y < 0 || y >= h) + return 0; + + tmp = (src[p1] - src[p2]) * sign; + + if (tmp >= 0) // local maximum found + break; + + luma2 -= tmp; + } + if (k > 0) luma2 /= k; + edge2 = k; + width += k; + + // for 45 degree directions approximate edge width in pixel units: 0.7 ~= sqrt(2)/2 + if (dir == DIRECTION_45UP || dir == DIRECTION_45DOWN) + width *= 0.7; + + return width; +} + +static float calculate_blur(BLRContext *s, int w, int h, int hsub, int vsub, + uint8_t* dir, int dir_linesize, + uint8_t* dst, int dst_linesize, + uint8_t* src, int src_linesize) +{ + float total_width = 0.0; + int block_count; + double block_total_width; + + int i, j; + int blkcnt = 0; + + float *blks = s->blks; + float block_pool_threshold = s->block_pct / 100.0; + + int block_width = AV_CEIL_RSHIFT(s->block_width, hsub); + int block_height = AV_CEIL_RSHIFT(s->block_height, vsub); + int brows = h / block_height; + int bcols = w / block_width; + + for (int blkj = 0; blkj < brows; blkj++) { + for (int blki = 0; blki < bcols; blki++) { + block_total_width = 0.0; + block_count = 0; + for (int inj = 0; inj < block_height; inj++) { + for (int ini = 0; ini < block_width; ini++) { + i = blki * block_width + ini; + j = blkj * block_height + inj; + + if (dst[j * dst_linesize + i] > 0) { + float width = edge_width(s, i, j, dir[j*dir_linesize+i], + w, h, dst[j*dst_linesize+i], + src, src_linesize); + if (width > 0.001) { // throw away zeros + block_count++; + block_total_width += width; + } + } + } + } + // if not enough edge pixels in a block, consider it smooth + if (block_total_width >= 2) { + blks[blkcnt] = block_total_width / block_count; + blkcnt++; + } + } + } + + // simple block pooling by sorting and keeping the sharper blocks + AV_QSORT(blks, blkcnt, float, comp); + blkcnt = ceil(blkcnt * block_pool_threshold); + for (int i = 0; i < blkcnt; i++) { + total_width += blks[i]; + } + + return total_width / blkcnt; +} + +static void set_meta(AVDictionary **metadata, const char *key, float d) +{ + char value[128]; + snprintf(value, sizeof(value), "%f", d); + av_dict_set(metadata, key, value, 0); +} + +static int blurdetect_filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx = inlink->dst; + BLRContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + + const int inw = inlink->w; + const int inh = inlink->h; + + uint8_t *tmpbuf = s->tmpbuf; + uint8_t *filterbuf = s->filterbuf; + uint16_t *gradients = s->gradients; + int8_t *directions = s->directions; + + float blur = 0.0f; + int nplanes = 0; + AVDictionary **metadata; + metadata = &in->metadata; + + for (int plane = 0; plane < s->nb_planes; plane++) { + int hsub = plane == 1 || plane == 2 ? s->hsub : 0; + int vsub = plane == 1 || plane == 2 ? s->vsub : 0; + int w = AV_CEIL_RSHIFT(inw, hsub); + int h = AV_CEIL_RSHIFT(inh, vsub); + + if (!((1 << plane) & s->planes)) + continue; + + nplanes++; + + // gaussian filter to reduce noise + ff_gaussian_blur(w, h, + filterbuf, w, + in->data[plane], in->linesize[plane]); + + // compute the 16-bits gradients and directions for the next step + ff_sobel(w, h, gradients, w, directions, w, filterbuf, w); + + // non_maximum_suppression() will actually keep & clip what's necessary and + // ignore the rest, so we need a clean output buffer + memset(tmpbuf, 0, inw * inh); + ff_non_maximum_suppression(w, h, tmpbuf, w, directions, w, gradients, w); + + + // keep high values, or low values surrounded by high values + ff_double_threshold(s->low_u8, s->high_u8, w, h, + tmpbuf, w, tmpbuf, w); + + blur += calculate_blur(s, w, h, hsub, vsub, directions, w, + tmpbuf, w, filterbuf, w); + } + + if (nplanes) + blur /= nplanes; + + s->blur_total += blur; + + // write stats + av_log(ctx, AV_LOG_VERBOSE, "blur: %.7f\n", blur); + + set_meta(metadata, "lavfi.blur", blur); + + s->nb_frames = inlink->frame_count_in; + + return ff_filter_frame(outlink, in); +} + +static av_cold void blurdetect_uninit(AVFilterContext *ctx) +{ + BLRContext *s = ctx->priv; + + if (s->nb_frames > 0) { + av_log(ctx, AV_LOG_INFO, "blur mean: %.7f\n", + s->blur_total / s->nb_frames); + } + + av_freep(&s->tmpbuf); + av_freep(&s->filterbuf); + av_freep(&s->gradients); + av_freep(&s->directions); + av_freep(&s->blks); +} + +static const enum AVPixelFormat pix_fmts[] = { + AV_PIX_FMT_GRAY8, + AV_PIX_FMT_GBRP, AV_PIX_FMT_GBRAP, + AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV420P, + AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUV440P, + AV_PIX_FMT_YUV411P, AV_PIX_FMT_YUV410P, + AV_PIX_FMT_YUVJ440P, AV_PIX_FMT_YUVJ411P, AV_PIX_FMT_YUVJ420P, + AV_PIX_FMT_YUVJ422P, AV_PIX_FMT_YUVJ444P, + AV_PIX_FMT_YUVA444P, AV_PIX_FMT_YUVA422P, AV_PIX_FMT_YUVA420P, + AV_PIX_FMT_NONE }; + +static const AVFilterPad blurdetect_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = blurdetect_config_input, + .filter_frame = blurdetect_filter_frame, + }, +}; + +static const AVFilterPad blurdetect_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + }, +}; + +const AVFilter ff_vf_blurdetect = { + .name = "blurdetect", + .description = NULL_IF_CONFIG_SMALL("Blurdetect filter."), + .priv_size = sizeof(BLRContext), + .init = blurdetect_init, + .uninit = blurdetect_uninit, + FILTER_PIXFMTS_ARRAY(pix_fmts), + FILTER_INPUTS(blurdetect_inputs), + FILTER_OUTPUTS(blurdetect_outputs), + .priv_class = &blurdetect_class, + .flags = AVFILTER_FLAG_METADATA_ONLY, +}; +
Am 23.04.22 um 15:32 schrieb Thilo Borgmann: > Hi, > >>>>> v3 updated to current HEAD. >>>>> >>>>> Named blurdetect filter now. >>>>> Minor fixes on allocation and removed -f option. >>>>> >>>> >>>> Please make this per plane filtering, with default to measure only first >>>> plane. >>> >>> done in v4. >>> >>> (Will add Changelog, version.h and fate test once the filter itself looks ok) >> >> Ping. > > v5 according to IRC comments. v6 (1/2 and 2/2) according to IRC comments. LGTM'd there, applying soon. Added Changelog, version.h, FATE. Thanks! Thilo From c90120255bff305ce76c7677c9063d94c3bb99f7 Mon Sep 17 00:00:00 2001 From: Thilo Borgmann <thilo.borgmann@mail.de> Date: Tue, 30 Nov 2021 00:16:52 +0100 Subject: [PATCH v6 1/2] lafi/vf_edgedetect: Move some common functions into seperate file --- libavfilter/Makefile | 2 +- libavfilter/edge_common.c | 181 +++++++++++++++++++++++++++++++++++ libavfilter/edge_common.h | 107 +++++++++++++++++++++ libavfilter/vf_edgedetect.c | 183 ++---------------------------------- 4 files changed, 296 insertions(+), 177 deletions(-) create mode 100644 libavfilter/edge_common.c create mode 100644 libavfilter/edge_common.h diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 32521a4836..38ca379e5a 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -266,7 +266,7 @@ OBJS-$(CONFIG_DRAWBOX_FILTER) += vf_drawbox.o OBJS-$(CONFIG_DRAWGRAPH_FILTER) += f_drawgraph.o OBJS-$(CONFIG_DRAWGRID_FILTER) += vf_drawbox.o OBJS-$(CONFIG_DRAWTEXT_FILTER) += vf_drawtext.o -OBJS-$(CONFIG_EDGEDETECT_FILTER) += vf_edgedetect.o +OBJS-$(CONFIG_EDGEDETECT_FILTER) += vf_edgedetect.o edge_common.o OBJS-$(CONFIG_ELBG_FILTER) += vf_elbg.o OBJS-$(CONFIG_ENTROPY_FILTER) += vf_entropy.o OBJS-$(CONFIG_EPX_FILTER) += vf_epx.o diff --git a/libavfilter/edge_common.c b/libavfilter/edge_common.c new file mode 100644 index 0000000000..d72e8521cd --- /dev/null +++ b/libavfilter/edge_common.c @@ -0,0 +1,181 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "edge_common.h" + +// Internal helper for ff_sobel() +static int get_rounded_direction(int gx, int gy) +{ + /* reference angles: + * tan( pi/8) = sqrt(2)-1 + * tan(3pi/8) = sqrt(2)+1 + * Gy/Gx is the tangent of the angle (theta), so Gy/Gx is compared against + * <ref-angle>, or more simply Gy against <ref-angle>*Gx + * + * Gx and Gy bounds = [-1020;1020], using 16-bit arithmetic: + * round((sqrt(2)-1) * (1<<16)) = 27146 + * round((sqrt(2)+1) * (1<<16)) = 158218 + */ + if (gx) { + int tanpi8gx, tan3pi8gx; + + if (gx < 0) + gx = -gx, gy = -gy; + gy *= (1 << 16); + tanpi8gx = 27146 * gx; + tan3pi8gx = 158218 * gx; + if (gy > -tan3pi8gx && gy < -tanpi8gx) return DIRECTION_45UP; + if (gy > -tanpi8gx && gy < tanpi8gx) return DIRECTION_HORIZONTAL; + if (gy > tanpi8gx && gy < tan3pi8gx) return DIRECTION_45DOWN; + } + return DIRECTION_VERTICAL; +} + +// Simple sobel operator to get rounded gradients +void ff_sobel(int w, int h, + uint16_t *dst, int dst_linesize, + int8_t *dir, int dir_linesize, + const uint8_t *src, int src_linesize) +{ + int i, j; + + for (j = 1; j < h - 1; j++) { + dst += dst_linesize; + dir += dir_linesize; + src += src_linesize; + for (i = 1; i < w - 1; i++) { + const int gx = + -1*src[-src_linesize + i-1] + 1*src[-src_linesize + i+1] + -2*src[ i-1] + 2*src[ i+1] + -1*src[ src_linesize + i-1] + 1*src[ src_linesize + i+1]; + const int gy = + -1*src[-src_linesize + i-1] + 1*src[ src_linesize + i-1] + -2*src[-src_linesize + i ] + 2*src[ src_linesize + i ] + -1*src[-src_linesize + i+1] + 1*src[ src_linesize + i+1]; + + dst[i] = FFABS(gx) + FFABS(gy); + dir[i] = get_rounded_direction(gx, gy); + } + } +} + +// Filters rounded gradients to drop all non-maxima +// Expects gradients generated by ff_sobel() +// Expects zero's destination buffer +void ff_non_maximum_suppression(int w, int h, + uint8_t *dst, int dst_linesize, + const int8_t *dir, int dir_linesize, + const uint16_t *src, int src_linesize) +{ + int i, j; + +#define COPY_MAXIMA(ay, ax, by, bx) do { \ + if (src[i] > src[(ay)*src_linesize + i+(ax)] && \ + src[i] > src[(by)*src_linesize + i+(bx)]) \ + dst[i] = av_clip_uint8(src[i]); \ +} while (0) + + for (j = 1; j < h - 1; j++) { + dst += dst_linesize; + dir += dir_linesize; + src += src_linesize; + for (i = 1; i < w - 1; i++) { + switch (dir[i]) { + case DIRECTION_45UP: COPY_MAXIMA( 1, -1, -1, 1); break; + case DIRECTION_45DOWN: COPY_MAXIMA(-1, -1, 1, 1); break; + case DIRECTION_HORIZONTAL: COPY_MAXIMA( 0, -1, 0, 1); break; + case DIRECTION_VERTICAL: COPY_MAXIMA(-1, 0, 1, 0); break; + } + } + } +} + +// Filter to keep all pixels > high, and keep all pixels > low where all surrounding pixels > high +void ff_double_threshold(int low, int high, int w, int h, + uint8_t *dst, int dst_linesize, + const uint8_t *src, int src_linesize) +{ + int i, j; + + for (j = 0; j < h; j++) { + for (i = 0; i < w; i++) { + if (src[i] > high) { + dst[i] = src[i]; + continue; + } + + if (!(!i || i == w - 1 || !j || j == h - 1) && + src[i] > low && + (src[-src_linesize + i-1] > high || + src[-src_linesize + i ] > high || + src[-src_linesize + i+1] > high || + src[ i-1] > high || + src[ i+1] > high || + src[ src_linesize + i-1] > high || + src[ src_linesize + i ] > high || + src[ src_linesize + i+1] > high)) + dst[i] = src[i]; + else + dst[i] = 0; + } + dst += dst_linesize; + src += src_linesize; + } +} + +// Applies gaussian blur, using 5x5 kernels, sigma = 1.4 +void ff_gaussian_blur(int w, int h, + uint8_t *dst, int dst_linesize, + const uint8_t *src, int src_linesize) +{ + int i, j; + + memcpy(dst, src, w); dst += dst_linesize; src += src_linesize; + memcpy(dst, src, w); dst += dst_linesize; src += src_linesize; + for (j = 2; j < h - 2; j++) { + dst[0] = src[0]; + dst[1] = src[1]; + for (i = 2; i < w - 2; i++) { + /* Gaussian mask of size 5x5 with sigma = 1.4 */ + dst[i] = ((src[-2*src_linesize + i-2] + src[2*src_linesize + i-2]) * 2 + + (src[-2*src_linesize + i-1] + src[2*src_linesize + i-1]) * 4 + + (src[-2*src_linesize + i ] + src[2*src_linesize + i ]) * 5 + + (src[-2*src_linesize + i+1] + src[2*src_linesize + i+1]) * 4 + + (src[-2*src_linesize + i+2] + src[2*src_linesize + i+2]) * 2 + + + (src[ -src_linesize + i-2] + src[ src_linesize + i-2]) * 4 + + (src[ -src_linesize + i-1] + src[ src_linesize + i-1]) * 9 + + (src[ -src_linesize + i ] + src[ src_linesize + i ]) * 12 + + (src[ -src_linesize + i+1] + src[ src_linesize + i+1]) * 9 + + (src[ -src_linesize + i+2] + src[ src_linesize + i+2]) * 4 + + + src[i-2] * 5 + + src[i-1] * 12 + + src[i ] * 15 + + src[i+1] * 12 + + src[i+2] * 5) / 159; + } + dst[i ] = src[i ]; + dst[i + 1] = src[i + 1]; + + dst += dst_linesize; + src += src_linesize; + } + memcpy(dst, src, w); dst += dst_linesize; src += src_linesize; + memcpy(dst, src, w); +} diff --git a/libavfilter/edge_common.h b/libavfilter/edge_common.h new file mode 100644 index 0000000000..87c143f2b8 --- /dev/null +++ b/libavfilter/edge_common.h @@ -0,0 +1,107 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * common functions for edge detection + */ + +#ifndef AVFILTER_EDGE_COMMON_H +#define AVFILTER_EDGE_COMMON_H + +#include "avfilter.h" + +/** + * @brief Rounded directions used in av_image_sobel() + */ +enum AVRoundedDirection { + DIRECTION_45UP, + DIRECTION_45DOWN, + DIRECTION_HORIZONTAL, + DIRECTION_VERTICAL, +}; + +/** + * Simple sobel operator to get rounded gradients + * + * @param w the width of the image in pixels + * @param h the height of the image in pixels + * @param dst data pointers to magnitude image + * @param dst_linesize linesizes for the magnitude image + * @param dir data pointers to direction image + * @param dir_linesize linesizes for the direction image + * @param src data pointers to source image + * @param src_linesize linesizes for the source image + */ +void ff_sobel(int w, int h, + uint16_t *dst, int dst_linesize, + int8_t *dir, int dir_linesize, + const uint8_t *src, int src_linesize); + +/** + * Filters rounded gradients to drop all non-maxima pixels in the magnitude image + * Expects gradients generated by av_image_sobel() + * Expects zero's in the destination buffer dst + * + * @param w the width of the image in pixels + * @param h the height of the image in pixels + * @param dst data pointers to magnitude image + * @param dst_linesize linesizes for the magnitude image + * @param dir data pointers to direction image + * @param dir_linesize linesizes for the direction image + * @param src data pointers to source image + * @param src_linesize linesizes for the source image + */ +void ff_non_maximum_suppression(int w, int h, + uint8_t *dst, int dst_linesize, + const int8_t *dir, int dir_linesize, + const uint16_t *src, int src_linesize); + +/** + * Filters all pixels in src to keep all pixels > high, + * and keep all pixels > low where all surrounding pixels > high + * + * @param low the low threshold value + * @param high the hegh threshold value + * @param w the width of the image in pixels + * @param h the height of the image in pixels + * @param dst data pointers to destination image + * @param dst_linesize linesizes for the destination image + * @param src data pointers to source image + * @param src_linesize linesizes for the source image + */ +void ff_double_threshold(int low, int high, int w, int h, + uint8_t *dst, int dst_linesize, + const uint8_t *src, int src_linesize); + +/** + * Applies gaussian blur. + * 5x5 kernels, sigma = 1.4 + * + * @param w the width of the image in pixels + * @param h the height of the image in pixels + * @param dst data pointers to destination image + * @param dst_linesize linesizes for the destination image + * @param src data pointers to source image + * @param src_linesize linesizes for the source image + */ +void ff_gaussian_blur(int w, int h, + uint8_t *dst, int dst_linesize, + const uint8_t *src, int src_linesize); + +#endif diff --git a/libavfilter/vf_edgedetect.c b/libavfilter/vf_edgedetect.c index 3eea34e325..90390ceb3e 100644 --- a/libavfilter/vf_edgedetect.c +++ b/libavfilter/vf_edgedetect.c @@ -32,6 +32,7 @@ #include "formats.h" #include "internal.h" #include "video.h" +#include "edge_common.h" #define PLANE_R 0x4 #define PLANE_G 0x1 @@ -139,176 +140,6 @@ static int config_props(AVFilterLink *inlink) return 0; } -static void gaussian_blur(AVFilterContext *ctx, int w, int h, - uint8_t *dst, int dst_linesize, - const uint8_t *src, int src_linesize) -{ - int i, j; - - memcpy(dst, src, w); dst += dst_linesize; src += src_linesize; - if (h > 1) { - memcpy(dst, src, w); dst += dst_linesize; src += src_linesize; - } - for (j = 2; j < h - 2; j++) { - dst[0] = src[0]; - if (w > 1) - dst[1] = src[1]; - for (i = 2; i < w - 2; i++) { - /* Gaussian mask of size 5x5 with sigma = 1.4 */ - dst[i] = ((src[-2*src_linesize + i-2] + src[2*src_linesize + i-2]) * 2 - + (src[-2*src_linesize + i-1] + src[2*src_linesize + i-1]) * 4 - + (src[-2*src_linesize + i ] + src[2*src_linesize + i ]) * 5 - + (src[-2*src_linesize + i+1] + src[2*src_linesize + i+1]) * 4 - + (src[-2*src_linesize + i+2] + src[2*src_linesize + i+2]) * 2 - - + (src[ -src_linesize + i-2] + src[ src_linesize + i-2]) * 4 - + (src[ -src_linesize + i-1] + src[ src_linesize + i-1]) * 9 - + (src[ -src_linesize + i ] + src[ src_linesize + i ]) * 12 - + (src[ -src_linesize + i+1] + src[ src_linesize + i+1]) * 9 - + (src[ -src_linesize + i+2] + src[ src_linesize + i+2]) * 4 - - + src[i-2] * 5 - + src[i-1] * 12 - + src[i ] * 15 - + src[i+1] * 12 - + src[i+2] * 5) / 159; - } - if (w > 2) - dst[i ] = src[i ]; - if (w > 3) - dst[i + 1] = src[i + 1]; - - dst += dst_linesize; - src += src_linesize; - } - if (h > 2) { - memcpy(dst, src, w); dst += dst_linesize; src += src_linesize; - } - if (h > 3) - memcpy(dst, src, w); -} - -enum { - DIRECTION_45UP, - DIRECTION_45DOWN, - DIRECTION_HORIZONTAL, - DIRECTION_VERTICAL, -}; - -static int get_rounded_direction(int gx, int gy) -{ - /* reference angles: - * tan( pi/8) = sqrt(2)-1 - * tan(3pi/8) = sqrt(2)+1 - * Gy/Gx is the tangent of the angle (theta), so Gy/Gx is compared against - * <ref-angle>, or more simply Gy against <ref-angle>*Gx - * - * Gx and Gy bounds = [-1020;1020], using 16-bit arithmetic: - * round((sqrt(2)-1) * (1<<16)) = 27146 - * round((sqrt(2)+1) * (1<<16)) = 158218 - */ - if (gx) { - int tanpi8gx, tan3pi8gx; - - if (gx < 0) - gx = -gx, gy = -gy; - gy *= (1 << 16); - tanpi8gx = 27146 * gx; - tan3pi8gx = 158218 * gx; - if (gy > -tan3pi8gx && gy < -tanpi8gx) return DIRECTION_45UP; - if (gy > -tanpi8gx && gy < tanpi8gx) return DIRECTION_HORIZONTAL; - if (gy > tanpi8gx && gy < tan3pi8gx) return DIRECTION_45DOWN; - } - return DIRECTION_VERTICAL; -} - -static void sobel(int w, int h, - uint16_t *dst, int dst_linesize, - int8_t *dir, int dir_linesize, - const uint8_t *src, int src_linesize) -{ - int i, j; - - for (j = 1; j < h - 1; j++) { - dst += dst_linesize; - dir += dir_linesize; - src += src_linesize; - for (i = 1; i < w - 1; i++) { - const int gx = - -1*src[-src_linesize + i-1] + 1*src[-src_linesize + i+1] - -2*src[ i-1] + 2*src[ i+1] - -1*src[ src_linesize + i-1] + 1*src[ src_linesize + i+1]; - const int gy = - -1*src[-src_linesize + i-1] + 1*src[ src_linesize + i-1] - -2*src[-src_linesize + i ] + 2*src[ src_linesize + i ] - -1*src[-src_linesize + i+1] + 1*src[ src_linesize + i+1]; - - dst[i] = FFABS(gx) + FFABS(gy); - dir[i] = get_rounded_direction(gx, gy); - } - } -} - -static void non_maximum_suppression(int w, int h, - uint8_t *dst, int dst_linesize, - const int8_t *dir, int dir_linesize, - const uint16_t *src, int src_linesize) -{ - int i, j; - -#define COPY_MAXIMA(ay, ax, by, bx) do { \ - if (src[i] > src[(ay)*src_linesize + i+(ax)] && \ - src[i] > src[(by)*src_linesize + i+(bx)]) \ - dst[i] = av_clip_uint8(src[i]); \ -} while (0) - - for (j = 1; j < h - 1; j++) { - dst += dst_linesize; - dir += dir_linesize; - src += src_linesize; - for (i = 1; i < w - 1; i++) { - switch (dir[i]) { - case DIRECTION_45UP: COPY_MAXIMA( 1, -1, -1, 1); break; - case DIRECTION_45DOWN: COPY_MAXIMA(-1, -1, 1, 1); break; - case DIRECTION_HORIZONTAL: COPY_MAXIMA( 0, -1, 0, 1); break; - case DIRECTION_VERTICAL: COPY_MAXIMA(-1, 0, 1, 0); break; - } - } - } -} - -static void double_threshold(int low, int high, int w, int h, - uint8_t *dst, int dst_linesize, - const uint8_t *src, int src_linesize) -{ - int i, j; - - for (j = 0; j < h; j++) { - for (i = 0; i < w; i++) { - if (src[i] > high) { - dst[i] = src[i]; - continue; - } - - if (!(!i || i == w - 1 || !j || j == h - 1) && - src[i] > low && - (src[-src_linesize + i-1] > high || - src[-src_linesize + i ] > high || - src[-src_linesize + i+1] > high || - src[ i-1] > high || - src[ i+1] > high || - src[ src_linesize + i-1] > high || - src[ src_linesize + i ] > high || - src[ src_linesize + i+1] > high)) - dst[i] = src[i]; - else - dst[i] = 0; - } - dst += dst_linesize; - src += src_linesize; - } -} - static void color_mix(int w, int h, uint8_t *dst, int dst_linesize, const uint8_t *src, int src_linesize) @@ -360,12 +191,12 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) } /* gaussian filter to reduce noise */ - gaussian_blur(ctx, width, height, - tmpbuf, width, - in->data[p], in->linesize[p]); + ff_gaussian_blur(width, height, + tmpbuf, width, + in->data[p], in->linesize[p]); /* compute the 16-bits gradients and directions for the next step */ - sobel(width, height, + ff_sobel(width, height, gradients, width, directions,width, tmpbuf, width); @@ -373,13 +204,13 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) /* non_maximum_suppression() will actually keep & clip what's necessary and * ignore the rest, so we need a clean output buffer */ memset(tmpbuf, 0, width * height); - non_maximum_suppression(width, height, + ff_non_maximum_suppression(width, height, tmpbuf, width, directions,width, gradients, width); /* keep high values, or low values surrounded by high values */ - double_threshold(edgedetect->low_u8, edgedetect->high_u8, + ff_double_threshold(edgedetect->low_u8, edgedetect->high_u8, width, height, out->data[p], out->linesize[p], tmpbuf, width);
Am 24.04.22 um 19:28 schrieb Thilo Borgmann: > Am 23.04.22 um 15:32 schrieb Thilo Borgmann: >> Hi, >> >>>>>> v3 updated to current HEAD. >>>>>> >>>>>> Named blurdetect filter now. >>>>>> Minor fixes on allocation and removed -f option. >>>>>> >>>>> >>>>> Please make this per plane filtering, with default to measure only first >>>>> plane. >>>> >>>> done in v4. >>>> >>>> (Will add Changelog, version.h and fate test once the filter itself looks ok) >>> >>> Ping. >> >> v5 according to IRC comments. > > v6 (1/2 and 2/2) according to IRC comments. LGTM'd there, applying soon. > > Added Changelog, version.h, FATE. v7: removed useless debug garbage. Still applying soon. -Thilo From 2879cfd94358aa977dd7f4314f931fb782750f96 Mon Sep 17 00:00:00 2001 From: Thilo Borgmann <thilo.borgmann@mail.de> Date: Mon, 25 Apr 2022 00:01:51 +0200 Subject: [PATCH v7 2/2] lavfi: Add blurdetect filter --- Changelog | 1 + doc/filters.texi | 52 +++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/version.h | 2 +- libavfilter/vf_blurdetect.c | 395 ++++++++++++++++++++ tests/fate/filter-video.mak | 3 + tests/ref/fate/filter-refcmp-blurdetect-yuv | 10 + 8 files changed, 464 insertions(+), 1 deletion(-) create mode 100644 libavfilter/vf_blurdetect.c create mode 100644 tests/ref/fate/filter-refcmp-blurdetect-yuv diff --git a/Changelog b/Changelog index 7a63e3d1ee..4d467eb741 100644 --- a/Changelog +++ b/Changelog @@ -13,6 +13,7 @@ version 5.1: - pixelize video filter - colormap video filter - colorchart video source filter +- blurdetect filter version 5.0: diff --git a/doc/filters.texi b/doc/filters.texi index c8699b9099..499f3adcd9 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -7997,6 +7997,58 @@ tblend=all_mode=grainextract @subsection Commands This filter supports same @ref{commands} as options. +@anchor{blurdetect} +@section blurdetect + +Determines blurriness of frames without altering the input frames. + +Based on Marziliano, Pina, et al. "A no-reference perceptual blur metric." +Allows for a block-based abbreviation. + +The filter accepts the following options: + +@table @option +@item low +@item high +Set low and high threshold values used by the Canny thresholding +algorithm. + +The high threshold selects the "strong" edge pixels, which are then +connected through 8-connectivity with the "weak" edge pixels selected +by the low threshold. + +@var{low} and @var{high} threshold values must be chosen in the range +[0,1], and @var{low} should be lesser or equal to @var{high}. + +Default value for @var{low} is @code{20/255}, and default value for @var{high} +is @code{50/255}. + +@item radius +Define the radius to search around an edge pixel for local maxima. + +@item block_pct +Determine blurriness only for the most significant blocks, given in percentage. + +@item block_width +Determine blurriness for blocks of width @var{block_width}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_height}. + +@item block_height +Determine blurriness for blocks of height @var{block_height}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_width}. + +@item planes +Set planes to filter. Default is first only. +@end table + +@subsection Examples + +@itemize +@item +Determine blur for 80% of most significant 32x32 blocks: +@example +blurdetect=block_width=32:block_height=32:block_pct=80 +@end example +@end itemize + @section bm3d Denoise frames using Block-Matching 3D algorithm. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 38ca379e5a..1db097b464 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -195,6 +195,7 @@ OBJS-$(CONFIG_BLACKDETECT_FILTER) += vf_blackdetect.o OBJS-$(CONFIG_BLACKFRAME_FILTER) += vf_blackframe.o OBJS-$(CONFIG_BLEND_FILTER) += vf_blend.o framesync.o OBJS-$(CONFIG_BLEND_VULKAN_FILTER) += vf_blend_vulkan.o framesync.o vulkan.o vulkan_filter.o +OBJS-$(CONFIG_BLURDETECT_FILTER) += vf_blurdetect.o edge_common.o OBJS-$(CONFIG_BM3D_FILTER) += vf_bm3d.o framesync.o OBJS-$(CONFIG_BOXBLUR_FILTER) += vf_boxblur.o boxblur.o OBJS-$(CONFIG_BOXBLUR_OPENCL_FILTER) += vf_avgblur_opencl.o opencl.o \ diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 36fa3ae8d7..2ad523fd0f 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -183,6 +183,7 @@ extern const AVFilter ff_vf_blackdetect; extern const AVFilter ff_vf_blackframe; extern const AVFilter ff_vf_blend; extern const AVFilter ff_vf_blend_vulkan; +extern const AVFilter ff_vf_blurdetect; extern const AVFilter ff_vf_bm3d; extern const AVFilter ff_vf_boxblur; extern const AVFilter ff_vf_boxblur_opencl; diff --git a/libavfilter/version.h b/libavfilter/version.h index 9add1658e5..8f1b16969a 100644 --- a/libavfilter/version.h +++ b/libavfilter/version.h @@ -31,7 +31,7 @@ #include "version_major.h" -#define LIBAVFILTER_VERSION_MINOR 36 +#define LIBAVFILTER_VERSION_MINOR 37 #define LIBAVFILTER_VERSION_MICRO 100 diff --git a/libavfilter/vf_blurdetect.c b/libavfilter/vf_blurdetect.c new file mode 100644 index 0000000000..7c0f5ce68a --- /dev/null +++ b/libavfilter/vf_blurdetect.c @@ -0,0 +1,395 @@ +/* + * Copyright (c) 2021 Thilo Borgmann <thilo.borgmann _at_ mail.de> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * No-reference blurdetect filter + * + * Implementing: + * Marziliano, Pina, et al. "A no-reference perceptual blur metric." Proceedings. + * International conference on image processing. Vol. 3. IEEE, 2002. + * https://infoscience.epfl.ch/record/111802/files/14%20A%20no-reference%20perceptual%20blur%20metric.pdf + * + * @author Thilo Borgmann <thilo.borgmann _at_ mail.de> + */ + +#include "libavutil/avassert.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "libavutil/pixelutils.h" +#include "libavutil/motion_vector.h" +#include "libavutil/qsort.h" +#include "avfilter.h" +#include "formats.h" +#include "internal.h" +#include "video.h" +#include "edge_common.h" + +static int comp(const float *a,const float *b) +{ + return FFDIFFSIGN(*a, *b); +} + +typedef struct BLRContext { + const AVClass *class; + + int hsub, vsub; + int nb_planes; + + float low, high; + uint8_t low_u8, high_u8; + int radius; // radius during local maxima detection + int block_pct; // percentage of "sharpest" blocks in the image to use for bluriness calculation + int block_width; // width for block abbreviation + int block_height; // height for block abbreviation + int planes; // number of planes to filter + + double blur_total; + uint64_t nb_frames; + + float *blks; + uint8_t *filterbuf; + uint8_t *tmpbuf; + uint16_t *gradients; + char *directions; +} BLRContext; + +#define OFFSET(x) offsetof(BLRContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM +static const AVOption blurdetect_options[] = { + { "high", "set high threshold", OFFSET(high), AV_OPT_TYPE_FLOAT, {.dbl=30/255.}, 0, 1, FLAGS }, + { "low", "set low threshold", OFFSET(low), AV_OPT_TYPE_FLOAT, {.dbl=15/255.}, 0, 1, FLAGS }, + { "radius", "search radius for maxima detection", OFFSET(radius), AV_OPT_TYPE_INT, {.i64=50}, 1, 100, FLAGS }, + { "block_pct", "block pooling threshold when calculating blurriness", OFFSET(block_pct), AV_OPT_TYPE_INT, {.i64=80}, 1, 100, FLAGS }, + { "block_width", "block size for block-based abbreviation of blurriness", OFFSET(block_width), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS }, + { "block_height", "block size for block-based abbreviation of blurriness", OFFSET(block_height), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS }, + { "planes", "set planes to filter", OFFSET(planes), AV_OPT_TYPE_INT, {.i64=1}, 0, 15, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(blurdetect); + +static av_cold int blurdetect_init(AVFilterContext *ctx) +{ + BLRContext *s = ctx->priv; + + s->low_u8 = s->low * 255. + .5; + s->high_u8 = s->high * 255. + .5; + + return 0; +} + +static int blurdetect_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + BLRContext *s = ctx->priv; + const int bufsize = inlink->w * inlink->h; + const AVPixFmtDescriptor *pix_desc; + + pix_desc = av_pix_fmt_desc_get(inlink->format); + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + s->nb_planes = av_pix_fmt_count_planes(inlink->format); + + if (s->block_width < 1 || s->block_height < 1) { + s->block_width = inlink->w; + s->block_height = inlink->h; + } + + s->tmpbuf = av_malloc(bufsize); + s->filterbuf = av_malloc(bufsize); + s->gradients = av_calloc(bufsize, sizeof(*s->gradients)); + s->directions = av_malloc(bufsize); + s->blks = av_calloc((inlink->w / s->block_width) * (inlink->h / s->block_height), + sizeof(*s->blks)); + + if (!s->tmpbuf || !s->filterbuf || !s->gradients || !s->directions || !s->blks) + return AVERROR(ENOMEM); + + return 0; +} + +// edge width is defined as the distance between surrounding maxima of the edge pixel +static float edge_width(BLRContext *blr, int i, int j, int8_t dir, int w, int h, + int edge, const uint8_t *src, int src_linesize) +{ + float width = 0; + int dX, dY; + int sign; + int tmp; + int p1; + int p2; + int k, x, y; + int edge1; + int edge2; + float luma1 = 0.0; // average luma difference per edge pixel + float luma2 = 0.0; + int radius = blr->radius; + + switch(dir) { + case DIRECTION_HORIZONTAL: dX = 1; dY = 0; break; + case DIRECTION_VERTICAL: dX = 0; dY = 1; break; + case DIRECTION_45UP: dX = 1; dY = -1; break; + case DIRECTION_45DOWN: dX = 1; dY = 1; break; + } + + // determines if search in direction dX/dY is looking for a maximum or minimum + sign = src[j * src_linesize + i] > src[(j - dY) * src_linesize + i - dX] ? 1 : -1; + + // search in -(dX/dY) direction + for (k = 0; k < radius; k++) { + x = i - k*dX; + y = j - k*dY; + p1 = y * src_linesize + x; + x -= dX; + y -= dY; + p2 = y * src_linesize + x; + if (x < 0 || x >= w || y < 0 || y >= h) + return 0; + + tmp = (src[p1] - src[p2]) * sign; + + if (tmp <= 0) // local maximum found + break; + + luma1 += tmp; + } + if (k > 0) luma1 /= k; + edge1 = k; + width += k; + + // search in +(dX/dY) direction + for (k = 0; k < radius; k++) { + x = i + k * dX; + y = j + k * dY; + p1 = y * src_linesize + x; + x += dX; + y += dY; + p2 = y * src_linesize + x; + if (x < 0 || x >= w || y < 0 || y >= h) + return 0; + + tmp = (src[p1] - src[p2]) * sign; + + if (tmp >= 0) // local maximum found + break; + + luma2 -= tmp; + } + if (k > 0) luma2 /= k; + edge2 = k; + width += k; + + // for 45 degree directions approximate edge width in pixel units: 0.7 ~= sqrt(2)/2 + if (dir == DIRECTION_45UP || dir == DIRECTION_45DOWN) + width *= 0.7; + + return width; +} + +static float calculate_blur(BLRContext *s, int w, int h, int hsub, int vsub, + uint8_t* dir, int dir_linesize, + uint8_t* dst, int dst_linesize, + uint8_t* src, int src_linesize) +{ + float total_width = 0.0; + int block_count; + double block_total_width; + + int i, j; + int blkcnt = 0; + + float *blks = s->blks; + float block_pool_threshold = s->block_pct / 100.0; + + int block_width = AV_CEIL_RSHIFT(s->block_width, hsub); + int block_height = AV_CEIL_RSHIFT(s->block_height, vsub); + int brows = h / block_height; + int bcols = w / block_width; + + for (int blkj = 0; blkj < brows; blkj++) { + for (int blki = 0; blki < bcols; blki++) { + block_total_width = 0.0; + block_count = 0; + for (int inj = 0; inj < block_height; inj++) { + for (int ini = 0; ini < block_width; ini++) { + i = blki * block_width + ini; + j = blkj * block_height + inj; + + if (dst[j * dst_linesize + i] > 0) { + float width = edge_width(s, i, j, dir[j*dir_linesize+i], + w, h, dst[j*dst_linesize+i], + src, src_linesize); + if (width > 0.001) { // throw away zeros + block_count++; + block_total_width += width; + } + } + } + } + // if not enough edge pixels in a block, consider it smooth + if (block_total_width >= 2) { + blks[blkcnt] = block_total_width / block_count; + blkcnt++; + } + } + } + + // simple block pooling by sorting and keeping the sharper blocks + AV_QSORT(blks, blkcnt, float, comp); + blkcnt = ceil(blkcnt * block_pool_threshold); + for (int i = 0; i < blkcnt; i++) { + total_width += blks[i]; + } + + return total_width / blkcnt; +} + +static void set_meta(AVDictionary **metadata, const char *key, float d) +{ + char value[128]; + snprintf(value, sizeof(value), "%f", d); + av_dict_set(metadata, key, value, 0); +} + +static int blurdetect_filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx = inlink->dst; + BLRContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + + const int inw = inlink->w; + const int inh = inlink->h; + + uint8_t *tmpbuf = s->tmpbuf; + uint8_t *filterbuf = s->filterbuf; + uint16_t *gradients = s->gradients; + int8_t *directions = s->directions; + + float blur = 0.0f; + int nplanes = 0; + AVDictionary **metadata; + metadata = &in->metadata; + + for (int plane = 0; plane < s->nb_planes; plane++) { + int hsub = plane == 1 || plane == 2 ? s->hsub : 0; + int vsub = plane == 1 || plane == 2 ? s->vsub : 0; + int w = AV_CEIL_RSHIFT(inw, hsub); + int h = AV_CEIL_RSHIFT(inh, vsub); + + if (!((1 << plane) & s->planes)) + continue; + + nplanes++; + + // gaussian filter to reduce noise + ff_gaussian_blur(w, h, + filterbuf, w, + in->data[plane], in->linesize[plane]); + + // compute the 16-bits gradients and directions for the next step + ff_sobel(w, h, gradients, w, directions, w, filterbuf, w); + + // non_maximum_suppression() will actually keep & clip what's necessary and + // ignore the rest, so we need a clean output buffer + memset(tmpbuf, 0, inw * inh); + ff_non_maximum_suppression(w, h, tmpbuf, w, directions, w, gradients, w); + + + // keep high values, or low values surrounded by high values + ff_double_threshold(s->low_u8, s->high_u8, w, h, + tmpbuf, w, tmpbuf, w); + + blur += calculate_blur(s, w, h, hsub, vsub, directions, w, + tmpbuf, w, filterbuf, w); + } + + if (nplanes) + blur /= nplanes; + + s->blur_total += blur; + + // write stats + av_log(ctx, AV_LOG_VERBOSE, "blur: %.7f\n", blur); + + set_meta(metadata, "lavfi.blur", blur); + + s->nb_frames = inlink->frame_count_in; + + return ff_filter_frame(outlink, in); +} + +static av_cold void blurdetect_uninit(AVFilterContext *ctx) +{ + BLRContext *s = ctx->priv; + + if (s->nb_frames > 0) { + av_log(ctx, AV_LOG_INFO, "blur mean: %.7f\n", + s->blur_total / s->nb_frames); + } + + av_freep(&s->tmpbuf); + av_freep(&s->filterbuf); + av_freep(&s->gradients); + av_freep(&s->directions); + av_freep(&s->blks); +} + +static const enum AVPixelFormat pix_fmts[] = { + AV_PIX_FMT_GRAY8, + AV_PIX_FMT_GBRP, AV_PIX_FMT_GBRAP, + AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV420P, + AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUV440P, + AV_PIX_FMT_YUV411P, AV_PIX_FMT_YUV410P, + AV_PIX_FMT_YUVJ440P, AV_PIX_FMT_YUVJ411P, AV_PIX_FMT_YUVJ420P, + AV_PIX_FMT_YUVJ422P, AV_PIX_FMT_YUVJ444P, + AV_PIX_FMT_YUVA444P, AV_PIX_FMT_YUVA422P, AV_PIX_FMT_YUVA420P, + AV_PIX_FMT_NONE +}; + +static const AVFilterPad blurdetect_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = blurdetect_config_input, + .filter_frame = blurdetect_filter_frame, + }, +}; + +static const AVFilterPad blurdetect_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + }, +}; + +const AVFilter ff_vf_blurdetect = { + .name = "blurdetect", + .description = NULL_IF_CONFIG_SMALL("Blurdetect filter."), + .priv_size = sizeof(BLRContext), + .init = blurdetect_init, + .uninit = blurdetect_uninit, + FILTER_PIXFMTS_ARRAY(pix_fmts), + FILTER_INPUTS(blurdetect_inputs), + FILTER_OUTPUTS(blurdetect_outputs), + .priv_class = &blurdetect_class, + .flags = AVFILTER_FLAG_METADATA_ONLY, +}; + diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak index 6c0d8df032..68f4c084f8 100644 --- a/tests/fate/filter-video.mak +++ b/tests/fate/filter-video.mak @@ -867,6 +867,9 @@ fate-filter-meta-4560-rotate0: CMD = framecrc -auto_conversion_filters -flags +b REFCMP_DEPS = FFMPEG LAVFI_INDEV TESTSRC2_FILTER AVGBLUR_FILTER METADATA_FILTER +FATE_FILTER-$(call ALLYES, $(REFCMP_DEPS) BLURDETECT_FILTER) += fate-filter-refcmp-blurdetect-yuv +fate-filter-refcmp-blurdetect-yuv: CMD = cmp_metadata blurdetect yuv420p 0.015 + FATE_FILTER-$(call ALLYES, $(REFCMP_DEPS) PSNR_FILTER) += fate-filter-refcmp-psnr-rgb fate-filter-refcmp-psnr-rgb: CMD = refcmp_metadata psnr rgb24 0.002 diff --git a/tests/ref/fate/filter-refcmp-blurdetect-yuv b/tests/ref/fate/filter-refcmp-blurdetect-yuv new file mode 100644 index 0000000000..a64a9a07ba --- /dev/null +++ b/tests/ref/fate/filter-refcmp-blurdetect-yuv @@ -0,0 +1,10 @@ +frame:0 pts:0 pts_time:0 +lavfi.blur=4.480640 +frame:1 pts:1 pts_time:1 +lavfi.blur=4.569398 +frame:2 pts:2 pts_time:2 +lavfi.blur=4.609386 +frame:3 pts:3 pts_time:3 +lavfi.blur=4.357340 +frame:4 pts:4 pts_time:4 +lavfi.blur=4.392871
Am 25.04.22 um 00:03 schrieb Thilo Borgmann: > Am 24.04.22 um 19:28 schrieb Thilo Borgmann: >> Am 23.04.22 um 15:32 schrieb Thilo Borgmann: >>> Hi, >>> >>>>>>> v3 updated to current HEAD. >>>>>>> >>>>>>> Named blurdetect filter now. >>>>>>> Minor fixes on allocation and removed -f option. >>>>>>> >>>>>> >>>>>> Please make this per plane filtering, with default to measure only first >>>>>> plane. >>>>> >>>>> done in v4. >>>>> >>>>> (Will add Changelog, version.h and fate test once the filter itself looks ok) >>>> >>>> Ping. >>> >>> v5 according to IRC comments. >> >> v6 (1/2 and 2/2) according to IRC comments. LGTM'd there, applying soon. >> >> Added Changelog, version.h, FATE. > > v7: removed useless debug garbage. Still applying soon. v8: and now even with adapted FATE. -Thilo From 72f1631e8994e15eac58c5c9a8c0fcf14ef627ad Mon Sep 17 00:00:00 2001 From: Thilo Borgmann <thilo.borgmann@mail.de> Date: Mon, 25 Apr 2022 00:07:04 +0200 Subject: [PATCH v8 2/2] lavfi: Add blurdetect filter --- Changelog | 1 + doc/filters.texi | 52 +++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/version.h | 2 +- libavfilter/vf_blurdetect.c | 395 ++++++++++++++++++++ tests/fate/filter-video.mak | 3 + tests/ref/fate/filter-refcmp-blurdetect-yuv | 10 + 8 files changed, 464 insertions(+), 1 deletion(-) create mode 100644 libavfilter/vf_blurdetect.c create mode 100644 tests/ref/fate/filter-refcmp-blurdetect-yuv diff --git a/Changelog b/Changelog index 7a63e3d1ee..4d467eb741 100644 --- a/Changelog +++ b/Changelog @@ -13,6 +13,7 @@ version 5.1: - pixelize video filter - colormap video filter - colorchart video source filter +- blurdetect filter version 5.0: diff --git a/doc/filters.texi b/doc/filters.texi index c8699b9099..499f3adcd9 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -7997,6 +7997,58 @@ tblend=all_mode=grainextract @subsection Commands This filter supports same @ref{commands} as options. +@anchor{blurdetect} +@section blurdetect + +Determines blurriness of frames without altering the input frames. + +Based on Marziliano, Pina, et al. "A no-reference perceptual blur metric." +Allows for a block-based abbreviation. + +The filter accepts the following options: + +@table @option +@item low +@item high +Set low and high threshold values used by the Canny thresholding +algorithm. + +The high threshold selects the "strong" edge pixels, which are then +connected through 8-connectivity with the "weak" edge pixels selected +by the low threshold. + +@var{low} and @var{high} threshold values must be chosen in the range +[0,1], and @var{low} should be lesser or equal to @var{high}. + +Default value for @var{low} is @code{20/255}, and default value for @var{high} +is @code{50/255}. + +@item radius +Define the radius to search around an edge pixel for local maxima. + +@item block_pct +Determine blurriness only for the most significant blocks, given in percentage. + +@item block_width +Determine blurriness for blocks of width @var{block_width}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_height}. + +@item block_height +Determine blurriness for blocks of height @var{block_height}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_width}. + +@item planes +Set planes to filter. Default is first only. +@end table + +@subsection Examples + +@itemize +@item +Determine blur for 80% of most significant 32x32 blocks: +@example +blurdetect=block_width=32:block_height=32:block_pct=80 +@end example +@end itemize + @section bm3d Denoise frames using Block-Matching 3D algorithm. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 38ca379e5a..1db097b464 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -195,6 +195,7 @@ OBJS-$(CONFIG_BLACKDETECT_FILTER) += vf_blackdetect.o OBJS-$(CONFIG_BLACKFRAME_FILTER) += vf_blackframe.o OBJS-$(CONFIG_BLEND_FILTER) += vf_blend.o framesync.o OBJS-$(CONFIG_BLEND_VULKAN_FILTER) += vf_blend_vulkan.o framesync.o vulkan.o vulkan_filter.o +OBJS-$(CONFIG_BLURDETECT_FILTER) += vf_blurdetect.o edge_common.o OBJS-$(CONFIG_BM3D_FILTER) += vf_bm3d.o framesync.o OBJS-$(CONFIG_BOXBLUR_FILTER) += vf_boxblur.o boxblur.o OBJS-$(CONFIG_BOXBLUR_OPENCL_FILTER) += vf_avgblur_opencl.o opencl.o \ diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 36fa3ae8d7..2ad523fd0f 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -183,6 +183,7 @@ extern const AVFilter ff_vf_blackdetect; extern const AVFilter ff_vf_blackframe; extern const AVFilter ff_vf_blend; extern const AVFilter ff_vf_blend_vulkan; +extern const AVFilter ff_vf_blurdetect; extern const AVFilter ff_vf_bm3d; extern const AVFilter ff_vf_boxblur; extern const AVFilter ff_vf_boxblur_opencl; diff --git a/libavfilter/version.h b/libavfilter/version.h index 9add1658e5..8f1b16969a 100644 --- a/libavfilter/version.h +++ b/libavfilter/version.h @@ -31,7 +31,7 @@ #include "version_major.h" -#define LIBAVFILTER_VERSION_MINOR 36 +#define LIBAVFILTER_VERSION_MINOR 37 #define LIBAVFILTER_VERSION_MICRO 100 diff --git a/libavfilter/vf_blurdetect.c b/libavfilter/vf_blurdetect.c new file mode 100644 index 0000000000..7c0f5ce68a --- /dev/null +++ b/libavfilter/vf_blurdetect.c @@ -0,0 +1,395 @@ +/* + * Copyright (c) 2021 Thilo Borgmann <thilo.borgmann _at_ mail.de> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * No-reference blurdetect filter + * + * Implementing: + * Marziliano, Pina, et al. "A no-reference perceptual blur metric." Proceedings. + * International conference on image processing. Vol. 3. IEEE, 2002. + * https://infoscience.epfl.ch/record/111802/files/14%20A%20no-reference%20perceptual%20blur%20metric.pdf + * + * @author Thilo Borgmann <thilo.borgmann _at_ mail.de> + */ + +#include "libavutil/avassert.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "libavutil/pixelutils.h" +#include "libavutil/motion_vector.h" +#include "libavutil/qsort.h" +#include "avfilter.h" +#include "formats.h" +#include "internal.h" +#include "video.h" +#include "edge_common.h" + +static int comp(const float *a,const float *b) +{ + return FFDIFFSIGN(*a, *b); +} + +typedef struct BLRContext { + const AVClass *class; + + int hsub, vsub; + int nb_planes; + + float low, high; + uint8_t low_u8, high_u8; + int radius; // radius during local maxima detection + int block_pct; // percentage of "sharpest" blocks in the image to use for bluriness calculation + int block_width; // width for block abbreviation + int block_height; // height for block abbreviation + int planes; // number of planes to filter + + double blur_total; + uint64_t nb_frames; + + float *blks; + uint8_t *filterbuf; + uint8_t *tmpbuf; + uint16_t *gradients; + char *directions; +} BLRContext; + +#define OFFSET(x) offsetof(BLRContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM +static const AVOption blurdetect_options[] = { + { "high", "set high threshold", OFFSET(high), AV_OPT_TYPE_FLOAT, {.dbl=30/255.}, 0, 1, FLAGS }, + { "low", "set low threshold", OFFSET(low), AV_OPT_TYPE_FLOAT, {.dbl=15/255.}, 0, 1, FLAGS }, + { "radius", "search radius for maxima detection", OFFSET(radius), AV_OPT_TYPE_INT, {.i64=50}, 1, 100, FLAGS }, + { "block_pct", "block pooling threshold when calculating blurriness", OFFSET(block_pct), AV_OPT_TYPE_INT, {.i64=80}, 1, 100, FLAGS }, + { "block_width", "block size for block-based abbreviation of blurriness", OFFSET(block_width), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS }, + { "block_height", "block size for block-based abbreviation of blurriness", OFFSET(block_height), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS }, + { "planes", "set planes to filter", OFFSET(planes), AV_OPT_TYPE_INT, {.i64=1}, 0, 15, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(blurdetect); + +static av_cold int blurdetect_init(AVFilterContext *ctx) +{ + BLRContext *s = ctx->priv; + + s->low_u8 = s->low * 255. + .5; + s->high_u8 = s->high * 255. + .5; + + return 0; +} + +static int blurdetect_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + BLRContext *s = ctx->priv; + const int bufsize = inlink->w * inlink->h; + const AVPixFmtDescriptor *pix_desc; + + pix_desc = av_pix_fmt_desc_get(inlink->format); + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + s->nb_planes = av_pix_fmt_count_planes(inlink->format); + + if (s->block_width < 1 || s->block_height < 1) { + s->block_width = inlink->w; + s->block_height = inlink->h; + } + + s->tmpbuf = av_malloc(bufsize); + s->filterbuf = av_malloc(bufsize); + s->gradients = av_calloc(bufsize, sizeof(*s->gradients)); + s->directions = av_malloc(bufsize); + s->blks = av_calloc((inlink->w / s->block_width) * (inlink->h / s->block_height), + sizeof(*s->blks)); + + if (!s->tmpbuf || !s->filterbuf || !s->gradients || !s->directions || !s->blks) + return AVERROR(ENOMEM); + + return 0; +} + +// edge width is defined as the distance between surrounding maxima of the edge pixel +static float edge_width(BLRContext *blr, int i, int j, int8_t dir, int w, int h, + int edge, const uint8_t *src, int src_linesize) +{ + float width = 0; + int dX, dY; + int sign; + int tmp; + int p1; + int p2; + int k, x, y; + int edge1; + int edge2; + float luma1 = 0.0; // average luma difference per edge pixel + float luma2 = 0.0; + int radius = blr->radius; + + switch(dir) { + case DIRECTION_HORIZONTAL: dX = 1; dY = 0; break; + case DIRECTION_VERTICAL: dX = 0; dY = 1; break; + case DIRECTION_45UP: dX = 1; dY = -1; break; + case DIRECTION_45DOWN: dX = 1; dY = 1; break; + } + + // determines if search in direction dX/dY is looking for a maximum or minimum + sign = src[j * src_linesize + i] > src[(j - dY) * src_linesize + i - dX] ? 1 : -1; + + // search in -(dX/dY) direction + for (k = 0; k < radius; k++) { + x = i - k*dX; + y = j - k*dY; + p1 = y * src_linesize + x; + x -= dX; + y -= dY; + p2 = y * src_linesize + x; + if (x < 0 || x >= w || y < 0 || y >= h) + return 0; + + tmp = (src[p1] - src[p2]) * sign; + + if (tmp <= 0) // local maximum found + break; + + luma1 += tmp; + } + if (k > 0) luma1 /= k; + edge1 = k; + width += k; + + // search in +(dX/dY) direction + for (k = 0; k < radius; k++) { + x = i + k * dX; + y = j + k * dY; + p1 = y * src_linesize + x; + x += dX; + y += dY; + p2 = y * src_linesize + x; + if (x < 0 || x >= w || y < 0 || y >= h) + return 0; + + tmp = (src[p1] - src[p2]) * sign; + + if (tmp >= 0) // local maximum found + break; + + luma2 -= tmp; + } + if (k > 0) luma2 /= k; + edge2 = k; + width += k; + + // for 45 degree directions approximate edge width in pixel units: 0.7 ~= sqrt(2)/2 + if (dir == DIRECTION_45UP || dir == DIRECTION_45DOWN) + width *= 0.7; + + return width; +} + +static float calculate_blur(BLRContext *s, int w, int h, int hsub, int vsub, + uint8_t* dir, int dir_linesize, + uint8_t* dst, int dst_linesize, + uint8_t* src, int src_linesize) +{ + float total_width = 0.0; + int block_count; + double block_total_width; + + int i, j; + int blkcnt = 0; + + float *blks = s->blks; + float block_pool_threshold = s->block_pct / 100.0; + + int block_width = AV_CEIL_RSHIFT(s->block_width, hsub); + int block_height = AV_CEIL_RSHIFT(s->block_height, vsub); + int brows = h / block_height; + int bcols = w / block_width; + + for (int blkj = 0; blkj < brows; blkj++) { + for (int blki = 0; blki < bcols; blki++) { + block_total_width = 0.0; + block_count = 0; + for (int inj = 0; inj < block_height; inj++) { + for (int ini = 0; ini < block_width; ini++) { + i = blki * block_width + ini; + j = blkj * block_height + inj; + + if (dst[j * dst_linesize + i] > 0) { + float width = edge_width(s, i, j, dir[j*dir_linesize+i], + w, h, dst[j*dst_linesize+i], + src, src_linesize); + if (width > 0.001) { // throw away zeros + block_count++; + block_total_width += width; + } + } + } + } + // if not enough edge pixels in a block, consider it smooth + if (block_total_width >= 2) { + blks[blkcnt] = block_total_width / block_count; + blkcnt++; + } + } + } + + // simple block pooling by sorting and keeping the sharper blocks + AV_QSORT(blks, blkcnt, float, comp); + blkcnt = ceil(blkcnt * block_pool_threshold); + for (int i = 0; i < blkcnt; i++) { + total_width += blks[i]; + } + + return total_width / blkcnt; +} + +static void set_meta(AVDictionary **metadata, const char *key, float d) +{ + char value[128]; + snprintf(value, sizeof(value), "%f", d); + av_dict_set(metadata, key, value, 0); +} + +static int blurdetect_filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx = inlink->dst; + BLRContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + + const int inw = inlink->w; + const int inh = inlink->h; + + uint8_t *tmpbuf = s->tmpbuf; + uint8_t *filterbuf = s->filterbuf; + uint16_t *gradients = s->gradients; + int8_t *directions = s->directions; + + float blur = 0.0f; + int nplanes = 0; + AVDictionary **metadata; + metadata = &in->metadata; + + for (int plane = 0; plane < s->nb_planes; plane++) { + int hsub = plane == 1 || plane == 2 ? s->hsub : 0; + int vsub = plane == 1 || plane == 2 ? s->vsub : 0; + int w = AV_CEIL_RSHIFT(inw, hsub); + int h = AV_CEIL_RSHIFT(inh, vsub); + + if (!((1 << plane) & s->planes)) + continue; + + nplanes++; + + // gaussian filter to reduce noise + ff_gaussian_blur(w, h, + filterbuf, w, + in->data[plane], in->linesize[plane]); + + // compute the 16-bits gradients and directions for the next step + ff_sobel(w, h, gradients, w, directions, w, filterbuf, w); + + // non_maximum_suppression() will actually keep & clip what's necessary and + // ignore the rest, so we need a clean output buffer + memset(tmpbuf, 0, inw * inh); + ff_non_maximum_suppression(w, h, tmpbuf, w, directions, w, gradients, w); + + + // keep high values, or low values surrounded by high values + ff_double_threshold(s->low_u8, s->high_u8, w, h, + tmpbuf, w, tmpbuf, w); + + blur += calculate_blur(s, w, h, hsub, vsub, directions, w, + tmpbuf, w, filterbuf, w); + } + + if (nplanes) + blur /= nplanes; + + s->blur_total += blur; + + // write stats + av_log(ctx, AV_LOG_VERBOSE, "blur: %.7f\n", blur); + + set_meta(metadata, "lavfi.blur", blur); + + s->nb_frames = inlink->frame_count_in; + + return ff_filter_frame(outlink, in); +} + +static av_cold void blurdetect_uninit(AVFilterContext *ctx) +{ + BLRContext *s = ctx->priv; + + if (s->nb_frames > 0) { + av_log(ctx, AV_LOG_INFO, "blur mean: %.7f\n", + s->blur_total / s->nb_frames); + } + + av_freep(&s->tmpbuf); + av_freep(&s->filterbuf); + av_freep(&s->gradients); + av_freep(&s->directions); + av_freep(&s->blks); +} + +static const enum AVPixelFormat pix_fmts[] = { + AV_PIX_FMT_GRAY8, + AV_PIX_FMT_GBRP, AV_PIX_FMT_GBRAP, + AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV420P, + AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUV440P, + AV_PIX_FMT_YUV411P, AV_PIX_FMT_YUV410P, + AV_PIX_FMT_YUVJ440P, AV_PIX_FMT_YUVJ411P, AV_PIX_FMT_YUVJ420P, + AV_PIX_FMT_YUVJ422P, AV_PIX_FMT_YUVJ444P, + AV_PIX_FMT_YUVA444P, AV_PIX_FMT_YUVA422P, AV_PIX_FMT_YUVA420P, + AV_PIX_FMT_NONE +}; + +static const AVFilterPad blurdetect_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = blurdetect_config_input, + .filter_frame = blurdetect_filter_frame, + }, +}; + +static const AVFilterPad blurdetect_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + }, +}; + +const AVFilter ff_vf_blurdetect = { + .name = "blurdetect", + .description = NULL_IF_CONFIG_SMALL("Blurdetect filter."), + .priv_size = sizeof(BLRContext), + .init = blurdetect_init, + .uninit = blurdetect_uninit, + FILTER_PIXFMTS_ARRAY(pix_fmts), + FILTER_INPUTS(blurdetect_inputs), + FILTER_OUTPUTS(blurdetect_outputs), + .priv_class = &blurdetect_class, + .flags = AVFILTER_FLAG_METADATA_ONLY, +}; + diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak index 6c0d8df032..68f4c084f8 100644 --- a/tests/fate/filter-video.mak +++ b/tests/fate/filter-video.mak @@ -867,6 +867,9 @@ fate-filter-meta-4560-rotate0: CMD = framecrc -auto_conversion_filters -flags +b REFCMP_DEPS = FFMPEG LAVFI_INDEV TESTSRC2_FILTER AVGBLUR_FILTER METADATA_FILTER +FATE_FILTER-$(call ALLYES, $(REFCMP_DEPS) BLURDETECT_FILTER) += fate-filter-refcmp-blurdetect-yuv +fate-filter-refcmp-blurdetect-yuv: CMD = cmp_metadata blurdetect yuv420p 0.015 + FATE_FILTER-$(call ALLYES, $(REFCMP_DEPS) PSNR_FILTER) += fate-filter-refcmp-psnr-rgb fate-filter-refcmp-psnr-rgb: CMD = refcmp_metadata psnr rgb24 0.002 diff --git a/tests/ref/fate/filter-refcmp-blurdetect-yuv b/tests/ref/fate/filter-refcmp-blurdetect-yuv new file mode 100644 index 0000000000..426321db0c --- /dev/null +++ b/tests/ref/fate/filter-refcmp-blurdetect-yuv @@ -0,0 +1,10 @@ +frame:0 pts:0 pts_time:0 +lavfi.blur=4.499666 +frame:1 pts:1 pts_time:1 +lavfi.blur=4.677492 +frame:2 pts:2 pts_time:2 +lavfi.blur=4.735711 +frame:3 pts:3 pts_time:3 +lavfi.blur=4.532343 +frame:4 pts:4 pts_time:4 +lavfi.blur=4.532660
Am 25.04.22 um 00:08 schrieb Thilo Borgmann: > Am 25.04.22 um 00:03 schrieb Thilo Borgmann: >> Am 24.04.22 um 19:28 schrieb Thilo Borgmann: >>> Am 23.04.22 um 15:32 schrieb Thilo Borgmann: >>>> Hi, >>>> >>>>>>>> v3 updated to current HEAD. >>>>>>>> >>>>>>>> Named blurdetect filter now. >>>>>>>> Minor fixes on allocation and removed -f option. >>>>>>>> >>>>>>> >>>>>>> Please make this per plane filtering, with default to measure only first >>>>>>> plane. >>>>>> >>>>>> done in v4. >>>>>> >>>>>> (Will add Changelog, version.h and fate test once the filter itself looks ok) >>>>> >>>>> Ping. >>>> >>>> v5 according to IRC comments. >>> >>> v6 (1/2 and 2/2) according to IRC comments. LGTM'd there, applying soon. >>> >>> Added Changelog, version.h, FATE. >> >> v7: removed useless debug garbage. Still applying soon. > > v8: and now even with adapted FATE. Pushed, thanks! -Thilo
diff --git a/doc/filters.texi b/doc/filters.texi index b4f888c14d..d066041e21 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -7712,6 +7712,61 @@ tblend=all_mode=grainextract @subsection Commands This filter supports same @ref{commands} as options. +@anchor{blurriness} +@section blurriness + +Determines blurriness of frames without altering the input frames. + +Based on Marziliano, Pina, et al. "A no-reference perceptual blur metric." +Allows for a block-based abbreviation. + +The filter accepts the following options: + +@table @option +@item low +@item high +Set low and high threshold values used by the Canny thresholding +algorithm. + +The high threshold selects the "strong" edge pixels, which are then +connected through 8-connectivity with the "weak" edge pixels selected +by the low threshold. + +@var{low} and @var{high} threshold values must be chosen in the range +[0,1], and @var{low} should be lesser or equal to @var{high}. + +Default value for @var{low} is @code{20/255}, and default value for @var{high} +is @code{50/255}. + +@item radius +Define the radius to search around an edge pixel for local maxima. + +@item block_pct +Determine blurriness only for the most significant blocks, given in percentage. + +@item block_width +Determine blurriness for blocks of width @var{block_width}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_height}. + +@item block_height +Determine blurriness for blocks of height @var{block_height}. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of @var{block_width}. + +@item stats_file, f +If specified the filter will use the named file to save the PSNR of +each individual frame. When filename equals "-" the data is sent to +standard output. +@end table + +@subsection Examples + +@itemize +@item +Determine blurriness for 80% of most significant 32x32 blocks and write to stdout: +@example +blurriness=block_width=32:block_height=32:block_pct=80:f=- +@end example +@end itemize + + @section bm3d Denoise frames using Block-Matching 3D algorithm. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 1b559539ba..8dd1b2a947 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -188,6 +188,7 @@ OBJS-$(CONFIG_BITPLANENOISE_FILTER) += vf_bitplanenoise.o OBJS-$(CONFIG_BLACKDETECT_FILTER) += vf_blackdetect.o OBJS-$(CONFIG_BLACKFRAME_FILTER) += vf_blackframe.o OBJS-$(CONFIG_BLEND_FILTER) += vf_blend.o framesync.o +OBJS-$(CONFIG_BLURRINESS_FILTER) += vf_blurriness.o edge_common.o OBJS-$(CONFIG_BM3D_FILTER) += vf_bm3d.o framesync.o OBJS-$(CONFIG_BOXBLUR_FILTER) += vf_boxblur.o boxblur.o OBJS-$(CONFIG_BOXBLUR_OPENCL_FILTER) += vf_avgblur_opencl.o opencl.o \ diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index f250020159..121e37391d 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -180,6 +180,7 @@ extern const AVFilter ff_vf_bitplanenoise; extern const AVFilter ff_vf_blackdetect; extern const AVFilter ff_vf_blackframe; extern const AVFilter ff_vf_blend; +extern const AVFilter ff_vf_blurriness; extern const AVFilter ff_vf_bm3d; extern const AVFilter ff_vf_boxblur; extern const AVFilter ff_vf_boxblur_opencl; diff --git a/libavfilter/vf_blurriness.c b/libavfilter/vf_blurriness.c new file mode 100644 index 0000000000..71b72d7211 --- /dev/null +++ b/libavfilter/vf_blurriness.c @@ -0,0 +1,366 @@ +/* + * Copyright (c) 2021 Thilo Borgmann <thilo.borgmann _at_ mail.de> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * No-reference blurriness metric filter + * + * Implementing: + * Marziliano, Pina, et al. "A no-reference perceptual blur metric." Proceedings. + * International conference on image processing. Vol. 3. IEEE, 2002. + * https://infoscience.epfl.ch/record/111802/files/14%20A%20no-reference%20perceptual%20blur%20metric.pdf + * + * @author Thilo Borgmann <thilo.borgmann _at_ mail.de> + */ + +#include "libavutil/avassert.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "libavutil/pixelutils.h" +#include "libavutil/motion_vector.h" +#include "avfilter.h" +#include "formats.h" +#include "internal.h" +#include "video.h" +#include "edge_common.h" + +static int comp(const void *a,const void *b) +{ + float x = *(float*)a; + float y = *(float*)b; + if (x > y) return 1; + if (x < y) return -1; + return 0; +} + +typedef struct BLRContext { + const AVClass *class; + + float low, high; + uint8_t low_u8, high_u8; + int radius; // radius during local maxima detection + int block_pct; // percentage of "sharpest" blocks in the image to use for bluriness calculation + int block_width; // width for block abbreviation + int block_height; // height for block abbreviation + + float blur_total; + uint64_t nb_frames; + + float *blks; + uint8_t *filterbuf; + uint8_t *tmpbuf; + uint16_t *gradients; + char *directions; +} BLRContext; + +#define OFFSET(x) offsetof(BLRContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM +static const AVOption blr_options[] = { + { "high", "set high threshold", OFFSET(high), AV_OPT_TYPE_FLOAT, {.dbl=30/255.}, 0, 1, FLAGS }, + { "low", "set low threshold", OFFSET(low), AV_OPT_TYPE_FLOAT, {.dbl=15/255.}, 0, 1, FLAGS }, + { "radius", "search radius for maxima detection", OFFSET(radius), AV_OPT_TYPE_INT, {.i64=50}, 1, 100, FLAGS }, + { "block_pct", "block pooling threshold when calculating blurriness", OFFSET(block_pct), AV_OPT_TYPE_INT, {.i64=80}, 1, 100, FLAGS }, + { "block_width", "block size for block-based abbreviation of blurriness", OFFSET(block_width), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS }, + { "block_height", "block size for block-based abbreviation of blurriness", OFFSET(block_height), AV_OPT_TYPE_INT, {.i64=-1}, -1, INT_MAX, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(blr); + +static av_cold int blr_init(AVFilterContext *ctx) +{ + BLRContext *blr = ctx->priv; + + blr->low_u8 = blr->low * 255. + .5; + blr->high_u8 = blr->high * 255. + .5; + + return 0; +} + +static int blr_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + BLRContext *blr = ctx->priv; + const int bufsize = inlink->w * inlink->h; + + if (blr->block_width < 1 || blr->block_height < 1) { + blr->block_width = inlink->w; + blr->block_height = inlink->h; + } + + blr->tmpbuf = av_malloc(bufsize); + blr->filterbuf = av_malloc(bufsize); + blr->gradients = av_calloc(bufsize, sizeof(*blr->gradients)); + blr->directions = av_malloc(bufsize); + if (!blr->tmpbuf || !blr->filterbuf || !blr->gradients || !blr->directions) + return AVERROR(ENOMEM); + + blr->blks = av_calloc((inlink->w / blr->block_width) * (inlink->h / blr->block_height), + sizeof(*blr->blks)); + + return 0; +} + +// edge width is defined as the distance between surrounding maxima of the edge pixel +static float edge_width(BLRContext *blr, int i, int j, int8_t dir, int w, int h, + int edge, const uint8_t *src, int src_linesize) +{ + float width = 0; + int dX, dY; + int sign; + int tmp; + int p1; + int p2; + int k, x, y; + int edge1; + int edge2; + float luma1 = 0.0; // average luma difference per edge pixel + float luma2 = 0.0; + int radius = blr->radius; + + switch(dir) + { + case DIRECTION_HORIZONTAL: dX = 1; dY = 0; break; + case DIRECTION_VERTICAL: dX = 0; dY = 1; break; + case DIRECTION_45UP: dX = 1; dY = -1; break; + case DIRECTION_45DOWN: dX = 1; dY = 1; break; + } + if (dir == DIRECTION_HORIZONTAL) return 0; + + // determines if search in direction dX/dY is looking for a maximum or minimum + sign = src[j * src_linesize + i] > src[(j - dY) * src_linesize + i - dX] ? 1 : -1; + + // search in -(dX/dY) direction + for (k = 0; k < radius; k++) + { + x = i - k*dX; + y = j - k*dY; + p1 = y * src_linesize + x; + x -= dX; + y -= dY; + p2 = y * src_linesize + x; + if (x < 0 || x >= w || y < 0 || y >= h) + return 0; + + tmp = (src[p1] - src[p2]) * sign; + + if (tmp <= 0) // local maximum found + break; + + luma1 += tmp; + } + if (k > 0) luma1 /= k; + edge1 = k; + width += k; + + // search in +(dX/dY) direction + for (k = 0; k < radius; k++) + { + x = i + k * dX; + y = j + k * dY; + p1 = y * src_linesize + x; + x += dX; + y += dY; + p2 = y * src_linesize + x; + if (x < 0 || x >= w || y < 0 || y >= h) + return 0; + + tmp = (src[p1] - src[p2]) * sign; + + if (tmp >= 0) // local maximum found + break; + + luma2 -= tmp; + } + if (k > 0) luma2 /= k; + edge2 = k; + width += k; + + // for 45 degree directions approximate edge width in pixel units: 0.7 ~= sqrt(2)/2 + if (dir == DIRECTION_45UP || dir == DIRECTION_45DOWN) + width *= 0.7; + + return width; +} + +static float calculate_blur(BLRContext *blr, int w, int h, + uint8_t* dir, int dir_linesize, + uint8_t* dst, int dst_linesize, + uint8_t* src, int src_linesize) +{ + float total_width = 0.0; + int block_count; + float block_total_width; + + int i, j; + int blkcnt = 0; + + float *blks = blr->blks; + int block_pct = blr->block_pct; + float block_pool_threshold = block_pct / 100.0; + + int block_width = blr->block_width; + int block_height = blr->block_height; + int brows = h / block_height; + int bcols = w / block_width; + + for (int blkj = 0; blkj < brows; blkj++) { + for (int blki = 0; blki < bcols; blki++) { + block_total_width = 0; + block_count = 0; + for (int inj = 0; inj < block_height; inj++) { + for (int ini = 0; ini < block_width; ini++) { + i = blki * block_width + ini; + j = blkj * block_height + inj; + + if (dst[j * dst_linesize + i] > 0) { + float width = edge_width(blr, i, j, dir[j*dir_linesize+i], + w, h, dst[j*dst_linesize+i], + src, src_linesize); + if (width > 0.001) { // throw away zeros + block_count++; + block_total_width += width; + } + } + } + } + // if not enough edge pixels in a block, consider it smooth + if (block_total_width >= 2) { + blks[blkcnt] = block_total_width / block_count; + blkcnt++; + } + } + } + + // simple block pooling by sorting and keeping the sharper blocks + qsort(blks, blkcnt, sizeof(*blks), comp); + blkcnt = ceil(blkcnt * block_pool_threshold); + for (int i = 0; i < blkcnt; i++) { + total_width += blks[i]; + } + + return total_width / blkcnt; +} + +static void set_meta(AVDictionary **metadata, const char *key, float d) +{ + char value[128]; + snprintf(value, sizeof(value), "%f", d); + av_dict_set(metadata, key, value, 0); +} + +static int blr_filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx = inlink->dst; + BLRContext *blr = ctx->priv; + + const int w = inlink->w; + const int h = inlink->h; + + uint8_t *tmpbuf = blr->tmpbuf; + uint8_t *filterbuf = blr->filterbuf; + uint16_t *gradients = blr->gradients; + int8_t *directions = blr->directions; + + float blur; + AVDictionary **metadata; + metadata = &in->metadata; + + // gaussian filter to reduce noise + ff_gaussian_blur(w, h, + filterbuf, w, + in->data[0], in->linesize[0]); + + // compute the 16-bits gradients and directions for the next step + ff_sobel(w, h, gradients, w, directions, w, filterbuf, w); + + // non_maximum_suppression() will actually keep & clip what's necessary and + // ignore the rest, so we need a clean output buffer + memset(tmpbuf, 0, w * h); + ff_non_maximum_suppression(w, h, tmpbuf, w, directions, w, gradients, w); + + + // keep high values, or low values surrounded by high values + ff_double_threshold(blr->low_u8, blr->high_u8, w, h, + tmpbuf, w, tmpbuf, w); + + + blur = calculate_blur(blr, w, h, directions, w, + tmpbuf, w, filterbuf, w); + + blr->blur_total += blur; + + // write stats + av_log(ctx, AV_LOG_VERBOSE, "blur: %.7f\n", blur); + + set_meta(metadata, "lavfi.blur", blur); + + blr->nb_frames++; + + return ff_filter_frame(ctx->outputs[0], in); +} + +static av_cold void blr_uninit(AVFilterContext *ctx) +{ + BLRContext *blr = ctx->priv; + + if (blr->nb_frames > 0) { + int nb_frames = blr->nb_frames; + av_log(ctx, AV_LOG_INFO, "blur mean: %.7f\n", + blr->blur_total / nb_frames); + } + + av_freep(&blr->tmpbuf); + av_freep(&blr->filterbuf); + av_freep(&blr->gradients); + av_freep(&blr->directions); + av_freep(&blr->blks); +} + +static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_YUV420P, AV_PIX_FMT_GRAY8, AV_PIX_FMT_NONE }; + +static const AVFilterPad blr_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = blr_config_input, + .filter_frame = blr_filter_frame, + }, +}; + +static const AVFilterPad blr_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + }, +}; + +AVFilter ff_vf_blurriness = { + .name = "blurriness", + .description = NULL_IF_CONFIG_SMALL("Blurriness metric filter."), + .priv_size = sizeof(BLRContext), + .init = blr_init, + .uninit = blr_uninit, + FILTER_PIXFMTS_ARRAY(pix_fmts), + FILTER_INPUTS(blr_inputs), + FILTER_OUTPUTS(blr_outputs), + .priv_class = &blr_class, + .flags = AVFILTER_FLAG_SUPPORT_TIMELINE_GENERIC, +};
Hi, $subject -Thilo From 0fa76024cf921df8a6df77a3860faab9669d0a2e Mon Sep 17 00:00:00 2001 From: Thilo Borgmann <thilo.borgmann@mail.de> Date: Tue, 30 Nov 2021 00:17:44 +0100 Subject: [PATCH 2/2] lavfi: Add blurriness filter --- doc/filters.texi | 55 ++++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_blurriness.c | 366 ++++++++++++++++++++++++++++++++++++ 4 files changed, 423 insertions(+) create mode 100644 libavfilter/vf_blurriness.c