From patchwork Fri Dec 27 11:11:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Wang X-Patchwork-Id: 16995 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id AA61E4498EC for ; Fri, 27 Dec 2019 13:11:30 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7D3AB68A9FC; Fri, 27 Dec 2019 13:11:30 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 34FA968A927 for ; Fri, 27 Dec 2019 13:11:23 +0200 (EET) Received: by mail-pg1-f172.google.com with SMTP id r11so14316967pgf.1 for ; Fri, 27 Dec 2019 03:11:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=652mARM9ZPe+wZ1dOYTw1EXjIQX/CGv9L6CCSCcsT+U=; b=UHznWPkjtmeikVRmDOH3xn4NqC1FuxHVUD6xG57e+N+qSLOmRvwm/FhrzxLFlFzpJu BrwVXx8iyQVdTBXJv9GEDUSbgCcjg78XBHDCES1ZBz8uwzFz8e5COcK9g5BZ8M/y4WnG k8zrQEYOTZZLqdNcqN1RbQB9q8A+hRZBEiWQIMpeq/CfE4XnRhwLsjPBP/UhKTTWckAt aq5KHDHFpab6I83lqyHxbXOoSbWSaDteE7yOX4Pi5YsmrvZjTJ93MMzyedETnRCo1Dih FvHHmJ+QAIGOhaRYrvpImckcTNla3zDzLGnjaLEQ8QeB+VFknEAtMYntnfHjpAZDbaKr 7SJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=652mARM9ZPe+wZ1dOYTw1EXjIQX/CGv9L6CCSCcsT+U=; b=fB3BsVFEd2NzV7Jz7USf/8WINJlBeUj2i0jLdi4sZagET6ro7h01E8Jsm3AoZbQQTC hYEo6wc8H9ROdHhQI3lfI/MvuXxmDtDS0qwfk8Jz+X8PQttdU5jlr1uJE4Zj1tbYykLD 2dxWnxgTw/luCrJEmSHN+31CddVDlSjLxrRhKAYdI5dqK+IGs54Ox+nF6p9qO9mbbKUA 1bovf6oJl8EMcNzv23D9Wf95BbfCF+4WuA44LMYfc7romFimM2ZfqKlfOA3Clmjqtfco OZ3Y2t0cycwYx4sYqi2cfFuQ5IfeKPVNmc3ggGfsYUQI0d08lS4+Pkkpu+WhOQK78EAJ kArw== X-Gm-Message-State: APjAAAXuMrrCeVVoABe1wlr7CSt7sTpQ8DVj5RLzVnUzMY0liIAQu6Ov 59n0c/ppD7KdaUwItWUzBKRxbWEF X-Google-Smtp-Source: APXvYqwCIZ/zBde9rmWUJvW2DzA99OVwu6cifDleM5HsFuYK7wB6SKQmJoKVlwVkXrNhE5BltCkHiA== X-Received: by 2002:aa7:9ec9:: with SMTP id r9mr53882505pfq.85.1577445080915; Fri, 27 Dec 2019 03:11:20 -0800 (PST) Received: from vpn.localdomain ([47.90.99.151]) by smtp.gmail.com with ESMTPSA id q8sm37594578pgg.92.2019.12.27.03.11.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Dec 2019 03:11:20 -0800 (PST) From: lance.lmwang@gmail.com To: ffmpeg-devel@ffmpeg.org Date: Fri, 27 Dec 2019 19:11:14 +0800 Message-Id: <20191227111114.32557-1-lance.lmwang@gmail.com> X-Mailer: git-send-email 2.9.5 Subject: [FFmpeg-devel] [PATCH v1] avfilter: add colorstats, colorrgbstats, coloryuvstats video filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Limin Wang MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Limin Wang Signed-off-by: Limin Wang --- doc/filters.texi | 74 ++++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 3 + libavfilter/vf_colorstats.c | 461 ++++++++++++++++++++++++++++++++++++ 4 files changed, 539 insertions(+) create mode 100644 libavfilter/vf_colorstats.c diff --git a/doc/filters.texi b/doc/filters.texi index 8c5d3a5760..81968b2c17 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -7695,6 +7695,80 @@ For example to convert the input to SMPTE-240M, use the command: colorspace=smpte240m @end example +@section colorstats, colorrgbstats, coloryuvstats +The filter provides statistical video measurements such as mean, minimum, maximum and +standard deviation for each frame. The user can check for unexpected/accidental errors +very quickly with them. + +@var{colorrgbstats} report the color stats for RGB input video, @var{coloryuvstats} +to an YUV input video. + +These filters accept the following parameters: +@table @option +@item planes +Set which planes to filter. Default is only the first plane. +@end table + +By default the filter will report these metadata values if the planes +are processed: + +@table @option +@item min.y, min.u, min.v, min.r, min.g, min.b, min.a +Display the minimal Y/U/V/R/G/B/A plane value contained within the input frame. +Expressed in range of [0, 1<priv; + const enum AVPixelFormat *pix_fmts = s->force_fmt == 1 ? rgb_pix_fmts : + s->force_fmt == 2 ? yuv_pix_fmts : + all_pix_fmts; + + AVFilterFormats *fmts_list = ff_make_format_list(pix_fmts); + if (!fmts_list) + return AVERROR(ENOMEM); + return ff_set_common_formats(ctx, fmts_list); +} + +#define DECLARE_STATS_PLANAR_FUNC(nbits, div) \ +static int stats_slice_planar_##nbits(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) \ +{ \ + const ColorStatsContext *s = ctx->priv; \ + ThreadData *td = arg; \ + const AVFrame *in = td->in; \ + int64_t sum[4] = { 0 }, sum2[4] = { 0 }; \ + int32_t count[4] = { 0 }; \ + double min_value[4] = { s->max }; \ + double max_value[4] = { 0 }; \ + \ + for (int i = 0; i < s->nb_components; i++) { \ + const int width = s->width[i]; \ + const int height = s->height[i]; \ + const int slice_start = (height * jobnr ) / nb_jobs; \ + const int slice_end = (height * (jobnr + 1)) / nb_jobs; \ + int linesize = in->linesize[i] / div; \ + uint##nbits##_t *src = (uint##nbits##_t*)in->data[i] + slice_start * linesize; \ + \ + if (!(s->planes & (1 << i))) \ + continue; \ + for (int j = slice_start; j < slice_end; j++) { \ + for (int x = 0; x < width; x++) { \ + sum[i] += src[x]; \ + sum2[i] += src[x] * src[x]; \ + if (src[i] > max_value[i]) max_value[i] = src[i]; \ + if (src[i] < min_value[i]) min_value[i] = src[i]; \ + } \ + count[i] += width; \ + src += linesize; \ + } \ + \ + s->mean[i][jobnr] = (double)(sum[i] + count[i] / 2) / count[i]; \ + s->stdev[i][jobnr] = sqrt((sum2[i] - sum[i] * (double)sum[i] / count[i]) / count[i]); \ + s->min_value[i][jobnr] = min_value[i]; \ + s->max_value[i][jobnr] = max_value[i]; \ + } \ + \ + return 0; \ +} +DECLARE_STATS_PLANAR_FUNC(8, 1) +DECLARE_STATS_PLANAR_FUNC(16, 2) + +#define DECLARE_STATS_PACKED_FUNC(nbits, div) \ +static int stats_slice_packed_##nbits(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) \ +{ \ + const ColorStatsContext *s = ctx->priv; \ + ThreadData *td = arg; \ + const AVFrame *in = td->in; \ + int64_t sum[4] = { 0 }, sum2[4] = { 0 }; \ + double min_value[4] = { s->max }; \ + double max_value[4] = { 0 }; \ + int32_t count[4] = { 0 }; \ + const int width = in->width; \ + const int height = in->height; \ + const int slice_start = (height * jobnr ) / nb_jobs; \ + const int slice_end = (height * (jobnr + 1)) / nb_jobs; \ + int linesize = in->linesize[0] / div; \ + uint##nbits##_t *src = (uint##nbits##_t*)in->data[0] + slice_start * linesize; \ + const uint8_t ro = s->rgba_map[R]; \ + const uint8_t go = s->rgba_map[G]; \ + const uint8_t bo = s->rgba_map[B]; \ + const uint8_t ao = s->rgba_map[A]; \ + \ + for (int y = slice_start; y < slice_end; y++) { \ + for (int x = 0; x < width * s->step; x += s->step) { \ + const int r = src[x + ro]; \ + const int g = src[x + go]; \ + const int b = src[x + bo]; \ + const int a = src[x + ao]; \ + \ + sum[ro] += r; \ + sum[go] += g; \ + sum[bo] += b; \ + sum2[ro] += r * r; \ + sum2[go] += g * g; \ + sum2[bo] += b * b; \ + \ + if (r > max_value[ro]) max_value[ro] = r; \ + if (r < min_value[ro]) min_value[ro] = r; \ + if (g > max_value[go]) max_value[go] = g; \ + if (g < min_value[go]) min_value[go] = g; \ + if (b > max_value[bo]) max_value[bo] = b; \ + if (b < min_value[bo]) min_value[bo] = b; \ + if (s->step == 4) { \ + sum2[ao] += a * a; \ + sum[ao] += a; \ + if (a > max_value[ao]) max_value[ao] = a; \ + if (a < min_value[ao]) min_value[ao] = a; \ + } \ + } \ + count[ro] += width; \ + count[go] += width; \ + count[bo] += width; \ + if (s->step == 4) \ + count[ao] += width; \ + src += linesize; \ + } \ + \ + for (int p = 0; p < s->nb_components; p++) { \ + int ci = s->is_rgb ? s->rgba_map[p] : p; \ + double variance; \ + \ + s->mean[ci][jobnr] = (double)(sum[ci] + count[ci] / 2) / count[ci]; \ + variance = (sum2[ci] - sum[ci] * (double)sum[ci] / count[ci]) / count[ci]; \ + s->stdev[ci][jobnr] = sqrt(variance); \ + s->min_value[ci][jobnr] = min_value[ci]; \ + s->max_value[ci][jobnr] = max_value[ci]; \ + } \ + \ + return 0; \ +} +DECLARE_STATS_PACKED_FUNC(8, 1) +DECLARE_STATS_PACKED_FUNC(16, 2) + +static av_cold void uninit(AVFilterContext *ctx) +{ + ColorStatsContext *s = ctx->priv; + + for (int i = 0; i < s->nb_components; i++) { + av_freep(&s->mean[i]); + av_freep(&s->stdev[i]); + av_freep(&s->min_value[i]); + av_freep(&s->max_value[i]); + } +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + ColorStatsContext *s = ctx->priv; + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(inlink->format); + + s->nb_components = desc->nb_components; + s->bitdepth = desc->comp[0].depth; + s->is_16bit = s->bitdepth > 8; + s->step = av_get_padded_bits_per_pixel(desc) >> (3 + s->is_16bit); + s->max = 1 << s->bitdepth - 1; + + s->is_rgb = ff_fill_rgba_map(s->rgba_map, inlink->format) >= 0; + s->comps[0] = s->is_rgb ? 'r' : 'y' ; + s->comps[1] = s->is_rgb ? 'g' : 'u' ; + s->comps[2] = s->is_rgb ? 'b' : 'v' ; + s->comps[3] = 'a'; + + s->thread_count = FFMAX(1, FFMIN(inlink->h, ff_filter_get_nb_threads(ctx))); + for (int i = 0; i < s->nb_components; i++) { + ptrdiff_t line_size = av_image_get_linesize(inlink->format, inlink->w, i); + + s->width[i] = line_size >> (s->bitdepth > 8); + s->height[i] = inlink->h >> ((i == 1 || i == 2) ? desc->log2_chroma_h : 0); + + s->mean[i] = av_mallocz_array(s->thread_count, sizeof(*s->mean[i])); + s->stdev[i] = av_mallocz_array(s->thread_count, sizeof(*s->stdev[i])); + s->max_value[i] = av_mallocz_array(s->thread_count, sizeof(*s->max_value[i])); + s->min_value[i] = av_mallocz_array(s->thread_count, sizeof(*s->min_value[i])); + if (!s->mean[i] || !s->stdev[i] || !s->max_value[i] || !s->min_value[i]) + return AVERROR(ENOMEM); + for (int j = 0; j < s->thread_count; j++) { + s->min_value[i][j] = (1 << s->bitdepth); + s->max_value[i][j] = 0; + } + } + + if (desc->flags & AV_PIX_FMT_FLAG_PLANAR) + s->stats_slice = s->bitdepth <= 8 ? stats_slice_planar_8 : stats_slice_planar_16; + else + s->stats_slice = s->bitdepth <= 8 ? stats_slice_packed_8 : stats_slice_packed_16; + + return 0; +} + +static void set_meta_float(AVDictionary **metadata, const char *key, char c, float d) +{ + char value[128]; + char key2[128]; + + snprintf(value, sizeof(value), "%.2f", d); + if (c) + snprintf(key2, sizeof(key2), "lavf.colorstats.%s.%c", key, c); + else + snprintf(key2, sizeof(key2), "lavf.colorstats.%s", key); + av_dict_set(metadata, key2, value, 0); +} + +static void set_meta_int(AVDictionary **metadata, const char *key, char c, int d) +{ + char value[128]; + char key2[128]; + + snprintf(value, sizeof(value), "%d", d); + if (c) + snprintf(key2, sizeof(key2), "lavf.colorstats.%s.%c", key, c); + else + snprintf(key2, sizeof(key2), "lavf.colorstats.%s", key); + av_dict_set(metadata, key2, value, 0); +} + +static void report_detect_result(AVFilterContext *ctx, AVFrame *in) +{ + const ColorStatsContext *s = ctx->priv; + double mean[4] = { 0 }; + double stdev[4] = { 0 }; + double min_value[4] = { s->max }; + double max_value[4] = { 0 }; + int cidx; + + for (int p = 0; p < s->nb_components; p++) { + cidx = s->is_rgb ? s->rgba_map[p] : p; + + if (!(s->planes & (1 << p))) + continue; + + for (int j = 0; j < s->thread_count; j++) { + mean[cidx] += s->mean[cidx][j]; + stdev[cidx] += s->stdev[cidx][j]; + if (s->min_value[cidx][j] < min_value[cidx]) + min_value[cidx] = s->min_value[cidx][j]; + if (s->max_value[cidx][j] > max_value[cidx]) + max_value[cidx] = s->max_value[cidx][j]; + } + mean[cidx] = mean[cidx] / s->thread_count; + stdev[cidx] = stdev[cidx] / s->thread_count; + + set_meta_int(&in->metadata, "min", s->comps[p], min_value[cidx]); + set_meta_int(&in->metadata, "max", s->comps[p], max_value[cidx]); + set_meta_int(&in->metadata, "mean", s->comps[p], mean[cidx]); + set_meta_int(&in->metadata, "stdev", s->comps[p], stdev[cidx]); + + set_meta_float(&in->metadata, "pmin", s->comps[p], min_value[cidx] / s->max); + set_meta_float(&in->metadata, "pmax", s->comps[p], max_value[cidx] / s->max); + set_meta_float(&in->metadata, "pmean", s->comps[p], mean[cidx] / s->max); + set_meta_float(&in->metadata, "pstdev", s->comps[p], stdev[cidx] / s->max); + } +} + +static int activate(AVFilterContext *ctx) +{ + int ret; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + ColorStatsContext *s = ctx->priv; + AVFrame *in; + ThreadData td; + + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + ret = ff_inlink_consume_frame(inlink, &in); + if (ret < 0) + return ret; + + if (in) { + td.in = in; + ctx->internal->execute(ctx, s->stats_slice, &td, NULL, s->thread_count); + + report_detect_result(ctx, in); + return ff_filter_frame(outlink, in); + } + + FF_FILTER_FORWARD_STATUS(inlink, outlink); + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input, + }, + { NULL } +}; + +static const AVFilterPad outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + }, + { NULL } +}; + +#define DEFINE_COLOR_FILTER(name_, description_) \ + AVFilter ff_vf_##name_ = { \ + .name = #name_, \ + .description = NULL_IF_CONFIG_SMALL(description_), \ + .priv_size = sizeof(ColorStatsContext), \ + .priv_class = &name_ ## _class, \ + .init = name_##_init, \ + .uninit = uninit, \ + .query_formats = query_formats, \ + .inputs = inputs, \ + .outputs = outputs, \ + .activate = activate, \ + .flags = AVFILTER_FLAG_SUPPORT_TIMELINE_GENERIC | \ + AVFILTER_FLAG_SLICE_THREADS, \ + } + +#if CONFIG_COLORSTATS_FILTER + +#define colorstats_options options +AVFILTER_DEFINE_CLASS(colorstats); + +static int colorstats_init(AVFilterContext *ctx) +{ + return 0; +} + +DEFINE_COLOR_FILTER(colorstats, "Video color stats."); +#endif + +#if CONFIG_COLORRGBSTATS_FILTER + +#define colorrgbstats_options options +AVFILTER_DEFINE_CLASS(colorrgbstats); + +static int colorrgbstats_init(AVFilterContext *ctx) +{ + ColorStatsContext *s = ctx->priv; + + s->force_fmt = 1; + return 0; +} + +DEFINE_COLOR_FILTER(colorrgbstats, "Video RGB color stats."); +#endif + +#if CONFIG_COLORYUVSTATS_FILTER + +#define coloryuvstats_options options +AVFILTER_DEFINE_CLASS(coloryuvstats); + +static int coloryuvstats_init(AVFilterContext *ctx) +{ + ColorStatsContext *s = ctx->priv; + + s->force_fmt = 2; + return 0; +} + +DEFINE_COLOR_FILTER(coloryuvstats, "Video YUV color stats."); +#endif