From patchwork Sat Aug 28 15:17:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Buxton X-Patchwork-Id: 29846 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp2983705iov; Sun, 29 Aug 2021 02:05:43 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwO3tpXQFhiygt5Aw8nhMTA5uEuxkbCbZ3PH1Brg/jnxrREzg37MhE0zknUdjXE91ESI4Mn X-Received: by 2002:a17:906:720e:: with SMTP id m14mr19163146ejk.500.1630227943463; Sun, 29 Aug 2021 02:05:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630227943; cv=none; d=google.com; s=arc-20160816; b=W6BlI/1zW2+qqI0+SM/uLZhjdkWH8vIltu2IidHLHJY+v6Lhw243E5kUlxxZIHDopu rJX9+A5+JV/8Cp2uSA56d901XVCWWskSJYtyrEWynYsxCqHqu7cPPL/vQhrVQoEA6hSp i01rczVlIqQMpS3O1qg1PhjaAnwcVh1Twnm8fw4LU8FdwxJjbWvFkjhwjkKvZZoYTpw2 SiLu6EVRAFVpaBdTgIohkuawGhKSMz8kx2urhuKWh+PRKCdsawYCfFslFxASZ7QjTaKl Oxg6qFMDbJyrc2kc/5EwXg63YfWUJuVvRefVKo1BS0ZpoiQ3S6Km6QX9oaRvYaPLes6m DJVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to; bh=9tOcqx9cqeakYA6CyEXGoHXiBPxm+hofFFO5A5DqBVo=; b=h+u/rpHe7v269oKWui3V0FG7pMXg25sZxZ3IVW+/UAXpRQN7eMGJUx570xZ+tcm+ED gqPjTPysajFOEjFEm34xfRHJgYIryH8TQnxSZC4t4zZfBLUMh+rJYvMwzzjSU3ycuHXw +xkhdqbCeS2yH6qs5E7MgXokdnuDpPZoKMViXhFvUySaXzAAAUVpAH7Ky5RPLjNL5ykc 9SBXfkipqXnBf6evrZWa3l55MC61KyUsHhJx4Zofsvji1LMb/NYK8FMLoEEPJRJ9MkwN JIgC7cZuRQAFKkWr7kNqZ0790k1F/hM5vMQSFg+awrxC4nGwqgf8pPWYTctMZ3KA4OEJ d4Lw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@googlemail.com header.s=20161025 header.b=DVvm7vX1; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id di1si11375468ejc.420.2021.08.29.02.05.42; Sun, 29 Aug 2021 02:05:43 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@googlemail.com header.s=20161025 header.b=DVvm7vX1; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A72926899DA; Sun, 29 Aug 2021 12:05:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0C5016898E6 for ; Sun, 29 Aug 2021 12:05:30 +0300 (EEST) Received: by mail-wm1-f53.google.com with SMTP id k20-20020a05600c0b5400b002e87ad6956eso7330098wmr.1 for ; Sun, 29 Aug 2021 02:05:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=gZYUBWPeCWxVKSkSEUOWDPES8IznXJ9iQXJMMe1PENI=; b=DVvm7vX1gYC3WJTnfNfLd0kyXI+uGFlRt0S7zBr/+hi4GcGUZMg0ApVn2Qes+u65qV cQ2HozXR6wfgcPKwjxLrhsAMJoguWBwGD3IlPQtmnz47qtVaTHkbuOjKu0Gso2upmoJq gLnK8dH8oJJpJRkvBRrveZJr3wfMLxlPNyJKlBhyXmWUdy1ky5IM8oaaLxxwf1rtZdiH t0OJkAweuuUhh6g+uQY0lTCM9VbszXGSj+1VIT+YczeoZQs4/QK6ZTGxuYr1ttZoF2bk oUK5rj2JkYWP9oB09cIhkTJrZyu6YZZt2VNqwm0g3p17pLcY2TnSxPYt2HiHqw1bahZV pYgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=gZYUBWPeCWxVKSkSEUOWDPES8IznXJ9iQXJMMe1PENI=; b=t7Pz3z+lZfc5+GPqgMBZALT7fjnsMUoy4SoivoJnVecWPWZTJqyoyXJH66574acqEA kROE1qQ1zA9L8n5d81anM79JdVivpXf03VOWtc46QyN4AIgEh28BqQ6aLIL0bY3munLS c7z4I3dKRtdpllrG6+Tsy8ocCA1FPLIpg/fGmWPeIlywx0HkZy3tgecyOMHUSasefA1r lEHLPDB1KP6WaaA9gE0p0yBRrRzv207ZpUWFwLvv1dQ15vFGvOvt33MOv2E0g6Okcv7U LQJzjwTHUB9mfT9ooW7dOhBdgzeq14+hRjYJ9FuXpP0xjuAmcp0/KgMIBjc88HYEMk5Z g4dA== X-Gm-Message-State: AOAM5334wiWYC7nBX5RxJh+BV+GfT3qlsW1qFut6PzwgI+Dg+leIOOD0 hl0Adt/+u5oXXN9qZ2ybzLjZHwc6KXE= X-Received: by 2002:a1c:ed13:: with SMTP id l19mr28256254wmh.48.1630227929971; Sun, 29 Aug 2021 02:05:29 -0700 (PDT) Received: from DESKTOP-D00LNAQ.localdomain (cpc119508-heme14-2-0-cust207.9-1.cable.virginm.net. [82.14.71.208]) by smtp.gmail.com with ESMTPSA id a6sm3721530wrh.97.2021.08.29.02.05.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Aug 2021 02:05:29 -0700 (PDT) From: Paul Buxton To: ffmpeg-devel@ffmpeg.org Date: Sat, 28 Aug 2021 16:17:27 +0100 Message-Id: <20210828151727.22669-1-paulbuxton.mail@googlemail.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [v7 PATCH 1/1] libavfilter/vf_grayworld: Add gray world filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Paul Buxton MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: dfaVIYl8D+Un Implements a gray world color correction algorithm using a log scale LAB colorspace Signed-off-by: Paul Buxton --- doc/filters.texi | 12 ++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/version.h | 2 +- libavfilter/vf_grayworld.c | 352 +++++++++++++++++++++++++++++++++++++ 5 files changed, 367 insertions(+), 1 deletion(-) create mode 100644 libavfilter/vf_grayworld.c diff --git a/doc/filters.texi b/doc/filters.texi index 5a17d00ece..6d1a13dffd 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -13091,6 +13091,18 @@ Set upper limit for video rate of output stream, Default value is @var{25}. This guarantee that output video frame rate will not be higher than this value. @end table +@section grayworld +A color constancy filter that applies color correction based on the grayworld assumption + +See: @url{https://www.researchgate.net/publication/275213614_A_New_Color_Correction_Method_for_Underwater_Imaging} + +The algorithm uses linear light, so input +data should be linearized beforehand (and possibly correctly tagged). + +@example +ffmpeg -i INPUT -vf zscale=transfer=linear,grayworld,zscale=transfer=bt709,format=yuv420p OUTPUT +@end example + @section greyedge A color constancy variation filter which estimates scene illumination via grey edge algorithm and corrects the scene colors accordingly. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 2b58325fee..af957a5ac0 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -289,6 +289,7 @@ OBJS-$(CONFIG_GBLUR_FILTER) += vf_gblur.o OBJS-$(CONFIG_GEQ_FILTER) += vf_geq.o OBJS-$(CONFIG_GRADFUN_FILTER) += vf_gradfun.o OBJS-$(CONFIG_GRAPHMONITOR_FILTER) += f_graphmonitor.o +OBJS-$(CONFIG_GRAYWORLD_FILTER) += vf_grayworld.o OBJS-$(CONFIG_GREYEDGE_FILTER) += vf_colorconstancy.o OBJS-$(CONFIG_GUIDED_FILTER) += vf_guided.o OBJS-$(CONFIG_HALDCLUT_FILTER) += vf_lut3d.o framesync.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 6f6677546d..0c6b2347c8 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -274,6 +274,7 @@ extern const AVFilter ff_vf_gblur; extern const AVFilter ff_vf_geq; extern const AVFilter ff_vf_gradfun; extern const AVFilter ff_vf_graphmonitor; +extern const AVFilter ff_vf_grayworld; extern const AVFilter ff_vf_greyedge; extern const AVFilter ff_vf_guided; extern const AVFilter ff_vf_haldclut; diff --git a/libavfilter/version.h b/libavfilter/version.h index 78c62a5e75..ff12ff9f8f 100644 --- a/libavfilter/version.h +++ b/libavfilter/version.h @@ -30,7 +30,7 @@ #include "libavutil/version.h" #define LIBAVFILTER_VERSION_MAJOR 8 -#define LIBAVFILTER_VERSION_MINOR 6 +#define LIBAVFILTER_VERSION_MINOR 7 #define LIBAVFILTER_VERSION_MICRO 100 diff --git a/libavfilter/vf_grayworld.c b/libavfilter/vf_grayworld.c new file mode 100644 index 0000000000..e1eb94ba0f --- /dev/null +++ b/libavfilter/vf_grayworld.c @@ -0,0 +1,352 @@ +/* + * Copyright (c) 2021 Paul Buxton + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + + /** + * @file + * Color correction filter based on + * https://www.researchgate.net/publication/275213614_A_New_Color_Correction_Method_for_Underwater_Imaging + * + */ + +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" +#include "libavutil/intreadwrite.h" + +#include "avfilter.h" +#include "drawutils.h" +#include "formats.h" +#include "internal.h" +#include "video.h" + +typedef struct ThreadData { + AVFrame *in, *out; + float l_avg; + float a_avg; + float b_avg; +} ThreadData; + +typedef struct GrayWorldContext { + const AVClass *class; + float *tmpplab; + int *line_count_pels; + float *line_sum; +} GrayWorldContext; + +#define OFFSET(x) offsetof(GrayWorldContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_RUNTIME_PARAM +static const AVOption grayworld_options[] = { NULL }; + +AVFILTER_DEFINE_CLASS(grayworld); + +static void compute_correction(GrayWorldContext *s, ThreadData *td); +static void apply_matrix(const float matrix[3][3], const float input[3], float output[3]); +static void rgb2lab(const float rgb[3], float lab[3]); +static void lab2rgb(const float lab[3], float rgb[3]); + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] = { + AV_PIX_FMT_GBRPF32, + AV_PIX_FMT_GBRAPF32, + AV_PIX_FMT_NONE, + }; + return ff_set_common_formats_from_list(ctx, pix_fmts); +} + +static void apply_matrix(const float matrix[3][3], const float input[3], float output[3]) +{ + for (int row = 0; row < 3; ++row) { + output[row] = matrix[row][0] * input[0] + matrix[row][1] * input[1] + matrix[row][2] * input[2]; + } +} + +static const float lms2lab[3][3] = { + {0.5774, 0.5774, 0.5774}, + {0.40825, 0.40825, -0.816458}, + {0.707, -0.707, 0} +}; + +static const float lab2lms[3][3] = { + {0.57735, 0.40825, 0.707}, + {0.57735, 0.40825, -0.707}, + {0.57735, -0.8165, 0} +}; + +static const float rgb2lms[3][3] = { + {0.3811, 0.5783, 0.0402}, + {0.1967, 0.7244, 0.0782}, + {0.0241, 0.1288, 0.8444} +}; + +static const float lms2rgb[3][3] = { + {4.4679, -3.5873, 0.1193}, + {-1.2186, 2.3809, -0.1624}, + {0.0497, -0.2439, 1.2045} +}; + +/** + * Convert from Linear RGB to logspace LAB + * + * @param rgb Input array of rgb components + * @param lab output array of lab components + */ +static void rgb2lab(const float rgb[3], float lab[3]) +{ + float lms[3]; + apply_matrix(rgb2lms, rgb, lms); + lms[0] = logf(lms[0]); + lms[1] = logf(lms[1]); + lms[2] = logf(lms[2]); + apply_matrix(lms2lab, lms, lab); +} + +/** + * Convert from Logspace LAB to Linear RGB + * + * @param lab input array of lab components + * @param rgb output array of rgb components + */ +static void lab2rgb(const float lab[3], float rgb[3]) +{ + float lms[3]; + apply_matrix(lab2lms, lab, lms); + lms[0] = expf(lms[0]); + lms[1] = expf(lms[1]); + lms[2] = expf(lms[2]); + apply_matrix(lms2rgb, lms, rgb); +} + +/** + * Convert a frame from linear RGB to logspace LAB, and accumulate channel totals for each row + * Convert from RGB -> lms using equation 4 in color transfer paper. + * + * @param ctx Filter context + * @param arg Thread data pointer + * @param jobnr job number + * @param nb_jobs number of jobs + */ +static int convert_frame(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) +{ + GrayWorldContext *s = ctx->priv; + ThreadData *td = arg; + AVFrame *in = td->in; + AVFrame *out = td->out; + AVFilterLink *outlink = ctx->outputs[0]; + const int slice_start = (out->height * jobnr) / nb_jobs; + const int slice_end = (out->height * (jobnr + 1)) / nb_jobs; + float rgb[3]; + float lab[3]; + + for (unsigned int i = slice_start; i < slice_end; i++) { + float *b_in_row = (float*)(in->data[1] + i * in->linesize[1]); + float *g_in_row = (float*)(in->data[0] + i * in->linesize[0]); + float *r_in_row = (float*)(in->data[2] + i * in->linesize[2]); + float *acur = s->tmpplab + i * outlink->w + outlink->w * outlink->h; + float *bcur = s->tmpplab + i * outlink->w + 2 * outlink->w * outlink->h; + float *lcur = s->tmpplab + i * outlink->w; + + s->line_sum[i] = 0.0f; + s->line_sum[i + outlink->h] = 0.0f; + s->line_count_pels[i] = 0; + + for (unsigned int j = 0; j < outlink->w; j++) { + rgb[0] = r_in_row[j]; + rgb[1] = g_in_row[j]; + rgb[2] = b_in_row[j]; + rgb2lab(rgb, lab); + *(lcur++) = lab[0]; + *(acur++) = lab[1]; + *(bcur++) = lab[2]; + s->line_sum[i] += lab[1]; + s->line_sum[i + outlink->h] += lab[2]; + s->line_count_pels[i]++; + } + } + return 0; +} + +/** + * Sum the channel totals and compute the mean for each channel + * + * @param s Frame context + * @param td thread data + */ +static void compute_correction(GrayWorldContext *s, ThreadData *td) +{ + float asum = 0.0f; + float bsum = 0.0f; + float pixels = 0; + for (int y = 0; y < td->out->height; ++y) { + asum += s->line_sum[y]; + bsum += s->line_sum[y + td->out->height]; + pixels += s->line_count_pels[y]; + } + td->a_avg = asum / pixels; + td->b_avg = bsum / pixels; +} + +/** + * Subtract the mean logspace AB values from each pixel. + * + * @param ctx Filter context + * @param arg Thread data pointer + * @param jobnr job number + * @param nb_jobs number of jobs + */ +static int correct_frame(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) +{ + GrayWorldContext *s = ctx->priv; + ThreadData *td = arg; + AVFrame *out = td->out; + AVFilterLink *outlink = ctx->outputs[0]; + const int slice_start = (out->height * jobnr) / nb_jobs; + const int slice_end = (out->height * (jobnr + 1)) / nb_jobs; + float rgb[3]; + float lab[3]; + + for (int i = slice_start; i < slice_end; i++) { + float *g_out_row = (float*)(out->data[0] + i * out->linesize[0]); + float *b_out_row = (float*)(out->data[1] + i * out->linesize[1]); + float *r_out_row = (float*)(out->data[2] + i * out->linesize[2]); + float *lcur = s->tmpplab + i * outlink->w; + float *acur = s->tmpplab + i * outlink->w + outlink->w * outlink->h; + float *bcur = s->tmpplab + i * outlink->w + 2 * outlink->w * outlink->h; + + for (int j = 0; j < outlink->w; j++) { + lab[0] = *lcur++; + lab[1] = *acur++; + lab[2] = *bcur++; + + // subtract the average for the color channels + lab[1] -= td->a_avg; + lab[2] -= td->b_avg; + + //convert back to linear rgb + lab2rgb(lab, rgb); + r_out_row[j] = rgb[0]; + g_out_row[j] = rgb[1]; + b_out_row[j] = rgb[2]; + } + } + return 0; +} + +static int config_input(AVFilterLink *inlink) +{ + GrayWorldContext *s = inlink->dst->priv; + + FF_ALLOC_TYPED_ARRAY(s->tmpplab, inlink->h * inlink->w * 3); + if (!s->tmpplab) { + return AVERROR(ENOMEM); + } + FF_ALLOC_TYPED_ARRAY(s->line_count_pels, inlink->h); + if (!s->line_count_pels) { + return AVERROR(ENOMEM); + } + FF_ALLOC_TYPED_ARRAY(s->line_sum, inlink->h * 2); + if (!s->line_sum) { + return AVERROR(ENOMEM); + } + + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + GrayWorldContext *s = ctx->priv; + + av_freep(&s->tmpplab); + av_freep(&s->line_count_pels); + av_freep(&s->line_sum); +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx = inlink->dst; + GrayWorldContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + ThreadData td; + AVFrame *out; + + if (av_frame_is_writable(in)) { + out = in; + } else { + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) { + av_frame_free(&in); + return AVERROR(ENOMEM); + } + av_frame_copy_props(out, in); + } + /* input and output transfer will be linear */ + if (in->color_trc == AVCOL_TRC_UNSPECIFIED) { + av_log(s, AV_LOG_WARNING, "Untagged transfer, assuming linear light\n"); + out->color_trc = AVCOL_TRC_LINEAR; + } else if (in->color_trc != AVCOL_TRC_LINEAR) { + av_log(s, AV_LOG_WARNING, "Grey world color correction works on linear light only\n"); + } + + td.in = in; + td.out = out; + + ff_filter_execute(ctx, convert_frame, &td, NULL, FFMIN(outlink->h, ff_filter_get_nb_threads(ctx))); + compute_correction(s, &td); + ff_filter_execute(ctx, correct_frame, &td, NULL, FFMIN(outlink->h, ff_filter_get_nb_threads(ctx))); + + if (in != out) { + // copy alpha channel from input frame to output + av_image_copy_plane(out->data[3], out->linesize[3], + in->data[3], in->linesize[3], + out->linesize[3], outlink->h); + // free the input frame + av_frame_free(&in); + } + + return ff_filter_frame(outlink, out); +} + +static const AVFilterPad grayworld_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + .config_props = config_input, + } +}; + +static const AVFilterPad grayworld_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + } +}; + +const AVFilter ff_vf_grayworld = { + .name = "grayworld", + .description = NULL_IF_CONFIG_SMALL("Adjust white balance using LAB gray world algorithm"), + .priv_size = sizeof(GrayWorldContext), + .priv_class = &grayworld_class, + .query_formats = query_formats, + FILTER_INPUTS(grayworld_inputs), + FILTER_OUTPUTS(grayworld_outputs), + .flags = AVFILTER_FLAG_SUPPORT_TIMELINE_GENERIC | AVFILTER_FLAG_SLICE_THREADS, + .uninit = uninit, +};