From patchwork Wed Jan 11 14:05:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: yizhuo liu X-Patchwork-Id: 39965 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:bc95:b0:ad:ade2:bfd2 with SMTP id fx21csp4777468pzb; Wed, 11 Jan 2023 06:06:16 -0800 (PST) X-Google-Smtp-Source: AMrXdXtzVdeQva7OZzKGwl80IqGMyz/fsU3xz/gqisJyqA24arI8ZOlQz/UsumAN7xSDE+HS/q1B X-Received: by 2002:a05:6512:2828:b0:4b5:98ca:548 with SMTP id cf40-20020a056512282800b004b598ca0548mr24685080lfb.39.1673445975613; Wed, 11 Jan 2023 06:06:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673445975; cv=none; d=google.com; s=arc-20160816; b=tBYq/kfkabnkGk1qKbuI8O4ii2kxaGd2v9SCHRdNLlmW1Z7D6AP17Cc/7XKpvRadMf i1185AD/sT0Yv3KCHEToWqQiD5l4m90P+CpZW+xp03cdV00D5Wy5H7F2AxLOLLJQsDwy OhNGzOKImUmmf2tME41umpsFmCDHTewqpdaEh4PzewxPe3cmMXsPh8Duc7LmISIlsNCZ iPA83zYgbzNVIbvjuI3ZMRNF/AZRcwVo9jaU0RNROIoeDpjb7e+7UkIjXEolpbXMgIR/ XHXLXgVrjkfi+0YB/03fXSWi+oMaI1Q/EdlE5KxfyTLSyuaGUnp1sW2sUp/8l6L1+r58 teLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=ex1wSMFXCd+iEZ2Gnc67TsjgDfMi5qWzDe2P6IY/X6Q=; b=zVGaArD4C8sViEgyCllSyJS3zMGprFQaITkp4YwppQ4akfSSoJIek8z8LmJCAT9H/d jMnwHXYDfOCT+uio491CN5TaeyUgN1uQEjTUckssEasadcNrYFwdWdH/stKxMoIrKrim MJj2RWXjjTzzFAuJfczGxtBYUDJp5RNPAzFyCcm9bCE+KbCpZwcD8fFNaBUsQcVgYVlq CRrRZxviPjBuef/dFDGDXv/Muk3FQi1CoUMEGFcsV2ZgC6KC5fszdoYrXLeIijn8bGs+ 3P1sCtqUUQqjMbkt/eK8+Cf+VL9Q3t//iviv94IIW1ga3s4RDqN828o8uOIbtzLYM5Mk AZJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=fRovmqk0; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id fi12-20020a056402550c00b004833b85ba26si13802886edb.403.2023.01.11.06.05.55; Wed, 11 Jan 2023 06:06:15 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=fRovmqk0; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7E75C68BC47; Wed, 11 Jan 2023 16:05:51 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B4E8C68A701 for ; Wed, 11 Jan 2023 16:05:44 +0200 (EET) Received: by mail-pl1-f169.google.com with SMTP id d9so16859588pll.9 for ; Wed, 11 Jan 2023 06:05:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=IXZlOu1buq84Yi9iQVdVO4jz83G5RYcIzma7IvgYo1w=; b=fRovmqk0bJRLwzvCBuQgWj8tia+P0EXXsZxgekcbXjRQNBdczFy4WQcb+p6q1qii8U CPeE1bBmGwcQFzWjIwmefHxlm6SOEuQb+CTR8+/XsCQcUt8GwCkJDh7llDwOrE4S5hU7 gxwRQ/FMGc7A8ih9ds7ZF9Dm9uld5CpACoyx7c9kmVl4OMVqE0VRyCxSvKBa/+l+xOI6 wE3IMK/+Q4CjuD2ADg1M0Ww0pdq6V9Io1xBqr7iYqzj/Czj2G0rbk6INWlG/NEZRDZlQ KY5dnLiXy2Xk1bBbhh03HWn6LFgsGj2Eg5fwDpP63+m6Rke9WZEsL1CP58LKW3bNDT6X w27A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IXZlOu1buq84Yi9iQVdVO4jz83G5RYcIzma7IvgYo1w=; b=QTWhzRo9+CjCZwLiAInZSZQtb459mn2EovSV+fwVs51m3QNu+juRT2p0IWeaHAt1WN KIE/JvJ9sff6D8BzBBjLLOwe5tjqp2CSVuNSuWzP+kwUCVPB9qCz/Od7q5at/vTaKJag SiBbLYUldtOMHcfJ/foLqIi501I+Dd6A3sTlAQROVnZywm6zncAO2KFJp7E0qMWzon6F BeVPjFgOsvmfIwH67IrB+FsgqfMTPIzuIxgEEHiy5DebRJhMnc8Q3rjljRtEdr1ZTwWd O5SYT3Ma5KZCuG/vKBy/8aOSsYleP70/hrPDDcOv4Uq/v9kdeQ4D3GN5xgf0QlNHcDdx ihfA== X-Gm-Message-State: AFqh2koGJlxTaKQBiglmUDDOJ2gBxeTPUlSd9b8ArVHM/9KA6J/SZ/ua haqQ8H66Ioddy0Y0JEILtc+HoTFikr84EkRn X-Received: by 2002:a17:902:6bcb:b0:192:eb8d:4d62 with SMTP id m11-20020a1709026bcb00b00192eb8d4d62mr23714967plt.13.1673445942120; Wed, 11 Jan 2023 06:05:42 -0800 (PST) Received: from localhost.localdomain ([111.108.111.139]) by smtp.gmail.com with ESMTPSA id z17-20020a170903019100b00192849d1209sm2877545plg.96.2023.01.11.06.05.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 11 Jan 2023 06:05:41 -0800 (PST) From: liuyizhuo To: ffmpeg-devel@ffmpeg.org Date: Wed, 11 Jan 2023 22:05:32 +0800 Message-Id: <20230111140532.26467-1-yizhuo.liu753@gmail.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [add video filter hsl(Hue, Saturation, Lightness)] X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: wangwei1237@gmail.com, "2498228118@qq.com" <2498228118@qq.com> Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: kgCkcJ0+P5Ib From: "2498228118@qq.com" <2498228118@qq.com> --- libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_hsl.c | 310 ++++++++++++++++++++++++++++++++++++ tests/fate-run.sh | 2 + tests/fate/filter-video.mak | 3 + tests/ref/fate/filter-hsl | 1 + 6 files changed, 318 insertions(+) create mode 100644 libavfilter/vf_hsl.c create mode 100644 tests/ref/fate/filter-hsl diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 5783be281d..b9ed9e065a 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -182,6 +182,7 @@ OBJS-$(CONFIG_SINE_FILTER) += asrc_sine.o OBJS-$(CONFIG_ANULLSINK_FILTER) += asink_anullsink.o # video filters +OBJS-$(CONFIG_HSL_FILTER) += vf_hsl.o OBJS-$(CONFIG_ADDROI_FILTER) += vf_addroi.o OBJS-$(CONFIG_ALPHAEXTRACT_FILTER) += vf_extractplanes.o OBJS-$(CONFIG_ALPHAMERGE_FILTER) += vf_alphamerge.o framesync.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 52741b60e4..6910189ce4 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -169,6 +169,7 @@ extern const AVFilter ff_asrc_sine; extern const AVFilter ff_asink_anullsink; +extern const AVFilter ff_vf_hsl; extern const AVFilter ff_vf_addroi; extern const AVFilter ff_vf_alphaextract; extern const AVFilter ff_vf_alphamerge; diff --git a/libavfilter/vf_hsl.c b/libavfilter/vf_hsl.c new file mode 100644 index 0000000000..47c9af9120 --- /dev/null +++ b/libavfilter/vf_hsl.c @@ -0,0 +1,310 @@ +/* + * Copyright (c) 2022 Wang Wei + * Copyright (c) 2022 Liu yizhuo + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * Calculate the HSL color space information. + * HSL is a color space that is more perceptually uniform than RGB : + * https://en.wikipedia.org/wiki/HSL_and_HSV + */ + +#include + +#include "libavutil/imgutils.h" +#include "libavutil/internal.h" +#include "libavutil/opt.h" +#include "libswscale/swscale.h" + +#include "avfilter.h" +#include "formats.h" +#include "internal.h" +#include "video.h" + +typedef float num; + +typedef struct HSL_COLOR { + num H; + num S; + num L; +} HSL_COLOR; + +typedef struct HSLContext { + const AVClass *class; + int width, height; + uint64_t nb_frames; + float max_hue, max_sat, max_light; + float min_hue, min_sat, min_light; + float sum_hue, sum_sat, sum_light; + int print_summary; + AVFrame *rgbFrame; + enum AVPixelFormat rgb_format; +} HSLContext; + +static const enum AVPixelFormat pix_fmts[] = { + AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P, + AV_PIX_FMT_YUVJ420P, AV_PIX_FMT_YUVJ422P, + AV_PIX_FMT_YUV420P10, AV_PIX_FMT_YUV422P10, + AV_PIX_FMT_NONE +}; + +static av_cold int init(AVFilterContext *ctx) +{ + // User options but no input data + HSLContext *s = ctx->priv; + s->max_hue = 0; + s->max_sat = 0; + s->max_light = 0; + + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + HSLContext *s = ctx->priv; + + if (s->print_summary) { + float avg_hue = s->sum_hue / s->nb_frames; + float avg_sat = s->sum_sat / s->nb_frames; + float avg_light = s->sum_light / s->nb_frames; + av_log(ctx, AV_LOG_INFO, + "HSL Summary:\nTotal frames: %"PRId64"\n\n" + "Hue Information:\nAverage: %.1f\nMax: %.1f\nMin: %.1f\n\n" + "Saturation Information:\nAverage: %.1f\nMax: %.1f\nMin: %.1f\n\n" + "Lightness Information:\nAverage: %.1f\nMax: %.1f\nMin: %.1f\n\n", + s->nb_frames, + avg_hue, s->max_hue, s->min_hue, + avg_sat * 100, s->max_sat * 100, s->min_sat * 100, + avg_light * 100, s->max_light * 100, s->min_light * 100 + ); + } + + av_frame_free(&s->rgbFrame); +} + +static int config_input(AVFilterLink *inlink) +{ + // Video input data avilable + AVFilterContext *ctx = inlink->dst; + HSLContext *s = ctx->priv; + + // free previous buffers in case they are allocated already + av_frame_free(&s->rgbFrame); + s->width = inlink->w; + s->height = inlink->h; + s->rgb_format = AV_PIX_FMT_RGB24; + s->rgbFrame = av_frame_alloc(); + if (!s->rgbFrame) { + return AVERROR(ENOMEM); + } + + s->rgbFrame->format = s->rgb_format; + s->rgbFrame->width = s->width; + s->rgbFrame->height = s->height; + if (av_frame_get_buffer(s->rgbFrame, 0)) { + return AVERROR(ENOMEM); + } + + return 0; +} + +static void set_meta(AVDictionary **metadata, const char *key, float d) +{ + char value[128]; + snprintf(value, sizeof(value), "%0.1f", d); + av_dict_set(metadata, key, value, 0); +} + +static void YUV2RGB(const AVFrame* src, enum AVPixelFormat dstFormat, AVFrame* dst) +{ + int width = src->width; + int height = src->height; + + struct SwsContext* conversion = NULL; + conversion = sws_getContext(width, + height, + (enum AVPixelFormat)src->format, + width, + height, + dstFormat, + SWS_FAST_BILINEAR, + NULL, + NULL, + NULL); + sws_scale(conversion, src->data, src->linesize, 0, height, dst->data, dst->linesize); + sws_freeContext(conversion); +} + +static const float EPSILON = 1e-9; + +/** @brief Min of A and B */ +#define MIN(A,B) (((A) <= (B)) ? (A) : (B)) + +/** @brief Max of A and B */ +#define MAX(A,B) (((A) >= (B)) ? (A) : (B)) + +/** @brief Min of A, B, and C */ +#define MIN3(A,B,C) (((A) <= (B)) ? MIN(A,C) : MIN(B,C)) + +/** @brief Max of A, B, and C */ +#define MAX3(A,B,C) (((A) >= (B)) ? MAX(A,C) : MAX(B,C)) + +/** @brief Equal of A and B */ +#define EQ(A,B) ((fabs((A) - (B)) < EPSILON) ? 1 : 0) + + +/** + * @brief Convert an sRGB color to Hue-Saturation-Lightness (HSL) + * + * @param H, S, L pointers to hold the result + * @param R, G, B the input sRGB values scaled in [0,1] + * + * This routine transforms from sRGB to the double hexcone HSL color space + * The sRGB values are assumed to be between 0 and 1. The outputs are + * H = hexagonal hue angle (0 <= H < 360), + * S = { C/(2L) if L <= 1/2 (0 <= S <= 1), + * { C/(2 - 2L) if L > 1/2 + * L = (max(R',G',B') + min(R',G',B'))/2 (0 <= L <= 1), + * where C = max(R',G',B') - min(R',G',B'). + * + * Wikipedia: http://en.wikipedia.org/wiki/HSL_and_HSV + */ +static void RGB2HSL(num *H, num *S, num *L, num R, num G, num B) +{ + num Max = MAX3(R, G, B); + num Min = MIN3(R, G, B); + num C = Max - Min; + + *L = (Max + Min) / 2; + + if (C > 0) { + if (EQ(Max, R)) { + *H = (G - B) / C; + + if (G < B) { + *H += 6; + } + } else if (EQ(Max, G)) { + *H = 2 + (B - R) / C; + } else { + *H = 4 + (R - G) / C; + } + + *H *= 60; + *S = (*L <= 0.5) ? (C/(2*(*L))) : (C/(2 - 2*(*L))); + } else { + *H = *S = 0; + } +} + +static void calcHSL(HSLContext *ctx, AVFrame *frame, HSL_COLOR *hsl) +{ + // Convert to RGB + YUV2RGB(frame, ctx->rgb_format, ctx->rgbFrame); + + int cnt = frame->width * frame->height; + num R = 0, G = 0, B = 0; + num H = 0, S = 0, L = 0; + num H_sum = 0, S_sum = 0, L_sum = 0; + + for (int i = 0; i < cnt; i++) { + R = ctx->rgbFrame->data[0][3 * i]; + G = ctx->rgbFrame->data[0][3 * i + 1]; + B = ctx->rgbFrame->data[0][3 * i + 2]; + RGB2HSL(&H, &S, &L, R/255, G/255, B/255); + H_sum += H; + S_sum += S; + L_sum += L; + } + + hsl->H = H_sum / cnt; + hsl->S = S_sum / cnt; + hsl->L = L_sum / cnt; + + // av_log(ctx, AV_LOG_INFO, "H: %.1f, S: %.1f, L: %.1f\n", + // hsl->H, hsl->S * 100, hsl->L * 100); +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + HSLContext *s = ctx->priv; + s->nb_frames++; + + HSL_COLOR hsl; + calcHSL(s, frame, &hsl); + + // Calculate statistics + s->max_hue = fmaxf(hsl.H, s->max_hue); + s->max_sat = fmaxf(hsl.S, s->max_sat); + s->max_light = fmaxf(hsl.L, s->max_light); + s->sum_hue += hsl.H; + s->sum_sat += hsl.S; + s->sum_light += hsl.L; + s->min_hue = s->nb_frames == 1 ? hsl.H : fminf(hsl.H, s->min_hue); + s->min_sat = s->nb_frames == 1 ? hsl.S : fminf(hsl.S, s->min_sat); + s->min_light = s->nb_frames == 1 ? hsl.L : fminf(hsl.L, s->min_light); + + // Set HSL information in frame metadata + set_meta(&frame->metadata, "lavfi.hsl.hue", hsl.H); + set_meta(&frame->metadata, "lavfi.hsl.sat", hsl.S * 100); + set_meta(&frame->metadata, "lavfi.hsl.light", hsl.L * 100); + + return ff_filter_frame(inlink->dst->outputs[0], frame); +} + +#define OFFSET(x) offsetof(HSLContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_FILTERING_PARAM + +static const AVOption hsl_options[] = { + { "print_summary", "Print summary showing average values", OFFSET(print_summary), AV_OPT_TYPE_BOOL, { .i64=0 }, 0, 1, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(hsl); + +static const AVFilterPad avfilter_vf_hsl_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input, + .filter_frame = filter_frame, + }, +}; + +static const AVFilterPad avfilter_vf_hsl_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO + }, +}; + +const AVFilter ff_vf_hsl = { + .name = "hsl", + .description = NULL_IF_CONFIG_SMALL("Calculate the HSL color space information."), + .priv_size = sizeof(HSLContext), + .priv_class = &hsl_class, + .init = init, + .uninit = uninit, + .flags = AVFILTER_FLAG_METADATA_ONLY, + FILTER_PIXFMTS_ARRAY(pix_fmts), + FILTER_INPUTS(avfilter_vf_hsl_inputs), + FILTER_OUTPUTS(avfilter_vf_hsl_outputs), +}; diff --git a/tests/fate-run.sh b/tests/fate-run.sh index 61cc59acc0..c350570f5b 100755 --- a/tests/fate-run.sh +++ b/tests/fate-run.sh @@ -1,5 +1,7 @@ #! /bin/sh +set -x + export LC_ALL=C base=$(dirname $0) diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak index 63873a7a07..aa9812aa18 100644 --- a/tests/fate/filter-video.mak +++ b/tests/fate/filter-video.mak @@ -408,6 +408,9 @@ fate-filter-scale200: CMD = video_filter "scale=w=200:h=200" FATE_FILTER_VSYNTH_VIDEO_FILTER-$(CONFIG_SCALE_FILTER) += fate-filter-scale500 fate-filter-scale500: CMD = video_filter "scale=w=500:h=500" +FATE_FILTER_VSYNTH_VIDEO_FILTER-$(CONFIG_SCALE_FILTER) += fate-filter-hsl +fate-filter-hsl: CMD = video_filter "hsl=print_summary=1" + FATE_FILTER_VSYNTH-$(call ALLYES, TESTSRC_FILTER SCALE2REF_FILTER NULLSINK_FILTER FRAMEMD5_MUXER FILE_PROTOCOL PIPE_PROTOCOL) += fate-filter-scale2ref_keep_aspect fate-filter-scale2ref_keep_aspect: tests/data/filtergraphs/scale2ref_keep_aspect fate-filter-scale2ref_keep_aspect: CMD = framemd5 -frames:v 5 -filter_complex_script $(TARGET_PATH)/tests/data/filtergraphs/scale2ref_keep_aspect -map "[main]" diff --git a/tests/ref/fate/filter-hsl b/tests/ref/fate/filter-hsl new file mode 100644 index 0000000000..ae0a064f46 --- /dev/null +++ b/tests/ref/fate/filter-hsl @@ -0,0 +1 @@ +hsl fcb007249fba9371fe84a61c974fcb00