From patchwork Sat Nov 5 15:26:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?Q2zDqW1lbnQgQsWTc2No?= X-Patchwork-Id: 39173 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp1085686pzb; Sat, 5 Nov 2022 08:28:04 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7HwGPQ6g8jQ8sX1DJhAqBHwr3wnC7Qqqf6JsNeuagy/UIxCBaglOdyAWU0DTRT4h5Aerrg X-Received: by 2002:a17:907:7e95:b0:78d:e9cf:82c7 with SMTP id qb21-20020a1709077e9500b0078de9cf82c7mr41006250ejc.724.1667662084682; Sat, 05 Nov 2022 08:28:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667662084; cv=none; d=google.com; s=arc-20160816; b=gWHI1dDOastW/uRP8w9CKTX8gL35PRv1BNkpZlotPJHu9kAMCbqwrE8KDpGAVR0n/Q ObLQxpoBQHDYzS8F89c1K5hM6Y/xgdYXEM/mCDatPKW98iNYR1Z4ke7VfqR9tr1QSWtx oe3ob4+7CwCJhvWWe0RnCJSWmNBWvPGqf99F3SAj8ozjFaJIB0NquV0X4exVz2ZYKndl X79UvcMQNM4gty9Q3q8hNJANVl+k+seKcvtZZy8JZafSZtEywnxIM5FhTaiEhlkH0szm /zDRjNDCg6rKEO50XIHFXMNRfZYqzZ/Ryo+GtMIu4E3TDsTL1m6tzrpt/s8X+XXarAMv oygw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=96vTgnOkEmt8PbF8i664h8QiEshtakrBEwUqkL8HSSQ=; b=yzaYM4IeUhSX9ZsQzzZNf4AKZ6Z9IjjuJ967bGYK9oDzXe+D7rRSXybTAA1oA9MuBu RKWScjh8bC53n+lksX2S5ErmV2oEcP8j3Q8zDDEJZh8UFRsP/76EIWg7MYMj+5JOnK5W VwS5YethBo3xoCeOovl+gsOiNsWFIwKwcz3K0s3iVeJA4XGdCCfytvYAgkSF9kiZGwP4 W8e0u+mDt65VZ9E093WqzFvk1mECcYOH6FZ+l2piW7i3BbABJQlGoc1sP6YlfjF2bpOF IibBbYldljL35MzYFg1EHjBLDq5hsXzZYj/xhp+W3XREgP3XOYeivrmkz17b98Xsphay ihkQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@pkh.me header.s=selector1 header.b=PO8TAcsW; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=pkh.me Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id k24-20020aa7d2d8000000b004646bcdd9f9si2470879edr.486.2022.11.05.08.28.04; Sat, 05 Nov 2022 08:28:04 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@pkh.me header.s=selector1 header.b=PO8TAcsW; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=pkh.me Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9084F68B371; Sat, 5 Nov 2022 17:26:45 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from ssq0.pkh.me (laubervilliers-656-1-228-164.w92-154.abo.wanadoo.fr [92.154.28.164]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 506FF68B469 for ; Sat, 5 Nov 2022 17:26:41 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pkh.me; s=selector1; t=1667661982; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QN0jTJzp3X7b8KCMHO0OIGFUbzN9O4j4GCOluwq6RvQ=; b=PO8TAcsW8HmrBiEGZESSwykJ/7XSHLBvhNLmlydIEgN48U4VXr+9ZI0VtIhVx0gYXcmNsy GRWujYQTR7NMRLa89OagzG74uFJtDyD59iijGh+54R2Rt1LS9JwTBo8M6Zy7VBaIiOm47N BNZ4zr0v5CnT6NQnVm2RKUdcIxVqRzM= Received: from localhost (ssq0.pkh.me [local]) by ssq0.pkh.me (OpenSMTPD) with ESMTPA id f512bc37; Sat, 5 Nov 2022 15:26:22 +0000 (UTC) From: =?utf-8?b?Q2zDqW1lbnQgQsWTc2No?= To: ffmpeg-devel@ffmpeg.org Date: Sat, 5 Nov 2022 16:26:14 +0100 Message-Id: <20221105152617.1809282-13-u@pkh.me> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221105152617.1809282-1-u@pkh.me> References: <20221105152617.1809282-1-u@pkh.me> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 12/15] avfilter/palettegen: base split decision on a perceptual model X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: =?utf-8?b?Q2zDqW1lbnQgQsWTc2No?= Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 8Cu/Rg7CCG02 Similar to the change in paletteuse, we rely on a perceptual model to decide how and where to split the box. --- libavfilter/Makefile | 2 +- libavfilter/vf_palettegen.c | 79 ++++++++++++++++-------------- tests/ref/fate/filter-palettegen-1 | 2 +- tests/ref/fate/filter-palettegen-2 | 2 +- 4 files changed, 44 insertions(+), 41 deletions(-) diff --git a/libavfilter/Makefile b/libavfilter/Makefile index e6b6d59d2d..0a31b76c6a 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -401,7 +401,7 @@ OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) += vf_overlay_vulkan.o vulkan.o vul OBJS-$(CONFIG_OWDENOISE_FILTER) += vf_owdenoise.o OBJS-$(CONFIG_PAD_FILTER) += vf_pad.o OBJS-$(CONFIG_PAD_OPENCL_FILTER) += vf_pad_opencl.o opencl.o opencl/pad.o -OBJS-$(CONFIG_PALETTEGEN_FILTER) += vf_palettegen.o +OBJS-$(CONFIG_PALETTEGEN_FILTER) += vf_palettegen.o palette.o OBJS-$(CONFIG_PALETTEUSE_FILTER) += vf_paletteuse.o framesync.o palette.o OBJS-$(CONFIG_PERMS_FILTER) += f_perms.o OBJS-$(CONFIG_PERSPECTIVE_FILTER) += vf_perspective.o diff --git a/libavfilter/vf_palettegen.c b/libavfilter/vf_palettegen.c index b8e4463539..4c2bcba7f7 100644 --- a/libavfilter/vf_palettegen.c +++ b/libavfilter/vf_palettegen.c @@ -23,6 +23,8 @@ * Generate one palette for a whole video stream. */ +#include + #include "libavutil/avassert.h" #include "libavutil/internal.h" #include "libavutil/opt.h" @@ -35,13 +37,14 @@ /* Reference a color and how much it's used */ struct color_ref { uint32_t color; + struct Lab lab; uint64_t count; }; /* Store a range of colors */ struct range_box { uint32_t color; // average color - int64_t variance; // overall variance of the box (how much the colors are spread) + double variance; // overall variance of the box (how much the colors are spread) int start; // index in PaletteGenContext->refs int len; // number of referenced colors int sorted_by; // whether range of colors is sorted by red (0), green (1) or blue (2) @@ -109,20 +112,19 @@ static int query_formats(AVFilterContext *ctx) typedef int (*cmp_func)(const void *, const void *); -#define DECLARE_CMP_FUNC(name, pos) \ +#define DECLARE_CMP_FUNC(name) \ static int cmp_##name(const void *pa, const void *pb) \ { \ const struct color_ref * const *a = pa; \ const struct color_ref * const *b = pb; \ - return (int)((*a)->color >> (8 * (2 - (pos))) & 0xff) \ - - (int)((*b)->color >> (8 * (2 - (pos))) & 0xff); \ + return FFDIFFSIGN((*a)->lab.name, (*b)->lab.name); \ } -DECLARE_CMP_FUNC(r, 0) -DECLARE_CMP_FUNC(g, 1) -DECLARE_CMP_FUNC(b, 2) +DECLARE_CMP_FUNC(L) +DECLARE_CMP_FUNC(a) +DECLARE_CMP_FUNC(b) -static const cmp_func cmp_funcs[] = {cmp_r, cmp_g, cmp_b}; +static const cmp_func cmp_funcs[] = {cmp_L, cmp_a, cmp_b}; /** * Simple color comparison for sorting the final palette @@ -134,19 +136,19 @@ static int cmp_color(const void *a, const void *b) return FFDIFFSIGN(box1->color , box2->color); } -static av_always_inline int diff(const uint32_t a, const uint32_t b) +static av_always_inline float diff(const uint32_t a, const uint32_t b) { - const uint8_t c1[] = {a >> 16 & 0xff, a >> 8 & 0xff, a & 0xff}; - const uint8_t c2[] = {b >> 16 & 0xff, b >> 8 & 0xff, b & 0xff}; - const int dr = c1[0] - c2[0]; - const int dg = c1[1] - c2[1]; - const int db = c1[2] - c2[2]; - return dr*dr + dg*dg + db*db; + const struct Lab lab0 = ff_srgb_u8_to_oklab(a); + const struct Lab lab1 = ff_srgb_u8_to_oklab(b); + const float dL = lab0.L - lab1.L; + const float da = lab0.a - lab1.a; + const float db = lab0.b - lab1.b; + return dL*dL + da*da + db*db; } static void compute_box_variance(PaletteGenContext *s, struct range_box *box) { - int64_t variance = 0; + double variance = 0.0; for (int i = 0; i < box->len; i++) { const struct color_ref *ref = s->refs[box->start + i]; @@ -179,7 +181,7 @@ static void compute_box_variance(PaletteGenContext *s, struct range_box *box) static int get_next_box_id_to_split(PaletteGenContext *s) { int box_id, best_box_id = -1; - int64_t max_variance = -1; + double max_variance = -1.0; if (s->nb_boxes == s->max_colors - s->reserve_transparent) return -1; @@ -188,14 +190,14 @@ static int get_next_box_id_to_split(PaletteGenContext *s) struct range_box *box = &s->boxes[box_id]; if (s->boxes[box_id].len >= 2) { - if (box->variance == -1) + if (box->variance == -1.0) compute_box_variance(s, box); if (box->variance > max_variance) { best_box_id = box_id; max_variance = box->variance; } } else { - box->variance = -1; + box->variance = -1.0; } } return best_box_id; @@ -245,8 +247,8 @@ static void split_box(PaletteGenContext *s, struct range_box *box, int n) box->color = get_avg_color(s->refs, box); new_box->color = get_avg_color(s->refs, new_box); - box->variance = -1; - new_box->variance = -1; + box->variance = -1.0; + new_box->variance = -1.0; } /** @@ -343,39 +345,39 @@ static AVFrame *get_palette_frame(AVFilterContext *ctx) box->len = s->nb_refs; box->sorted_by = -1; box->color = get_avg_color(s->refs, box); - box->variance = -1; + box->variance = -1.0; s->nb_boxes = 1; while (box && box->len > 1) { - int i, rr, gr, br, longest; + int i, longest; + double Lr, ar, br; uint64_t median, box_weight = 0; /* compute the box weight (sum all the weights of the colors in the * range) and its boundings */ - uint8_t min[3] = {0xff, 0xff, 0xff}; - uint8_t max[3] = {0x00, 0x00, 0x00}; + float min[3] = {FLT_MAX, FLT_MAX, FLT_MAX}; + float max[3] = {-FLT_MAX, -FLT_MAX, -FLT_MAX}; for (i = box->start; i < box->start + box->len; i++) { const struct color_ref *ref = s->refs[i]; - const uint32_t rgb = ref->color; - const uint8_t r = rgb >> 16 & 0xff, g = rgb >> 8 & 0xff, b = rgb & 0xff; - min[0] = FFMIN(r, min[0]), max[0] = FFMAX(r, max[0]); - min[1] = FFMIN(g, min[1]), max[1] = FFMAX(g, max[1]); - min[2] = FFMIN(b, min[2]), max[2] = FFMAX(b, max[2]); + const struct Lab lab = ref->lab; + min[0] = FFMIN(lab.L, min[0]), max[0] = FFMAX(lab.L, max[0]); + min[1] = FFMIN(lab.a, min[1]), max[1] = FFMAX(lab.a, max[1]); + min[2] = FFMIN(lab.b, min[2]), max[2] = FFMAX(lab.b, max[2]); box_weight += ref->count; } /* define the axis to sort by according to the widest range of colors */ - rr = max[0] - min[0]; - gr = max[1] - min[1]; + Lr = max[0] - min[0]; + ar = max[1] - min[1]; br = max[2] - min[2]; - longest = 1; // pick green by default (the color the eye is the most sensitive to) - if (br >= rr && br >= gr) longest = 2; - if (rr >= gr && rr >= br) longest = 0; - if (gr >= rr && gr >= br) longest = 1; // prefer green again + longest = 0; + if (br >= Lr && br >= ar) longest = 2; + if (ar >= Lr && ar >= br) longest = 1; + if (Lr >= ar && Lr >= br) longest = 0; - ff_dlog(ctx, "box #%02X [%6d..%-6d] (%6d) w:%-6"PRIu64" ranges:[%2x %2x %2x] sort by %c (already sorted:%c) ", + ff_dlog(ctx, "box #%02X [%6d..%-6d] (%6d) w:%-6"PRIu64" ranges:[%.3f %.3f %.3f] sort by %c (already sorted:%c) ", box_id, box->start, box->start + box->len - 1, box->len, box_weight, - rr, gr, br, "rgb"[longest], box->sorted_by == longest ? 'y':'n'); + Lr, ar, br, "Lab"[longest], box->sorted_by == longest ? 'y':'n'); /* sort the range by its longest axis if it's not already sorted */ if (box->sorted_by != longest) { @@ -449,6 +451,7 @@ static int color_inc(struct hist_node *hist, uint32_t color) if (!e) return AVERROR(ENOMEM); e->color = color; + e->lab = ff_srgb_u8_to_oklab(color); e->count = 1; return 1; } diff --git a/tests/ref/fate/filter-palettegen-1 b/tests/ref/fate/filter-palettegen-1 index df3b714ebb..7b7ce98b76 100644 --- a/tests/ref/fate/filter-palettegen-1 +++ b/tests/ref/fate/filter-palettegen-1 @@ -3,4 +3,4 @@ #codec_id 0: rawvideo #dimensions 0: 16x16 #sar 0: 1/1 -0, 0, 0, 1, 1024, 0x69ec37aa +0, 0, 0, 1, 1024, 0xf1fb64c1 diff --git a/tests/ref/fate/filter-palettegen-2 b/tests/ref/fate/filter-palettegen-2 index 08320a8359..b856a79273 100644 --- a/tests/ref/fate/filter-palettegen-2 +++ b/tests/ref/fate/filter-palettegen-2 @@ -3,4 +3,4 @@ #codec_id 0: rawvideo #dimensions 0: 16x16 #sar 0: 1/1 -0, 0, 0, 1, 1024, 0x76078b2e +0, 0, 0, 1, 1024, 0xe84a671a