From patchwork Thu Oct 5 14:43:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Bjorn Roche X-Patchwork-Id: 5427 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.161.76 with SMTP id m12csp2115397jah; Thu, 5 Oct 2017 07:51:02 -0700 (PDT) X-Received: by 10.28.174.67 with SMTP id x64mr21023144wme.82.1507215062263; Thu, 05 Oct 2017 07:51:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507215062; cv=none; d=google.com; s=arc-20160816; b=Ug9aJ2T8wqehMxRxQ+/lDrvYwZZsb3zhzNyO/jlXO32lNY/nYNQtXVNDonwGDtl73u mnEpiOK3ma2L05MC07iUMfWeyf7VSdP19Tkp3QG0Fbo14U9JbQ0fY0NJzkKQgwZKVn33 Zp4+CaGKWp4ro4gDb694q2vh20auP1lhFxh9IlcSNnwihUoJtZBGDD+01g8Y9wcW+Jom Rbdlpj6QnQehQPFtnFiJPcQmc3Zt3X1MPBPPc+gslSNgMJVXostxqivEYgS1vn+QGlw/ JdaMOh3Oqf1CBAWdsJyxbMLc6VxyHaphNgNeA4V7dNk7nwigb4s+tDzWLhyzRrfwCXlI 2yiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject:to :message-id:date:from:references:in-reply-to:mime-version :dkim-signature:delivered-to:arc-authentication-results; bh=AYHvf9RHQmfqEZJXaMx+fe2+eYrC+5rAbrlf3LJJuIs=; b=m7I1U78EC8Kh+NTIVIRr9UEm8AKM1RkErguL7ViNd4d64RGS0gWsZ9alZXtLVknZzU ibPOzibhPzr45XEjZ3aBUTH1vc3ABAUjCDPzCfuzOCyi/CMkLCXJtCXO3zzwFFhyHSQz mn7v2YxE1zGsLvOwV7JMQ3LWVRSS8JY7YAkJZ0stWaRBHk09O8Zcn6zWkwWrj9cjHRjA sSH0CeRMbjrUs74vTaVoMVsrCBjjK9Z8s36r2iyGlQp4dGEp65SIJs0hqT0qn9817TUu 6CFc5Z4aeUHiQwgKa5du13Xj9EzW1hnaoSCuLaoaPNbqa46u7jPym53DkrS4Hl2k10kT cN1A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@giphy.com header.s=google header.b=lQo5fS5t; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id k73si2452417wrc.84.2017.10.05.07.51.01; Thu, 05 Oct 2017 07:51:02 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@giphy.com header.s=google header.b=lQo5fS5t; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2732568A350; Thu, 5 Oct 2017 17:50:43 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oi0-f45.google.com (mail-oi0-f45.google.com [209.85.218.45]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2E2F768A329 for ; Thu, 5 Oct 2017 17:50:36 +0300 (EEST) Received: by mail-oi0-f45.google.com with SMTP id c77so22962114oig.0 for ; Thu, 05 Oct 2017 07:50:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=giphy.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=WMd+uGgnPnDQORQyMfwfXyYa6bAxYGz1jAVHp+cKsDo=; b=lQo5fS5tRKy1a5EafnoqeGKhJBbY0iWzo+O34EPFI0wTUG2xrGs7056mLUyF/FUpUQ FmIwUvjkK2dTeQm2CI//fN4kji1Ks4CW5eCrmzXR6f9aLRaW0CWE1VlvXu+jEPaAuKN5 Md++yfTqQ7B/T6VpSP7spn5otYpM8ZlzhepBw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=WMd+uGgnPnDQORQyMfwfXyYa6bAxYGz1jAVHp+cKsDo=; b=HMQZ6nt+iVXshHABzqt7JwCrZf9OOd14YATyLMmdl4C5G17S9M+M02OoBjs75GIZyJ FPk6EfQ2YY8lYRKovxlnfYivBGnnfovgtHSoTm9odkGIqTGz7L56CcyZkV3unKFitD/H Z57qwA3+yHNaueLaH4085uXGpbnNTJBJcFNl1pCAWNAhpDuI0nF/+O306z93WWLjQ/Jo c7o6Akj0UNyr0M0gO0k9vTeVP+vlbZW2BChc/F7E8Sj3MGq4VlDruD4yxtBSZBx5DDb4 nykTgebv0G2U/YzhxBBMXwI7i36h/utWN/y1aejlDorGR3OVxf0/F/MISoiFK8KxloFD RDxw== X-Gm-Message-State: AMCzsaWzd0IlyFhYE4sj1p9CBOs5uMxRmBMz2zb9vqVx6Ns5g4pF/l5u jErVLb2eAzexoSH9BdRwoZgK4vzYeI2H3PNL/V94rKin X-Google-Smtp-Source: AOwi7QDFXp17yy5HUh/N+7TDFP6CsinpO6CUMY/9UnUF66LqvDKZcmxp2EWsbWOFe+tbo1i8DDXmNKV4Vp13bVP98Ec= X-Received: by 10.157.35.11 with SMTP id j11mr6916607otb.167.1507214614688; Thu, 05 Oct 2017 07:43:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.17.72 with HTTP; Thu, 5 Oct 2017 07:43:04 -0700 (PDT) In-Reply-To: References: <20171002172439.65628-1-bjorn@giphy.com> <20171002172439.65628-4-bjorn@giphy.com> From: Bjorn Roche Date: Thu, 5 Oct 2017 10:43:04 -0400 Message-ID: To: FFmpeg development discussions and patches X-Content-Filtered-By: Mailman/MimeDel 2.1.20 Subject: Re: [FFmpeg-devel] [PATCH 3/7] Removing some debugging X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" On Tue, Oct 3, 2017 at 4:02 AM, Carl Eugen Hoyos wrote: > 2017-10-03 1:52 GMT+02:00 Bjorn Roche : > > > Attached is a patch for paletteuse only. > > I tested the following with and without your patch: > $ ffmpeg -i fate-suite/lena.pnm -vf palettegen pal.png > $ ffmpeg -i fate-suite/lena.pnm -i pal.png -lavfi paletteuse out.png > > out.png changes with your patch: Is that intended? > Not intended, but not unexpected -- the ordering of colors in the palette changed to make it easier to deal with transparency. However, there are visual differences, so that's obviously a bug. I've supplied a new patch (below) that fixes that bug, and maintains the order so it's easier to test for that. > The input frame contains no transparency. > > Your patch still contains printfs that should be removed > (change them to av_log(DEBUG) if you think they are > useful) and a new warning is shown at compilation: > libavfilter/vf_paletteuse.c: In function ‘colormap_nearest_recursive’: > libavfilter/vf_paletteuse.c:242:5: warning: ISO C90 forbids mixed > declarations and code > > Please fix the style: Space after "if", no space after "if (" > Thank you for your attention. This should also be resolved in the attached patch. I have tested fate, all dithering (including none, which I added) and all color search algorithms. diff --git a/libavfilter/vf_paletteuse.c b/libavfilter/vf_paletteuse.c index ffd37bf1da..7cb5a3db02 100644 --- a/libavfilter/vf_paletteuse.c +++ b/libavfilter/vf_paletteuse.c @@ -56,7 +56,7 @@ enum diff_mode { }; struct color_node { - uint8_t val[3]; + uint8_t val[4]; uint8_t palette_id; int split; int left_id, right_id; @@ -86,6 +86,7 @@ typedef struct PaletteUseContext { struct cache_node cache[CACHE_SIZE]; /* lookup cache */ struct color_node map[AVPALETTE_COUNT]; /* 3D-Tree (KD-Tree with K=3) for reverse colormap */ uint32_t palette[AVPALETTE_COUNT]; + int transparency_index; /* index in the palette of transparency. -1 if there isn't a transparency. */ int palette_loaded; int dither; int new; @@ -108,6 +109,7 @@ typedef struct PaletteUseContext { #define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM static const AVOption paletteuse_options[] = { { "dither", "select dithering mode", OFFSET(dither), AV_OPT_TYPE_INT, {.i64=DITHERING_SIERRA2_4A}, 0, NB_DITHERING-1, FLAGS, "dithering_mode" }, + { "none", "no dither", 0, AV_OPT_TYPE_CONST, {.i64=DITHERING_NONE}, INT_MIN, INT_MAX, FLAGS, "dithering_mode" }, { "bayer", "ordered 8x8 bayer dithering (deterministic)", 0, AV_OPT_TYPE_CONST, {.i64=DITHERING_BAYER}, INT_MIN, INT_MAX, FLAGS, "dithering_mode" }, { "heckbert", "dithering as defined by Paul Heckbert in 1982 (simple error diffusion)", 0, AV_OPT_TYPE_CONST, {.i64=DITHERING_HECKBERT}, INT_MIN, INT_MAX, FLAGS, "dithering_mode" }, { "floyd_steinberg", "Floyd and Steingberg dithering (error diffusion)", 0, AV_OPT_TYPE_CONST, {.i64=DITHERING_FLOYD_STEINBERG}, INT_MIN, INT_MAX, FLAGS, "dithering_mode" }, @@ -157,7 +159,8 @@ static int query_formats(AVFilterContext *ctx) static av_always_inline int dither_color(uint32_t px, int er, int eg, int eb, int scale, int shift) { - return av_clip_uint8((px >> 16 & 0xff) + ((er * scale) / (1<> 24 & 0xff) ) << 24 + | av_clip_uint8((px >> 16 & 0xff) + ((er * scale) / (1<> 8 & 0xff) + ((eg * scale) / (1<>24 & 0xff, palette[i]>>16 & 0xff, palette[i]>> 8 & 0xff, palette[i] & 0xff, }; - const int d = diff(palrgb, rgb); + const int d = diff(palargb, rgb); if (d < min_dist) { pal_id = i; min_dist = d; } } } + return pal_id; } @@ -325,14 +338,15 @@ end: * Note: r, g, and b are the component of c but are passed as well to avoid * recomputing them (they are generally computed by the caller for other uses). */ -static av_always_inline int color_get(struct cache_node *cache, uint32_t color, - uint8_t r, uint8_t g, uint8_t b, +static av_always_inline int color_get(struct cache_node *cache, uint32_t argb, + uint8_t a, uint8_t r, uint8_t g, uint8_t b, + int transparency_index, const struct color_node *map, const uint32_t *palette, const enum color_search_method search_method) { int i; - const uint8_t rgb[] = {r, g, b}; + const uint8_t argb_elts[] = {a, r, g, b}; const uint8_t rhash = r & ((1<= 0) { + return transparency_index; + } + for (i = 0; i < node->nb_entries; i++) { e = &node->entries[i]; - if (e->color == color) + if (e->color == argb) return e->pal_entry; } @@ -350,21 +369,24 @@ static av_always_inline int color_get(struct cache_node *cache, uint32_t color, sizeof(*node->entries), NULL); if (!e) return AVERROR(ENOMEM); - e->color = color; - e->pal_entry = COLORMAP_NEAREST(search_method, palette, map, rgb); + e->color = argb; + e->pal_entry = COLORMAP_NEAREST(search_method, palette, map, argb_elts); + return e->pal_entry; } static av_always_inline int get_dst_color_err(struct cache_node *cache, - uint32_t c, const struct color_node *map, + uint32_t argb, const struct color_node *map, const uint32_t *palette, + int transparency_index, int *er, int *eg, int *eb, const enum color_search_method search_method) { - const uint8_t r = c >> 16 & 0xff; - const uint8_t g = c >> 8 & 0xff; - const uint8_t b = c & 0xff; - const int dstx = color_get(cache, c, r, g, b, map, palette, search_method); + const uint8_t a = argb >> 24 & 0xff; + const uint8_t r = argb >> 16 & 0xff; + const uint8_t g = argb >> 8 & 0xff; + const uint8_t b = argb & 0xff; + const int dstx = color_get(cache, argb, a, r, g, b, transparency_index, map, palette, search_method); const uint32_t dstc = palette[dstx]; *er = r - (dstc >> 16 & 0xff); *eg = g - (dstc >> 8 & 0xff); @@ -385,6 +407,7 @@ static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram const int dst_linesize = out->linesize[0]; uint32_t *src = ((uint32_t *)in ->data[0]) + y_start*src_linesize; uint8_t *dst = out->data[0] + y_start*dst_linesize; + int transparency_index = s->transparency_index; w += x_start; h += y_start; @@ -395,14 +418,14 @@ static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram if (dither == DITHERING_BAYER) { const int d = s->ordered_dither[(y & 7)<<3 | (x & 7)]; + const uint8_t a8 = src[x] >> 24 & 0xff; const uint8_t r8 = src[x] >> 16 & 0xff; const uint8_t g8 = src[x] >> 8 & 0xff; const uint8_t b8 = src[x] & 0xff; const uint8_t r = av_clip_uint8(r8 + d); const uint8_t g = av_clip_uint8(g8 + d); const uint8_t b = av_clip_uint8(b8 + d); - const uint32_t c = r<<16 | g<<8 | b; - const int color = color_get(cache, c, r, g, b, map, palette, search_method); + const int color = color_get(cache, src[x], a8, r, g, b, transparency_index, map, palette, search_method); if (color < 0) return color; @@ -410,7 +433,7 @@ static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram } else if (dither == DITHERING_HECKBERT) { const int right = x < w - 1, down = y < h - 1; - const int color = get_dst_color_err(cache, src[x], map, palette, &er, &eg, &eb, search_method); + const int color = get_dst_color_err(cache, src[x], map, palette, transparency_index, &er, &eg, &eb, search_method); if (color < 0) return color; @@ -422,7 +445,7 @@ static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram } else if (dither == DITHERING_FLOYD_STEINBERG) { const int right = x < w - 1, down = y < h - 1, left = x > x_start; - const int color = get_dst_color_err(cache, src[x], map, palette, &er, &eg, &eb, search_method); + const int color = get_dst_color_err(cache, src[x], map, palette, transparency_index, &er, &eg, &eb, search_method); if (color < 0) return color; @@ -436,7 +459,7 @@ static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram } else if (dither == DITHERING_SIERRA2) { const int right = x < w - 1, down = y < h - 1, left = x > x_start; const int right2 = x < w - 2, left2 = x > x_start + 1; - const int color = get_dst_color_err(cache, src[x], map, palette, &er, &eg, &eb, search_method); + const int color = get_dst_color_err(cache, src[x], map, palette, transparency_index, &er, &eg, &eb, search_method); if (color < 0) return color; @@ -455,7 +478,7 @@ static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram } else if (dither == DITHERING_SIERRA2_4A) { const int right = x < w - 1, down = y < h - 1, left = x > x_start; - const int color = get_dst_color_err(cache, src[x], map, palette, &er, &eg, &eb, search_method); + const int color = get_dst_color_err(cache, src[x], map, palette, transparency_index, &er, &eg, &eb, search_method); if (color < 0) return color; @@ -466,10 +489,11 @@ static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram if ( down) src[src_linesize + x ] = dither_color(src[src_linesize + x ], er, eg, eb, 1, 2); } else { + const uint8_t a = src[x] >> 24 & 0xff; const uint8_t r = src[x] >> 16 & 0xff; const uint8_t g = src[x] >> 8 & 0xff; const uint8_t b = src[x] & 0xff; - const int color = color_get(cache, src[x] & 0xffffff, r, g, b, map, palette, search_method); + const int color = color_get(cache, src[x], a, r, g, b, transparency_index, map, palette, search_method); if (color < 0) return color; @@ -489,19 +513,19 @@ static void disp_node(AVBPrint *buf, int depth) { const struct color_node *node = &map[node_id]; - const uint32_t fontcolor = node->val[0] > 0x50 && - node->val[1] > 0x50 && - node->val[2] > 0x50 ? 0 : 0xffffff; + const uint32_t fontcolor = node->val[1] > 0x50 && + node->val[2] > 0x50 && + node->val[3] > 0x50 ? 0 : 0xffffff; av_bprintf(buf, "%*cnode%d [" "label=\"%c%02X%c%02X%c%02X%c\" " "fillcolor=\"#%02x%02x%02x\" " "fontcolor=\"#%06"PRIX32"\"]\n", depth*INDENT, ' ', node->palette_id, - "[ "[node->split], node->val[0], - "][ "[node->split], node->val[1], - " ]["[node->split], node->val[2], + "[ "[node->split], node->val[1], + "][ "[node->split], node->val[2], + " ]["[node->split], node->val[3], " ]"[node->split], - node->val[0], node->val[1], node->val[2], + node->val[1], node->val[2], node->val[3], fontcolor); if (parent_id != -1) av_bprintf(buf, "%*cnode%d -> node%d\n", depth*INDENT, ' ', @@ -584,17 +608,18 @@ static int cmp_##name(const void *pa, const void *pb) \ { \ const struct color *a = pa; \ const struct color *b = pb; \ - return (a->value >> (8 * (2 - (pos))) & 0xff) \ - - (b->value >> (8 * (2 - (pos))) & 0xff); \ + return (a->value >> (8 * (3 - (pos))) & 0xff) \ + - (b->value >> (8 * (3 - (pos))) & 0xff); \ } -DECLARE_CMP_FUNC(r, 0) -DECLARE_CMP_FUNC(g, 1) -DECLARE_CMP_FUNC(b, 2) +DECLARE_CMP_FUNC(a, 0) +DECLARE_CMP_FUNC(r, 1) +DECLARE_CMP_FUNC(g, 2) +DECLARE_CMP_FUNC(b, 3) -static const cmp_func cmp_funcs[] = {cmp_r, cmp_g, cmp_b}; +static const cmp_func cmp_funcs[] = {cmp_a, cmp_r, cmp_g, cmp_b}; -static int get_next_color(const uint8_t *color_used, const uint32_t *palette, +static int get_next_color(const uint8_t *color_used, const uint32_t *palette, const int nb_colors, int *component, const struct color_rect *box) { int wr, wg, wb; @@ -607,12 +632,17 @@ static int get_next_color(const uint8_t *color_used, const uint32_t *palette, ranges.min[0] = ranges.min[1] = ranges.min[2] = 0xff; ranges.max[0] = ranges.max[1] = ranges.max[2] = 0x00; - for (i = 0; i < AVPALETTE_COUNT; i++) { + for (i = 0; i < nb_colors; i++) { const uint32_t c = palette[i]; + const uint8_t a = c >> 24 & 0xff; const uint8_t r = c >> 16 & 0xff; const uint8_t g = c >> 8 & 0xff; const uint8_t b = c & 0xff; + if (a != 0xff) { + continue; + } + if (color_used[i] || r < box->min[0] || g < box->min[1] || b < box->min[2] || r > box->max[0] || g > box->max[1] || b > box->max[2]) @@ -639,9 +669,9 @@ static int get_next_color(const uint8_t *color_used, const uint32_t *palette, wr = ranges.max[0] - ranges.min[0]; wg = ranges.max[1] - ranges.min[1]; wb = ranges.max[2] - ranges.min[2]; - if (wr >= wg && wr >= wb) longest = 0; - if (wg >= wr && wg >= wb) longest = 1; - if (wb >= wr && wb >= wg) longest = 2; + if (wr >= wg && wr >= wb) longest = 1; + if (wg >= wr && wg >= wb) longest = 2; + if (wb >= wr && wb >= wg) longest = 3; cmpf = cmp_funcs[longest]; *component = longest; @@ -655,6 +685,7 @@ static int colormap_insert(struct color_node *map, uint8_t *color_used, int *nb_used, const uint32_t *palette, + const int nb_colors, const struct color_rect *box) { uint32_t c; @@ -662,7 +693,7 @@ static int colormap_insert(struct color_node *map, int node_left_id = -1, node_right_id = -1; struct color_node *node; struct color_rect box1, box2; - const int pal_id = get_next_color(color_used, palette, &component, box); + const int pal_id = get_next_color(color_used, palette, nb_colors, &component, box); if (pal_id < 0) return -1; @@ -673,21 +704,22 @@ static int colormap_insert(struct color_node *map, node = &map[cur_id]; node->split = component; node->palette_id = pal_id; - node->val[0] = c>>16 & 0xff; - node->val[1] = c>> 8 & 0xff; - node->val[2] = c & 0xff; + node->val[0] = c>>24 & 0xff; + node->val[1] = c>>16 & 0xff; + node->val[2] = c>> 8 & 0xff; + node->val[3] = c & 0xff; color_used[pal_id] = 1; /* get the two boxes this node creates */ box1 = box2 = *box; - box1.max[component] = node->val[component]; - box2.min[component] = node->val[component] + 1; + box1.max[component-1] = node->val[component]; + box2.min[component-1] = node->val[component] + 1; - node_left_id = colormap_insert(map, color_used, nb_used, palette, &box1); + node_left_id = colormap_insert(map, color_used, nb_used, palette, nb_colors, &box1); - if (box2.min[component] <= box2.max[component]) - node_right_id = colormap_insert(map, color_used, nb_used, palette, &box2); + if (box2.min[component-1] <= box2.max[component-1]) + node_right_id = colormap_insert(map, color_used, nb_used, palette, nb_colors, &box2); node->left_id = node_left_id; node->right_id = node_right_id; @@ -702,7 +734,7 @@ static int cmp_pal_entry(const void *a, const void *b) return c1 - c2; } -static void load_colormap(PaletteUseContext *s) +static void load_colormap(PaletteUseContext *s, int nb_colors) { int i, nb_used = 0; uint8_t color_used[AVPALETTE_COUNT] = {0}; @@ -710,8 +742,8 @@ static void load_colormap(PaletteUseContext *s) struct color_rect box; /* disable transparent colors and dups */ - qsort(s->palette, AVPALETTE_COUNT, sizeof(*s->palette), cmp_pal_entry); - for (i = 0; i < AVPALETTE_COUNT; i++) { + qsort(s->palette, nb_colors, sizeof(*s->palette), cmp_pal_entry); + for (i = 0; i < nb_colors; i++) { const uint32_t c = s->palette[i]; if (i != 0 && c == last_color) { color_used[i] = 1; @@ -727,7 +759,7 @@ static void load_colormap(PaletteUseContext *s) box.min[0] = box.min[1] = box.min[2] = 0x00; box.max[0] = box.max[1] = box.max[2] = 0xff; - colormap_insert(s->map, color_used, &nb_used, s->palette, &box); + colormap_insert(s->map, color_used, &nb_used, s->palette, nb_colors, &box); if (s->dot_filename) disp_tree(s->map, s->dot_filename); @@ -937,10 +969,12 @@ static int config_input_palette(AVFilterLink *inlink) static void load_palette(PaletteUseContext *s, const AVFrame *palette_frame) { - int i, x, y; + int pal_index, i, x, y; const uint32_t *p = (const uint32_t *)palette_frame->data[0]; const int p_linesize = palette_frame->linesize[0] >> 2; + s->transparency_index = -1; + if (s->new) { memset(s->palette, 0, sizeof(s->palette)); memset(s->map, 0, sizeof(s->map)); @@ -949,14 +983,19 @@ static void load_palette(PaletteUseContext *s, const AVFrame *palette_frame) memset(s->cache, 0, sizeof(s->cache)); } - i = 0; + pal_index = 0; for (y = 0; y < palette_frame->height; y++) { - for (x = 0; x < palette_frame->width; x++) - s->palette[i++] = p[x]; + for (x = 0; x < palette_frame->width; x++) { + s->palette[pal_index] = p[x]; + if ((p[x]>>24 & 0xff) == 0) { + s->transparency_index = pal_index; // we are assuming art most one transparent color + } + ++pal_index; + } p += p_linesize; } - load_colormap(s); + load_colormap(s, pal_index); if (!s->new) s->palette_loaded = 1;