[FFmpeg-devel,3/7] Removing some debugging

Submitted by Bjorn Roche on Oct. 5, 2017, 2:43 p.m.

Details

Message ID CAGiq_kPDxB1k1zTYFxa7D0z2mO7xH2vy=TDr5hS-QfCFZYejgg@mail.gmail.com
State New
Headers show

Commit Message

Bjorn Roche Oct. 5, 2017, 2:43 p.m.
On Tue, Oct 3, 2017 at 4:02 AM, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:

> 2017-10-03 1:52 GMT+02:00 Bjorn Roche <bjorn@giphy.com>:
>
> > Attached is a patch for paletteuse only.
>
> I tested the following with and without your patch:
> $ ffmpeg -i fate-suite/lena.pnm -vf palettegen pal.png
> $ ffmpeg -i fate-suite/lena.pnm -i pal.png -lavfi paletteuse out.png
>
> out.png changes with your patch: Is that intended?
>

Not intended, but not unexpected -- the ordering of colors in the palette
changed to make it easier to deal with transparency. However, there are
visual differences, so that's obviously a bug. I've supplied a new patch
(below) that fixes that bug, and maintains the order so it's easier to test
for that.


> The input frame contains no transparency.
>
> Your patch still contains printfs that should be removed
> (change them to av_log(DEBUG) if you think they are
> useful) and a new warning is shown at compilation:
> libavfilter/vf_paletteuse.c: In function ‘colormap_nearest_recursive’:
> libavfilter/vf_paletteuse.c:242:5: warning: ISO C90 forbids mixed
> declarations and code
>
> Please fix the style: Space after "if", no space after "if ("
>

Thank you for your attention. This should also be resolved in the attached
patch.

I have tested fate, all dithering (including none, which I added) and all
color search algorithms.

Comments

Michael Niedermayer Oct. 5, 2017, 6:47 p.m.
On Thu, Oct 05, 2017 at 10:43:04AM -0400, Bjorn Roche wrote:
> On Tue, Oct 3, 2017 at 4:02 AM, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
> 
> > 2017-10-03 1:52 GMT+02:00 Bjorn Roche <bjorn@giphy.com>:
> >
> > > Attached is a patch for paletteuse only.
> >
> > I tested the following with and without your patch:
> > $ ffmpeg -i fate-suite/lena.pnm -vf palettegen pal.png
> > $ ffmpeg -i fate-suite/lena.pnm -i pal.png -lavfi paletteuse out.png
> >
> > out.png changes with your patch: Is that intended?
> >
> 
> Not intended, but not unexpected -- the ordering of colors in the palette
> changed to make it easier to deal with transparency. However, there are
> visual differences, so that's obviously a bug. I've supplied a new patch
> (below) that fixes that bug, and maintains the order so it's easier to test
> for that.
> 
> 
> > The input frame contains no transparency.
> >
> > Your patch still contains printfs that should be removed
> > (change them to av_log(DEBUG) if you think they are
> > useful) and a new warning is shown at compilation:
> > libavfilter/vf_paletteuse.c: In function ‘colormap_nearest_recursive’:
> > libavfilter/vf_paletteuse.c:242:5: warning: ISO C90 forbids mixed
> > declarations and code
> >
> > Please fix the style: Space after "if", no space after "if ("
> >
> 
> Thank you for your attention. This should also be resolved in the attached
> patch.
> 
> I have tested fate, all dithering (including none, which I added) and all
> color search algorithms.

>  vf_paletteuse.c |  163 ++++++++++++++++++++++++++++++++++----------------------
>  1 file changed, 101 insertions(+), 62 deletions(-)
> 3d215b0705c5352700ed11c6ab8fd8638c9fea48  palette-use-transparency2.diff
> diff --git a/libavfilter/vf_paletteuse.c b/libavfilter/vf_paletteuse.c
> index ffd37bf1da..7cb5a3db02 100644
> --- a/libavfilter/vf_paletteuse.c
> +++ b/libavfilter/vf_paletteuse.c
> @@ -56,7 +56,7 @@ enum diff_mode {

This is not a git patch with commit message, all patches should
contain a commit message

[...]
Carl Eugen Hoyos Oct. 5, 2017, 7:44 p.m.
2017-10-05 16:43 GMT+02:00 Bjorn Roche <bjorn@giphy.com>:
> On Tue, Oct 3, 2017 at 4:02 AM, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>
>> 2017-10-03 1:52 GMT+02:00 Bjorn Roche <bjorn@giphy.com>:
>>
>> > Attached is a patch for paletteuse only.
>>
>> I tested the following with and without your patch:
>> $ ffmpeg -i fate-suite/lena.pnm -vf palettegen pal.png
>> $ ffmpeg -i fate-suite/lena.pnm -i pal.png -lavfi paletteuse out.png
>>
>> out.png changes with your patch: Is that intended?
>
> Not intended, but not unexpected -- the ordering of colors in the palette
> changed to make it easier to deal with transparency. However, there are
> visual differences, so that's obviously a bug. I've supplied a new patch
> (below) that fixes that bug, and maintains the order so it's easier to test
> for that.
>
>> The input frame contains no transparency.
>>
>> Your patch still contains printfs that should be removed
>> (change them to av_log(DEBUG) if you think they are
>> useful) and a new warning is shown at compilation:
>> libavfilter/vf_paletteuse.c: In function ‘colormap_nearest_recursive’:
>> libavfilter/vf_paletteuse.c:242:5: warning: ISO C90 forbids mixed
>> declarations and code
>>
>> Please fix the style: Space after "if", no space after "if ("
>>
>
> Thank you for your attention. This should also be resolved
> in the attached patch.
>
> I have tested fate, all dithering (including none, which I added)
> and all color search algorithms.

I tested the following with the sample from ticket #503:
$ ffmpeg -i rgba64le-lzw.tif -vf palettegen pal.png
$ ffmpeg -i rgba64le-lzw.tif -i pal.png -lavfi paletteuse out.gif

The output file has its background (that is supposed to be
transparent) converted to white (which I believe is good)
but the respective colour white (that is the 256th and last
colour in the palette) is not transparent but opaque (0xff).
Instead the first colour (0x00ff00) is the only transparent
colour in the palette. The output gif is opaque.

I confirmed that if I force the first palette entry to opaque
(0xFF00FF00) and the last one to transparent (0xFFFFFF)
in apply_palette() I get the expected transparent output with
above command line.

Thank you, Carl Eugen
Carl Eugen Hoyos Oct. 5, 2017, 11:14 p.m.
2017-10-05 21:44 GMT+02:00 Carl Eugen Hoyos <ceffmpeg@gmail.com>:

> I tested the following with the sample from ticket #503:
> $ ffmpeg -i rgba64le-lzw.tif -vf palettegen pal.png
> $ ffmpeg -i rgba64le-lzw.tif -i pal.png -lavfi paletteuse out.gif
>
> The output file has its background (that is supposed to be
> transparent) converted to white (which I believe is good)
> but the respective colour white (that is the 256th and last
> colour in the palette) is not transparent but opaque (0xff).
> Instead the first colour (0x00ff00) is the only transparent
> colour in the palette. The output gif is opaque.
>
> I confirmed that if I force the first palette entry to opaque
> (0xFF00FF00) and the last one to transparent (0xFFFFFF)
> in apply_palette() I get the expected transparent output with
> above command line.

Forgot to mention:
I can reproduce this exact behaviour writing a png image:
$ ffmpeg -i rgba64le-lzw.tif -i pal.png -lavfi paletteuse out.png

Carl Eugen

Patch hide | download patch | download mbox

diff --git a/libavfilter/vf_paletteuse.c b/libavfilter/vf_paletteuse.c
index ffd37bf1da..7cb5a3db02 100644
--- a/libavfilter/vf_paletteuse.c
+++ b/libavfilter/vf_paletteuse.c
@@ -56,7 +56,7 @@  enum diff_mode {
 };
 
 struct color_node {
-    uint8_t val[3];
+    uint8_t val[4];
     uint8_t palette_id;
     int split;
     int left_id, right_id;
@@ -86,6 +86,7 @@  typedef struct PaletteUseContext {
     struct cache_node cache[CACHE_SIZE];    /* lookup cache */
     struct color_node map[AVPALETTE_COUNT]; /* 3D-Tree (KD-Tree with K=3) for reverse colormap */
     uint32_t palette[AVPALETTE_COUNT];
+    int transparency_index; /* index in the palette of transparency. -1 if there isn't a transparency. */
     int palette_loaded;
     int dither;
     int new;
@@ -108,6 +109,7 @@  typedef struct PaletteUseContext {
 #define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM
 static const AVOption paletteuse_options[] = {
     { "dither", "select dithering mode", OFFSET(dither), AV_OPT_TYPE_INT, {.i64=DITHERING_SIERRA2_4A}, 0, NB_DITHERING-1, FLAGS, "dithering_mode" },
+        { "none",            "no dither",                                                              0, AV_OPT_TYPE_CONST, {.i64=DITHERING_NONE},            INT_MIN, INT_MAX, FLAGS, "dithering_mode" },
         { "bayer",           "ordered 8x8 bayer dithering (deterministic)",                            0, AV_OPT_TYPE_CONST, {.i64=DITHERING_BAYER},           INT_MIN, INT_MAX, FLAGS, "dithering_mode" },
         { "heckbert",        "dithering as defined by Paul Heckbert in 1982 (simple error diffusion)", 0, AV_OPT_TYPE_CONST, {.i64=DITHERING_HECKBERT},        INT_MIN, INT_MAX, FLAGS, "dithering_mode" },
         { "floyd_steinberg", "Floyd and Steingberg dithering (error diffusion)",                       0, AV_OPT_TYPE_CONST, {.i64=DITHERING_FLOYD_STEINBERG}, INT_MIN, INT_MAX, FLAGS, "dithering_mode" },
@@ -157,7 +159,8 @@  static int query_formats(AVFilterContext *ctx)
 
 static av_always_inline int dither_color(uint32_t px, int er, int eg, int eb, int scale, int shift)
 {
-    return av_clip_uint8((px >> 16 & 0xff) + ((er * scale) / (1<<shift))) << 16
+    return av_clip_uint8((px >> 24 & 0xff)                              ) << 24
+         | av_clip_uint8((px >> 16 & 0xff) + ((er * scale) / (1<<shift))) << 16
          | av_clip_uint8((px >>  8 & 0xff) + ((eg * scale) / (1<<shift))) <<  8
          | av_clip_uint8((px       & 0xff) + ((eb * scale) / (1<<shift)));
 }
@@ -165,10 +168,18 @@  static av_always_inline int dither_color(uint32_t px, int er, int eg, int eb, in
 static av_always_inline int diff(const uint8_t *c1, const uint8_t *c2)
 {
     // XXX: try L*a*b with CIE76 (dL*dL + da*da + db*db)
-    const int dr = c1[0] - c2[0];
-    const int dg = c1[1] - c2[1];
-    const int db = c1[2] - c2[2];
-    return dr*dr + dg*dg + db*db;
+    const static int max_diff = 255*255 + 255*255 + 255*255;
+    const int dr = c1[1] - c2[1];
+    const int dg = c1[2] - c2[2];
+    const int db = c1[3] - c2[3];
+
+    if (c1[0] == 0 && c2[0] == 0) {
+        return 0;
+    } else if (c1[0] == c2[0]) {
+        return dr*dr + dg*dg + db*db;
+    } else {
+        return max_diff;
+    }
 }
 
 static av_always_inline uint8_t colormap_nearest_bruteforce(const uint32_t *palette, const uint8_t *rgb)
@@ -179,18 +190,20 @@  static av_always_inline uint8_t colormap_nearest_bruteforce(const uint32_t *pale
         const uint32_t c = palette[i];
 
         if ((c & 0xff000000) == 0xff000000) { // ignore transparent entry
-            const uint8_t palrgb[] = {
+            const uint8_t palargb[] = {
+                palette[i]>>24 & 0xff,
                 palette[i]>>16 & 0xff,
                 palette[i]>> 8 & 0xff,
                 palette[i]     & 0xff,
             };
-            const int d = diff(palrgb, rgb);
+            const int d = diff(palargb, rgb);
             if (d < min_dist) {
                 pal_id = i;
                 min_dist = d;
             }
         }
     }
+
     return pal_id;
 }
 
@@ -325,14 +338,15 @@  end:
  * Note: r, g, and b are the component of c but are passed as well to avoid
  * recomputing them (they are generally computed by the caller for other uses).
  */
-static av_always_inline int color_get(struct cache_node *cache, uint32_t color,
-                                      uint8_t r, uint8_t g, uint8_t b,
+static av_always_inline int color_get(struct cache_node *cache, uint32_t argb,
+                                      uint8_t a, uint8_t r, uint8_t g, uint8_t b,
+                                      int transparency_index,
                                       const struct color_node *map,
                                       const uint32_t *palette,
                                       const enum color_search_method search_method)
 {
     int i;
-    const uint8_t rgb[] = {r, g, b};
+    const uint8_t argb_elts[] = {a, r, g, b};
     const uint8_t rhash = r & ((1<<NBITS)-1);
     const uint8_t ghash = g & ((1<<NBITS)-1);
     const uint8_t bhash = b & ((1<<NBITS)-1);
@@ -340,9 +354,14 @@  static av_always_inline int color_get(struct cache_node *cache, uint32_t color,
     struct cache_node *node = &cache[hash];
     struct cached_color *e;
 
+    // first, check for transparency
+    if (a == 0 && transparency_index >= 0) {
+        return transparency_index;
+    }
+
     for (i = 0; i < node->nb_entries; i++) {
         e = &node->entries[i];
-        if (e->color == color)
+        if (e->color == argb)
             return e->pal_entry;
     }
 
@@ -350,21 +369,24 @@  static av_always_inline int color_get(struct cache_node *cache, uint32_t color,
                          sizeof(*node->entries), NULL);
     if (!e)
         return AVERROR(ENOMEM);
-    e->color = color;
-    e->pal_entry = COLORMAP_NEAREST(search_method, palette, map, rgb);
+    e->color = argb;
+    e->pal_entry = COLORMAP_NEAREST(search_method, palette, map, argb_elts);
+
     return e->pal_entry;
 }
 
 static av_always_inline int get_dst_color_err(struct cache_node *cache,
-                                              uint32_t c, const struct color_node *map,
+                                              uint32_t argb, const struct color_node *map,
                                               const uint32_t *palette,
+                                              int transparency_index,
                                               int *er, int *eg, int *eb,
                                               const enum color_search_method search_method)
 {
-    const uint8_t r = c >> 16 & 0xff;
-    const uint8_t g = c >>  8 & 0xff;
-    const uint8_t b = c       & 0xff;
-    const int dstx = color_get(cache, c, r, g, b, map, palette, search_method);
+    const uint8_t a = argb >> 24 & 0xff;
+    const uint8_t r = argb >> 16 & 0xff;
+    const uint8_t g = argb >>  8 & 0xff;
+    const uint8_t b = argb       & 0xff;
+    const int dstx = color_get(cache, argb, a, r, g, b, transparency_index, map, palette, search_method);
     const uint32_t dstc = palette[dstx];
     *er = r - (dstc >> 16 & 0xff);
     *eg = g - (dstc >>  8 & 0xff);
@@ -385,6 +407,7 @@  static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram
     const int dst_linesize = out->linesize[0];
     uint32_t *src = ((uint32_t *)in ->data[0]) + y_start*src_linesize;
     uint8_t  *dst =              out->data[0]  + y_start*dst_linesize;
+    int transparency_index = s->transparency_index;
 
     w += x_start;
     h += y_start;
@@ -395,14 +418,14 @@  static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram
 
             if (dither == DITHERING_BAYER) {
                 const int d = s->ordered_dither[(y & 7)<<3 | (x & 7)];
+                const uint8_t a8 = src[x] >> 24 & 0xff;
                 const uint8_t r8 = src[x] >> 16 & 0xff;
                 const uint8_t g8 = src[x] >>  8 & 0xff;
                 const uint8_t b8 = src[x]       & 0xff;
                 const uint8_t r = av_clip_uint8(r8 + d);
                 const uint8_t g = av_clip_uint8(g8 + d);
                 const uint8_t b = av_clip_uint8(b8 + d);
-                const uint32_t c = r<<16 | g<<8 | b;
-                const int color = color_get(cache, c, r, g, b, map, palette, search_method);
+                const int color = color_get(cache, src[x], a8, r, g, b, transparency_index, map, palette, search_method);
 
                 if (color < 0)
                     return color;
@@ -410,7 +433,7 @@  static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram
 
             } else if (dither == DITHERING_HECKBERT) {
                 const int right = x < w - 1, down = y < h - 1;
-                const int color = get_dst_color_err(cache, src[x], map, palette, &er, &eg, &eb, search_method);
+                const int color = get_dst_color_err(cache, src[x], map, palette, transparency_index, &er, &eg, &eb, search_method);
 
                 if (color < 0)
                     return color;
@@ -422,7 +445,7 @@  static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram
 
             } else if (dither == DITHERING_FLOYD_STEINBERG) {
                 const int right = x < w - 1, down = y < h - 1, left = x > x_start;
-                const int color = get_dst_color_err(cache, src[x], map, palette, &er, &eg, &eb, search_method);
+                const int color = get_dst_color_err(cache, src[x], map, palette, transparency_index, &er, &eg, &eb, search_method);
 
                 if (color < 0)
                     return color;
@@ -436,7 +459,7 @@  static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram
             } else if (dither == DITHERING_SIERRA2) {
                 const int right  = x < w - 1, down  = y < h - 1, left  = x > x_start;
                 const int right2 = x < w - 2,                    left2 = x > x_start + 1;
-                const int color = get_dst_color_err(cache, src[x], map, palette, &er, &eg, &eb, search_method);
+                const int color = get_dst_color_err(cache, src[x], map, palette, transparency_index, &er, &eg, &eb, search_method);
 
                 if (color < 0)
                     return color;
@@ -455,7 +478,7 @@  static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram
 
             } else if (dither == DITHERING_SIERRA2_4A) {
                 const int right = x < w - 1, down = y < h - 1, left = x > x_start;
-                const int color = get_dst_color_err(cache, src[x], map, palette, &er, &eg, &eb, search_method);
+                const int color = get_dst_color_err(cache, src[x], map, palette, transparency_index, &er, &eg, &eb, search_method);
 
                 if (color < 0)
                     return color;
@@ -466,10 +489,11 @@  static av_always_inline int set_frame(PaletteUseContext *s, AVFrame *out, AVFram
                 if (         down) src[src_linesize + x    ] = dither_color(src[src_linesize + x    ], er, eg, eb, 1, 2);
 
             } else {
+                const uint8_t a = src[x] >> 24 & 0xff;
                 const uint8_t r = src[x] >> 16 & 0xff;
                 const uint8_t g = src[x] >>  8 & 0xff;
                 const uint8_t b = src[x]       & 0xff;
-                const int color = color_get(cache, src[x] & 0xffffff, r, g, b, map, palette, search_method);
+                const int color = color_get(cache, src[x], a, r, g, b, transparency_index, map, palette, search_method);
 
                 if (color < 0)
                     return color;
@@ -489,19 +513,19 @@  static void disp_node(AVBPrint *buf,
                       int depth)
 {
     const struct color_node *node = &map[node_id];
-    const uint32_t fontcolor = node->val[0] > 0x50 &&
-                               node->val[1] > 0x50 &&
-                               node->val[2] > 0x50 ? 0 : 0xffffff;
+    const uint32_t fontcolor = node->val[1] > 0x50 &&
+                               node->val[2] > 0x50 &&
+                               node->val[3] > 0x50 ? 0 : 0xffffff;
     av_bprintf(buf, "%*cnode%d ["
                "label=\"%c%02X%c%02X%c%02X%c\" "
                "fillcolor=\"#%02x%02x%02x\" "
                "fontcolor=\"#%06"PRIX32"\"]\n",
                depth*INDENT, ' ', node->palette_id,
-               "[  "[node->split], node->val[0],
-               "][ "[node->split], node->val[1],
-               " ]["[node->split], node->val[2],
+               "[  "[node->split], node->val[1],
+               "][ "[node->split], node->val[2],
+               " ]["[node->split], node->val[3],
                "  ]"[node->split],
-               node->val[0], node->val[1], node->val[2],
+               node->val[1], node->val[2], node->val[3],
                fontcolor);
     if (parent_id != -1)
         av_bprintf(buf, "%*cnode%d -> node%d\n", depth*INDENT, ' ',
@@ -584,17 +608,18 @@  static int cmp_##name(const void *pa, const void *pb)   \
 {                                                       \
     const struct color *a = pa;                         \
     const struct color *b = pb;                         \
-    return   (a->value >> (8 * (2 - (pos))) & 0xff)     \
-           - (b->value >> (8 * (2 - (pos))) & 0xff);    \
+    return   (a->value >> (8 * (3 - (pos))) & 0xff)     \
+           - (b->value >> (8 * (3 - (pos))) & 0xff);    \
 }
 
-DECLARE_CMP_FUNC(r, 0)
-DECLARE_CMP_FUNC(g, 1)
-DECLARE_CMP_FUNC(b, 2)
+DECLARE_CMP_FUNC(a, 0)
+DECLARE_CMP_FUNC(r, 1)
+DECLARE_CMP_FUNC(g, 2)
+DECLARE_CMP_FUNC(b, 3)
 
-static const cmp_func cmp_funcs[] = {cmp_r, cmp_g, cmp_b};
+static const cmp_func cmp_funcs[] = {cmp_a, cmp_r, cmp_g, cmp_b};
 
-static int get_next_color(const uint8_t *color_used, const uint32_t *palette,
+static int get_next_color(const uint8_t *color_used, const uint32_t *palette, const int nb_colors,
                           int *component, const struct color_rect *box)
 {
     int wr, wg, wb;
@@ -607,12 +632,17 @@  static int get_next_color(const uint8_t *color_used, const uint32_t *palette,
     ranges.min[0] = ranges.min[1] = ranges.min[2] = 0xff;
     ranges.max[0] = ranges.max[1] = ranges.max[2] = 0x00;
 
-    for (i = 0; i < AVPALETTE_COUNT; i++) {
+    for (i = 0; i < nb_colors; i++) {
         const uint32_t c = palette[i];
+        const uint8_t a = c >> 24 & 0xff;
         const uint8_t r = c >> 16 & 0xff;
         const uint8_t g = c >>  8 & 0xff;
         const uint8_t b = c       & 0xff;
 
+        if (a != 0xff) {
+            continue;
+        }
+
         if (color_used[i] ||
             r < box->min[0] || g < box->min[1] || b < box->min[2] ||
             r > box->max[0] || g > box->max[1] || b > box->max[2])
@@ -639,9 +669,9 @@  static int get_next_color(const uint8_t *color_used, const uint32_t *palette,
     wr = ranges.max[0] - ranges.min[0];
     wg = ranges.max[1] - ranges.min[1];
     wb = ranges.max[2] - ranges.min[2];
-    if (wr >= wg && wr >= wb) longest = 0;
-    if (wg >= wr && wg >= wb) longest = 1;
-    if (wb >= wr && wb >= wg) longest = 2;
+    if (wr >= wg && wr >= wb) longest = 1;
+    if (wg >= wr && wg >= wb) longest = 2;
+    if (wb >= wr && wb >= wg) longest = 3;
     cmpf = cmp_funcs[longest];
     *component = longest;
 
@@ -655,6 +685,7 @@  static int colormap_insert(struct color_node *map,
                            uint8_t *color_used,
                            int *nb_used,
                            const uint32_t *palette,
+                           const int nb_colors,
                            const struct color_rect *box)
 {
     uint32_t c;
@@ -662,7 +693,7 @@  static int colormap_insert(struct color_node *map,
     int node_left_id = -1, node_right_id = -1;
     struct color_node *node;
     struct color_rect box1, box2;
-    const int pal_id = get_next_color(color_used, palette, &component, box);
+    const int pal_id = get_next_color(color_used, palette, nb_colors, &component, box);
 
     if (pal_id < 0)
         return -1;
@@ -673,21 +704,22 @@  static int colormap_insert(struct color_node *map,
     node = &map[cur_id];
     node->split = component;
     node->palette_id = pal_id;
-    node->val[0] = c>>16 & 0xff;
-    node->val[1] = c>> 8 & 0xff;
-    node->val[2] = c     & 0xff;
+    node->val[0] = c>>24 & 0xff;
+    node->val[1] = c>>16 & 0xff;
+    node->val[2] = c>> 8 & 0xff;
+    node->val[3] = c     & 0xff;
 
     color_used[pal_id] = 1;
 
     /* get the two boxes this node creates */
     box1 = box2 = *box;
-    box1.max[component] = node->val[component];
-    box2.min[component] = node->val[component] + 1;
+    box1.max[component-1] = node->val[component];
+    box2.min[component-1] = node->val[component] + 1;
 
-    node_left_id = colormap_insert(map, color_used, nb_used, palette, &box1);
+    node_left_id = colormap_insert(map, color_used, nb_used, palette, nb_colors, &box1);
 
-    if (box2.min[component] <= box2.max[component])
-        node_right_id = colormap_insert(map, color_used, nb_used, palette, &box2);
+    if (box2.min[component-1] <= box2.max[component-1])
+        node_right_id = colormap_insert(map, color_used, nb_used, palette, nb_colors, &box2);
 
     node->left_id  = node_left_id;
     node->right_id = node_right_id;
@@ -702,7 +734,7 @@  static int cmp_pal_entry(const void *a, const void *b)
     return c1 - c2;
 }
 
-static void load_colormap(PaletteUseContext *s)
+static void load_colormap(PaletteUseContext *s, int nb_colors)
 {
     int i, nb_used = 0;
     uint8_t color_used[AVPALETTE_COUNT] = {0};
@@ -710,8 +742,8 @@  static void load_colormap(PaletteUseContext *s)
     struct color_rect box;
 
     /* disable transparent colors and dups */
-    qsort(s->palette, AVPALETTE_COUNT, sizeof(*s->palette), cmp_pal_entry);
-    for (i = 0; i < AVPALETTE_COUNT; i++) {
+    qsort(s->palette, nb_colors, sizeof(*s->palette), cmp_pal_entry);
+    for (i = 0; i < nb_colors; i++) {
         const uint32_t c = s->palette[i];
         if (i != 0 && c == last_color) {
             color_used[i] = 1;
@@ -727,7 +759,7 @@  static void load_colormap(PaletteUseContext *s)
     box.min[0] = box.min[1] = box.min[2] = 0x00;
     box.max[0] = box.max[1] = box.max[2] = 0xff;
 
-    colormap_insert(s->map, color_used, &nb_used, s->palette, &box);
+    colormap_insert(s->map, color_used, &nb_used, s->palette, nb_colors, &box);
 
     if (s->dot_filename)
         disp_tree(s->map, s->dot_filename);
@@ -937,10 +969,12 @@  static int config_input_palette(AVFilterLink *inlink)
 
 static void load_palette(PaletteUseContext *s, const AVFrame *palette_frame)
 {
-    int i, x, y;
+    int pal_index, i, x, y;
     const uint32_t *p = (const uint32_t *)palette_frame->data[0];
     const int p_linesize = palette_frame->linesize[0] >> 2;
 
+    s->transparency_index = -1;
+
     if (s->new) {
         memset(s->palette, 0, sizeof(s->palette));
         memset(s->map, 0, sizeof(s->map));
@@ -949,14 +983,19 @@  static void load_palette(PaletteUseContext *s, const AVFrame *palette_frame)
         memset(s->cache, 0, sizeof(s->cache));
     }
 
-    i = 0;
+    pal_index = 0;
     for (y = 0; y < palette_frame->height; y++) {
-        for (x = 0; x < palette_frame->width; x++)
-            s->palette[i++] = p[x];
+        for (x = 0; x < palette_frame->width; x++) {
+            s->palette[pal_index] = p[x];
+            if ((p[x]>>24 & 0xff) == 0) {
+                s->transparency_index = pal_index; // we are assuming art most one transparent color            
+            }
+            ++pal_index;
+        }
         p += p_linesize;
     }
 
-    load_colormap(s);
+    load_colormap(s, pal_index);
 
     if (!s->new)
         s->palette_loaded = 1;