diff mbox

[FFmpeg-devel] lavfi/mergeplanes: Fix little endian formats >8 bit.

Message ID alpine.DEB.2.11.1610300040590.6395@rainbow.studorg.tuwien.ac.at
State Accepted
Headers show

Commit Message

Carl Eugen Hoyos Oct. 29, 2016, 10:43 p.m. UTC
Hi!

Attached patch fixes ticket #5916 for (for example) yuv444p9le.

Please comment, Carl Eugen

Comments

Paul B Mahol Oct. 30, 2016, 7:55 a.m. UTC | #1
On 10/30/16, Carl Eugen Hoyos <cehoyos@ag.or.at> wrote:
> Hi!
>
> Attached patch fixes ticket #5916 for (for example) yuv444p9le.
>
> Please comment, Carl Eugen

Same as for another mergeplanes patch from you, please explain why
this is correct. Are patches dependend on each other?
Carl Eugen Hoyos Oct. 30, 2016, 2:23 p.m. UTC | #2
> Am 30.10.2016 um 00:55 schrieb Paul B Mahol <onemda@gmail.com>:
> 
>> On 10/30/16, Carl Eugen Hoyos <cehoyos@ag.or.at> wrote:
>> Hi!
>> 
>> Attached patch fixes ticket #5916 for (for example) yuv444p9le.
> 
> Same as for another mergeplanes patch from you, please explain why
> this is correct.

I think that for little endian yuv formats >8bit but <16bit, using the mergeplanes filter currently implies an intermediate format which must have the same endianness as the input format but this is not guaranteed by the current code.
Even if the issue can be worked-around differently, this patch should be applied anyway imo.

> Are patches dependend on each other?

The issue described here can be reproduced (seen) without applying the other patch, that's what the OP means with "left are strange" in his report on the bug tracker.

Carl Eugen
Paul B Mahol Oct. 30, 2016, 4:15 p.m. UTC | #3
On 10/30/16, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>
>
>> Am 30.10.2016 um 00:55 schrieb Paul B Mahol <onemda@gmail.com>:
>>
>>> On 10/30/16, Carl Eugen Hoyos <cehoyos@ag.or.at> wrote:
>>> Hi!
>>>
>>> Attached patch fixes ticket #5916 for (for example) yuv444p9le.
>>
>> Same as for another mergeplanes patch from you, please explain why
>> this is correct.
>
> I think that for little endian yuv formats >8bit but <16bit, using the
> mergeplanes filter currently implies an intermediate format which must have
> the same endianness as the input format but this is not guaranteed by the
> current code.
> Even if the issue can be worked-around differently, this patch should be
> applied anyway imo.
>
>> Are patches dependend on each other?
>
> The issue described here can be reproduced (seen) without applying the other
> patch, that's what the OP means with "left are strange" in his report on the
> bug tracker.
>
> Carl Eugen
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>

OK, try with non 444 cases, like with 420 high bit depth and if it
works apply both patches.
Carl Eugen Hoyos Nov. 2, 2016, 12:44 a.m. UTC | #4
2016-10-30 17:15 GMT+01:00 Paul B Mahol <onemda@gmail.com>:

> OK, try with non 444 cases, like with 420 high bit depth and if it
> works apply both patches.

Tested and applied.

Thank you, Carl Eugen
diff mbox

Patch

From e50c9ee93ce6a8ceb21d3319667eb1b392c1d2da Mon Sep 17 00:00:00 2001
From: Carl Eugen Hoyos <cehoyos@ag.or.at>
Date: Sun, 30 Oct 2016 00:39:19 +0200
Subject: [PATCH] lavfi/mergeplanes: Fix little endian formats >8 bit.

Fixes ticket #5916.
---
 libavfilter/vf_mergeplanes.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/libavfilter/vf_mergeplanes.c b/libavfilter/vf_mergeplanes.c
index c4948cc..c211043 100644
--- a/libavfilter/vf_mergeplanes.c
+++ b/libavfilter/vf_mergeplanes.c
@@ -122,6 +122,7 @@  static int query_formats(AVFilterContext *ctx)
     for (i = 0; av_pix_fmt_desc_get(i); i++) {
         const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(i);
         if (desc->comp[0].depth == s->outdesc->comp[0].depth &&
+            (desc->comp[0].depth <= 8 || (desc->flags & AV_PIX_FMT_FLAG_BE) == (s->outdesc->flags & AV_PIX_FMT_FLAG_BE)) &&
             av_pix_fmt_count_planes(i) == desc->nb_components &&
             (ret = ff_add_format(&formats, i)) < 0)
                 return ret;
-- 
1.8.4.5