[FFmpeg-devel,v2] Fix visual glitch with XvMC, caused by wrong idct permutation.

Submitted by Ivan Kalvachev on Oct. 9, 2017, 11:53 p.m.

Details

Message ID CABA=pqdE2zfhYiJpocnnP3FfJ3T_wRCEMAasRX7VJXrm9gRJ8A@mail.gmail.com
State New
Headers show

Commit Message

Ivan Kalvachev Oct. 9, 2017, 11:53 p.m.
On 10/9/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> On Mon, Oct 09, 2017 at 09:02:38AM -0400, Ronald S. Bultje wrote:
>> Hi,
>>
>> On Mon, Oct 9, 2017 at 6:46 AM, Ivan Kalvachev <ikalvachev@gmail.com>
>> wrote:
>>
>> > On 10/9/17, Ronald S. Bultje <rsbultje@gmail.com> wrote:
>> > > On Sun, Oct 8, 2017 at 6:52 PM, Ivan Kalvachev <ikalvachev@gmail.com>
>> > wrote:
>> > > [..]
>> > >
>> > > Indentation is off in the second hunk, can you fix that?
>> >
>> > You want it 4 spaces to the right

Done.

>>
>> Yes, please.
>>
>> BTW, I think it would be better to use "127" number.
>> >
>>
>> I don't really mind either way. The number 128 suggests it may have been
>> intended as a bitmask. Michael is probably better positioned to comment
>> on
>> this.
>
> I don't really remember but i think 128 was chosen for ABI
> compatibility with additions to it from libav. So it should no longer
> matter what values are used on additions

Then I'm using the next free number "24".

Please, commit when you think it is appropriate.

Best Regards
   Ivan Kalvachev

Patch hide | download patch | download mbox

From 8842a69091b5eb5cf9b704b3ff504d21db4aad9b Mon Sep 17 00:00:00 2001
From: Ivan Kalvachev <ikalvachev@gmail.com>
Date: Mon, 9 Oct 2017 01:25:00 +0300
Subject: [PATCH] Fix visual glitch with XvMC, caused by wrong idct
 permutation.

In the past XvMC forced simple_idct since
it was using FF_IDCT_PERM_NONE.
However now we have SIMD variants of simple_idct that
are using FF_IDCT_PERM_TRANSPOSE and if they are selected
XvMC would get coefficients in the wrong order.

The patch creates new FF_IDCT_NONE that
is used only for this kind of hardware decoding
and that fallbacks to the old C only simple idct.

Signed-off-by: Ivan Kalvachev <ikalvachev@gmail.com>
---
 libavcodec/avcodec.h   | 1 +
 libavcodec/idctdsp.c   | 1 +
 libavcodec/mpeg12dec.c | 2 +-
 3 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
index 52cc5b0ca..18c3e3ea1 100644
--- a/libavcodec/avcodec.h
+++ b/libavcodec/avcodec.h
@@ -3146,6 +3146,7 @@  typedef struct AVCodecContext {
 #if FF_API_ARCH_ALPHA
 #define FF_IDCT_SIMPLEALPHA   23
 #endif
+#define FF_IDCT_NONE          24 /* Used by XvMC to extract IDCT coefficients with FF_IDCT_PERM_NONE */
 #define FF_IDCT_SIMPLEAUTO    128
 
     /**
diff --git a/libavcodec/idctdsp.c b/libavcodec/idctdsp.c
index d596aed1a..0122d29ef 100644
--- a/libavcodec/idctdsp.c
+++ b/libavcodec/idctdsp.c
@@ -279,6 +279,7 @@  av_cold void ff_idctdsp_init(IDCTDSPContext *c, AVCodecContext *avctx)
                 c->perm_type = FF_IDCT_PERM_NONE;
 #endif /* CONFIG_FAANIDCT */
             } else { // accurate/default
+                /* Be sure FF_IDCT_NONE will select this one, since it uses FF_IDCT_PERM_NONE */
                 c->idct_put  = ff_simple_idct_put_8;
                 c->idct_add  = ff_simple_idct_add_8;
                 c->idct      = ff_simple_idct_8;
diff --git a/libavcodec/mpeg12dec.c b/libavcodec/mpeg12dec.c
index 22c29c150..4e68be27f 100644
--- a/libavcodec/mpeg12dec.c
+++ b/libavcodec/mpeg12dec.c
@@ -1217,7 +1217,7 @@  static void setup_hwaccel_for_pixfmt(AVCodecContext *avctx)
 #endif
         )
         if (avctx->idct_algo == FF_IDCT_AUTO)
-            avctx->idct_algo = FF_IDCT_SIMPLE;
+            avctx->idct_algo = FF_IDCT_NONE;
 
     if (avctx->hwaccel && avctx->pix_fmt == AV_PIX_FMT_XVMC) {
         Mpeg1Context *s1 = avctx->priv_data;
-- 
2.14.1