diff mbox

[FFmpeg-devel,3/9] avcodec/videotoolbox: h264 decoder on iOS is unable to decode interlaced video

Message ID 20170926003631.55571-3-ffmpeg@tmm1.net
State New
Headers show

Commit Message

Aman Karmani Sept. 26, 2017, 12:36 a.m. UTC 
From: Aman Gupta <aman@tmm1.net>

VideoToolbox's support for interlaced H264 is quite poor.

On macOS, VTSessionCopySupportedPropertyDictionary() will show that
kVTDecompressionPropertyKey_FieldMode is supported. Possible values for
this option include DeinterlaceFields and BothFields.

However, files that use MBAFF interlacing are not deinterlaced on macOS
even if the FieldMode=DeinterlacedFields option is specified. Although
it doesn't always deinterlace, the macOS version of VideoToolbox will
always decode and return frame data even when the H264 source is
interlaced.

On iOS, FieldMode is not a valid option and interlaced H264 is not
supported at all. You can create a valid decompression session, but no
frames are returned and almost every DecodeFrame() call returns a
malfunction error.

I opened rdar://30669495 about this, and Apple's response was:

> Correct and intentional. Please stop using interlaced video.

So this commit forces the VideoToolbox hwaccel to fail early when
interlaced H264 is encountered. Thus the API user can easily detect the
failure and use a different decoder.

Here are some sample files I tested on iOS 10 and 11:

https://s3.amazonaws.com/tmm1/videotoolbox/interlaced.ts
https://s3.amazonaws.com/tmm1/videotoolbox/interlaced2.ts
https://s3.amazonaws.com/tmm1/videotoolbox/interlaced3.ts

Decoding interlaced2.ts with VideoToolbox never produces any frames.
Decoding interlaced3.ts ocassional produces frames, but most slices
cause errors.

Decoding interlaced.ts actually works as expected, even though its SPS
matches the other two samples which fail. This means my test is not
comprehensive, and it is not possible to fully detect if a file is
compatible with VideoToolbox using its SPS alone. Still, though this
method produces false positives, it does not produce false negatives.
---
 libavcodec/videotoolbox.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

Comments

Carl Eugen Hoyos Sept. 26, 2017, 11:20 a.m. UTC | #1 
2017-09-26 2:36 GMT+02:00 Aman Gupta <ffmpeg@tmm1.net>:

> +   case kCMVideoCodecType_H264 : {
> +      H264Context *h = avctx->priv_data;
> +      if (TARGET_OS_IPHONE && h->ps.sps->frame_mbs_only_flag == 0) {

I believe that for DVB 1080 transmissions the flag is
never set even for progressive frames, so this has
to be conditional depending on a new user option.
Imo, the default should be not to abort but others may
disagree.

Carl Eugen
Aman Karmani Sept. 4, 2020, 9:44 p.m. UTC | #2 
On Tue, Sep 26, 2017 at 4:20 AM Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:

> 2017-09-26 2:36 GMT+02:00 Aman Gupta <ffmpeg@tmm1.net>:
>
>
>
> > +   case kCMVideoCodecType_H264 : {
>
> > +      H264Context *h = avctx->priv_data;
>
> > +      if (TARGET_OS_IPHONE && h->ps.sps->frame_mbs_only_flag == 0) {
>
>
>
> I believe that for DVB 1080 transmissions the flag is
>
> never set even for progressive frames, so this has


Do you know where I could find a sample of such a transmission?

Aman


>
> to be conditional depending on a new user option.
>
> Imo, the default should be not to abort but others may
>
> disagree.
>
>
>
> Carl Eugen
>
> _______________________________________________
>
> ffmpeg-devel mailing list
>
> ffmpeg-devel@ffmpeg.org
>
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
>
diff mbox

Patch

diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c
index 7dfcf14c00..8ed56392cc 100644
--- a/libavcodec/videotoolbox.c
+++ b/libavcodec/videotoolbox.c
@@ -496,11 +496,18 @@  static CFDictionaryRef videotoolbox_decoder_config_create(CMVideoCodecType codec
             if (data)
                 CFDictionarySetValue(avc_info, CFSTR("esds"), data);
             break;
-        case kCMVideoCodecType_H264 :
+        case kCMVideoCodecType_H264 : {
+            H264Context *h = avctx->priv_data;
+            if (TARGET_OS_IPHONE && h->ps.sps->frame_mbs_only_flag == 0) {
+                av_log(avctx, AV_LOG_ERROR, "VideoToolbox cannot decode interlaced fields on iOS\n");
+                CFRelease(avc_info);
+                goto fail;
+            }
             data = ff_videotoolbox_avcc_extradata_create(avctx);
             if (data)
                 CFDictionarySetValue(avc_info, CFSTR("avcC"), data);
             break;
+        }
         default:
             break;
         }
@@ -515,6 +522,10 @@  static CFDictionaryRef videotoolbox_decoder_config_create(CMVideoCodecType codec
         CFRelease(avc_info);
     }
     return config_info;
+
+fail:
+    CFRelease(config_info);
+    return NULL;
 }
 
 static CFDictionaryRef videotoolbox_buffer_attributes_create(int width,