[FFmpeg-devel] avcodec/h264, videotoolbox: handle streams with multiple/changing PPS

Submitted by Aman Gupta on Feb. 15, 2017, 1:14 a.m.

Details

Message ID 20170215011437.49702-1-ffmpeg@tmm1.net
State New
Headers show

Commit Message

Aman Gupta Feb. 15, 2017, 1:14 a.m.
From: Aman Gupta <aman@tmm1.net>

The videotoolbox hwaccel only receives SLICE and IDR_SLICE NALUs. This
works fine most of the time, but some streams fail to decode because
changes in PPS are not propagated to the VT decoder.

The failures in this case are incredibly annoying, as VTDecodeFrame()
still returns noErr. Simiarly the decoder callback is invoked with noErr,
and a NULL imageBuffer. Even though all the VT apis indicate success, no
frames are actually decoded.

When running ffmpeg via lldb however, some internal VT errors and
warnings show up all of a sudden. These suggest that the bitstream is
failing some internal consistency checks.

    $ ffmpeg -y -loglevel error -threads 1 -hwaccel videotoolbox \
             -i http://tmm1.s3.amazonaws.com/h264.ts -map v -f null /dev/null
    ...
    [h264 @ 0x7fdadc000000] hardware accelerator failed to decode picture
    Error while decoding stream #0:0: Unknown error occurred
    vt decoder cb: output image buffer is null
    ...

    $ lldb -- ffmpeg ...
    ...
    ffmpeg[49384:2009219] GVA error: AVC_RBSP::parseSliceHeader error
    ffmpeg[49384:2009219] GVA error: pushPicture parseSliceHeader
    vt decoder cb: output image buffer is null
    ffmpeg[49384:2009219] GVA warning: OutputQueueReadyCallback status = 1, buffer == 0x0
    [h264 @ 0x10300a200] hardware accelerator failed to decode picture: Unknown error occurred
    Error while decoding stream #0:0: Unknown error occurred
    ...

After this patch, there are no more errors and the sample decodes as
expected.
---
 libavcodec/h264dec.c      | 7 +++++++
 libavcodec/videotoolbox.c | 8 +++++---
 2 files changed, 12 insertions(+), 3 deletions(-)

Comments

Aman Gupta Feb. 15, 2017, 5:33 a.m.
On Tue, Feb 14, 2017 at 5:14 PM Aman Gupta <ffmpeg@tmm1.net> wrote:

> From: Aman Gupta <aman@tmm1.net>
>
> The videotoolbox hwaccel only receives SLICE and IDR_SLICE NALUs. This
> works fine most of the time, but some streams fail to decode because
> changes in PPS are not propagated to the VT decoder.


Perhaps SPS can change mid-stream, and should be passed into the VT decoder
as well? I'm not familiar enough with h264 to know.


>
> The failures in this case are incredibly annoying, as VTDecodeFrame()
> still returns noErr. Simiarly the decoder callback is invoked with noErr,
> and a NULL imageBuffer. Even though all the VT apis indicate success, no
> frames are actually decoded.
>
> When running ffmpeg via lldb however, some internal VT errors and
> warnings show up all of a sudden. These suggest that the bitstream is
> failing some internal consistency checks.
>
>     $ ffmpeg -y -loglevel error -threads 1 -hwaccel videotoolbox \
>              -i http://tmm1.s3.amazonaws.com/h264.ts -map v -f null
> /dev/null
>     ...
>     [h264 @ 0x7fdadc000000] hardware accelerator failed to decode picture
>     Error while decoding stream #0:0: Unknown error occurred
>     vt decoder cb: output image buffer is null
>     ...
>
>     $ lldb -- ffmpeg ...
>     ...
>     ffmpeg[49384:2009219] GVA error: AVC_RBSP::parseSliceHeader error
>     ffmpeg[49384:2009219] GVA error: pushPicture parseSliceHeader
>     vt decoder cb: output image buffer is null
>     ffmpeg[49384:2009219] GVA warning: OutputQueueReadyCallback status =
> 1, buffer == 0x0
>     [h264 @ 0x10300a200] hardware accelerator failed to decode picture:
> Unknown error occurred
>     Error while decoding stream #0:0: Unknown error occurred
>     ...
>
> After this patch, there are no more errors and the sample decodes as
> expected.
> ---
>  libavcodec/h264dec.c      | 7 +++++++
>  libavcodec/videotoolbox.c | 8 +++++---
>  2 files changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
> index 41c0964..af8b256 100644
> --- a/libavcodec/h264dec.c
> +++ b/libavcodec/h264dec.c
> @@ -755,6 +755,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
>                                                         nal->size_bits);
>              if (ret < 0 && (h->avctx->err_recognition & AV_EF_EXPLODE))
>                  goto end;
> +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt ==
> AV_PIX_FMT_VIDEOTOOLBOX) {
> +                ret = avctx->hwaccel->decode_slice(avctx,
> +                                                   nal->raw_data,
> +                                                   nal->raw_size);
> +                if (ret < 0)
> +                    goto end;
> +            }
>              break;
>          case H264_NAL_AUD:
>          case H264_NAL_END_SEQUENCE:
> diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c
> index 1288aa5..1b5dffd 100644
> --- a/libavcodec/videotoolbox.c
> +++ b/libavcodec/videotoolbox.c
> @@ -138,8 +138,6 @@ int ff_videotoolbox_h264_start_frame(AVCodecContext
> *avctx,
>      VTContext *vtctx = avctx->internal->hwaccel_priv_data;
>      H264Context *h  = avctx->priv_data;
>
> -    vtctx->bitstream_size = 0;
> -
>      if (h->is_avc == 1) {
>          return videotoolbox_buffer_copy(vtctx, buffer, size);
>      }
> @@ -373,8 +371,12 @@ static int videotoolbox_h264_end_frame(AVCodecContext
> *avctx)
>  {
>      H264Context *h = avctx->priv_data;
>      AVFrame *frame = h->cur_pic_ptr->f;
> +    VTContext *vtctx = avctx->internal->hwaccel_priv_data;
> +    int ret;
>
> -    return videotoolbox_common_end_frame(avctx, frame);
> +    ret = videotoolbox_common_end_frame(avctx, frame);
> +    vtctx->bitstream_size = 0;
> +    return ret;
>  }
>
>  static int videotoolbox_mpeg_start_frame(AVCodecContext *avctx,
> --
> 2.10.1 (Apple Git-78)
>
>
wm4 Feb. 15, 2017, 6:04 a.m.
On Wed, 15 Feb 2017 05:33:38 +0000
Aman Gupta <ffmpeg@tmm1.net> wrote:

> On Tue, Feb 14, 2017 at 5:14 PM Aman Gupta <ffmpeg@tmm1.net> wrote:
> 
> > From: Aman Gupta <aman@tmm1.net>
> >
> > The videotoolbox hwaccel only receives SLICE and IDR_SLICE NALUs. This
> > works fine most of the time, but some streams fail to decode because
> > changes in PPS are not propagated to the VT decoder.  
> 
> 
> Perhaps SPS can change mid-stream, and should be passed into the VT decoder
> as well? I'm not familiar enough with h264 to know.

Yes, it can. In theory, it should work, because a SPS/PPS change (in a
slice) will trigger a configuration change and reinit the hwaccel (so
ff_videotoolbox_avcc_extradata_create() will be called and update with
the new SPS). But there's some partial change-detection, which prevents
reconfiguration changes if the sw decoder doesn't need them, so this is
imperfect. (Also assumes slices can reference only 1 SPS and 1 PPS,
which is probably true.)

Patch hide | download patch | download mbox

diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
index 41c0964..af8b256 100644
--- a/libavcodec/h264dec.c
+++ b/libavcodec/h264dec.c
@@ -755,6 +755,13 @@  FF_ENABLE_DEPRECATION_WARNINGS
                                                        nal->size_bits);
             if (ret < 0 && (h->avctx->err_recognition & AV_EF_EXPLODE))
                 goto end;
+            if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
+                ret = avctx->hwaccel->decode_slice(avctx,
+                                                   nal->raw_data,
+                                                   nal->raw_size);
+                if (ret < 0)
+                    goto end;
+            }
             break;
         case H264_NAL_AUD:
         case H264_NAL_END_SEQUENCE:
diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c
index 1288aa5..1b5dffd 100644
--- a/libavcodec/videotoolbox.c
+++ b/libavcodec/videotoolbox.c
@@ -138,8 +138,6 @@  int ff_videotoolbox_h264_start_frame(AVCodecContext *avctx,
     VTContext *vtctx = avctx->internal->hwaccel_priv_data;
     H264Context *h  = avctx->priv_data;
 
-    vtctx->bitstream_size = 0;
-
     if (h->is_avc == 1) {
         return videotoolbox_buffer_copy(vtctx, buffer, size);
     }
@@ -373,8 +371,12 @@  static int videotoolbox_h264_end_frame(AVCodecContext *avctx)
 {
     H264Context *h = avctx->priv_data;
     AVFrame *frame = h->cur_pic_ptr->f;
+    VTContext *vtctx = avctx->internal->hwaccel_priv_data;
+    int ret;
 
-    return videotoolbox_common_end_frame(avctx, frame);
+    ret = videotoolbox_common_end_frame(avctx, frame);
+    vtctx->bitstream_size = 0;
+    return ret;
 }
 
 static int videotoolbox_mpeg_start_frame(AVCodecContext *avctx,