| Message ID | 20170215011437.49702-1-ffmpeg@tmm1.net |
|---|---|
| State | New |
| Headers | show |
On Tue, Feb 14, 2017 at 5:14 PM Aman Gupta <ffmpeg@tmm1.net> wrote: > From: Aman Gupta <aman@tmm1.net> > > The videotoolbox hwaccel only receives SLICE and IDR_SLICE NALUs. This > works fine most of the time, but some streams fail to decode because > changes in PPS are not propagated to the VT decoder. Perhaps SPS can change mid-stream, and should be passed into the VT decoder as well? I'm not familiar enough with h264 to know. > > The failures in this case are incredibly annoying, as VTDecodeFrame() > still returns noErr. Simiarly the decoder callback is invoked with noErr, > and a NULL imageBuffer. Even though all the VT apis indicate success, no > frames are actually decoded. > > When running ffmpeg via lldb however, some internal VT errors and > warnings show up all of a sudden. These suggest that the bitstream is > failing some internal consistency checks. > > $ ffmpeg -y -loglevel error -threads 1 -hwaccel videotoolbox \ > -i http://tmm1.s3.amazonaws.com/h264.ts -map v -f null > /dev/null > ... > [h264 @ 0x7fdadc000000] hardware accelerator failed to decode picture > Error while decoding stream #0:0: Unknown error occurred > vt decoder cb: output image buffer is null > ... > > $ lldb -- ffmpeg ... > ... > ffmpeg[49384:2009219] GVA error: AVC_RBSP::parseSliceHeader error > ffmpeg[49384:2009219] GVA error: pushPicture parseSliceHeader > vt decoder cb: output image buffer is null > ffmpeg[49384:2009219] GVA warning: OutputQueueReadyCallback status = > 1, buffer == 0x0 > [h264 @ 0x10300a200] hardware accelerator failed to decode picture: > Unknown error occurred > Error while decoding stream #0:0: Unknown error occurred > ... > > After this patch, there are no more errors and the sample decodes as > expected. > --- > libavcodec/h264dec.c | 7 +++++++ > libavcodec/videotoolbox.c | 8 +++++--- > 2 files changed, 12 insertions(+), 3 deletions(-) > > diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c > index 41c0964..af8b256 100644 > --- a/libavcodec/h264dec.c > +++ b/libavcodec/h264dec.c > @@ -755,6 +755,13 @@ FF_ENABLE_DEPRECATION_WARNINGS > nal->size_bits); > if (ret < 0 && (h->avctx->err_recognition & AV_EF_EXPLODE)) > goto end; > + if (avctx->hwaccel && avctx->hwaccel->pix_fmt == > AV_PIX_FMT_VIDEOTOOLBOX) { > + ret = avctx->hwaccel->decode_slice(avctx, > + nal->raw_data, > + nal->raw_size); > + if (ret < 0) > + goto end; > + } > break; > case H264_NAL_AUD: > case H264_NAL_END_SEQUENCE: > diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c > index 1288aa5..1b5dffd 100644 > --- a/libavcodec/videotoolbox.c > +++ b/libavcodec/videotoolbox.c > @@ -138,8 +138,6 @@ int ff_videotoolbox_h264_start_frame(AVCodecContext > *avctx, > VTContext *vtctx = avctx->internal->hwaccel_priv_data; > H264Context *h = avctx->priv_data; > > - vtctx->bitstream_size = 0; > - > if (h->is_avc == 1) { > return videotoolbox_buffer_copy(vtctx, buffer, size); > } > @@ -373,8 +371,12 @@ static int videotoolbox_h264_end_frame(AVCodecContext > *avctx) > { > H264Context *h = avctx->priv_data; > AVFrame *frame = h->cur_pic_ptr->f; > + VTContext *vtctx = avctx->internal->hwaccel_priv_data; > + int ret; > > - return videotoolbox_common_end_frame(avctx, frame); > + ret = videotoolbox_common_end_frame(avctx, frame); > + vtctx->bitstream_size = 0; > + return ret; > } > > static int videotoolbox_mpeg_start_frame(AVCodecContext *avctx, > -- > 2.10.1 (Apple Git-78) > >
On Wed, 15 Feb 2017 05:33:38 +0000 Aman Gupta <ffmpeg@tmm1.net> wrote: > On Tue, Feb 14, 2017 at 5:14 PM Aman Gupta <ffmpeg@tmm1.net> wrote: > > > From: Aman Gupta <aman@tmm1.net> > > > > The videotoolbox hwaccel only receives SLICE and IDR_SLICE NALUs. This > > works fine most of the time, but some streams fail to decode because > > changes in PPS are not propagated to the VT decoder. > > > Perhaps SPS can change mid-stream, and should be passed into the VT decoder > as well? I'm not familiar enough with h264 to know. Yes, it can. In theory, it should work, because a SPS/PPS change (in a slice) will trigger a configuration change and reinit the hwaccel (so ff_videotoolbox_avcc_extradata_create() will be called and update with the new SPS). But there's some partial change-detection, which prevents reconfiguration changes if the sw decoder doesn't need them, so this is imperfect. (Also assumes slices can reference only 1 SPS and 1 PPS, which is probably true.)
diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c index 41c0964..af8b256 100644 --- a/libavcodec/h264dec.c +++ b/libavcodec/h264dec.c @@ -755,6 +755,13 @@ FF_ENABLE_DEPRECATION_WARNINGS nal->size_bits); if (ret < 0 && (h->avctx->err_recognition & AV_EF_EXPLODE)) goto end; + if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) { + ret = avctx->hwaccel->decode_slice(avctx, + nal->raw_data, + nal->raw_size); + if (ret < 0) + goto end; + } break; case H264_NAL_AUD: case H264_NAL_END_SEQUENCE: diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index 1288aa5..1b5dffd 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -138,8 +138,6 @@ int ff_videotoolbox_h264_start_frame(AVCodecContext *avctx, VTContext *vtctx = avctx->internal->hwaccel_priv_data; H264Context *h = avctx->priv_data; - vtctx->bitstream_size = 0; - if (h->is_avc == 1) { return videotoolbox_buffer_copy(vtctx, buffer, size); } @@ -373,8 +371,12 @@ static int videotoolbox_h264_end_frame(AVCodecContext *avctx) { H264Context *h = avctx->priv_data; AVFrame *frame = h->cur_pic_ptr->f; + VTContext *vtctx = avctx->internal->hwaccel_priv_data; + int ret; - return videotoolbox_common_end_frame(avctx, frame); + ret = videotoolbox_common_end_frame(avctx, frame); + vtctx->bitstream_size = 0; + return ret; } static int videotoolbox_mpeg_start_frame(AVCodecContext *avctx,
From: Aman Gupta <aman@tmm1.net> The videotoolbox hwaccel only receives SLICE and IDR_SLICE NALUs. This works fine most of the time, but some streams fail to decode because changes in PPS are not propagated to the VT decoder. The failures in this case are incredibly annoying, as VTDecodeFrame() still returns noErr. Simiarly the decoder callback is invoked with noErr, and a NULL imageBuffer. Even though all the VT apis indicate success, no frames are actually decoded. When running ffmpeg via lldb however, some internal VT errors and warnings show up all of a sudden. These suggest that the bitstream is failing some internal consistency checks. $ ffmpeg -y -loglevel error -threads 1 -hwaccel videotoolbox \ -i http://tmm1.s3.amazonaws.com/h264.ts -map v -f null /dev/null ... [h264 @ 0x7fdadc000000] hardware accelerator failed to decode picture Error while decoding stream #0:0: Unknown error occurred vt decoder cb: output image buffer is null ... $ lldb -- ffmpeg ... ... ffmpeg[49384:2009219] GVA error: AVC_RBSP::parseSliceHeader error ffmpeg[49384:2009219] GVA error: pushPicture parseSliceHeader vt decoder cb: output image buffer is null ffmpeg[49384:2009219] GVA warning: OutputQueueReadyCallback status = 1, buffer == 0x0 [h264 @ 0x10300a200] hardware accelerator failed to decode picture: Unknown error occurred Error while decoding stream #0:0: Unknown error occurred ... After this patch, there are no more errors and the sample decodes as expected. --- libavcodec/h264dec.c | 7 +++++++ libavcodec/videotoolbox.c | 8 +++++--- 2 files changed, 12 insertions(+), 3 deletions(-)