[FFmpeg-devel,1/3] avcodec/h264, videotoolbox: pass SPS changes into the VT decoder

Submitted by Aman Gupta on Feb. 16, 2017, 6:29 p.m.

Details

Message ID 20170216182938.55393-1-ffmpeg@tmm1.net
State New
Headers show

Commit Message

Aman Gupta Feb. 16, 2017, 6:29 p.m.
From: Aman Gupta <aman@tmm1.net>

This fixes playback of h264 streams with SPS changes. One such sample
is available at http://tmm1.s3.amazonaws.com/videotoolbox/spschange.ts.
It switches mid-stream from level 4.0 to level 3.2.

Previously, playing this sample with the VT hwaccel on iOS would crash.
After this patch, it plays back as expected.

On macOS however, feeding in new SPS into an existing decompression
session does not always work, so this patch is only a partial fix.
---
 libavcodec/h264dec.c | 7 +++++++
 1 file changed, 7 insertions(+)

Comments

Mark Thompson Feb. 16, 2017, 7:46 p.m.
On 16/02/17 18:29, Aman Gupta wrote:
> From: Aman Gupta <aman@tmm1.net>
> 
> This fixes playback of h264 streams with SPS changes. One such sample
> is available at http://tmm1.s3.amazonaws.com/videotoolbox/spschange.ts.
> It switches mid-stream from level 4.0 to level 3.2.
> 
> Previously, playing this sample with the VT hwaccel on iOS would crash.
> After this patch, it plays back as expected.
> 
> On macOS however, feeding in new SPS into an existing decompression
> session does not always work, so this patch is only a partial fix.
> ---
>  libavcodec/h264dec.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
> index 41c0964..e521c52 100644
> --- a/libavcodec/h264dec.c
> +++ b/libavcodec/h264dec.c
> @@ -740,6 +740,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
>              break;
>          case H264_NAL_SPS: {
>              GetBitContext tmp_gb = nal->gb;
> +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
> +                ret = avctx->hwaccel->decode_slice(avctx,
> +                                                   nal->data,
> +                                                   nal->size);
> +                if (ret < 0)
> +                    goto end;
> +            }
>              if (ff_h264_decode_seq_parameter_set(&tmp_gb, avctx, &h->ps, 0) >= 0)
>                  break;
>              av_log(h->avctx, AV_LOG_DEBUG,
> 

The SPS isn't a slice - calling decode_slice() on it is confusing.  I realise you are really using it as "upload blob" and that does basically make sense, but changes like this are moving it further away from being maintainable as a hwaccel.  (Compare how many DXVA- or VAAPI-specific code paths currently exist in the decoder outside the actual hwaccel functions.)  The previous patches doing the same thing with the PPS and fiddling something funny inside the reference picture lists are similar - in themselves not obviously unreasonable, but making the code trickier in ways which may make things more difficult later.

So, I think this needs a step back to consider whether this should actually be implemented as a hwaccel at all if changes like this are needed:

It looks like you are now supplying it all video-significant packets - what would need to be implemented to do this without being inside hwaccel infrastructure?  (As a standalone full-bitstream decoder, but possibly calling in to the existing H.264 code if that helps.)

On the other hand, if it is going to continue being a hwaccel, can we add make some changes to the internal hwaccel API to avoid having to make changes like this in the generic decoder?  (Also thinking of other codecs - you're changing H.264 here, are other codecs going to run into similar issues, particularly H.265 which is structurally very similar in these aspects?)
wm4 Feb. 17, 2017, 6:02 a.m.
On Thu, 16 Feb 2017 10:29:36 -0800
Aman Gupta <ffmpeg@tmm1.net> wrote:

> From: Aman Gupta <aman@tmm1.net>
> 
> This fixes playback of h264 streams with SPS changes. One such sample
> is available at http://tmm1.s3.amazonaws.com/videotoolbox/spschange.ts.
> It switches mid-stream from level 4.0 to level 3.2.
> 
> Previously, playing this sample with the VT hwaccel on iOS would crash.
> After this patch, it plays back as expected.
> 
> On macOS however, feeding in new SPS into an existing decompression
> session does not always work, so this patch is only a partial fix.
> ---
>  libavcodec/h264dec.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
> index 41c0964..e521c52 100644
> --- a/libavcodec/h264dec.c
> +++ b/libavcodec/h264dec.c
> @@ -740,6 +740,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
>              break;
>          case H264_NAL_SPS: {
>              GetBitContext tmp_gb = nal->gb;
> +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
> +                ret = avctx->hwaccel->decode_slice(avctx,
> +                                                   nal->data,
> +                                                   nal->size);
> +                if (ret < 0)
> +                    goto end;
> +            }
>              if (ff_h264_decode_seq_parameter_set(&tmp_gb, avctx, &h->ps, 0) >= 0)
>                  break;
>              av_log(h->avctx, AV_LOG_DEBUG,

A bit ugly but ok IMHO. Maybe it would be better to add a new hwaccel
callback here, even if it's used by VT only.

You should probably wait for approval by michaelni.
wm4 Feb. 17, 2017, 6:06 a.m.
On Thu, 16 Feb 2017 19:46:08 +0000
Mark Thompson <sw@jkqxz.net> wrote:

> On 16/02/17 18:29, Aman Gupta wrote:
> > From: Aman Gupta <aman@tmm1.net>
> > 
> > This fixes playback of h264 streams with SPS changes. One such sample
> > is available at http://tmm1.s3.amazonaws.com/videotoolbox/spschange.ts.
> > It switches mid-stream from level 4.0 to level 3.2.
> > 
> > Previously, playing this sample with the VT hwaccel on iOS would crash.
> > After this patch, it plays back as expected.
> > 
> > On macOS however, feeding in new SPS into an existing decompression
> > session does not always work, so this patch is only a partial fix.
> > ---
> >  libavcodec/h264dec.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
> > index 41c0964..e521c52 100644
> > --- a/libavcodec/h264dec.c
> > +++ b/libavcodec/h264dec.c
> > @@ -740,6 +740,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
> >              break;
> >          case H264_NAL_SPS: {
> >              GetBitContext tmp_gb = nal->gb;
> > +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
> > +                ret = avctx->hwaccel->decode_slice(avctx,
> > +                                                   nal->data,
> > +                                                   nal->size);
> > +                if (ret < 0)
> > +                    goto end;
> > +            }
> >              if (ff_h264_decode_seq_parameter_set(&tmp_gb, avctx, &h->ps, 0) >= 0)
> >                  break;
> >              av_log(h->avctx, AV_LOG_DEBUG,
> >   
> 
> The SPS isn't a slice - calling decode_slice() on it is confusing.  I realise you are really using it as "upload blob" and that does basically make sense, but changes like this are moving it further away from being maintainable as a hwaccel.  (Compare how many DXVA- or VAAPI-specific code paths currently exist in the decoder outside the actual hwaccel functions.)  The previous patches doing the same thing with the PPS and fiddling something funny inside the reference picture lists are similar - in themselves not obviously unreasonable, but making the code trickier in ways which may make things more difficult later.
> 
> So, I think this needs a step back to consider whether this should actually be implemented as a hwaccel at all if changes like this are needed:
> 
> It looks like you are now supplying it all video-significant packets - what would need to be implemented to do this without being inside hwaccel infrastructure?  (As a standalone full-bitstream decoder, but possibly calling in to the existing H.264 code if that helps.)
> 
> On the other hand, if it is going to continue being a hwaccel, can we add make some changes to the internal hwaccel API to avoid having to make changes like this in the generic decoder?  (Also thinking of other codecs - you're changing H.264 here, are other codecs going to run into similar issues, particularly H.265 which is structurally very similar in these aspects?)

Yeah, it's not clear at all how this should be done. Clearly using it
as hwaccel has advantages, but is also a bit hacky. We also need to
consider things like deinterlacing (which VT does in the decoder), and
async API (needed to make it actually fast). At least it looks like
deinterlacing doesn't actually work, so we might not need to worry
about that one.
Michael Niedermayer Feb. 17, 2017, 8:15 p.m.
On Fri, Feb 17, 2017 at 07:02:10AM +0100, wm4 wrote:
> On Thu, 16 Feb 2017 10:29:36 -0800
> Aman Gupta <ffmpeg@tmm1.net> wrote:
> 
> > From: Aman Gupta <aman@tmm1.net>
> > 
> > This fixes playback of h264 streams with SPS changes. One such sample
> > is available at http://tmm1.s3.amazonaws.com/videotoolbox/spschange.ts.
> > It switches mid-stream from level 4.0 to level 3.2.
> > 
> > Previously, playing this sample with the VT hwaccel on iOS would crash.
> > After this patch, it plays back as expected.
> > 
> > On macOS however, feeding in new SPS into an existing decompression
> > session does not always work, so this patch is only a partial fix.
> > ---
> >  libavcodec/h264dec.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
> > index 41c0964..e521c52 100644
> > --- a/libavcodec/h264dec.c
> > +++ b/libavcodec/h264dec.c
> > @@ -740,6 +740,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
> >              break;
> >          case H264_NAL_SPS: {
> >              GetBitContext tmp_gb = nal->gb;
> > +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
> > +                ret = avctx->hwaccel->decode_slice(avctx,
> > +                                                   nal->data,
> > +                                                   nal->size);
> > +                if (ret < 0)
> > +                    goto end;
> > +            }
> >              if (ff_h264_decode_seq_parameter_set(&tmp_gb, avctx, &h->ps, 0) >= 0)
> >                  break;
> >              av_log(h->avctx, AV_LOG_DEBUG,
> 
> A bit ugly but ok IMHO. Maybe it would be better to add a new hwaccel
> callback here, even if it's used by VT only.
> 
> You should probably wait for approval by michaelni.

i dont really have an oppinion on hwaccel, thats not so much my
area
though i find special cases for specific hwaccel a bit ugly, i dont
object to it, just saying i would be in favor of not having special
cases if that is possible

[...]
wm4 March 2, 2017, 9:35 a.m.
On Fri, 17 Feb 2017 21:15:56 +0100
Michael Niedermayer <michael@niedermayer.cc> wrote:

> On Fri, Feb 17, 2017 at 07:02:10AM +0100, wm4 wrote:
> > On Thu, 16 Feb 2017 10:29:36 -0800
> > Aman Gupta <ffmpeg@tmm1.net> wrote:
> >   
> > > From: Aman Gupta <aman@tmm1.net>
> > > 
> > > This fixes playback of h264 streams with SPS changes. One such sample
> > > is available at http://tmm1.s3.amazonaws.com/videotoolbox/spschange.ts.
> > > It switches mid-stream from level 4.0 to level 3.2.
> > > 
> > > Previously, playing this sample with the VT hwaccel on iOS would crash.
> > > After this patch, it plays back as expected.
> > > 
> > > On macOS however, feeding in new SPS into an existing decompression
> > > session does not always work, so this patch is only a partial fix.
> > > ---
> > >  libavcodec/h264dec.c | 7 +++++++
> > >  1 file changed, 7 insertions(+)
> > > 
> > > diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
> > > index 41c0964..e521c52 100644
> > > --- a/libavcodec/h264dec.c
> > > +++ b/libavcodec/h264dec.c
> > > @@ -740,6 +740,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
> > >              break;
> > >          case H264_NAL_SPS: {
> > >              GetBitContext tmp_gb = nal->gb;
> > > +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
> > > +                ret = avctx->hwaccel->decode_slice(avctx,
> > > +                                                   nal->data,
> > > +                                                   nal->size);
> > > +                if (ret < 0)
> > > +                    goto end;
> > > +            }
> > >              if (ff_h264_decode_seq_parameter_set(&tmp_gb, avctx, &h->ps, 0) >= 0)
> > >                  break;
> > >              av_log(h->avctx, AV_LOG_DEBUG,  
> > 
> > A bit ugly but ok IMHO. Maybe it would be better to add a new hwaccel
> > callback here, even if it's used by VT only.
> > 
> > You should probably wait for approval by michaelni.  
> 
> i dont really have an oppinion on hwaccel, thats not so much my
> area
> though i find special cases for specific hwaccel a bit ugly, i dont
> object to it, just saying i would be in favor of not having special
> cases if that is possible

So do you think this is tolerable in the current state or not?
Hendrik Leppkes March 2, 2017, 9:47 a.m.
On Thu, Mar 2, 2017 at 10:35 AM, wm4 <nfxjfg@googlemail.com> wrote:
> On Fri, 17 Feb 2017 21:15:56 +0100
> Michael Niedermayer <michael@niedermayer.cc> wrote:
>
>> On Fri, Feb 17, 2017 at 07:02:10AM +0100, wm4 wrote:
>> > On Thu, 16 Feb 2017 10:29:36 -0800
>> > Aman Gupta <ffmpeg@tmm1.net> wrote:
>> >
>> > > From: Aman Gupta <aman@tmm1.net>
>> > >
>> > > This fixes playback of h264 streams with SPS changes. One such sample
>> > > is available at http://tmm1.s3.amazonaws.com/videotoolbox/spschange.ts.
>> > > It switches mid-stream from level 4.0 to level 3.2.
>> > >
>> > > Previously, playing this sample with the VT hwaccel on iOS would crash.
>> > > After this patch, it plays back as expected.
>> > >
>> > > On macOS however, feeding in new SPS into an existing decompression
>> > > session does not always work, so this patch is only a partial fix.
>> > > ---
>> > >  libavcodec/h264dec.c | 7 +++++++
>> > >  1 file changed, 7 insertions(+)
>> > >
>> > > diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
>> > > index 41c0964..e521c52 100644
>> > > --- a/libavcodec/h264dec.c
>> > > +++ b/libavcodec/h264dec.c
>> > > @@ -740,6 +740,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
>> > >              break;
>> > >          case H264_NAL_SPS: {
>> > >              GetBitContext tmp_gb = nal->gb;
>> > > +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
>> > > +                ret = avctx->hwaccel->decode_slice(avctx,
>> > > +                                                   nal->data,
>> > > +                                                   nal->size);
>> > > +                if (ret < 0)
>> > > +                    goto end;
>> > > +            }
>> > >              if (ff_h264_decode_seq_parameter_set(&tmp_gb, avctx, &h->ps, 0) >= 0)
>> > >                  break;
>> > >              av_log(h->avctx, AV_LOG_DEBUG,
>> >
>> > A bit ugly but ok IMHO. Maybe it would be better to add a new hwaccel
>> > callback here, even if it's used by VT only.
>> >
>> > You should probably wait for approval by michaelni.
>>
>> i dont really have an oppinion on hwaccel, thats not so much my
>> area
>> though i find special cases for specific hwaccel a bit ugly, i dont
>> object to it, just saying i would be in favor of not having special
>> cases if that is possible
>
> So do you think this is tolerable in the current state or not?

I agree with Michael, the number of VT-specific hacks seem to be
piling up. Maybe it should be re-designed entirely instead of piling
hacks on top of hacks to keep it barely working.

- Hendrik
wm4 March 2, 2017, 10:04 a.m.
On Thu, 2 Mar 2017 10:47:23 +0100
Hendrik Leppkes <h.leppkes@gmail.com> wrote:

> On Thu, Mar 2, 2017 at 10:35 AM, wm4 <nfxjfg@googlemail.com> wrote:
> > On Fri, 17 Feb 2017 21:15:56 +0100
> > Michael Niedermayer <michael@niedermayer.cc> wrote:
> >  
> >> On Fri, Feb 17, 2017 at 07:02:10AM +0100, wm4 wrote:  
> >> > On Thu, 16 Feb 2017 10:29:36 -0800
> >> > Aman Gupta <ffmpeg@tmm1.net> wrote:
> >> >  
> >> > > From: Aman Gupta <aman@tmm1.net>
> >> > >
> >> > > This fixes playback of h264 streams with SPS changes. One such sample
> >> > > is available at http://tmm1.s3.amazonaws.com/videotoolbox/spschange.ts.
> >> > > It switches mid-stream from level 4.0 to level 3.2.
> >> > >
> >> > > Previously, playing this sample with the VT hwaccel on iOS would crash.
> >> > > After this patch, it plays back as expected.
> >> > >
> >> > > On macOS however, feeding in new SPS into an existing decompression
> >> > > session does not always work, so this patch is only a partial fix.
> >> > > ---
> >> > >  libavcodec/h264dec.c | 7 +++++++
> >> > >  1 file changed, 7 insertions(+)
> >> > >
> >> > > diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
> >> > > index 41c0964..e521c52 100644
> >> > > --- a/libavcodec/h264dec.c
> >> > > +++ b/libavcodec/h264dec.c
> >> > > @@ -740,6 +740,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
> >> > >              break;
> >> > >          case H264_NAL_SPS: {
> >> > >              GetBitContext tmp_gb = nal->gb;
> >> > > +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
> >> > > +                ret = avctx->hwaccel->decode_slice(avctx,
> >> > > +                                                   nal->data,
> >> > > +                                                   nal->size);
> >> > > +                if (ret < 0)
> >> > > +                    goto end;
> >> > > +            }
> >> > >              if (ff_h264_decode_seq_parameter_set(&tmp_gb, avctx, &h->ps, 0) >= 0)
> >> > >                  break;
> >> > >              av_log(h->avctx, AV_LOG_DEBUG,  
> >> >
> >> > A bit ugly but ok IMHO. Maybe it would be better to add a new hwaccel
> >> > callback here, even if it's used by VT only.
> >> >
> >> > You should probably wait for approval by michaelni.  
> >>
> >> i dont really have an oppinion on hwaccel, thats not so much my
> >> area
> >> though i find special cases for specific hwaccel a bit ugly, i dont
> >> object to it, just saying i would be in favor of not having special
> >> cases if that is possible  
> >
> > So do you think this is tolerable in the current state or not?  
> 
> I agree with Michael, the number of VT-specific hacks seem to be
> piling up. Maybe it should be re-designed entirely instead of piling
> hacks on top of hacks to keep it barely working.

Would that mean reimplementing it as a full-stream decoder?

That's my favorite choice as well, but apparently the VT hwaccel
wouldn't require much more changes to work reasonably well.
Aman Gupta March 2, 2017, 5:56 p.m.
On Thu, Mar 2, 2017 at 2:04 AM, wm4 <nfxjfg@googlemail.com> wrote:

> On Thu, 2 Mar 2017 10:47:23 +0100
> Hendrik Leppkes <h.leppkes@gmail.com> wrote:
>
> > On Thu, Mar 2, 2017 at 10:35 AM, wm4 <nfxjfg@googlemail.com> wrote:
> > > On Fri, 17 Feb 2017 21:15:56 +0100
> > > Michael Niedermayer <michael@niedermayer.cc> wrote:
> > >
> > >> On Fri, Feb 17, 2017 at 07:02:10AM +0100, wm4 wrote:
> > >> > On Thu, 16 Feb 2017 10:29:36 -0800
> > >> > Aman Gupta <ffmpeg@tmm1.net> wrote:
> > >> >
> > >> > > From: Aman Gupta <aman@tmm1.net>
> > >> > >
> > >> > > This fixes playback of h264 streams with SPS changes. One such
> sample
> > >> > > is available at http://tmm1.s3.amazonaws.com/
> videotoolbox/spschange.ts.
> > >> > > It switches mid-stream from level 4.0 to level 3.2.
> > >> > >
> > >> > > Previously, playing this sample with the VT hwaccel on iOS would
> crash.
> > >> > > After this patch, it plays back as expected.
> > >> > >
> > >> > > On macOS however, feeding in new SPS into an existing
> decompression
> > >> > > session does not always work, so this patch is only a partial fix.
> > >> > > ---
> > >> > >  libavcodec/h264dec.c | 7 +++++++
> > >> > >  1 file changed, 7 insertions(+)
> > >> > >
> > >> > > diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
> > >> > > index 41c0964..e521c52 100644
> > >> > > --- a/libavcodec/h264dec.c
> > >> > > +++ b/libavcodec/h264dec.c
> > >> > > @@ -740,6 +740,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
> > >> > >              break;
> > >> > >          case H264_NAL_SPS: {
> > >> > >              GetBitContext tmp_gb = nal->gb;
> > >> > > +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt ==
> AV_PIX_FMT_VIDEOTOOLBOX) {
> > >> > > +                ret = avctx->hwaccel->decode_slice(avctx,
> > >> > > +                                                   nal->data,
> > >> > > +                                                   nal->size);
> > >> > > +                if (ret < 0)
> > >> > > +                    goto end;
> > >> > > +            }
> > >> > >              if (ff_h264_decode_seq_parameter_set(&tmp_gb,
> avctx, &h->ps, 0) >= 0)
> > >> > >                  break;
> > >> > >              av_log(h->avctx, AV_LOG_DEBUG,
> > >> >
> > >> > A bit ugly but ok IMHO. Maybe it would be better to add a new
> hwaccel
> > >> > callback here, even if it's used by VT only.
> > >> >
> > >> > You should probably wait for approval by michaelni.
> > >>
> > >> i dont really have an oppinion on hwaccel, thats not so much my
> > >> area
> > >> though i find special cases for specific hwaccel a bit ugly, i dont
> > >> object to it, just saying i would be in favor of not having special
> > >> cases if that is possible
> > >
> > > So do you think this is tolerable in the current state or not?
> >
> > I agree with Michael, the number of VT-specific hacks seem to be
> > piling up. Maybe it should be re-designed entirely instead of piling
> > hacks on top of hacks to keep it barely working.
>
> Would that mean reimplementing it as a full-stream decoder?
>
> That's my favorite choice as well, but apparently the VT hwaccel
> wouldn't require much more changes to work reasonably well.
>

FWIW, after these patches the VT hwaccel has been working very reliably for
me across hundreds of test streams.

I agree that re-implementing as a decoder is the correct long term
solution, but the VT api presents several technical hurdles before that can
become a viable alternative.

For this particular patch, I think it would make sense to add a new hwaccel
callback that's only used by VT. If we can agree on a name I can work up a
patch. Maybe hwaccel->upload_nal() ?

Aman


> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
Michael Niedermayer March 2, 2017, 10:49 p.m.
On Thu, Mar 02, 2017 at 10:35:21AM +0100, wm4 wrote:
> On Fri, 17 Feb 2017 21:15:56 +0100
> Michael Niedermayer <michael@niedermayer.cc> wrote:
> 
> > On Fri, Feb 17, 2017 at 07:02:10AM +0100, wm4 wrote:
> > > On Thu, 16 Feb 2017 10:29:36 -0800
> > > Aman Gupta <ffmpeg@tmm1.net> wrote:
> > >   
> > > > From: Aman Gupta <aman@tmm1.net>
> > > > 
> > > > This fixes playback of h264 streams with SPS changes. One such sample
> > > > is available at http://tmm1.s3.amazonaws.com/videotoolbox/spschange.ts.
> > > > It switches mid-stream from level 4.0 to level 3.2.
> > > > 
> > > > Previously, playing this sample with the VT hwaccel on iOS would crash.
> > > > After this patch, it plays back as expected.
> > > > 
> > > > On macOS however, feeding in new SPS into an existing decompression
> > > > session does not always work, so this patch is only a partial fix.
> > > > ---
> > > >  libavcodec/h264dec.c | 7 +++++++
> > > >  1 file changed, 7 insertions(+)
> > > > 
> > > > diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
> > > > index 41c0964..e521c52 100644
> > > > --- a/libavcodec/h264dec.c
> > > > +++ b/libavcodec/h264dec.c
> > > > @@ -740,6 +740,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
> > > >              break;
> > > >          case H264_NAL_SPS: {
> > > >              GetBitContext tmp_gb = nal->gb;
> > > > +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
> > > > +                ret = avctx->hwaccel->decode_slice(avctx,
> > > > +                                                   nal->data,
> > > > +                                                   nal->size);
> > > > +                if (ret < 0)
> > > > +                    goto end;
> > > > +            }
> > > >              if (ff_h264_decode_seq_parameter_set(&tmp_gb, avctx, &h->ps, 0) >= 0)
> > > >                  break;
> > > >              av_log(h->avctx, AV_LOG_DEBUG,  
> > > 
> > > A bit ugly but ok IMHO. Maybe it would be better to add a new hwaccel
> > > callback here, even if it's used by VT only.
> > > 
> > > You should probably wait for approval by michaelni.  
> > 
> > i dont really have an oppinion on hwaccel, thats not so much my
> > area
> > though i find special cases for specific hwaccel a bit ugly, i dont
> > object to it, just saying i would be in favor of not having special
> > cases if that is possible
> 
> So do you think this is tolerable in the current state or not?

i leave this entirely to the people working on hwaccel. i dont
object to it nor to a alternative

Patch hide | download patch | download mbox

diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
index 41c0964..e521c52 100644
--- a/libavcodec/h264dec.c
+++ b/libavcodec/h264dec.c
@@ -740,6 +740,13 @@  FF_ENABLE_DEPRECATION_WARNINGS
             break;
         case H264_NAL_SPS: {
             GetBitContext tmp_gb = nal->gb;
+            if (avctx->hwaccel && avctx->hwaccel->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
+                ret = avctx->hwaccel->decode_slice(avctx,
+                                                   nal->data,
+                                                   nal->size);
+                if (ret < 0)
+                    goto end;
+            }
             if (ff_h264_decode_seq_parameter_set(&tmp_gb, avctx, &h->ps, 0) >= 0)
                 break;
             av_log(h->avctx, AV_LOG_DEBUG,