mbox series

[FFmpeg-devel,0/6] Add support for utilizing av1c as extradata

Message ID 20201123191046.28887-1-jeebjp@gmail.com
Headers show
Series Add support for utilizing av1c as extradata
Related show

Message

Jan Ekström Nov. 23, 2020, 7:10 p.m. UTC
Until now an API user with a non-libavformat MP4 or Matroska demuxer
would have had to manually filter the extradata to skip the first four
bytes when passing data to libavcodec, so that only the OBUs would
remain. Now the AV1CodecConfigurationRecord structure can be passed
to libavcodec as-is. This is lets it to be dealt with in a similar
manner to AVC or HEVC.

This is done in the following steps:
1. A nice helper by Mark is added so that it can be utilized in stead
   of ff_cbs_read from both the decoder and parser after the CBS gains
   the capability to handle AV1CodecConfigurationRecord structures.
2. Support is added to AV1 CBS for handling AV1CodecConfigurationRecord
   in a similar vein to AVC/HEVC's avcc/hvcc handling.
3. AV1 decoder and parser are switched to utilize the newly added
   helper functionality.
4. Support is added for writing out the passed AV1CodecConfigurationRecord
   in ff_isom_write_av1c, which is called from the MP4 and Matroska
   writers.
5. Finally, both MP4 and Matroska demuxers are changed to expose the
   full AV1CodecConfigurationRecord as extradata. 

Jan Ekström (5):
  avcodec/cbs_av1: add support for standard MP4/Matroska extradata
  avcodec/av1{dec,parser}: move to ff_cbs_read_extradata_from_codec
  avformat/av1: add support for passing through MP4/Matroska av1c
  avformat/mov: remove special handling of av1c extradata
  avformat/matroskadec: remove special handling of av1c extradata

Mark Thompson (1):
  cbs: Add function to read extradata from an AVCodecContext

 libavcodec/av1_parser.c   |  2 +-
 libavcodec/av1dec.c       |  5 ++-
 libavcodec/cbs.c          | 77 ++++++++++++++++++++-------------------
 libavcodec/cbs.h          | 11 ++++++
 libavcodec/cbs_av1.c      | 34 +++++++++++++++++
 libavformat/av1.c         | 14 +++++++
 libavformat/matroskadec.c |  4 --
 libavformat/mov.c         | 30 +--------------
 8 files changed, 103 insertions(+), 74 deletions(-)

Comments

James Almer Nov. 23, 2020, 8:03 p.m. UTC | #1
On 11/23/2020 4:10 PM, Jan Ekström wrote:
> Until now an API user with a non-libavformat MP4 or Matroska demuxer
> would have had to manually filter the extradata to skip the first four
> bytes when passing data to libavcodec, so that only the OBUs would
> remain. Now the AV1CodecConfigurationRecord structure can be passed
> to libavcodec as-is. This is lets it to be dealt with in a similar
> manner to AVC or HEVC.
> 
> This is done in the following steps:
> 1. A nice helper by Mark is added so that it can be utilized in stead
>     of ff_cbs_read from both the decoder and parser after the CBS gains
>     the capability to handle AV1CodecConfigurationRecord structures.
> 2. Support is added to AV1 CBS for handling AV1CodecConfigurationRecord
>     in a similar vein to AVC/HEVC's avcc/hvcc handling.
> 3. AV1 decoder and parser are switched to utilize the newly added
>     helper functionality.
> 4. Support is added for writing out the passed AV1CodecConfigurationRecord
>     in ff_isom_write_av1c, which is called from the MP4 and Matroska
>     writers.
> 5. Finally, both MP4 and Matroska demuxers are changed to expose the
>     full AV1CodecConfigurationRecord as extradata.
> 
> Jan Ekström (5):
>    avcodec/cbs_av1: add support for standard MP4/Matroska extradata
>    avcodec/av1{dec,parser}: move to ff_cbs_read_extradata_from_codec
>    avformat/av1: add support for passing through MP4/Matroska av1c
>    avformat/mov: remove special handling of av1c extradata
>    avformat/matroskadec: remove special handling of av1c extradata
> 
> Mark Thompson (1):
>    cbs: Add function to read extradata from an AVCodecContext
> 
>   libavcodec/av1_parser.c   |  2 +-
>   libavcodec/av1dec.c       |  5 ++-
>   libavcodec/cbs.c          | 77 ++++++++++++++++++++-------------------
>   libavcodec/cbs.h          | 11 ++++++
>   libavcodec/cbs_av1.c      | 34 +++++++++++++++++
>   libavformat/av1.c         | 14 +++++++
>   libavformat/matroskadec.c |  4 --
>   libavformat/mov.c         | 30 +--------------
>   8 files changed, 103 insertions(+), 74 deletions(-)

Patches 1 to 4 LGTM. Patches 5 and 6 should IMO wait until after a bump 
since they change what the demuxers export as extradata, and existing 
compiled software linking to for ex 4.3 should not have to deal with 
this change in behavior if you use a lavf from an hypothetical pre-bump 
4.4 release at runtime.
Jan Ekström Nov. 24, 2020, 8:37 a.m. UTC | #2
On Mon, Nov 23, 2020 at 10:11 PM James Almer <jamrial@gmail.com> wrote:
>
> On 11/23/2020 4:10 PM, Jan Ekström wrote:
> > Until now an API user with a non-libavformat MP4 or Matroska demuxer
> > would have had to manually filter the extradata to skip the first four
> > bytes when passing data to libavcodec, so that only the OBUs would
> > remain. Now the AV1CodecConfigurationRecord structure can be passed
> > to libavcodec as-is. This is lets it to be dealt with in a similar
> > manner to AVC or HEVC.
> >
> > This is done in the following steps:
> > 1. A nice helper by Mark is added so that it can be utilized in stead
> >     of ff_cbs_read from both the decoder and parser after the CBS gains
> >     the capability to handle AV1CodecConfigurationRecord structures.
> > 2. Support is added to AV1 CBS for handling AV1CodecConfigurationRecord
> >     in a similar vein to AVC/HEVC's avcc/hvcc handling.
> > 3. AV1 decoder and parser are switched to utilize the newly added
> >     helper functionality.
> > 4. Support is added for writing out the passed AV1CodecConfigurationRecord
> >     in ff_isom_write_av1c, which is called from the MP4 and Matroska
> >     writers.
> > 5. Finally, both MP4 and Matroska demuxers are changed to expose the
> >     full AV1CodecConfigurationRecord as extradata.
> >
> > Jan Ekström (5):
> >    avcodec/cbs_av1: add support for standard MP4/Matroska extradata
> >    avcodec/av1{dec,parser}: move to ff_cbs_read_extradata_from_codec
> >    avformat/av1: add support for passing through MP4/Matroska av1c
> >    avformat/mov: remove special handling of av1c extradata
> >    avformat/matroskadec: remove special handling of av1c extradata
> >
> > Mark Thompson (1):
> >    cbs: Add function to read extradata from an AVCodecContext
> >
> >   libavcodec/av1_parser.c   |  2 +-
> >   libavcodec/av1dec.c       |  5 ++-
> >   libavcodec/cbs.c          | 77 ++++++++++++++++++++-------------------
> >   libavcodec/cbs.h          | 11 ++++++
> >   libavcodec/cbs_av1.c      | 34 +++++++++++++++++
> >   libavformat/av1.c         | 14 +++++++
> >   libavformat/matroskadec.c |  4 --
> >   libavformat/mov.c         | 30 +--------------
> >   8 files changed, 103 insertions(+), 74 deletions(-)
>
> Patches 1 to 4 LGTM. Patches 5 and 6 should IMO wait until after a bump
> since they change what the demuxers export as extradata, and existing
> compiled software linking to for ex 4.3 should not have to deal with
> this change in behavior if you use a lavf from an hypothetical pre-bump
> 4.4 release at runtime.

Thanks, applied 1-4 as:
7bacf60ae5df75954a538563d19f6001aa598b3f..4688017d9a1c2633c5888de5a0580d29e7680d54

I would prefer we switch over and simplify the demuxers sooner than
later, as I am not sure it was meant from our side that the interface
was defined as "only OBUs from this specific structure", while for
every other major format (AVC/HEVC) our demuxers output the whole
structure, and our decoders would accept that (even if it contained
some extraneous information such as the HEVC extradata).

But for now at least one side has been fixed.

Jan