[FFmpeg-devel,5/5] avformat/movenc: add support for AV1 streams

Submitted by James Almer on July 9, 2018, 6:26 p.m.

Details

Message ID 20180709182654.9996-6-jamrial@gmail.com
State New
Headers show

Commit Message

James Almer July 9, 2018, 6:26 p.m.
Signed-off-by: James Almer <jamrial@gmail.com>
---
ff_av1_filter_obus() could eventually be replaced by an autoinserted
filter_units bsf, assuming it doesn't slow down the muxing process
too much (CBS is fast reading packets, but not so much assembling and
writing packets).
ff_isom_write_av1c() however can't be replaced given filter_units
doesn't handle extradata (either codecpar or packet side data).

 libavformat/Makefile |   2 +-
 libavformat/av1.c    | 107 +++++++++++++++++++++++++++++++++++++++++++
 libavformat/av1.h    |  70 ++++++++++++++++++++++++++++
 libavformat/movenc.c |  36 ++++++++++++---
 4 files changed, 207 insertions(+), 8 deletions(-)
 create mode 100644 libavformat/av1.c
 create mode 100644 libavformat/av1.h

Comments

Michael Niedermayer July 9, 2018, 9:08 p.m.
On Mon, Jul 09, 2018 at 03:26:54PM -0300, James Almer wrote:
> Signed-off-by: James Almer <jamrial@gmail.com>
> ---
> ff_av1_filter_obus() could eventually be replaced by an autoinserted
> filter_units bsf, assuming it doesn't slow down the muxing process
> too much (CBS is fast reading packets, but not so much assembling and
> writing packets).
> ff_isom_write_av1c() however can't be replaced given filter_units
> doesn't handle extradata (either codecpar or packet side data).
> 
[...]
> diff --git a/libavformat/av1.h b/libavformat/av1.h
> new file mode 100644
> index 0000000000..733034c12d
> --- /dev/null
> +++ b/libavformat/av1.h
> @@ -0,0 +1,70 @@
> +/*
> + * AV1 helper functions for muxers
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#ifndef AVFORMAT_AV1_H
> +#define AVFORMAT_AV1_H
> +
> +#include <stdint.h>
> +
> +#include "avio.h"
> +
> +/**
> + * Filter out AV1 OBUs not meant to be present in ISOBMFF sample data and write
> + * the resulting bitstream to the provided AVIOContext.
> + *
> + * @param pb pointer to the AVIOContext where the filtered bitstream shall be
> + *           written
> + * @param buf input data buffer
> + * @param size size of the input data buffer
> + *
> + * @return the amount of bytes written in case of success, a negative AVERROR
> + *         code in case of failure
> + */
> +int ff_av1_filter_obus(AVIOContext *pb, const uint8_t *buf, int size);
> +
> +/**
> + * Filter out AV1 OBUs not meant to be present in ISOBMFF sample data and write
> + * the resulting bitstream to a newly allocated data buffer.
> + *
> + * @param pb pointer to the AVIOContext where the filtered bitstream shall be
> + *           written
> + * @param buf input data buffer

> + * @param out pointer to pointer that will hold the allocated data buffer
> + * @param size size of the input data buffer. The size of the resulting output
> +               data buffer will be written here
> + *
> + * @return the amount of bytes written in case of success, a negative AVERROR
> + *         code in case of failure

this leaves it unspecified what happens to out/size in case of errors
are they 0/null are they undefined, left as before ?


> + */
> +int ff_av1_filter_obus_buf(const uint8_t *buf, uint8_t **out, int *size);
> +
> +/**
> + * Writes AV1 extradata (Sequence Header and Metadata OBUs) to the provided
> + * AVIOContext.
> + *
> + * @param pb pointer to the AVIOContext where the hvcC shall be written
> + * @param buf input data buffer

> + * @param size size of the input data buffer

very minor nitpick but you could add "in bytes"


> + *
> + * @return 0 in case of success, a negative AVERROR code in case of failure

if >= 0 is defined as success then its possible to use this in the future
for some additional information without the need to review all callers

i guess most of this doesnt matter much as its not public API ...



[...]
James Almer July 9, 2018, 9:26 p.m.
On 7/9/2018 6:08 PM, Michael Niedermayer wrote:
> On Mon, Jul 09, 2018 at 03:26:54PM -0300, James Almer wrote:
>> Signed-off-by: James Almer <jamrial@gmail.com>
>> ---
>> ff_av1_filter_obus() could eventually be replaced by an autoinserted
>> filter_units bsf, assuming it doesn't slow down the muxing process
>> too much (CBS is fast reading packets, but not so much assembling and
>> writing packets).
>> ff_isom_write_av1c() however can't be replaced given filter_units
>> doesn't handle extradata (either codecpar or packet side data).
>>
> [...]
>> diff --git a/libavformat/av1.h b/libavformat/av1.h
>> new file mode 100644
>> index 0000000000..733034c12d
>> --- /dev/null
>> +++ b/libavformat/av1.h
>> @@ -0,0 +1,70 @@
>> +/*
>> + * AV1 helper functions for muxers
>> + *
>> + * This file is part of FFmpeg.
>> + *
>> + * FFmpeg is free software; you can redistribute it and/or
>> + * modify it under the terms of the GNU Lesser General Public
>> + * License as published by the Free Software Foundation; either
>> + * version 2.1 of the License, or (at your option) any later version.
>> + *
>> + * FFmpeg is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> + * Lesser General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU Lesser General Public
>> + * License along with FFmpeg; if not, write to the Free Software
>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
>> + */
>> +
>> +#ifndef AVFORMAT_AV1_H
>> +#define AVFORMAT_AV1_H
>> +
>> +#include <stdint.h>
>> +
>> +#include "avio.h"
>> +
>> +/**
>> + * Filter out AV1 OBUs not meant to be present in ISOBMFF sample data and write
>> + * the resulting bitstream to the provided AVIOContext.
>> + *
>> + * @param pb pointer to the AVIOContext where the filtered bitstream shall be
>> + *           written
>> + * @param buf input data buffer
>> + * @param size size of the input data buffer
>> + *
>> + * @return the amount of bytes written in case of success, a negative AVERROR
>> + *         code in case of failure
>> + */
>> +int ff_av1_filter_obus(AVIOContext *pb, const uint8_t *buf, int size);
>> +
>> +/**
>> + * Filter out AV1 OBUs not meant to be present in ISOBMFF sample data and write
>> + * the resulting bitstream to a newly allocated data buffer.
>> + *
>> + * @param pb pointer to the AVIOContext where the filtered bitstream shall be
>> + *           written
>> + * @param buf input data buffer
> 
>> + * @param out pointer to pointer that will hold the allocated data buffer
>> + * @param size size of the input data buffer. The size of the resulting output
>> +               data buffer will be written here
>> + *
>> + * @return the amount of bytes written in case of success, a negative AVERROR
>> + *         code in case of failure
> 
> this leaves it unspecified what happens to out/size in case of errors
> are they 0/null are they undefined, left as before ?

Depending on when in the function an error happens, i will either be
untouched or *out will be set to NULL.
I can make it so the latter always happens in case of failure and
document it as such if that's preferred, but the current callers pass a
pointer to a pointer set to NULL, so that's already guaranteed.

This is based on ff_avc_parse_nal_units_buf() and
ff_hevc_annexb2mp4_buf(), for that matter, so those could also be fixed.

> 
> 
>> + */
>> +int ff_av1_filter_obus_buf(const uint8_t *buf, uint8_t **out, int *size);
>> +
>> +/**
>> + * Writes AV1 extradata (Sequence Header and Metadata OBUs) to the provided
>> + * AVIOContext.
>> + *
>> + * @param pb pointer to the AVIOContext where the hvcC shall be written
>> + * @param buf input data buffer
> 
>> + * @param size size of the input data buffer
> 
> very minor nitpick but you could add "in bytes"

Sure.

> 
> 
>> + *
>> + * @return 0 in case of success, a negative AVERROR code in case of failure
> 
> if >= 0 is defined as success then its possible to use this in the future
> for some additional information without the need to review all callers

I can make this change, but as you well mention below it doesn't really
matter since it's internal, so anything defined now can be changed later.

> 
> i guess most of this doesnt matter much as its not public API ...
> 
> 
> 
> [...]
> 
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
Baptiste Coudurier July 9, 2018, 10:23 p.m.
Hi James,

On Mon, Jul 9, 2018 at 11:26 AM, James Almer <jamrial@gmail.com> wrote:

[...]


> @@ -5438,7 +5461,7 @@ int ff_mov_write_packet(AVFormatContext *s,
> AVPacket *pkt)
>          av_log(s, AV_LOG_WARNING, "pts has no value\n");
>          pkt->pts = pkt->dts;
>      }
> -    if (pkt->dts != pkt->pts)
> +    if (pkt->dts != pkt->pts && par->codec_id != AV_CODEC_ID_AV1)
>          trk->flags |= MOV_TRACK_CTTS;
>      trk->cluster[trk->entry].cts   = pkt->pts - pkt->dts;
>      trk->cluster[trk->entry].flags = 0;
>

[...]

Interesting, is there a spec document that would explicitly mention that
AV1 forbids "ctts" atom ?

Thanks!
Hendrik Leppkes July 9, 2018, 10:35 p.m.
On Tue, Jul 10, 2018 at 12:29 AM Baptiste Coudurier
<baptiste.coudurier@gmail.com> wrote:
>
> Hi James,
>
> On Mon, Jul 9, 2018 at 11:26 AM, James Almer <jamrial@gmail.com> wrote:
>
> [...]
>
>
> > @@ -5438,7 +5461,7 @@ int ff_mov_write_packet(AVFormatContext *s,
> > AVPacket *pkt)
> >          av_log(s, AV_LOG_WARNING, "pts has no value\n");
> >          pkt->pts = pkt->dts;
> >      }
> > -    if (pkt->dts != pkt->pts)
> > +    if (pkt->dts != pkt->pts && par->codec_id != AV_CODEC_ID_AV1)
> >          trk->flags |= MOV_TRACK_CTTS;
> >      trk->cluster[trk->entry].cts   = pkt->pts - pkt->dts;
> >      trk->cluster[trk->entry].flags = 0;
> >
>
> [...]
>
> Interesting, is there a spec document that would explicitly mention that
> AV1 forbids "ctts" atom ?
>

The draft spec does:
https://aomediacodec.github.io/av1-isobmff/#sampleformat

Unlike many video standards, AV1 does not distinguish the display
order from the decoding order, but achieves similar effects by
grouping multiple frames within a sample. Therefore, composition
offsets are not used. In tracks using the AV1SampleEntry, the ctts box
and composition offsets in movie fragments SHALL NOT be used.

- Hendrik
James Almer July 9, 2018, 10:35 p.m.
On 7/9/2018 7:23 PM, Baptiste Coudurier wrote:
> Hi James,
> 
> On Mon, Jul 9, 2018 at 11:26 AM, James Almer <jamrial@gmail.com> wrote:
> 
> [...]
> 
> 
>> @@ -5438,7 +5461,7 @@ int ff_mov_write_packet(AVFormatContext *s,
>> AVPacket *pkt)
>>          av_log(s, AV_LOG_WARNING, "pts has no value\n");
>>          pkt->pts = pkt->dts;
>>      }
>> -    if (pkt->dts != pkt->pts)
>> +    if (pkt->dts != pkt->pts && par->codec_id != AV_CODEC_ID_AV1)
>>          trk->flags |= MOV_TRACK_CTTS;
>>      trk->cluster[trk->entry].cts   = pkt->pts - pkt->dts;
>>      trk->cluster[trk->entry].flags = 0;
>>
> 
> [...]
> 
> Interesting, is there a spec document that would explicitly mention that
> AV1 forbids "ctts" atom ?
> 
> Thanks!


Yes, https://aomediacodec.github.io/av1-isobmff/#sampleformat

"Unlike many video standards, AV1 does not distinguish the display order
from the decoding order, but achieves similar effects by grouping
multiple frames within a sample. Therefore, composition offsets are not
used. In tracks using the AV1SampleEntry, the ctts box and composition
offsets in movie fragments SHALL NOT be used. Similarly, the is_leading
flag, if used, SHALL be set to 0 or 2."
Michael Niedermayer July 12, 2018, 11:26 a.m.
On Mon, Jul 09, 2018 at 03:26:54PM -0300, James Almer wrote:
> Signed-off-by: James Almer <jamrial@gmail.com>
> ---
> ff_av1_filter_obus() could eventually be replaced by an autoinserted
> filter_units bsf, assuming it doesn't slow down the muxing process
> too much (CBS is fast reading packets, but not so much assembling and
> writing packets).
> ff_isom_write_av1c() however can't be replaced given filter_units
> doesn't handle extradata (either codecpar or packet side data).
[...]
> +#endif /* AVFORMAT_AV1_H */
> diff --git a/libavformat/movenc.c b/libavformat/movenc.c
> index fe0a244a8f..784df6d08d 100644
> --- a/libavformat/movenc.c
> +++ b/libavformat/movenc.c
> @@ -30,6 +30,7 @@
>  #include "riff.h"
>  #include "avio.h"
>  #include "isom.h"
> +#include "av1.h"
>  #include "avc.h"
>  #include "libavcodec/ac3_parser_internal.h"
>  #include "libavcodec/dnxhddata.h"
> @@ -1163,6 +1164,19 @@ static int mov_write_d263_tag(AVIOContext *pb)
>      return 0xf;
>  }
>  
> +static int mov_write_av1c_tag(AVIOContext *pb, MOVTrack *track)
> +{
> +    int64_t pos = avio_tell(pb);
> +
> +    avio_wb32(pb, 0);
> +    ffio_wfourcc(pb, "av1C");
> +    avio_w8(pb, 0); /* version */
> +    avio_wb24(pb, 0); /* flags */
> +    avio_w8(pb, 0); /* reserved (3), initial_presentation_delay_present (1), initial_presentation_delay_minus_one/reserved (4) */
> +    ff_isom_write_av1c(pb, track->vos_data, track->vos_len);
> +    return update_size(pb, pos);
> +}
> +
>  static int mov_write_avcc_tag(AVIOContext *pb, MOVTrack *track)
>  {
>      int64_t pos = avio_tell(pb);
> @@ -2009,6 +2023,8 @@ static int mov_write_video_tag(AVIOContext *pb, MOVMuxContext *mov, MOVTrack *tr
>              mov_write_uuid_tag_ipod(pb);
>      } else if (track->par->codec_id == AV_CODEC_ID_VP9) {
>          mov_write_vpcc_tag(mov->fc, pb, track);
> +    } else if (track->par->codec_id == AV_CODEC_ID_AV1) {
> +        mov_write_av1c_tag(pb, track);
>      } else if (track->par->codec_id == AV_CODEC_ID_VC1 && track->vos_len > 0)
>          mov_write_dvc1_tag(pb, track);
>      else if (track->par->codec_id == AV_CODEC_ID_VP6F ||
> @@ -5319,6 +5335,13 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt)
>          } else {
>              size = ff_hevc_annexb2mp4(pb, pkt->data, pkt->size, 0, NULL);
>          }
> +    } else if (par->codec_id == AV_CODEC_ID_AV1) {
> +        if (trk->hint_track >= 0 && trk->hint_track < mov->nb_streams) {
> +            ff_av1_filter_obus_buf(pkt->data, &reformatted_data, &size);
> +            avio_write(pb, reformatted_data, size);
> +        } else {
> +            size = ff_av1_filter_obus(pb, pkt->data, pkt->size);
> +        }
>  #if CONFIG_AC3_PARSER
>      } else if (par->codec_id == AV_CODEC_ID_EAC3) {
>          size = handle_eac3(mov, pkt, trk);

> @@ -5438,7 +5461,7 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt)
>          av_log(s, AV_LOG_WARNING, "pts has no value\n");
>          pkt->pts = pkt->dts;
>      }
> -    if (pkt->dts != pkt->pts)
> +    if (pkt->dts != pkt->pts && par->codec_id != AV_CODEC_ID_AV1)

When is dts != pts && par->codec_id == AV_CODEC_ID_AV1 ?

Iam asking because if it never is then this check is not needed
and if it is sometimes true then i suspect the pts values will be lost
that is the demuxer input would differ from the muxer output, which doesnt
seem right
Maybe iam missing something

[...]
James Almer July 12, 2018, 1:59 p.m.
On 7/12/2018 8:26 AM, Michael Niedermayer wrote:
> On Mon, Jul 09, 2018 at 03:26:54PM -0300, James Almer wrote:
>> Signed-off-by: James Almer <jamrial@gmail.com>
>> ---
>> ff_av1_filter_obus() could eventually be replaced by an autoinserted
>> filter_units bsf, assuming it doesn't slow down the muxing process
>> too much (CBS is fast reading packets, but not so much assembling and
>> writing packets).
>> ff_isom_write_av1c() however can't be replaced given filter_units
>> doesn't handle extradata (either codecpar or packet side data).
> [...]
>> +#endif /* AVFORMAT_AV1_H */
>> diff --git a/libavformat/movenc.c b/libavformat/movenc.c
>> index fe0a244a8f..784df6d08d 100644
>> --- a/libavformat/movenc.c
>> +++ b/libavformat/movenc.c
>> @@ -30,6 +30,7 @@
>>  #include "riff.h"
>>  #include "avio.h"
>>  #include "isom.h"
>> +#include "av1.h"
>>  #include "avc.h"
>>  #include "libavcodec/ac3_parser_internal.h"
>>  #include "libavcodec/dnxhddata.h"
>> @@ -1163,6 +1164,19 @@ static int mov_write_d263_tag(AVIOContext *pb)
>>      return 0xf;
>>  }
>>  
>> +static int mov_write_av1c_tag(AVIOContext *pb, MOVTrack *track)
>> +{
>> +    int64_t pos = avio_tell(pb);
>> +
>> +    avio_wb32(pb, 0);
>> +    ffio_wfourcc(pb, "av1C");
>> +    avio_w8(pb, 0); /* version */
>> +    avio_wb24(pb, 0); /* flags */
>> +    avio_w8(pb, 0); /* reserved (3), initial_presentation_delay_present (1), initial_presentation_delay_minus_one/reserved (4) */
>> +    ff_isom_write_av1c(pb, track->vos_data, track->vos_len);
>> +    return update_size(pb, pos);
>> +}
>> +
>>  static int mov_write_avcc_tag(AVIOContext *pb, MOVTrack *track)
>>  {
>>      int64_t pos = avio_tell(pb);
>> @@ -2009,6 +2023,8 @@ static int mov_write_video_tag(AVIOContext *pb, MOVMuxContext *mov, MOVTrack *tr
>>              mov_write_uuid_tag_ipod(pb);
>>      } else if (track->par->codec_id == AV_CODEC_ID_VP9) {
>>          mov_write_vpcc_tag(mov->fc, pb, track);
>> +    } else if (track->par->codec_id == AV_CODEC_ID_AV1) {
>> +        mov_write_av1c_tag(pb, track);
>>      } else if (track->par->codec_id == AV_CODEC_ID_VC1 && track->vos_len > 0)
>>          mov_write_dvc1_tag(pb, track);
>>      else if (track->par->codec_id == AV_CODEC_ID_VP6F ||
>> @@ -5319,6 +5335,13 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt)
>>          } else {
>>              size = ff_hevc_annexb2mp4(pb, pkt->data, pkt->size, 0, NULL);
>>          }
>> +    } else if (par->codec_id == AV_CODEC_ID_AV1) {
>> +        if (trk->hint_track >= 0 && trk->hint_track < mov->nb_streams) {
>> +            ff_av1_filter_obus_buf(pkt->data, &reformatted_data, &size);
>> +            avio_write(pb, reformatted_data, size);
>> +        } else {
>> +            size = ff_av1_filter_obus(pb, pkt->data, pkt->size);
>> +        }
>>  #if CONFIG_AC3_PARSER
>>      } else if (par->codec_id == AV_CODEC_ID_EAC3) {
>>          size = handle_eac3(mov, pkt, trk);
> 
>> @@ -5438,7 +5461,7 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt)
>>          av_log(s, AV_LOG_WARNING, "pts has no value\n");
>>          pkt->pts = pkt->dts;
>>      }
>> -    if (pkt->dts != pkt->pts)
>> +    if (pkt->dts != pkt->pts && par->codec_id != AV_CODEC_ID_AV1)
> 
> When is dts != pts && par->codec_id == AV_CODEC_ID_AV1 ?
> 
> Iam asking because if it never is then this check is not needed
> and if it is sometimes true then i suspect the pts values will be lost
> that is the demuxer input would differ from the muxer output, which doesnt
> seem right
> Maybe iam missing something

It should never be. I just added this check for extra paranoid
precaution since the spec forbids the ctts box. I can remove it.
Michael Niedermayer July 13, 2018, 12:29 a.m.
On Thu, Jul 12, 2018 at 10:59:44AM -0300, James Almer wrote:
> On 7/12/2018 8:26 AM, Michael Niedermayer wrote:
> > On Mon, Jul 09, 2018 at 03:26:54PM -0300, James Almer wrote:
> >> Signed-off-by: James Almer <jamrial@gmail.com>
> >> ---
> >> ff_av1_filter_obus() could eventually be replaced by an autoinserted
> >> filter_units bsf, assuming it doesn't slow down the muxing process
> >> too much (CBS is fast reading packets, but not so much assembling and
> >> writing packets).
> >> ff_isom_write_av1c() however can't be replaced given filter_units
> >> doesn't handle extradata (either codecpar or packet side data).
> > [...]
> >> +#endif /* AVFORMAT_AV1_H */
> >> diff --git a/libavformat/movenc.c b/libavformat/movenc.c
> >> index fe0a244a8f..784df6d08d 100644
> >> --- a/libavformat/movenc.c
> >> +++ b/libavformat/movenc.c
> >> @@ -30,6 +30,7 @@
> >>  #include "riff.h"
> >>  #include "avio.h"
> >>  #include "isom.h"
> >> +#include "av1.h"
> >>  #include "avc.h"
> >>  #include "libavcodec/ac3_parser_internal.h"
> >>  #include "libavcodec/dnxhddata.h"
> >> @@ -1163,6 +1164,19 @@ static int mov_write_d263_tag(AVIOContext *pb)
> >>      return 0xf;
> >>  }
> >>  
> >> +static int mov_write_av1c_tag(AVIOContext *pb, MOVTrack *track)
> >> +{
> >> +    int64_t pos = avio_tell(pb);
> >> +
> >> +    avio_wb32(pb, 0);
> >> +    ffio_wfourcc(pb, "av1C");
> >> +    avio_w8(pb, 0); /* version */
> >> +    avio_wb24(pb, 0); /* flags */
> >> +    avio_w8(pb, 0); /* reserved (3), initial_presentation_delay_present (1), initial_presentation_delay_minus_one/reserved (4) */
> >> +    ff_isom_write_av1c(pb, track->vos_data, track->vos_len);
> >> +    return update_size(pb, pos);
> >> +}
> >> +
> >>  static int mov_write_avcc_tag(AVIOContext *pb, MOVTrack *track)
> >>  {
> >>      int64_t pos = avio_tell(pb);
> >> @@ -2009,6 +2023,8 @@ static int mov_write_video_tag(AVIOContext *pb, MOVMuxContext *mov, MOVTrack *tr
> >>              mov_write_uuid_tag_ipod(pb);
> >>      } else if (track->par->codec_id == AV_CODEC_ID_VP9) {
> >>          mov_write_vpcc_tag(mov->fc, pb, track);
> >> +    } else if (track->par->codec_id == AV_CODEC_ID_AV1) {
> >> +        mov_write_av1c_tag(pb, track);
> >>      } else if (track->par->codec_id == AV_CODEC_ID_VC1 && track->vos_len > 0)
> >>          mov_write_dvc1_tag(pb, track);
> >>      else if (track->par->codec_id == AV_CODEC_ID_VP6F ||
> >> @@ -5319,6 +5335,13 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt)
> >>          } else {
> >>              size = ff_hevc_annexb2mp4(pb, pkt->data, pkt->size, 0, NULL);
> >>          }
> >> +    } else if (par->codec_id == AV_CODEC_ID_AV1) {
> >> +        if (trk->hint_track >= 0 && trk->hint_track < mov->nb_streams) {
> >> +            ff_av1_filter_obus_buf(pkt->data, &reformatted_data, &size);
> >> +            avio_write(pb, reformatted_data, size);
> >> +        } else {
> >> +            size = ff_av1_filter_obus(pb, pkt->data, pkt->size);
> >> +        }
> >>  #if CONFIG_AC3_PARSER
> >>      } else if (par->codec_id == AV_CODEC_ID_EAC3) {
> >>          size = handle_eac3(mov, pkt, trk);
> > 
> >> @@ -5438,7 +5461,7 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt)
> >>          av_log(s, AV_LOG_WARNING, "pts has no value\n");
> >>          pkt->pts = pkt->dts;
> >>      }
> >> -    if (pkt->dts != pkt->pts)
> >> +    if (pkt->dts != pkt->pts && par->codec_id != AV_CODEC_ID_AV1)
> > 
> > When is dts != pts && par->codec_id == AV_CODEC_ID_AV1 ?
> > 
> > Iam asking because if it never is then this check is not needed
> > and if it is sometimes true then i suspect the pts values will be lost
> > that is the demuxer input would differ from the muxer output, which doesnt
> > seem right
> > Maybe iam missing something
> 
> It should never be. I just added this check for extra paranoid
> precaution since the spec forbids the ctts box. I can remove it.

I dont think this is the ideal behavior then

some random thoughts:

For example if pts!= dts, ommiting the ctts complies to that one rule
but why did they mismatch and is teh result actually a well playable
file.
The user also wont know about any of this happening as there will be
no warning i think. 
And if pts != dts, which should be use dts or pts or neither ?
also iam not sure if the muxer is the best place for this.





> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
James Almer July 13, 2018, 12:44 a.m.
On 7/12/2018 9:29 PM, Michael Niedermayer wrote:
> On Thu, Jul 12, 2018 at 10:59:44AM -0300, James Almer wrote:
>> On 7/12/2018 8:26 AM, Michael Niedermayer wrote:
>>> On Mon, Jul 09, 2018 at 03:26:54PM -0300, James Almer wrote:
>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>> ---
>>>> ff_av1_filter_obus() could eventually be replaced by an autoinserted
>>>> filter_units bsf, assuming it doesn't slow down the muxing process
>>>> too much (CBS is fast reading packets, but not so much assembling and
>>>> writing packets).
>>>> ff_isom_write_av1c() however can't be replaced given filter_units
>>>> doesn't handle extradata (either codecpar or packet side data).
>>> [...]
>>>> +#endif /* AVFORMAT_AV1_H */
>>>> diff --git a/libavformat/movenc.c b/libavformat/movenc.c
>>>> index fe0a244a8f..784df6d08d 100644
>>>> --- a/libavformat/movenc.c
>>>> +++ b/libavformat/movenc.c
>>>> @@ -30,6 +30,7 @@
>>>>  #include "riff.h"
>>>>  #include "avio.h"
>>>>  #include "isom.h"
>>>> +#include "av1.h"
>>>>  #include "avc.h"
>>>>  #include "libavcodec/ac3_parser_internal.h"
>>>>  #include "libavcodec/dnxhddata.h"
>>>> @@ -1163,6 +1164,19 @@ static int mov_write_d263_tag(AVIOContext *pb)
>>>>      return 0xf;
>>>>  }
>>>>  
>>>> +static int mov_write_av1c_tag(AVIOContext *pb, MOVTrack *track)
>>>> +{
>>>> +    int64_t pos = avio_tell(pb);
>>>> +
>>>> +    avio_wb32(pb, 0);
>>>> +    ffio_wfourcc(pb, "av1C");
>>>> +    avio_w8(pb, 0); /* version */
>>>> +    avio_wb24(pb, 0); /* flags */
>>>> +    avio_w8(pb, 0); /* reserved (3), initial_presentation_delay_present (1), initial_presentation_delay_minus_one/reserved (4) */
>>>> +    ff_isom_write_av1c(pb, track->vos_data, track->vos_len);
>>>> +    return update_size(pb, pos);
>>>> +}
>>>> +
>>>>  static int mov_write_avcc_tag(AVIOContext *pb, MOVTrack *track)
>>>>  {
>>>>      int64_t pos = avio_tell(pb);
>>>> @@ -2009,6 +2023,8 @@ static int mov_write_video_tag(AVIOContext *pb, MOVMuxContext *mov, MOVTrack *tr
>>>>              mov_write_uuid_tag_ipod(pb);
>>>>      } else if (track->par->codec_id == AV_CODEC_ID_VP9) {
>>>>          mov_write_vpcc_tag(mov->fc, pb, track);
>>>> +    } else if (track->par->codec_id == AV_CODEC_ID_AV1) {
>>>> +        mov_write_av1c_tag(pb, track);
>>>>      } else if (track->par->codec_id == AV_CODEC_ID_VC1 && track->vos_len > 0)
>>>>          mov_write_dvc1_tag(pb, track);
>>>>      else if (track->par->codec_id == AV_CODEC_ID_VP6F ||
>>>> @@ -5319,6 +5335,13 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt)
>>>>          } else {
>>>>              size = ff_hevc_annexb2mp4(pb, pkt->data, pkt->size, 0, NULL);
>>>>          }
>>>> +    } else if (par->codec_id == AV_CODEC_ID_AV1) {
>>>> +        if (trk->hint_track >= 0 && trk->hint_track < mov->nb_streams) {
>>>> +            ff_av1_filter_obus_buf(pkt->data, &reformatted_data, &size);
>>>> +            avio_write(pb, reformatted_data, size);
>>>> +        } else {
>>>> +            size = ff_av1_filter_obus(pb, pkt->data, pkt->size);
>>>> +        }
>>>>  #if CONFIG_AC3_PARSER
>>>>      } else if (par->codec_id == AV_CODEC_ID_EAC3) {
>>>>          size = handle_eac3(mov, pkt, trk);
>>>
>>>> @@ -5438,7 +5461,7 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt)
>>>>          av_log(s, AV_LOG_WARNING, "pts has no value\n");
>>>>          pkt->pts = pkt->dts;
>>>>      }
>>>> -    if (pkt->dts != pkt->pts)
>>>> +    if (pkt->dts != pkt->pts && par->codec_id != AV_CODEC_ID_AV1)
>>>
>>> When is dts != pts && par->codec_id == AV_CODEC_ID_AV1 ?
>>>
>>> Iam asking because if it never is then this check is not needed
>>> and if it is sometimes true then i suspect the pts values will be lost
>>> that is the demuxer input would differ from the muxer output, which doesnt
>>> seem right
>>> Maybe iam missing something
>>
>> It should never be. I just added this check for extra paranoid
>> precaution since the spec forbids the ctts box. I can remove it.
> 
> I dont think this is the ideal behavior then
> 
> some random thoughts:
> 
> For example if pts!= dts, ommiting the ctts complies to that one rule
> but why did they mismatch and is teh result actually a well playable
> file.
> The user also wont know about any of this happening as there will be
> no warning i think. 
> And if pts != dts, which should be use dts or pts or neither ?
> also iam not sure if the muxer is the best place for this.

A file reporting such timestamps would be invalid and something that
should be handled at the demuxer level, yes. This check can be removed
as it was a bogus addition to blindly enforce a spec rule.

Patch hide | download patch | download mbox

diff --git a/libavformat/Makefile b/libavformat/Makefile
index 8fb075f06f..f2f3aabdc2 100644
--- a/libavformat/Makefile
+++ b/libavformat/Makefile
@@ -302,7 +302,7 @@  OBJS-$(CONFIG_MM_DEMUXER)                += mm.o
 OBJS-$(CONFIG_MMF_DEMUXER)               += mmf.o
 OBJS-$(CONFIG_MMF_MUXER)                 += mmf.o rawenc.o
 OBJS-$(CONFIG_MOV_DEMUXER)               += mov.o mov_chan.o mov_esds.o replaygain.o
-OBJS-$(CONFIG_MOV_MUXER)                 += movenc.o avc.o hevc.o vpcc.o \
+OBJS-$(CONFIG_MOV_MUXER)                 += movenc.o av1.o avc.o hevc.o vpcc.o \
                                             movenchint.o mov_chan.o rtp.o \
                                             movenccenc.o rawutils.o
 OBJS-$(CONFIG_MP2_MUXER)                 += rawenc.o
diff --git a/libavformat/av1.c b/libavformat/av1.c
new file mode 100644
index 0000000000..3b200176a5
--- /dev/null
+++ b/libavformat/av1.c
@@ -0,0 +1,107 @@ 
+/*
+ * AV1 helper functions for muxers
+ * Copyright (c) 2018 James Almer <jamrial@gmail.com>
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include "libavutil/mem.h"
+#include "libavcodec/av1.h"
+#include "libavcodec/av1_parse.h"
+#include "av1.h"
+#include "avio.h"
+
+int ff_av1_filter_obus(AVIOContext *pb, const uint8_t *buf, int size)
+{
+    const uint8_t *end = buf + size;
+    int64_t obu_size;
+    int start_pos, type, temporal_id, spatial_id;
+
+    size = 0;
+    while (buf < end) {
+        int ret = parse_obu_header(buf, end - buf, &obu_size, &start_pos,
+                                   &type, &temporal_id, &spatial_id);
+        if (ret < 0)
+            return ret;
+
+        obu_size += start_pos;
+        if (obu_size > INT_MAX)
+            return AVERROR_INVALIDDATA;
+
+        switch (type) {
+        case AV1_OBU_TEMPORAL_DELIMITER:
+        case AV1_OBU_REDUNDANT_FRAME_HEADER:
+        case AV1_OBU_PADDING:
+            break;
+        default:
+            avio_write(pb, buf, obu_size);
+            size += obu_size;
+            break;
+        }
+        buf += obu_size;
+    }
+
+    return size;
+}
+
+int ff_av1_filter_obus_buf(const uint8_t *buf, uint8_t **out, int *size)
+{
+    AVIOContext *pb;
+    int ret;
+
+    ret = avio_open_dyn_buf(&pb);
+    if (ret < 0)
+        return ret;
+
+    ret = ff_av1_filter_obus(pb, buf, *size);
+    *size = avio_close_dyn_buf(pb, out);
+
+    if (ret < 0)
+        av_freep(out);
+
+    return ret;
+}
+
+int ff_isom_write_av1c(AVIOContext *pb, const uint8_t *buf, int size)
+{
+    int64_t obu_size;
+    int start_pos, type, temporal_id, spatial_id;
+
+    while (size > 0) {
+        int ret = parse_obu_header(buf, size, &obu_size, &start_pos,
+                                   &type, &temporal_id, &spatial_id);
+        if (ret < 0)
+            return ret;
+
+        obu_size += start_pos;
+        if (obu_size > INT_MAX)
+            return AVERROR_INVALIDDATA;
+
+        switch (type) {
+        case AV1_OBU_SEQUENCE_HEADER:
+        case AV1_OBU_METADATA:
+            avio_write(pb, buf, obu_size);
+            size -= obu_size;
+            break;
+        default:
+            break;
+        }
+        buf += obu_size;
+    }
+
+    return 0;
+}
diff --git a/libavformat/av1.h b/libavformat/av1.h
new file mode 100644
index 0000000000..733034c12d
--- /dev/null
+++ b/libavformat/av1.h
@@ -0,0 +1,70 @@ 
+/*
+ * AV1 helper functions for muxers
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef AVFORMAT_AV1_H
+#define AVFORMAT_AV1_H
+
+#include <stdint.h>
+
+#include "avio.h"
+
+/**
+ * Filter out AV1 OBUs not meant to be present in ISOBMFF sample data and write
+ * the resulting bitstream to the provided AVIOContext.
+ *
+ * @param pb pointer to the AVIOContext where the filtered bitstream shall be
+ *           written
+ * @param buf input data buffer
+ * @param size size of the input data buffer
+ *
+ * @return the amount of bytes written in case of success, a negative AVERROR
+ *         code in case of failure
+ */
+int ff_av1_filter_obus(AVIOContext *pb, const uint8_t *buf, int size);
+
+/**
+ * Filter out AV1 OBUs not meant to be present in ISOBMFF sample data and write
+ * the resulting bitstream to a newly allocated data buffer.
+ *
+ * @param pb pointer to the AVIOContext where the filtered bitstream shall be
+ *           written
+ * @param buf input data buffer
+ * @param out pointer to pointer that will hold the allocated data buffer
+ * @param size size of the input data buffer. The size of the resulting output
+               data buffer will be written here
+ *
+ * @return the amount of bytes written in case of success, a negative AVERROR
+ *         code in case of failure
+ */
+int ff_av1_filter_obus_buf(const uint8_t *buf, uint8_t **out, int *size);
+
+/**
+ * Writes AV1 extradata (Sequence Header and Metadata OBUs) to the provided
+ * AVIOContext.
+ *
+ * @param pb pointer to the AVIOContext where the hvcC shall be written
+ * @param buf input data buffer
+ * @param size size of the input data buffer
+ *
+ * @return 0 in case of success, a negative AVERROR code in case of failure
+ */
+int ff_isom_write_av1c(AVIOContext *pb, const uint8_t *buf, int size);
+
+#endif /* AVFORMAT_AV1_H */
diff --git a/libavformat/movenc.c b/libavformat/movenc.c
index fe0a244a8f..784df6d08d 100644
--- a/libavformat/movenc.c
+++ b/libavformat/movenc.c
@@ -30,6 +30,7 @@ 
 #include "riff.h"
 #include "avio.h"
 #include "isom.h"
+#include "av1.h"
 #include "avc.h"
 #include "libavcodec/ac3_parser_internal.h"
 #include "libavcodec/dnxhddata.h"
@@ -1163,6 +1164,19 @@  static int mov_write_d263_tag(AVIOContext *pb)
     return 0xf;
 }
 
+static int mov_write_av1c_tag(AVIOContext *pb, MOVTrack *track)
+{
+    int64_t pos = avio_tell(pb);
+
+    avio_wb32(pb, 0);
+    ffio_wfourcc(pb, "av1C");
+    avio_w8(pb, 0); /* version */
+    avio_wb24(pb, 0); /* flags */
+    avio_w8(pb, 0); /* reserved (3), initial_presentation_delay_present (1), initial_presentation_delay_minus_one/reserved (4) */
+    ff_isom_write_av1c(pb, track->vos_data, track->vos_len);
+    return update_size(pb, pos);
+}
+
 static int mov_write_avcc_tag(AVIOContext *pb, MOVTrack *track)
 {
     int64_t pos = avio_tell(pb);
@@ -2009,6 +2023,8 @@  static int mov_write_video_tag(AVIOContext *pb, MOVMuxContext *mov, MOVTrack *tr
             mov_write_uuid_tag_ipod(pb);
     } else if (track->par->codec_id == AV_CODEC_ID_VP9) {
         mov_write_vpcc_tag(mov->fc, pb, track);
+    } else if (track->par->codec_id == AV_CODEC_ID_AV1) {
+        mov_write_av1c_tag(pb, track);
     } else if (track->par->codec_id == AV_CODEC_ID_VC1 && track->vos_len > 0)
         mov_write_dvc1_tag(pb, track);
     else if (track->par->codec_id == AV_CODEC_ID_VP6F ||
@@ -5319,6 +5335,13 @@  int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt)
         } else {
             size = ff_hevc_annexb2mp4(pb, pkt->data, pkt->size, 0, NULL);
         }
+    } else if (par->codec_id == AV_CODEC_ID_AV1) {
+        if (trk->hint_track >= 0 && trk->hint_track < mov->nb_streams) {
+            ff_av1_filter_obus_buf(pkt->data, &reformatted_data, &size);
+            avio_write(pb, reformatted_data, size);
+        } else {
+            size = ff_av1_filter_obus(pb, pkt->data, pkt->size);
+        }
 #if CONFIG_AC3_PARSER
     } else if (par->codec_id == AV_CODEC_ID_EAC3) {
         size = handle_eac3(mov, pkt, trk);
@@ -5438,7 +5461,7 @@  int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt)
         av_log(s, AV_LOG_WARNING, "pts has no value\n");
         pkt->pts = pkt->dts;
     }
-    if (pkt->dts != pkt->pts)
+    if (pkt->dts != pkt->pts && par->codec_id != AV_CODEC_ID_AV1)
         trk->flags |= MOV_TRACK_CTTS;
     trk->cluster[trk->entry].cts   = pkt->pts - pkt->dts;
     trk->cluster[trk->entry].flags = 0;
@@ -5512,6 +5535,7 @@  static int mov_write_single_packet(AVFormatContext *s, AVPacket *pkt)
 
         if (trk->par->codec_id == AV_CODEC_ID_MP4ALS ||
             trk->par->codec_id == AV_CODEC_ID_AAC ||
+            trk->par->codec_id == AV_CODEC_ID_AV1 ||
             trk->par->codec_id == AV_CODEC_ID_FLAC) {
             int side_size = 0;
             uint8_t *side = av_packet_get_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, &side_size);
@@ -6209,15 +6233,12 @@  static int mov_init(AVFormatContext *s)
                         pix_fmt == AV_PIX_FMT_MONOWHITE ||
                         pix_fmt == AV_PIX_FMT_MONOBLACK;
             }
-            if (track->par->codec_id == AV_CODEC_ID_VP9) {
+            if (track->par->codec_id == AV_CODEC_ID_VP9 ||
+                track->par->codec_id == AV_CODEC_ID_AV1) {
                 if (track->mode != MODE_MP4) {
-                    av_log(s, AV_LOG_ERROR, "VP9 only supported in MP4.\n");
+                    av_log(s, AV_LOG_ERROR, "%s only supported in MP4.\n", avcodec_get_name(track->par->codec_id));
                     return AVERROR(EINVAL);
                 }
-            } else if (track->par->codec_id == AV_CODEC_ID_AV1) {
-                /* spec is not finished, so forbid for now */
-                av_log(s, AV_LOG_ERROR, "AV1 muxing is currently not supported.\n");
-                return AVERROR_PATCHWELCOME;
             } else if (track->par->codec_id == AV_CODEC_ID_VP8) {
                 /* altref frames handling is not defined in the spec as of version v1.0,
                  * so just forbid muxing VP8 streams altogether until a new version does */
@@ -6732,6 +6753,7 @@  const AVCodecTag codec_mp4_tags[] = {
     { AV_CODEC_ID_DIRAC       , MKTAG('d', 'r', 'a', 'c') },
     { AV_CODEC_ID_TSCC2       , MKTAG('m', 'p', '4', 'v') },
     { AV_CODEC_ID_VP9         , MKTAG('v', 'p', '0', '9') },
+    { AV_CODEC_ID_AV1         , MKTAG('a', 'v', '0', '1') },
     { AV_CODEC_ID_AAC         , MKTAG('m', 'p', '4', 'a') },
     { AV_CODEC_ID_MP4ALS      , MKTAG('m', 'p', '4', 'a') },
     { AV_CODEC_ID_MP3         , MKTAG('m', 'p', '4', 'a') },