[FFmpeg-devel,GSoC,5/6] avformat/utils: add av_packet_clean to remove AVPackets not needed in pktl

From: spartazhc <spartazhc@gmail.com>

Add av_packet_clean to remove AVPackets whose stream_index is not
in st_index list.

Generally s->internal->packet_buffer may have pkts from different
stream, and stream_index will be used to discard pkt that is not
needed. But in case of abr, several streams may pass the stream_index
check. So we need a function to remove AVPackets not needed in pktl
added by hls_read_header.

Signed-off-by: spartazhc <spartazhc@gmail.com>
---
 libavformat/avformat.h |  9 +++++++
 libavformat/internal.h | 15 +++++++++++
 libavformat/utils.c    | 59 ++++++++++++++++++++++++++++++++++++++++++
 libavformat/version.h  |  2 +-
 4 files changed, 84 insertions(+), 1 deletion(-)

Hongcheng Zhong:
> From: spartazhc <spartazhc@gmail.com>
> 
> Add av_packet_clean to remove AVPackets whose stream_index is not
> in st_index list.
> 
> Generally s->internal->packet_buffer may have pkts from different
> stream, and stream_index will be used to discard pkt that is not
> needed. But in case of abr, several streams may pass the stream_index
> check. So we need a function to remove AVPackets not needed in pktl
> added by hls_read_header.
> 
> Signed-off-by: spartazhc <spartazhc@gmail.com>
> ---
>  libavformat/avformat.h |  9 +++++++
>  libavformat/internal.h | 15 +++++++++++
>  libavformat/utils.c    | 59 ++++++++++++++++++++++++++++++++++++++++++
>  libavformat/version.h  |  2 +-
>  4 files changed, 84 insertions(+), 1 deletion(-)
> 

First: I do not know whether we even need this API or not; but I
nevertheless looked at your proposal.

> diff --git a/libavformat/avformat.h b/libavformat/avformat.h
> index e91e7f1d33..90dee0d075 100644
> --- a/libavformat/avformat.h
> +++ b/libavformat/avformat.h
> @@ -2474,6 +2474,15 @@ int avformat_seek_file(AVFormatContext *s, int stream_index, int64_t min_ts, int
>   */
>  int avformat_flush(AVFormatContext *s);
>  
> +/**
> + * Remove the AVPackets do not needed in the packet list.
> + * The behaviour is undefined if the packet list is empty.
> + *

This is completely wrong. It is even more wrong than for
ff_packet_list_clean, as the user has no way to check whether the packet
list is empty.

> + * @param s media file handle
> + * @param st_index the stream_index list which is needed
> + */
> +int av_packet_clean(AVFormatContext *s, int *st_index);

I don't like "clean" as this sounds as if the whole list were discarded;
something like "filter" sounds better.

> +
>  /**
>   * Start playing a network-based stream (e.g. RTSP stream) at the
>   * current position.
> diff --git a/libavformat/internal.h b/libavformat/internal.h
> index 17a6ab07d3..ac943b1441 100644
> --- a/libavformat/internal.h
> +++ b/libavformat/internal.h
> @@ -764,6 +764,21 @@ int ff_packet_list_put(AVPacketList **head, AVPacketList **tail,
>  int ff_packet_list_get(AVPacketList **head, AVPacketList **tail,
>                         AVPacket *pkt);
>  
> +/**
> + * Remove the AVPackets do not needed in the packet list.

Wrong English "do not needed"

> + * The behaviour is undefined if the packet list is empty.
> + *

Undefined if empty? That is very bad behaviour. If it is empty, then
there simply are no packets to discard and this function shouldn't do
anything.

> + * @note The pkt will be overwritten completely. The caller owns the
> + *       packet and must unref it by itself.
> + *

You obviously copied this (and the undefined if empty) from
ff_packet_list_get. It makes no sense at all here.

> + * @param head List head element
> + * @param tail List tail element
> + * @param st_index the stream_index list which is needed

This list needs a length argument (or a documented sentinel). You are
currently hardcoding AVMEDIA_TYPE_NB in an undocumented manner and this
is wrong.

> + * @return 0 on success. Success is guaranteed
> + *         if the packet list is not empty.
> + */
> +int ff_packet_list_clean(AVPacketList **head, AVPacketList **tail,
> +                       int *st_index);
>  /**
>   * Wipe the list and unref all the packets in it.
>   *
> diff --git a/libavformat/utils.c b/libavformat/utils.c
> index 45a4179552..26b5a08a19 100644
> --- a/libavformat/utils.c
> +++ b/libavformat/utils.c
> @@ -1565,6 +1565,65 @@ int ff_packet_list_get(AVPacketList **pkt_buffer,
>      return 0;
>  }
>  
> +/**
> + * return 1 if needed
> + */
> +static int ff_check_st_index(int st, int *st_index)
> +{
> +    for (int i = 0; i < AVMEDIA_TYPE_NB; ++i) {
> +        if (st_index[i] == st)
> +            return 1;
> +    }
> +    return 0;
> +}
> +
> +int ff_packet_list_clean(AVPacketList **pkt_buffer,
> +                         AVPacketList **pkt_buffer_end,
> +                         int           *st_index)
> +{
> +    AVPacketList *pktl, *pktn;
> +    av_assert0(*pkt_buffer);
> +    pktl = *pkt_buffer;
> +    pktn = pktl->next;
> +
> +    /* num >= 3 */

?

> +    while (pktn && pktn != *pkt_buffer_end) {
> +        if (!ff_check_st_index(pktn->pkt.stream_index, st_index)) {
> +            av_packet_unref(&pktn->pkt);
> +            pktl->next = pktn->next;
> +            av_freep(pktn);

This does not free pktn (which leaks -- use valgrind to see it for
yourself). It will instead av_free pktn->pkt.buf which is NULL because
of the av_packet_unref above.

> +            pktn = pktl->next;
> +        } else {
> +            pktl = pktn;
> +            pktn = pktn->next;
> +        }
> +    }
> +    /* last one*/
> +    if (pktn && !ff_check_st_index(pktn->pkt.stream_index, st_index)) {
> +        av_packet_unref(&pktn->pkt);
> +        av_freep(pktn);
> +        pktl->next = NULL;
> +        *pkt_buffer_end = pktl;
> +    }

Is handling the last one differently really necessary? Wouldn't it be
enough to remove pktn != *pkt_buffer_end from the above condition and do
*pkt_buffer_end = pktl; unconditionally after the loop?

> +    /* first one*/
> +    pktl = *pkt_buffer;
> +    if (!ff_check_st_index(pktl->pkt.stream_index, st_index)) {
> +        av_packet_unref(&pktl->pkt);
> +        *pkt_buffer = pktl->next;
> +        if (!pktl->next)
> +            *pkt_buffer_end = NULL;
> +        av_freep(&pktl);
> +    }
> +
> +    return 0;
> +}
> +
> +int av_packet_clean(AVFormatContext *s, int *st_index) {

{ belongs on a separate line here.

> +    int ret = ff_packet_list_clean(&s->internal->packet_buffer,
> +                                   &s->internal->packet_buffer_end, st_index);

The packet_buffer you are filtering here is only used if genpts is used
or at the beginning for avformat_find_stream_info. There are two other
lists that you are not filtering. Is this intended?

> +    return ret;
> +}
> +
>  static int64_t ts_to_samples(AVStream *st, int64_t ts)
>  {
>      return av_rescale(ts, st->time_base.num * st->codecpar->sample_rate, st->time_base.den);
> diff --git a/libavformat/version.h b/libavformat/version.h
> index 3c1957b00c..75c03fde0a 100644
> --- a/libavformat/version.h
> +++ b/libavformat/version.h
> @@ -32,7 +32,7 @@
>  // Major bumping may affect Ticket5467, 5421, 5451(compatibility with Chromium)
>  // Also please add any ticket numbers that you believe might be affected here
>  #define LIBAVFORMAT_VERSION_MAJOR  58
> -#define LIBAVFORMAT_VERSION_MINOR  47
> +#define LIBAVFORMAT_VERSION_MINOR  48
>  #define LIBAVFORMAT_VERSION_MICRO 100
>  
>  #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \
>

----- On Jul 5, 2020, at 9:14 PM, Andreas Rheinhardt andreas.rheinhardt@gmail.com wrote:

> Hongcheng Zhong:
>> From: spartazhc <spartazhc@gmail.com>
>> 
>> Add av_packet_clean to remove AVPackets whose stream_index is not
>> in st_index list.
>> 
>> Generally s->internal->packet_buffer may have pkts from different
>> stream, and stream_index will be used to discard pkt that is not
>> needed. But in case of abr, several streams may pass the stream_index
>> check. So we need a function to remove AVPackets not needed in pktl
>> added by hls_read_header.
>> 
>> Signed-off-by: spartazhc <spartazhc@gmail.com>
>> ---
>>  libavformat/avformat.h |  9 +++++++
>>  libavformat/internal.h | 15 +++++++++++
>>  libavformat/utils.c    | 59 ++++++++++++++++++++++++++++++++++++++++++
>>  libavformat/version.h  |  2 +-
>>  4 files changed, 84 insertions(+), 1 deletion(-)
>> 
> 
> First: I do not know whether we even need this API or not; but I
> nevertheless looked at your proposal.
> 
>> diff --git a/libavformat/avformat.h b/libavformat/avformat.h
>> index e91e7f1d33..90dee0d075 100644
>> --- a/libavformat/avformat.h
>> +++ b/libavformat/avformat.h
>> @@ -2474,6 +2474,15 @@ int avformat_seek_file(AVFormatContext *s, int
>> stream_index, int64_t min_ts, int
>>   */
>>  int avformat_flush(AVFormatContext *s);
>>  
>> +/**
>> + * Remove the AVPackets do not needed in the packet list.
>> + * The behaviour is undefined if the packet list is empty.
>> + *
> 
> This is completely wrong. It is even more wrong than for
> ff_packet_list_clean, as the user has no way to check whether the packet
> list is empty.
> 
>> + * @param s media file handle
>> + * @param st_index the stream_index list which is needed
>> + */
>> +int av_packet_clean(AVFormatContext *s, int *st_index);
> 
> I don't like "clean" as this sounds as if the whole list were discarded;
> something like "filter" sounds better.
> 
>> +
>>  /**
>>   * Start playing a network-based stream (e.g. RTSP stream) at the
>>   * current position.
>> diff --git a/libavformat/internal.h b/libavformat/internal.h
>> index 17a6ab07d3..ac943b1441 100644
>> --- a/libavformat/internal.h
>> +++ b/libavformat/internal.h
>> @@ -764,6 +764,21 @@ int ff_packet_list_put(AVPacketList **head, AVPacketList
>> **tail,
>>  int ff_packet_list_get(AVPacketList **head, AVPacketList **tail,
>>                         AVPacket *pkt);
>>  
>> +/**
>> + * Remove the AVPackets do not needed in the packet list.
> 
> Wrong English "do not needed"
> 
>> + * The behaviour is undefined if the packet list is empty.
>> + *
> 
> Undefined if empty? That is very bad behaviour. If it is empty, then
> there simply are no packets to discard and this function shouldn't do
> anything.
> 
>> + * @note The pkt will be overwritten completely. The caller owns the
>> + *       packet and must unref it by itself.
>> + *
> 
> You obviously copied this (and the undefined if empty) from
> ff_packet_list_get. It makes no sense at all here.
> 
>> + * @param head List head element
>> + * @param tail List tail element
>> + * @param st_index the stream_index list which is needed
> 
> This list needs a length argument (or a documented sentinel). You are
> currently hardcoding AVMEDIA_TYPE_NB in an undocumented manner and this
> is wrong.
> 
>> + * @return 0 on success. Success is guaranteed
>> + *         if the packet list is not empty.
>> + */
>> +int ff_packet_list_clean(AVPacketList **head, AVPacketList **tail,
>> +                       int *st_index);
>>  /**
>>   * Wipe the list and unref all the packets in it.
>>   *
>> diff --git a/libavformat/utils.c b/libavformat/utils.c
>> index 45a4179552..26b5a08a19 100644
>> --- a/libavformat/utils.c
>> +++ b/libavformat/utils.c
>> @@ -1565,6 +1565,65 @@ int ff_packet_list_get(AVPacketList **pkt_buffer,
>>      return 0;
>>  }
>>  
>> +/**
>> + * return 1 if needed
>> + */
>> +static int ff_check_st_index(int st, int *st_index)
>> +{
>> +    for (int i = 0; i < AVMEDIA_TYPE_NB; ++i) {
>> +        if (st_index[i] == st)
>> +            return 1;
>> +    }
>> +    return 0;
>> +}
>> +
>> +int ff_packet_list_clean(AVPacketList **pkt_buffer,
>> +                         AVPacketList **pkt_buffer_end,
>> +                         int           *st_index)
>> +{
>> +    AVPacketList *pktl, *pktn;
>> +    av_assert0(*pkt_buffer);
>> +    pktl = *pkt_buffer;
>> +    pktn = pktl->next;
>> +
>> +    /* num >= 3 */
> 
> ?
> 
>> +    while (pktn && pktn != *pkt_buffer_end) {
>> +        if (!ff_check_st_index(pktn->pkt.stream_index, st_index)) {
>> +            av_packet_unref(&pktn->pkt);
>> +            pktl->next = pktn->next;
>> +            av_freep(pktn);
> 
> This does not free pktn (which leaks -- use valgrind to see it for
> yourself). It will instead av_free pktn->pkt.buf which is NULL because
> of the av_packet_unref above.
> 
>> +            pktn = pktl->next;
>> +        } else {
>> +            pktl = pktn;
>> +            pktn = pktn->next;
>> +        }
>> +    }
>> +    /* last one*/
>> +    if (pktn && !ff_check_st_index(pktn->pkt.stream_index, st_index)) {
>> +        av_packet_unref(&pktn->pkt);
>> +        av_freep(pktn);
>> +        pktl->next = NULL;
>> +        *pkt_buffer_end = pktl;
>> +    }
> 
> Is handling the last one differently really necessary? Wouldn't it be
> enough to remove pktn != *pkt_buffer_end from the above condition and do
> *pkt_buffer_end = pktl; unconditionally after the loop?
> 
>> +    /* first one*/
>> +    pktl = *pkt_buffer;
>> +    if (!ff_check_st_index(pktl->pkt.stream_index, st_index)) {
>> +        av_packet_unref(&pktl->pkt);
>> +        *pkt_buffer = pktl->next;
>> +        if (!pktl->next)
>> +            *pkt_buffer_end = NULL;
>> +        av_freep(&pktl);
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +int av_packet_clean(AVFormatContext *s, int *st_index) {
> 
> { belongs on a separate line here.
> 
>> +    int ret = ff_packet_list_clean(&s->internal->packet_buffer,
>> +                                   &s->internal->packet_buffer_end, st_index);
> 
> The packet_buffer you are filtering here is only used if genpts is used
> or at the beginning for avformat_find_stream_info. There are two other
> lists that you are not filtering. Is this intended?
> 
>> +    return ret;
>> +}
>> +
>>  static int64_t ts_to_samples(AVStream *st, int64_t ts)
>>  {
>>      return av_rescale(ts, st->time_base.num * st->codecpar->sample_rate,
>>      st->time_base.den);
>> diff --git a/libavformat/version.h b/libavformat/version.h
>> index 3c1957b00c..75c03fde0a 100644
>> --- a/libavformat/version.h
>> +++ b/libavformat/version.h
>> @@ -32,7 +32,7 @@
>>  // Major bumping may affect Ticket5467, 5421, 5451(compatibility with Chromium)
>>  // Also please add any ticket numbers that you believe might be affected here
>>  #define LIBAVFORMAT_VERSION_MAJOR  58
>> -#define LIBAVFORMAT_VERSION_MINOR  47
>> +#define LIBAVFORMAT_VERSION_MINOR  48
>>  #define LIBAVFORMAT_VERSION_MICRO 100
>>  
>>  #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \
>> 
Thanks for your reply! Will check all these comments.
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".