[FFmpeg-devel,v2] avcodec: add a get_encoder_buffer() callback to AVCodecContext

This callback is functionally the same as get_buffer2() is for decoders, and
implements for the new encode API the functionality of the old encode API had
where the user could provide their own buffers.

Signed-off-by: James Almer <jamrial@gmail.com>
---
Used the names Lynne suggested this time, plus a line about how the callback
must be thread safe.

 libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
 libavcodec/codec.h   |  8 ++++---
 libavcodec/encode.c  | 54 +++++++++++++++++++++++++++++++++++++++++++-
 libavcodec/encode.h  |  8 +++++++
 libavcodec/options.c |  1 +
 5 files changed, 112 insertions(+), 4 deletions(-)

Feb 21, 2021, 18:35 by jamrial@gmail.com:

> This callback is functionally the same as get_buffer2() is for decoders, and
> implements for the new encode API the functionality of the old encode API had
> where the user could provide their own buffers.
>
> Signed-off-by: James Almer <jamrial@gmail.com>
> ---
> Used the names Lynne suggested this time, plus a line about how the callback
> must be thread safe.
>
>  libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>  libavcodec/codec.h   |  8 ++++---
>  libavcodec/encode.c  | 54 +++++++++++++++++++++++++++++++++++++++++++-
>  libavcodec/encode.h  |  8 +++++++
>  libavcodec/options.c |  1 +
>  5 files changed, 112 insertions(+), 4 deletions(-)
>
> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
> index 7dbf083a24..e60eb16ce1 100644
> --- a/libavcodec/avcodec.h
> +++ b/libavcodec/avcodec.h
> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>  */
>  #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>  
> +/**
> + * The encoder will keep a reference to the packet and may reuse it later.
> + */
> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
> +
>  struct AVCodecInternal;
>  
>  /**
> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>  * - encoding: set by user
>  */
>  int export_side_data;
> +
> +    /**
> +     * This callback is called at the beginning of each packet to get a data
> +     * buffer for it.
> +     *
> +     * The following field will be set in the packet before this callback is
> +     * called:
> +     * - size
> +     * This callback must use the above value to calculate the required buffer size,
> +     * which must padded by at least AV_INPUT_BUFFER_PADDING_SIZE bytes.
> +     *
> +     * This callback must fill the following fields in the packet:
> +     * - data
> +     * - buf must contain a pointer to an AVBufferRef structure. The packet's
> +     *   data pointer must be contained in it.
> +     *   See: av_buffer_create(), av_buffer_alloc(), and av_buffer_ref().
> +     *
> +     * If AV_CODEC_CAP_DR1 is not set then get_encoder_buffer() must call
> +     * avcodec_default_get_encoder_buffer() instead of providing a buffer allocated by
> +     * some other means.
> +     *
> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in flags then the packet may be reused
> +     * (read and/or written to if it is writable) later by libavcodec.
> +     *
> +     * This callback must be thread-safe, as when frame multithreading is used, it may
> +     * be called from multiple threads simultaneously.
> +     *
> +     * @see avcodec_default_get_encoder_buffer()
> +     *
> +     * - encoding: Set by libavcodec, user can override.
> +     * - decoding: unused
> +     */
> +    int (*get_encoder_buffer)(struct AVCodecContext *s, AVPacket *pkt, int flags);
>  } AVCodecContext;
>  
>  #if FF_API_CODEC_GET_SET
> @@ -2920,6 +2958,13 @@ void avsubtitle_free(AVSubtitle *sub);
>  */
>  int avcodec_default_get_buffer2(AVCodecContext *s, AVFrame *frame, int flags);
>  
> +/**
> + * The default callback for AVCodecContext.get_encoder_buffer(). It is made public so
> + * it can be called by custom get_encoder_buffer() implementations for encoders without
> + * AV_CODEC_CAP_DR1 set.
> + */
> +int avcodec_default_get_encoder_buffer(AVCodecContext *s, AVPacket *pkt, int flags);
> +
>  /**
>  * Modify width and height values so that they will result in a memory
>  * buffer that is acceptable for the codec if you do not use any horizontal
> diff --git a/libavcodec/codec.h b/libavcodec/codec.h
> index 0ccbf0eb19..a679fdc9e1 100644
> --- a/libavcodec/codec.h
> +++ b/libavcodec/codec.h
> @@ -43,9 +43,11 @@
>  */
>  #define AV_CODEC_CAP_DRAW_HORIZ_BAND     (1 <<  0)
>  /**
> - * Codec uses get_buffer() for allocating buffers and supports custom allocators.
> - * If not set, it might not use get_buffer() at all or use operations that
> - * assume the buffer was allocated by avcodec_default_get_buffer.
> + * Codec uses get_buffer() or get_encoder_buffer() for allocating buffers and
> + * supports custom allocators.
> + * If not set, it might not use get_buffer() or get_encoder_buffer() at all, or
> + * use operations that assume the buffer was allocated by
> + * avcodec_default_get_buffer2 or avcodec_default_get_encoder_buffer.
>  */
>  #define AV_CODEC_CAP_DR1                 (1 <<  1)
>  #define AV_CODEC_CAP_TRUNCATED           (1 <<  3)
> diff --git a/libavcodec/encode.c b/libavcodec/encode.c
> index 282337e453..f39c8d38ce 100644
> --- a/libavcodec/encode.c
> +++ b/libavcodec/encode.c
> @@ -56,6 +56,52 @@ int ff_alloc_packet2(AVCodecContext *avctx, AVPacket *avpkt, int64_t size, int64
>  return 0;
>  }
>  
> +int avcodec_default_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, int flags)
> +{
> +    int ret;
> +
> +    if (avpkt->data || avpkt->buf) {
> +        av_log(avctx, AV_LOG_ERROR, "avpkt->{data,buf} != NULL in avcodec_default_get_encoder_buffer()\n");
> +        return AVERROR(EINVAL);
> +    }
> +
> +    ret = av_new_packet(avpkt, avpkt->size);
> +    if (ret < 0)
> +        av_log(avctx, AV_LOG_ERROR, "Failed to allocate packet of size %d\n", avpkt->size);
> +
> +    return ret;
> +}
> +
> +int ff_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, int64_t size, int flags)
> +{
> +    int ret;
> +
> +    if (size < 0 || size > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE)
> +        return AVERROR(EINVAL);
> +
> +    av_assert0(!avpkt->data && !avpkt->buf);
> +
> +    avpkt->size = size;
> +    ret = avctx->get_encoder_buffer(avctx, avpkt, flags);
> +    if (ret < 0)
> +        goto fail;
> +
> +    if (!avpkt->data || !avpkt->buf) {
> +        av_log(avctx, AV_LOG_ERROR, "No buffer returned by get_encoder_buffer()\n");
> +        ret = AVERROR(EINVAL);
> +        goto fail;
> +    }
> +
> +    ret = 0;
> +fail:
> +    if (ret < 0) {
> +        av_log(avctx, AV_LOG_ERROR, "get_encoder_buffer() failed\n");
> +        av_packet_unref(avpkt);
> +    }
> +
> +    return ret;
> +}
> +
>  /**
>  * Pad last frame with silence.
>  */
> @@ -169,7 +215,7 @@ static int encode_simple_internal(AVCodecContext *avctx, AVPacket *avpkt)
>  emms_c();
>  
>  if (!ret && got_packet) {
> -        if (avpkt->data) {
> +        if (avpkt->data && !(avctx->codec->capabilities & AV_CODEC_CAP_DR1)) {
>  ret = av_packet_make_refcounted(avpkt);
>  if (ret < 0)
>  goto end;
> @@ -377,6 +423,12 @@ static int compat_encode(AVCodecContext *avctx, AVPacket *avpkt,
>  av_log(avctx, AV_LOG_WARNING, "AVFrame.width or height is not set\n");
>  }
>  
> +    if (avctx->codec->capabilities & AV_CODEC_CAP_DR1) {
> +        av_log(avctx, AV_LOG_WARNING, "The deprecated avcodec_encode_* API does not support "
> +                                      "AV_CODEC_CAP_DR1 encoders\n");
> +        return AVERROR(ENOSYS);
> +    }
> +
>  ret = avcodec_send_frame(avctx, frame);
>  if (ret == AVERROR_EOF)
>  ret = 0;
> diff --git a/libavcodec/encode.h b/libavcodec/encode.h
> index dfa9cb2d97..3192bd9e38 100644
> --- a/libavcodec/encode.h
> +++ b/libavcodec/encode.h
> @@ -24,6 +24,7 @@
>  #include "libavutil/frame.h"
>  
>  #include "avcodec.h"
> +#include "packet.h"
>  
>  /**
>  * Called by encoders to get the next frame for encoding.
> @@ -36,4 +37,11 @@
>  */
>  int ff_encode_get_frame(AVCodecContext *avctx, AVFrame *frame);
>  
> +/**
> + * Get a buffer for a packet. This is a wrapper around
> + * AVCodecContext.get_encoder_buffer() and should be used instead calling get_encoder_buffer()
> + * directly.
> + */
> +int ff_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, int64_t size, int flags);
> +
>  #endif /* AVCODEC_ENCODE_H */
> diff --git a/libavcodec/options.c b/libavcodec/options.c
> index 4bbf74ec7f..cd5fa6eb14 100644
> --- a/libavcodec/options.c
> +++ b/libavcodec/options.c
> @@ -130,6 +130,7 @@ static int init_context_defaults(AVCodecContext *s, const AVCodec *codec)
>  s->pkt_timebase        = (AVRational){ 0, 1 };
>  s->get_buffer2         = avcodec_default_get_buffer2;
>  s->get_format          = avcodec_default_get_format;
> +    s->get_encoder_buffer  = avcodec_default_get_encoder_buffer;
>  s->execute             = avcodec_default_execute;
>  s->execute2            = avcodec_default_execute2;
>  s->sample_aspect_ratio = (AVRational){0,1};
> -- 
> 2.30.0
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org>  with subject "unsubscribe".
>

I thought I suggested "encode", but I typed "encoder", sorry :(
"encoder" looks worse. It's a nit, you don't have to send a new patch for it,
but could you change everything to "encode" sometime before you push it?
Thanks.

With that change patch LGTM.

On 2/21/2021 2:59 PM, Lynne wrote:
> Feb 21, 2021, 18:35 by jamrial@gmail.com:
> 
>> This callback is functionally the same as get_buffer2() is for decoders, and
>> implements for the new encode API the functionality of the old encode API had
>> where the user could provide their own buffers.
>>
>> Signed-off-by: James Almer <jamrial@gmail.com>
>> ---
>> Used the names Lynne suggested this time, plus a line about how the callback
>> must be thread safe.
>>
>>   libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>   libavcodec/codec.h   |  8 ++++---
>>   libavcodec/encode.c  | 54 +++++++++++++++++++++++++++++++++++++++++++-
>>   libavcodec/encode.h  |  8 +++++++
>>   libavcodec/options.c |  1 +
>>   5 files changed, 112 insertions(+), 4 deletions(-)

[...]

> I thought I suggested "encode", but I typed "encoder", sorry :(
> "encoder" looks worse. It's a nit, you don't have to send a new patch for it,
> but could you change everything to "encode" sometime before you push it?

Sure.

> Thanks.
> 
> With that change patch LGTM.

Will wait a bit to see if others want to comment, then push it after 
adding an APIChanges entry and version bump.

On 21/02/2021 17:35, James Almer wrote:
> This callback is functionally the same as get_buffer2() is for decoders, and
> implements for the new encode API the functionality of the old encode API had
> where the user could provide their own buffers.
> 
> Signed-off-by: James Almer <jamrial@gmail.com>
> ---
> Used the names Lynne suggested this time, plus a line about how the callback
> must be thread safe.
> 
>   libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>   libavcodec/codec.h   |  8 ++++---
>   libavcodec/encode.c  | 54 +++++++++++++++++++++++++++++++++++++++++++-
>   libavcodec/encode.h  |  8 +++++++
>   libavcodec/options.c |  1 +
>   5 files changed, 112 insertions(+), 4 deletions(-)
> 
> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
> index 7dbf083a24..e60eb16ce1 100644
> --- a/libavcodec/avcodec.h
> +++ b/libavcodec/avcodec.h
> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>    */
>   #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>   
> +/**
> + * The encoder will keep a reference to the packet and may reuse it later.
> + */
> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
> +
>   struct AVCodecInternal;
>   
>   /**
> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>        * - encoding: set by user
>        */
>       int export_side_data;
> +
> +    /**
> +     * This callback is called at the beginning of each packet to get a data
> +     * buffer for it.
> +     *
> +     * The following field will be set in the packet before this callback is
> +     * called:
> +     * - size
> +     * This callback must use the above value to calculate the required buffer size,
> +     * which must padded by at least AV_INPUT_BUFFER_PADDING_SIZE bytes.
> +     *
> +     * This callback must fill the following fields in the packet:
> +     * - data

Is the data pointer allowed to be in write-only memory?  Does it have any alignment requirements?

> +     * - buf must contain a pointer to an AVBufferRef structure. The packet's
> +     *   data pointer must be contained in it.
> +     *   See: av_buffer_create(), av_buffer_alloc(), and av_buffer_ref().
> +     *
> +     * If AV_CODEC_CAP_DR1 is not set then get_encoder_buffer() must call
> +     * avcodec_default_get_encoder_buffer() instead of providing a buffer allocated by
> +     * some other means.
> +     *
> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in flags then the packet may be reused
> +     * (read and/or written to if it is writable) later by libavcodec.
> +     *
> +     * This callback must be thread-safe, as when frame multithreading is used, it may
> +     * be called from multiple threads simultaneously.

Allowing simulatenous calls feels unexpectedly tricky.  Is it really necessary?

> +     *
> +     * @see avcodec_default_get_encoder_buffer()
> +     *
> +     * - encoding: Set by libavcodec, user can override.
> +     * - decoding: unused
> +     */
> +    int (*get_encoder_buffer)(struct AVCodecContext *s, AVPacket *pkt, int flags);

Can the encoder ask for arbitrarily many packets?

Can the user return "not yet" somehow to this if they have a fixed output buffer pool but no buffer is currently available?

I don't much like the idea of the user suspending the thread in the callback until they have some available, which might work in some cases but might also deadlock if an avcodec_receive_packet() call is blocked by it.

(For get_buffer2 we have a bit of consideration of this problem for hardware frames which might have limited numbers via avcodec_get_hw_frames_parameters(), though the general case does not have a solution.)

>   } AVCodecContext;
>   
>   #if FF_API_CODEC_GET_SET
> @@ -2920,6 +2958,13 @@ void avsubtitle_free(AVSubtitle *sub);
>    */
>   int avcodec_default_get_buffer2(AVCodecContext *s, AVFrame *frame, int flags);
>   
> +/**
> + * The default callback for AVCodecContext.get_encoder_buffer(). It is made public so
> + * it can be called by custom get_encoder_buffer() implementations for encoders without
> + * AV_CODEC_CAP_DR1 set.
> + */
> +int avcodec_default_get_encoder_buffer(AVCodecContext *s, AVPacket *pkt, int flags);
> +
>   /**
>    * Modify width and height values so that they will result in a memory
>    * buffer that is acceptable for the codec if you do not use any horizontal
> diff --git a/libavcodec/codec.h b/libavcodec/codec.h
> index 0ccbf0eb19..a679fdc9e1 100644
> --- a/libavcodec/codec.h
> +++ b/libavcodec/codec.h
> @@ -43,9 +43,11 @@
>    */
>   #define AV_CODEC_CAP_DRAW_HORIZ_BAND     (1 <<  0)
>   /**
> - * Codec uses get_buffer() for allocating buffers and supports custom allocators.
> - * If not set, it might not use get_buffer() at all or use operations that
> - * assume the buffer was allocated by avcodec_default_get_buffer.
> + * Codec uses get_buffer() or get_encoder_buffer() for allocating buffers and
> + * supports custom allocators.
> + * If not set, it might not use get_buffer() or get_encoder_buffer() at all, or
> + * use operations that assume the buffer was allocated by
> + * avcodec_default_get_buffer2 or avcodec_default_get_encoder_buffer.
>    */
>   #define AV_CODEC_CAP_DR1                 (1 <<  1)
>   #define AV_CODEC_CAP_TRUNCATED           (1 <<  3)
> diff --git a/libavcodec/encode.c b/libavcodec/encode.c
> index 282337e453..f39c8d38ce 100644
> --- a/libavcodec/encode.c
> +++ b/libavcodec/encode.c
> @@ -56,6 +56,52 @@ int ff_alloc_packet2(AVCodecContext *avctx, AVPacket *avpkt, int64_t size, int64
>       return 0;
>   }
>   
> +int avcodec_default_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, int flags)
> +{
> +    int ret;
> +
> +    if (avpkt->data || avpkt->buf) {
> +        av_log(avctx, AV_LOG_ERROR, "avpkt->{data,buf} != NULL in avcodec_default_get_encoder_buffer()\n");
> +        return AVERROR(EINVAL);
> +    }
> +
> +    ret = av_new_packet(avpkt, avpkt->size);
> +    if (ret < 0)
> +        av_log(avctx, AV_LOG_ERROR, "Failed to allocate packet of size %d\n", avpkt->size);
> +
> +    return ret;
> +}
> +
> +int ff_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, int64_t size, int flags)
> +{
> +    int ret;
> +
> +    if (size < 0 || size > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE)
> +        return AVERROR(EINVAL);
> +
> +    av_assert0(!avpkt->data && !avpkt->buf);
> +
> +    avpkt->size = size;
> +    ret = avctx->get_encoder_buffer(avctx, avpkt, flags);
> +    if (ret < 0)
> +        goto fail;
> +
> +    if (!avpkt->data || !avpkt->buf) {
> +        av_log(avctx, AV_LOG_ERROR, "No buffer returned by get_encoder_buffer()\n");
> +        ret = AVERROR(EINVAL);
> +        goto fail;
> +    }
> +
> +    ret = 0;
> +fail:
> +    if (ret < 0) {
> +        av_log(avctx, AV_LOG_ERROR, "get_encoder_buffer() failed\n");
> +        av_packet_unref(avpkt);
> +    }
> +
> +    return ret;
> +}
> +
>   /**
>    * Pad last frame with silence.
>    */
> @@ -169,7 +215,7 @@ static int encode_simple_internal(AVCodecContext *avctx, AVPacket *avpkt)
>       emms_c();
>   
>       if (!ret && got_packet) {
> -        if (avpkt->data) {
> +        if (avpkt->data && !(avctx->codec->capabilities & AV_CODEC_CAP_DR1)) {
>               ret = av_packet_make_refcounted(avpkt);
>               if (ret < 0)
>                   goto end;
> @@ -377,6 +423,12 @@ static int compat_encode(AVCodecContext *avctx, AVPacket *avpkt,
>               av_log(avctx, AV_LOG_WARNING, "AVFrame.width or height is not set\n");
>       }
>   
> +    if (avctx->codec->capabilities & AV_CODEC_CAP_DR1) {
> +        av_log(avctx, AV_LOG_WARNING, "The deprecated avcodec_encode_* API does not support "
> +                                      "AV_CODEC_CAP_DR1 encoders\n");
> +        return AVERROR(ENOSYS);
> +    }
> +
>       ret = avcodec_send_frame(avctx, frame);
>       if (ret == AVERROR_EOF)
>           ret = 0;
> diff --git a/libavcodec/encode.h b/libavcodec/encode.h
> index dfa9cb2d97..3192bd9e38 100644
> --- a/libavcodec/encode.h
> +++ b/libavcodec/encode.h
> @@ -24,6 +24,7 @@
>   #include "libavutil/frame.h"
>   
>   #include "avcodec.h"
> +#include "packet.h"
>   
>   /**
>    * Called by encoders to get the next frame for encoding.
> @@ -36,4 +37,11 @@
>    */
>   int ff_encode_get_frame(AVCodecContext *avctx, AVFrame *frame);
>   
> +/**
> + * Get a buffer for a packet. This is a wrapper around
> + * AVCodecContext.get_encoder_buffer() and should be used instead calling get_encoder_buffer()
> + * directly.
> + */
> +int ff_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, int64_t size, int flags);

Why int64_t rather than size_t?

> +
>   #endif /* AVCODEC_ENCODE_H */
> diff --git a/libavcodec/options.c b/libavcodec/options.c
> index 4bbf74ec7f..cd5fa6eb14 100644
> --- a/libavcodec/options.c
> +++ b/libavcodec/options.c
> @@ -130,6 +130,7 @@ static int init_context_defaults(AVCodecContext *s, const AVCodec *codec)
>       s->pkt_timebase        = (AVRational){ 0, 1 };
>       s->get_buffer2         = avcodec_default_get_buffer2;
>       s->get_format          = avcodec_default_get_format;
> +    s->get_encoder_buffer  = avcodec_default_get_encoder_buffer;
>       s->execute             = avcodec_default_execute;
>       s->execute2            = avcodec_default_execute2;
>       s->sample_aspect_ratio = (AVRational){0,1};
> 

It might help to have an implementation for at least one codec and an example user for review.

Thanks,

- Mark

On 2/21/2021 4:13 PM, Mark Thompson wrote:
> On 21/02/2021 17:35, James Almer wrote:
>> This callback is functionally the same as get_buffer2() is for 
>> decoders, and
>> implements for the new encode API the functionality of the old encode 
>> API had
>> where the user could provide their own buffers.
>>
>> Signed-off-by: James Almer <jamrial@gmail.com>
>> ---
>> Used the names Lynne suggested this time, plus a line about how the 
>> callback
>> must be thread safe.
>>
>>   libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>   libavcodec/codec.h   |  8 ++++---
>>   libavcodec/encode.c  | 54 +++++++++++++++++++++++++++++++++++++++++++-
>>   libavcodec/encode.h  |  8 +++++++
>>   libavcodec/options.c |  1 +
>>   5 files changed, 112 insertions(+), 4 deletions(-)
>>
>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>> index 7dbf083a24..e60eb16ce1 100644
>> --- a/libavcodec/avcodec.h
>> +++ b/libavcodec/avcodec.h
>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>    */
>>   #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>> +/**
>> + * The encoder will keep a reference to the packet and may reuse it 
>> later.
>> + */
>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>> +
>>   struct AVCodecInternal;
>>   /**
>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>        * - encoding: set by user
>>        */
>>       int export_side_data;
>> +
>> +    /**
>> +     * This callback is called at the beginning of each packet to get 
>> a data
>> +     * buffer for it.
>> +     *
>> +     * The following field will be set in the packet before this 
>> callback is
>> +     * called:
>> +     * - size
>> +     * This callback must use the above value to calculate the 
>> required buffer size,
>> +     * which must padded by at least AV_INPUT_BUFFER_PADDING_SIZE bytes.
>> +     *
>> +     * This callback must fill the following fields in the packet:
>> +     * - data
> 
> Is the data pointer allowed to be in write-only memory?

I'm not sure what the use case for this would be, so probably no?

> Does it have any alignment requirements?

No, just padding. AVPacket doesn't require alignment for the payload.

> 
>> +     * - buf must contain a pointer to an AVBufferRef structure. The 
>> packet's
>> +     *   data pointer must be contained in it.
>> +     *   See: av_buffer_create(), av_buffer_alloc(), and 
>> av_buffer_ref().
>> +     *
>> +     * If AV_CODEC_CAP_DR1 is not set then get_encoder_buffer() must 
>> call
>> +     * avcodec_default_get_encoder_buffer() instead of providing a 
>> buffer allocated by
>> +     * some other means.
>> +     *
>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in flags then the 
>> packet may be reused
>> +     * (read and/or written to if it is writable) later by libavcodec.
>> +     *
>> +     * This callback must be thread-safe, as when frame 
>> multithreading is used, it may
>> +     * be called from multiple threads simultaneously.
> 
> Allowing simulatenous calls feels unexpectedly tricky.  Is it really 
> necessary?

This was a suggestion by Lynne, i personally don't know. We support 
frame threading encoding (For intra-only codecs), but currently 
ff_alloc_packet2() does not seem to be thread safe, seeing it calls 
av_fast_padded_malloc(), yet it's called by frame threaded encoders.
Should i remove this?

> 
>> +     *
>> +     * @see avcodec_default_get_encoder_buffer()
>> +     *
>> +     * - encoding: Set by libavcodec, user can override.
>> +     * - decoding: unused
>> +     */
>> +    int (*get_encoder_buffer)(struct AVCodecContext *s, AVPacket 
>> *pkt, int flags);
> 
> Can the encoder ask for arbitrarily many packets?
> 
> Can the user return "not yet" somehow to this if they have a fixed 
> output buffer pool but no buffer is currently available?

No, as is it can't. Return values < 0 are considered errors.

> 
> I don't much like the idea of the user suspending the thread in the 
> callback until they have some available, which might work in some cases 
> but might also deadlock if an avcodec_receive_packet() call is blocked 
> by it.

Can we make what's in essence a malloc() call return something like 
EAGAIN, and this in turn be propagated back to 
encode_receive_packet_internal()? Couldn't this potentially end up in 
the forbidden scenario of avcodec_send_frame() and 
avcodec_receive_packet() both returning EAGAIN?

> 
> (For get_buffer2 we have a bit of consideration of this problem for 
> hardware frames which might have limited numbers via 
> avcodec_get_hw_frames_parameters(), though the general case does not 
> have a solution.)
> 
>>   } AVCodecContext;
>>   #if FF_API_CODEC_GET_SET
>> @@ -2920,6 +2958,13 @@ void avsubtitle_free(AVSubtitle *sub);
>>    */
>>   int avcodec_default_get_buffer2(AVCodecContext *s, AVFrame *frame, 
>> int flags);
>> +/**
>> + * The default callback for AVCodecContext.get_encoder_buffer(). It 
>> is made public so
>> + * it can be called by custom get_encoder_buffer() implementations 
>> for encoders without
>> + * AV_CODEC_CAP_DR1 set.
>> + */
>> +int avcodec_default_get_encoder_buffer(AVCodecContext *s, AVPacket 
>> *pkt, int flags);
>> +
>>   /**
>>    * Modify width and height values so that they will result in a memory
>>    * buffer that is acceptable for the codec if you do not use any 
>> horizontal
>> diff --git a/libavcodec/codec.h b/libavcodec/codec.h
>> index 0ccbf0eb19..a679fdc9e1 100644
>> --- a/libavcodec/codec.h
>> +++ b/libavcodec/codec.h
>> @@ -43,9 +43,11 @@
>>    */
>>   #define AV_CODEC_CAP_DRAW_HORIZ_BAND     (1 <<  0)
>>   /**
>> - * Codec uses get_buffer() for allocating buffers and supports custom 
>> allocators.
>> - * If not set, it might not use get_buffer() at all or use operations 
>> that
>> - * assume the buffer was allocated by avcodec_default_get_buffer.
>> + * Codec uses get_buffer() or get_encoder_buffer() for allocating 
>> buffers and
>> + * supports custom allocators.
>> + * If not set, it might not use get_buffer() or get_encoder_buffer() 
>> at all, or
>> + * use operations that assume the buffer was allocated by
>> + * avcodec_default_get_buffer2 or avcodec_default_get_encoder_buffer.
>>    */
>>   #define AV_CODEC_CAP_DR1                 (1 <<  1)
>>   #define AV_CODEC_CAP_TRUNCATED           (1 <<  3)
>> diff --git a/libavcodec/encode.c b/libavcodec/encode.c
>> index 282337e453..f39c8d38ce 100644
>> --- a/libavcodec/encode.c
>> +++ b/libavcodec/encode.c
>> @@ -56,6 +56,52 @@ int ff_alloc_packet2(AVCodecContext *avctx, 
>> AVPacket *avpkt, int64_t size, int64
>>       return 0;
>>   }
>> +int avcodec_default_get_encoder_buffer(AVCodecContext *avctx, 
>> AVPacket *avpkt, int flags)
>> +{
>> +    int ret;
>> +
>> +    if (avpkt->data || avpkt->buf) {
>> +        av_log(avctx, AV_LOG_ERROR, "avpkt->{data,buf} != NULL in 
>> avcodec_default_get_encoder_buffer()\n");
>> +        return AVERROR(EINVAL);
>> +    }
>> +
>> +    ret = av_new_packet(avpkt, avpkt->size);
>> +    if (ret < 0)
>> +        av_log(avctx, AV_LOG_ERROR, "Failed to allocate packet of 
>> size %d\n", avpkt->size);
>> +
>> +    return ret;
>> +}
>> +
>> +int ff_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, 
>> int64_t size, int flags)
>> +{
>> +    int ret;
>> +
>> +    if (size < 0 || size > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE)
>> +        return AVERROR(EINVAL);
>> +
>> +    av_assert0(!avpkt->data && !avpkt->buf);
>> +
>> +    avpkt->size = size;
>> +    ret = avctx->get_encoder_buffer(avctx, avpkt, flags);
>> +    if (ret < 0)
>> +        goto fail;
>> +
>> +    if (!avpkt->data || !avpkt->buf) {
>> +        av_log(avctx, AV_LOG_ERROR, "No buffer returned by 
>> get_encoder_buffer()\n");
>> +        ret = AVERROR(EINVAL);
>> +        goto fail;
>> +    }
>> +
>> +    ret = 0;
>> +fail:
>> +    if (ret < 0) {
>> +        av_log(avctx, AV_LOG_ERROR, "get_encoder_buffer() failed\n");
>> +        av_packet_unref(avpkt);
>> +    }
>> +
>> +    return ret;
>> +}
>> +
>>   /**
>>    * Pad last frame with silence.
>>    */
>> @@ -169,7 +215,7 @@ static int encode_simple_internal(AVCodecContext 
>> *avctx, AVPacket *avpkt)
>>       emms_c();
>>       if (!ret && got_packet) {
>> -        if (avpkt->data) {
>> +        if (avpkt->data && !(avctx->codec->capabilities & 
>> AV_CODEC_CAP_DR1)) {
>>               ret = av_packet_make_refcounted(avpkt);
>>               if (ret < 0)
>>                   goto end;
>> @@ -377,6 +423,12 @@ static int compat_encode(AVCodecContext *avctx, 
>> AVPacket *avpkt,
>>               av_log(avctx, AV_LOG_WARNING, "AVFrame.width or height 
>> is not set\n");
>>       }
>> +    if (avctx->codec->capabilities & AV_CODEC_CAP_DR1) {
>> +        av_log(avctx, AV_LOG_WARNING, "The deprecated 
>> avcodec_encode_* API does not support "
>> +                                      "AV_CODEC_CAP_DR1 encoders\n");
>> +        return AVERROR(ENOSYS);
>> +    }
>> +
>>       ret = avcodec_send_frame(avctx, frame);
>>       if (ret == AVERROR_EOF)
>>           ret = 0;
>> diff --git a/libavcodec/encode.h b/libavcodec/encode.h
>> index dfa9cb2d97..3192bd9e38 100644
>> --- a/libavcodec/encode.h
>> +++ b/libavcodec/encode.h
>> @@ -24,6 +24,7 @@
>>   #include "libavutil/frame.h"
>>   #include "avcodec.h"
>> +#include "packet.h"
>>   /**
>>    * Called by encoders to get the next frame for encoding.
>> @@ -36,4 +37,11 @@
>>    */
>>   int ff_encode_get_frame(AVCodecContext *avctx, AVFrame *frame);
>> +/**
>> + * Get a buffer for a packet. This is a wrapper around
>> + * AVCodecContext.get_encoder_buffer() and should be used instead 
>> calling get_encoder_buffer()
>> + * directly.
>> + */
>> +int ff_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, 
>> int64_t size, int flags);
> 
> Why int64_t rather than size_t?

That's what ff_alloc_packet2() uses, so i figured it will make porting 
existing users much easier (basically just a sed replace, vs trying to 
find if any encoder is assuming int64_t, like exrenc).

> 
>> +
>>   #endif /* AVCODEC_ENCODE_H */
>> diff --git a/libavcodec/options.c b/libavcodec/options.c
>> index 4bbf74ec7f..cd5fa6eb14 100644
>> --- a/libavcodec/options.c
>> +++ b/libavcodec/options.c
>> @@ -130,6 +130,7 @@ static int init_context_defaults(AVCodecContext 
>> *s, const AVCodec *codec)
>>       s->pkt_timebase        = (AVRational){ 0, 1 };
>>       s->get_buffer2         = avcodec_default_get_buffer2;
>>       s->get_format          = avcodec_default_get_format;
>> +    s->get_encoder_buffer  = avcodec_default_get_encoder_buffer;
>>       s->execute             = avcodec_default_execute;
>>       s->execute2            = avcodec_default_execute2;
>>       s->sample_aspect_ratio = (AVRational){0,1};
>>
> 
> It might help to have an implementation for at least one codec and an 
> example user for review.

Sure, below is huffyuvenc adapted to use this API.

> diff --git a/libavcodec/huffyuvenc.c b/libavcodec/huffyuvenc.c
> index 28a5534535..709a5a9d0e 100644
> --- a/libavcodec/huffyuvenc.c
> +++ b/libavcodec/huffyuvenc.c
> @@ -29,6 +29,7 @@
>   */
> 
>  #include "avcodec.h"
> +#include "encode.h"
>  #include "huffyuv.h"
>  #include "huffman.h"
>  #include "huffyuvencdsp.h"
> @@ -762,7 +763,7 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
>      const AVFrame * const p = pict;
>      int i, j, size = 0, ret;
> 
> -    if ((ret = ff_alloc_packet2(avctx, pkt, width * height * 3 * 4 + AV_INPUT_BUFFER_MIN_SIZE, 0)) < 0)
> +    if ((ret = ff_get_encoder_buffer(avctx, pkt, width * height * 3 * 4 + AV_INPUT_BUFFER_MIN_SIZE, 0)) < 0)
>          return ret;
> 
>      if (s->context) {
> @@ -1111,7 +1112,7 @@ AVCodec ff_ffvhuff_encoder = {
>      .init           = encode_init,
>      .encode2        = encode_frame,
>      .close          = encode_end,
> -    .capabilities   = AV_CODEC_CAP_FRAME_THREADS,
> +    .capabilities   = AV_CODEC_CAP_FRAME_THREADS | AV_CODEC_CAP_DR1,
>      .priv_class     = &ff_class,
>      .pix_fmts       = (const enum AVPixelFormat[]){
>          AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUV411P,

On 21/02/2021 20:00, James Almer wrote:
> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>> On 21/02/2021 17:35, James Almer wrote:
>>> This callback is functionally the same as get_buffer2() is for decoders, and
>>> implements for the new encode API the functionality of the old encode API had
>>> where the user could provide their own buffers.
>>>
>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>> ---
>>> Used the names Lynne suggested this time, plus a line about how the callback
>>> must be thread safe.
>>>
>>>   libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>   libavcodec/codec.h   |  8 ++++---
>>>   libavcodec/encode.c  | 54 +++++++++++++++++++++++++++++++++++++++++++-
>>>   libavcodec/encode.h  |  8 +++++++
>>>   libavcodec/options.c |  1 +
>>>   5 files changed, 112 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>> index 7dbf083a24..e60eb16ce1 100644
>>> --- a/libavcodec/avcodec.h
>>> +++ b/libavcodec/avcodec.h
>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>    */
>>>   #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>> +/**
>>> + * The encoder will keep a reference to the packet and may reuse it later.
>>> + */
>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>> +
>>>   struct AVCodecInternal;
>>>   /**
>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>        * - encoding: set by user
>>>        */
>>>       int export_side_data;
>>> +
>>> +    /**
>>> +     * This callback is called at the beginning of each packet to get a data
>>> +     * buffer for it.
>>> +     *
>>> +     * The following field will be set in the packet before this callback is
>>> +     * called:
>>> +     * - size
>>> +     * This callback must use the above value to calculate the required buffer size,
>>> +     * which must padded by at least AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>> +     *
>>> +     * This callback must fill the following fields in the packet:
>>> +     * - data
>>
>> Is the data pointer allowed to be in write-only memory?
> 
> I'm not sure what the use case for this would be, so probably no?

The two use-cases I see for this API are:

* You want to avoid a copy when combining the output with something else.  E.g. you pass a pointer to the block of memory following where you are going to put your header data (for something you are going to send over the network, say).

* You want to avoid a copy when passing the output directly to something external.  E.g. you pass a pointer to a memory-mapped device buffer (such as a V4L2 buffer, say).

In the second case, write-only memory on an external device seems possible, as does memory which is, say, readable but uncached, so reading it is a really bad idea.

>> Does it have any alignment requirements?
> 
> No, just padding. AVPacket doesn't require alignment for the payload.

I think say that explicitly.  avcodec_default_get_encoder_buffer() does give you aligned memory, even though it isn't needed.

>>> +     * - buf must contain a pointer to an AVBufferRef structure. The packet's
>>> +     *   data pointer must be contained in it.
>>> +     *   See: av_buffer_create(), av_buffer_alloc(), and av_buffer_ref().
>>> +     *
>>> +     * If AV_CODEC_CAP_DR1 is not set then get_encoder_buffer() must call
>>> +     * avcodec_default_get_encoder_buffer() instead of providing a buffer allocated by
>>> +     * some other means.
>>> +     *
>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in flags then the packet may be reused
>>> +     * (read and/or written to if it is writable) later by libavcodec.
>>> +     *
>>> +     * This callback must be thread-safe, as when frame multithreading is used, it may
>>> +     * be called from multiple threads simultaneously.
>>
>> Allowing simulatenous calls feels unexpectedly tricky.  Is it really necessary?
> 
> This was a suggestion by Lynne, i personally don't know. We support frame threading encoding (For intra-only codecs), but currently ff_alloc_packet2() does not seem to be thread safe, seeing it calls av_fast_padded_malloc(), yet it's called by frame threaded encoders.
> Should i remove this?

I don't know, I was asking only because it sounds tricky.  For cases with a limited number of buffers available (like memory-mapped devices) you are going to need locking anyway, so maybe rentrancy adds no additional inconvenience.

>>> +     *
>>> +     * @see avcodec_default_get_encoder_buffer()
>>> +     *
>>> +     * - encoding: Set by libavcodec, user can override.
>>> +     * - decoding: unused
>>> +     */
>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s, AVPacket *pkt, int flags);
>>
>> Can the encoder ask for arbitrarily many packets?
>>
>> Can the user return "not yet" somehow to this if they have a fixed output buffer pool but no buffer is currently available?
> 
> No, as is it can't. Return values < 0 are considered errors.
> 
>>
>> I don't much like the idea of the user suspending the thread in the callback until they have some available, which might work in some cases but might also deadlock if an avcodec_receive_packet() call is blocked by it.
> 
> Can we make what's in essence a malloc() call return something like EAGAIN, and this in turn be propagated back to encode_receive_packet_internal()?

Maybe, or if it has many threads maybe it could wait for something else to finish first.

> Couldn't this potentially end up in the forbidden scenario of avcodec_send_frame() and avcodec_receive_packet() both returning EAGAIN?

Yes.  If the forbidden case happens then the encoder is stuck anyway and can't make any forward progress so we need to error out properly, but the EAGAIN return isn't needed if there is something else to do on another thread.

>>
>> (For get_buffer2 we have a bit of consideration of this problem for hardware frames which might have limited numbers via avcodec_get_hw_frames_parameters(), though the general case does not have a solution.)
>>
>>>   } AVCodecContext;
>>>   #if FF_API_CODEC_GET_SET
>>> @@ -2920,6 +2958,13 @@ void avsubtitle_free(AVSubtitle *sub);
>>>    */
>>>   int avcodec_default_get_buffer2(AVCodecContext *s, AVFrame *frame, int flags);
>>> +/**
>>> + * The default callback for AVCodecContext.get_encoder_buffer(). It is made public so
>>> + * it can be called by custom get_encoder_buffer() implementations for encoders without
>>> + * AV_CODEC_CAP_DR1 set.
>>> + */
>>> +int avcodec_default_get_encoder_buffer(AVCodecContext *s, AVPacket *pkt, int flags);
>>> +
>>>   /**
>>>    * Modify width and height values so that they will result in a memory
>>>    * buffer that is acceptable for the codec if you do not use any horizontal
>>> diff --git a/libavcodec/codec.h b/libavcodec/codec.h
>>> index 0ccbf0eb19..a679fdc9e1 100644
>>> --- a/libavcodec/codec.h
>>> +++ b/libavcodec/codec.h
>>> @@ -43,9 +43,11 @@
>>>    */
>>>   #define AV_CODEC_CAP_DRAW_HORIZ_BAND     (1 <<  0)
>>>   /**
>>> - * Codec uses get_buffer() for allocating buffers and supports custom allocators.
>>> - * If not set, it might not use get_buffer() at all or use operations that
>>> - * assume the buffer was allocated by avcodec_default_get_buffer.
>>> + * Codec uses get_buffer() or get_encoder_buffer() for allocating buffers and
>>> + * supports custom allocators.
>>> + * If not set, it might not use get_buffer() or get_encoder_buffer() at all, or
>>> + * use operations that assume the buffer was allocated by
>>> + * avcodec_default_get_buffer2 or avcodec_default_get_encoder_buffer.
>>>    */
>>>   #define AV_CODEC_CAP_DR1                 (1 <<  1)
>>>   #define AV_CODEC_CAP_TRUNCATED           (1 <<  3)
>>> diff --git a/libavcodec/encode.c b/libavcodec/encode.c
>>> index 282337e453..f39c8d38ce 100644
>>> --- a/libavcodec/encode.c
>>> +++ b/libavcodec/encode.c
>>> @@ -56,6 +56,52 @@ int ff_alloc_packet2(AVCodecContext *avctx, AVPacket *avpkt, int64_t size, int64
>>>       return 0;
>>>   }
>>> +int avcodec_default_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, int flags)
>>> +{
>>> +    int ret;
>>> +
>>> +    if (avpkt->data || avpkt->buf) {
>>> +        av_log(avctx, AV_LOG_ERROR, "avpkt->{data,buf} != NULL in avcodec_default_get_encoder_buffer()\n");
>>> +        return AVERROR(EINVAL);
>>> +    }
>>> +
>>> +    ret = av_new_packet(avpkt, avpkt->size);
>>> +    if (ret < 0)
>>> +        av_log(avctx, AV_LOG_ERROR, "Failed to allocate packet of size %d\n", avpkt->size);
>>> +
>>> +    return ret;
>>> +}
>>> +
>>> +int ff_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, int64_t size, int flags)
>>> +{
>>> +    int ret;
>>> +
>>> +    if (size < 0 || size > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE)
>>> +        return AVERROR(EINVAL);
>>> +
>>> +    av_assert0(!avpkt->data && !avpkt->buf);
>>> +
>>> +    avpkt->size = size;
>>> +    ret = avctx->get_encoder_buffer(avctx, avpkt, flags);
>>> +    if (ret < 0)
>>> +        goto fail;
>>> +
>>> +    if (!avpkt->data || !avpkt->buf) {
>>> +        av_log(avctx, AV_LOG_ERROR, "No buffer returned by get_encoder_buffer()\n");
>>> +        ret = AVERROR(EINVAL);
>>> +        goto fail;
>>> +    }
>>> +
>>> +    ret = 0;
>>> +fail:
>>> +    if (ret < 0) {
>>> +        av_log(avctx, AV_LOG_ERROR, "get_encoder_buffer() failed\n");
>>> +        av_packet_unref(avpkt);
>>> +    }
>>> +
>>> +    return ret;
>>> +}
>>> +
>>>   /**
>>>    * Pad last frame with silence.
>>>    */
>>> @@ -169,7 +215,7 @@ static int encode_simple_internal(AVCodecContext *avctx, AVPacket *avpkt)
>>>       emms_c();
>>>       if (!ret && got_packet) {
>>> -        if (avpkt->data) {
>>> +        if (avpkt->data && !(avctx->codec->capabilities & AV_CODEC_CAP_DR1)) {
>>>               ret = av_packet_make_refcounted(avpkt);
>>>               if (ret < 0)
>>>                   goto end;
>>> @@ -377,6 +423,12 @@ static int compat_encode(AVCodecContext *avctx, AVPacket *avpkt,
>>>               av_log(avctx, AV_LOG_WARNING, "AVFrame.width or height is not set\n");
>>>       }
>>> +    if (avctx->codec->capabilities & AV_CODEC_CAP_DR1) {
>>> +        av_log(avctx, AV_LOG_WARNING, "The deprecated avcodec_encode_* API does not support "
>>> +                                      "AV_CODEC_CAP_DR1 encoders\n");
>>> +        return AVERROR(ENOSYS);
>>> +    }
>>> +
>>>       ret = avcodec_send_frame(avctx, frame);
>>>       if (ret == AVERROR_EOF)
>>>           ret = 0;
>>> diff --git a/libavcodec/encode.h b/libavcodec/encode.h
>>> index dfa9cb2d97..3192bd9e38 100644
>>> --- a/libavcodec/encode.h
>>> +++ b/libavcodec/encode.h
>>> @@ -24,6 +24,7 @@
>>>   #include "libavutil/frame.h"
>>>   #include "avcodec.h"
>>> +#include "packet.h"
>>>   /**
>>>    * Called by encoders to get the next frame for encoding.
>>> @@ -36,4 +37,11 @@
>>>    */
>>>   int ff_encode_get_frame(AVCodecContext *avctx, AVFrame *frame);
>>> +/**
>>> + * Get a buffer for a packet. This is a wrapper around
>>> + * AVCodecContext.get_encoder_buffer() and should be used instead calling get_encoder_buffer()
>>> + * directly.
>>> + */
>>> +int ff_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, int64_t size, int flags);
>>
>> Why int64_t rather than size_t?
> 
> That's what ff_alloc_packet2() uses, so i figured it will make porting existing users much easier (basically just a sed replace, vs trying to find if any encoder is assuming int64_t, like exrenc).

Fair enough.

>>> +
>>>   #endif /* AVCODEC_ENCODE_H */
>>> diff --git a/libavcodec/options.c b/libavcodec/options.c
>>> index 4bbf74ec7f..cd5fa6eb14 100644
>>> --- a/libavcodec/options.c
>>> +++ b/libavcodec/options.c
>>> @@ -130,6 +130,7 @@ static int init_context_defaults(AVCodecContext *s, const AVCodec *codec)
>>>       s->pkt_timebase        = (AVRational){ 0, 1 };
>>>       s->get_buffer2         = avcodec_default_get_buffer2;
>>>       s->get_format          = avcodec_default_get_format;
>>> +    s->get_encoder_buffer  = avcodec_default_get_encoder_buffer;
>>>       s->execute             = avcodec_default_execute;
>>>       s->execute2            = avcodec_default_execute2;
>>>       s->sample_aspect_ratio = (AVRational){0,1};
>>>
>>
>> It might help to have an implementation for at least one codec and an example user for review.
> 
> Sure, below is huffyuvenc adapted to use this API.
> 
>> diff --git a/libavcodec/huffyuvenc.c b/libavcodec/huffyuvenc.c
>> index 28a5534535..709a5a9d0e 100644
>> --- a/libavcodec/huffyuvenc.c
>> +++ b/libavcodec/huffyuvenc.c
>> @@ -29,6 +29,7 @@
>>   */
>>
>>  #include "avcodec.h"
>> +#include "encode.h"
>>  #include "huffyuv.h"
>>  #include "huffman.h"
>>  #include "huffyuvencdsp.h"
>> @@ -762,7 +763,7 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
>>      const AVFrame * const p = pict;
>>      int i, j, size = 0, ret;
>>
>> -    if ((ret = ff_alloc_packet2(avctx, pkt, width * height * 3 * 4 + AV_INPUT_BUFFER_MIN_SIZE, 0)) < 0)
>> +    if ((ret = ff_get_encoder_buffer(avctx, pkt, width * height * 3 * 4 + AV_INPUT_BUFFER_MIN_SIZE, 0)) < 0)
>>          return ret;
>>
>>      if (s->context) {
>> @@ -1111,7 +1112,7 @@ AVCodec ff_ffvhuff_encoder = {
>>      .init           = encode_init,
>>      .encode2        = encode_frame,
>>      .close          = encode_end,
>> -    .capabilities   = AV_CODEC_CAP_FRAME_THREADS,
>> +    .capabilities   = AV_CODEC_CAP_FRAME_THREADS | AV_CODEC_CAP_DR1,

(That's applying to ffvhuff only, not huffyuv.)

>>      .priv_class     = &ff_class,
>>      .pix_fmts       = (const enum AVPixelFormat[]){
>>          AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUV411P,
Yeah, I guess it's actually very easy for most encoders.  Did you have a particular use-case in mind for the user side?

Thanks,

- Mark

On 2/21/2021 5:29 PM, Mark Thompson wrote:
> On 21/02/2021 20:00, James Almer wrote:
>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>> On 21/02/2021 17:35, James Almer wrote:
>>>> This callback is functionally the same as get_buffer2() is for 
>>>> decoders, and
>>>> implements for the new encode API the functionality of the old 
>>>> encode API had
>>>> where the user could provide their own buffers.
>>>>
>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>> ---
>>>> Used the names Lynne suggested this time, plus a line about how the 
>>>> callback
>>>> must be thread safe.
>>>>
>>>>   libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>   libavcodec/codec.h   |  8 ++++---
>>>>   libavcodec/encode.c  | 54 
>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>   libavcodec/encode.h  |  8 +++++++
>>>>   libavcodec/options.c |  1 +
>>>>   5 files changed, 112 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>> index 7dbf083a24..e60eb16ce1 100644
>>>> --- a/libavcodec/avcodec.h
>>>> +++ b/libavcodec/avcodec.h
>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>    */
>>>>   #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>> +/**
>>>> + * The encoder will keep a reference to the packet and may reuse it 
>>>> later.
>>>> + */
>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>> +
>>>>   struct AVCodecInternal;
>>>>   /**
>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>        * - encoding: set by user
>>>>        */
>>>>       int export_side_data;
>>>> +
>>>> +    /**
>>>> +     * This callback is called at the beginning of each packet to 
>>>> get a data
>>>> +     * buffer for it.
>>>> +     *
>>>> +     * The following field will be set in the packet before this 
>>>> callback is
>>>> +     * called:
>>>> +     * - size
>>>> +     * This callback must use the above value to calculate the 
>>>> required buffer size,
>>>> +     * which must padded by at least AV_INPUT_BUFFER_PADDING_SIZE 
>>>> bytes.
>>>> +     *
>>>> +     * This callback must fill the following fields in the packet:
>>>> +     * - data
>>>
>>> Is the data pointer allowed to be in write-only memory?
>>
>> I'm not sure what the use case for this would be, so probably no?
> 
> The two use-cases I see for this API are:
> 
> * You want to avoid a copy when combining the output with something 
> else.  E.g. you pass a pointer to the block of memory following where 
> you are going to put your header data (for something you are going to 
> send over the network, say).
> 
> * You want to avoid a copy when passing the output directly to something 
> external.  E.g. you pass a pointer to a memory-mapped device buffer 
> (such as a V4L2 buffer, say).
> 
> In the second case, write-only memory on an external device seems 
> possible, as does memory which is, say, readable but uncached, so 
> reading it is a really bad idea.

Allowing the second case would depend on how encoders behave. Some may 
attempt to read data already written to the output packet. It's not like 
all of them allocate the packet, do a memcpy from an internal buffer, 
then return.
There is also the flag meant to signal that the encoder will keep a 
reference to the packet around, which more or less implies it will be 
read later in the encoding process.

The doxy for avcodec_encode_video2(), which allowed the user to provide 
their own buffers in the output packet, does not mention any kind of 
requirement for the data pointer, so I don't think we can say it's an 
allowed scenario here either.

> 
>>> Does it have any alignment requirements?
>>
>> No, just padding. AVPacket doesn't require alignment for the payload.
> 
> I think say that explicitly.  avcodec_default_get_encoder_buffer() does 
> give you aligned memory, even though it isn't needed.

Would saying "There's no alignment requirement for the data pointer" add 
anything of value to the doxy? If i don't mention any kind of alignment 
requirement, it's because there isn't any, and it's implicit.
I listed the requirements the user needs to keep in mind, like the 
padding and the need for an AVBufferRef. But if you think it's worth 
adding, then sure.

> 
>>>> +     * - buf must contain a pointer to an AVBufferRef structure. 
>>>> The packet's
>>>> +     *   data pointer must be contained in it.
>>>> +     *   See: av_buffer_create(), av_buffer_alloc(), and 
>>>> av_buffer_ref().
>>>> +     *
>>>> +     * If AV_CODEC_CAP_DR1 is not set then get_encoder_buffer() 
>>>> must call
>>>> +     * avcodec_default_get_encoder_buffer() instead of providing a 
>>>> buffer allocated by
>>>> +     * some other means.
>>>> +     *
>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in flags then the 
>>>> packet may be reused
>>>> +     * (read and/or written to if it is writable) later by libavcodec.
>>>> +     *
>>>> +     * This callback must be thread-safe, as when frame 
>>>> multithreading is used, it may
>>>> +     * be called from multiple threads simultaneously.
>>>
>>> Allowing simulatenous calls feels unexpectedly tricky.  Is it really 
>>> necessary?
>>
>> This was a suggestion by Lynne, i personally don't know. We support 
>> frame threading encoding (For intra-only codecs), but currently 
>> ff_alloc_packet2() does not seem to be thread safe, seeing it calls 
>> av_fast_padded_malloc(), yet it's called by frame threaded encoders.
>> Should i remove this?
> 
> I don't know, I was asking only because it sounds tricky.  For cases 
> with a limited number of buffers available (like memory-mapped devices) 
> you are going to need locking anyway, so maybe rentrancy adds no 
> additional inconvenience.
> 
>>>> +     *
>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>> +     *
>>>> +     * - encoding: Set by libavcodec, user can override.
>>>> +     * - decoding: unused
>>>> +     */
>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s, AVPacket 
>>>> *pkt, int flags);
>>>
>>> Can the encoder ask for arbitrarily many packets?
>>>
>>> Can the user return "not yet" somehow to this if they have a fixed 
>>> output buffer pool but no buffer is currently available?
>>
>> No, as is it can't. Return values < 0 are considered errors.
>>
>>>
>>> I don't much like the idea of the user suspending the thread in the 
>>> callback until they have some available, which might work in some 
>>> cases but might also deadlock if an avcodec_receive_packet() call is 
>>> blocked by it.
>>
>> Can we make what's in essence a malloc() call return something like 
>> EAGAIN, and this in turn be propagated back to 
>> encode_receive_packet_internal()?
> 
> Maybe, or if it has many threads maybe it could wait for something else 
> to finish first.
> 
>> Couldn't this potentially end up in the forbidden scenario of 
>> avcodec_send_frame() and avcodec_receive_packet() both returning EAGAIN?
> 
> Yes.  If the forbidden case happens then the encoder is stuck anyway and 
> can't make any forward progress so we need to error out properly, but 
> the EAGAIN return isn't needed if there is something else to do on 
> another thread.

Ok, but I'm not familiar or knowledgeable enough with the frame thread 
encoder code to implement this.

> 
>>>
>>> (For get_buffer2 we have a bit of consideration of this problem for 
>>> hardware frames which might have limited numbers via 
>>> avcodec_get_hw_frames_parameters(), though the general case does not 
>>> have a solution.)
>>>
>>>>   } AVCodecContext;
>>>>   #if FF_API_CODEC_GET_SET
>>>> @@ -2920,6 +2958,13 @@ void avsubtitle_free(AVSubtitle *sub);
>>>>    */
>>>>   int avcodec_default_get_buffer2(AVCodecContext *s, AVFrame *frame, 
>>>> int flags);
>>>> +/**
>>>> + * The default callback for AVCodecContext.get_encoder_buffer(). It 
>>>> is made public so
>>>> + * it can be called by custom get_encoder_buffer() implementations 
>>>> for encoders without
>>>> + * AV_CODEC_CAP_DR1 set.
>>>> + */
>>>> +int avcodec_default_get_encoder_buffer(AVCodecContext *s, AVPacket 
>>>> *pkt, int flags);
>>>> +
>>>>   /**
>>>>    * Modify width and height values so that they will result in a 
>>>> memory
>>>>    * buffer that is acceptable for the codec if you do not use any 
>>>> horizontal
>>>> diff --git a/libavcodec/codec.h b/libavcodec/codec.h
>>>> index 0ccbf0eb19..a679fdc9e1 100644
>>>> --- a/libavcodec/codec.h
>>>> +++ b/libavcodec/codec.h
>>>> @@ -43,9 +43,11 @@
>>>>    */
>>>>   #define AV_CODEC_CAP_DRAW_HORIZ_BAND     (1 <<  0)
>>>>   /**
>>>> - * Codec uses get_buffer() for allocating buffers and supports 
>>>> custom allocators.
>>>> - * If not set, it might not use get_buffer() at all or use 
>>>> operations that
>>>> - * assume the buffer was allocated by avcodec_default_get_buffer.
>>>> + * Codec uses get_buffer() or get_encoder_buffer() for allocating 
>>>> buffers and
>>>> + * supports custom allocators.
>>>> + * If not set, it might not use get_buffer() or 
>>>> get_encoder_buffer() at all, or
>>>> + * use operations that assume the buffer was allocated by
>>>> + * avcodec_default_get_buffer2 or avcodec_default_get_encoder_buffer.
>>>>    */
>>>>   #define AV_CODEC_CAP_DR1                 (1 <<  1)
>>>>   #define AV_CODEC_CAP_TRUNCATED           (1 <<  3)
>>>> diff --git a/libavcodec/encode.c b/libavcodec/encode.c
>>>> index 282337e453..f39c8d38ce 100644
>>>> --- a/libavcodec/encode.c
>>>> +++ b/libavcodec/encode.c
>>>> @@ -56,6 +56,52 @@ int ff_alloc_packet2(AVCodecContext *avctx, 
>>>> AVPacket *avpkt, int64_t size, int64
>>>>       return 0;
>>>>   }
>>>> +int avcodec_default_get_encoder_buffer(AVCodecContext *avctx, 
>>>> AVPacket *avpkt, int flags)
>>>> +{
>>>> +    int ret;
>>>> +
>>>> +    if (avpkt->data || avpkt->buf) {
>>>> +        av_log(avctx, AV_LOG_ERROR, "avpkt->{data,buf} != NULL in 
>>>> avcodec_default_get_encoder_buffer()\n");
>>>> +        return AVERROR(EINVAL);
>>>> +    }
>>>> +
>>>> +    ret = av_new_packet(avpkt, avpkt->size);
>>>> +    if (ret < 0)
>>>> +        av_log(avctx, AV_LOG_ERROR, "Failed to allocate packet of 
>>>> size %d\n", avpkt->size);
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +int ff_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, 
>>>> int64_t size, int flags)
>>>> +{
>>>> +    int ret;
>>>> +
>>>> +    if (size < 0 || size > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE)
>>>> +        return AVERROR(EINVAL);
>>>> +
>>>> +    av_assert0(!avpkt->data && !avpkt->buf);
>>>> +
>>>> +    avpkt->size = size;
>>>> +    ret = avctx->get_encoder_buffer(avctx, avpkt, flags);
>>>> +    if (ret < 0)
>>>> +        goto fail;
>>>> +
>>>> +    if (!avpkt->data || !avpkt->buf) {
>>>> +        av_log(avctx, AV_LOG_ERROR, "No buffer returned by 
>>>> get_encoder_buffer()\n");
>>>> +        ret = AVERROR(EINVAL);
>>>> +        goto fail;
>>>> +    }
>>>> +
>>>> +    ret = 0;
>>>> +fail:
>>>> +    if (ret < 0) {
>>>> +        av_log(avctx, AV_LOG_ERROR, "get_encoder_buffer() failed\n");
>>>> +        av_packet_unref(avpkt);
>>>> +    }
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>>   /**
>>>>    * Pad last frame with silence.
>>>>    */
>>>> @@ -169,7 +215,7 @@ static int encode_simple_internal(AVCodecContext 
>>>> *avctx, AVPacket *avpkt)
>>>>       emms_c();
>>>>       if (!ret && got_packet) {
>>>> -        if (avpkt->data) {
>>>> +        if (avpkt->data && !(avctx->codec->capabilities & 
>>>> AV_CODEC_CAP_DR1)) {
>>>>               ret = av_packet_make_refcounted(avpkt);
>>>>               if (ret < 0)
>>>>                   goto end;
>>>> @@ -377,6 +423,12 @@ static int compat_encode(AVCodecContext *avctx, 
>>>> AVPacket *avpkt,
>>>>               av_log(avctx, AV_LOG_WARNING, "AVFrame.width or height 
>>>> is not set\n");
>>>>       }
>>>> +    if (avctx->codec->capabilities & AV_CODEC_CAP_DR1) {
>>>> +        av_log(avctx, AV_LOG_WARNING, "The deprecated 
>>>> avcodec_encode_* API does not support "
>>>> +                                      "AV_CODEC_CAP_DR1 encoders\n");
>>>> +        return AVERROR(ENOSYS);
>>>> +    }
>>>> +
>>>>       ret = avcodec_send_frame(avctx, frame);
>>>>       if (ret == AVERROR_EOF)
>>>>           ret = 0;
>>>> diff --git a/libavcodec/encode.h b/libavcodec/encode.h
>>>> index dfa9cb2d97..3192bd9e38 100644
>>>> --- a/libavcodec/encode.h
>>>> +++ b/libavcodec/encode.h
>>>> @@ -24,6 +24,7 @@
>>>>   #include "libavutil/frame.h"
>>>>   #include "avcodec.h"
>>>> +#include "packet.h"
>>>>   /**
>>>>    * Called by encoders to get the next frame for encoding.
>>>> @@ -36,4 +37,11 @@
>>>>    */
>>>>   int ff_encode_get_frame(AVCodecContext *avctx, AVFrame *frame);
>>>> +/**
>>>> + * Get a buffer for a packet. This is a wrapper around
>>>> + * AVCodecContext.get_encoder_buffer() and should be used instead 
>>>> calling get_encoder_buffer()
>>>> + * directly.
>>>> + */
>>>> +int ff_get_encoder_buffer(AVCodecContext *avctx, AVPacket *avpkt, 
>>>> int64_t size, int flags);
>>>
>>> Why int64_t rather than size_t?
>>
>> That's what ff_alloc_packet2() uses, so i figured it will make porting 
>> existing users much easier (basically just a sed replace, vs trying to 
>> find if any encoder is assuming int64_t, like exrenc).
> 
> Fair enough.
> 
>>>> +
>>>>   #endif /* AVCODEC_ENCODE_H */
>>>> diff --git a/libavcodec/options.c b/libavcodec/options.c
>>>> index 4bbf74ec7f..cd5fa6eb14 100644
>>>> --- a/libavcodec/options.c
>>>> +++ b/libavcodec/options.c
>>>> @@ -130,6 +130,7 @@ static int init_context_defaults(AVCodecContext 
>>>> *s, const AVCodec *codec)
>>>>       s->pkt_timebase        = (AVRational){ 0, 1 };
>>>>       s->get_buffer2         = avcodec_default_get_buffer2;
>>>>       s->get_format          = avcodec_default_get_format;
>>>> +    s->get_encoder_buffer  = avcodec_default_get_encoder_buffer;
>>>>       s->execute             = avcodec_default_execute;
>>>>       s->execute2            = avcodec_default_execute2;
>>>>       s->sample_aspect_ratio = (AVRational){0,1};
>>>>
>>>
>>> It might help to have an implementation for at least one codec and an 
>>> example user for review.
>>
>> Sure, below is huffyuvenc adapted to use this API.
>>
>>> diff --git a/libavcodec/huffyuvenc.c b/libavcodec/huffyuvenc.c
>>> index 28a5534535..709a5a9d0e 100644
>>> --- a/libavcodec/huffyuvenc.c
>>> +++ b/libavcodec/huffyuvenc.c
>>> @@ -29,6 +29,7 @@
>>>   */
>>>
>>>  #include "avcodec.h"
>>> +#include "encode.h"
>>>  #include "huffyuv.h"
>>>  #include "huffman.h"
>>>  #include "huffyuvencdsp.h"
>>> @@ -762,7 +763,7 @@ static int encode_frame(AVCodecContext *avctx, 
>>> AVPacket *pkt,
>>>      const AVFrame * const p = pict;
>>>      int i, j, size = 0, ret;
>>>
>>> -    if ((ret = ff_alloc_packet2(avctx, pkt, width * height * 3 * 4 + 
>>> AV_INPUT_BUFFER_MIN_SIZE, 0)) < 0)
>>> +    if ((ret = ff_get_encoder_buffer(avctx, pkt, width * height * 3 
>>> * 4 + AV_INPUT_BUFFER_MIN_SIZE, 0)) < 0)
>>>          return ret;
>>>
>>>      if (s->context) {
>>> @@ -1111,7 +1112,7 @@ AVCodec ff_ffvhuff_encoder = {
>>>      .init           = encode_init,
>>>      .encode2        = encode_frame,
>>>      .close          = encode_end,
>>> -    .capabilities   = AV_CODEC_CAP_FRAME_THREADS,
>>> +    .capabilities   = AV_CODEC_CAP_FRAME_THREADS | AV_CODEC_CAP_DR1,
> 
> (That's applying to ffvhuff only, not huffyuv.)
> 
>>>      .priv_class     = &ff_class,
>>>      .pix_fmts       = (const enum AVPixelFormat[]){
>>>          AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV444P, 
>>> AV_PIX_FMT_YUV411P,
> Yeah, I guess it's actually very easy for most encoders.  Did you have a 
> particular use-case in mind for the user side?

No, it was suggested by Anton so i figured I'd implement it. It lets 
users of the new decoupled send/receive encode API provide their own 
buffers like they could with the old encode API, so the one missing 
attractive feature of the old API is now also available in the new.

> 
> Thanks,
> 
> - Mark
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Feb 21, 2021, 22:04 by jamrial@gmail.com:

> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>
>> On 21/02/2021 20:00, James Almer wrote:
>>
>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>
>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>
>>>>> This callback is functionally the same as get_buffer2() is for decoders, and
>>>>> implements for the new encode API the functionality of the old encode API had
>>>>> where the user could provide their own buffers.
>>>>>
>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>> ---
>>>>> Used the names Lynne suggested this time, plus a line about how the callback
>>>>> must be thread safe.
>>>>>
>>>>>   libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>   libavcodec/codec.h   |  8 ++++---
>>>>>   libavcodec/encode.c  | 54 +++++++++++++++++++++++++++++++++++++++++++-
>>>>>   libavcodec/encode.h  |  8 +++++++
>>>>>   libavcodec/options.c |  1 +
>>>>>   5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>> --- a/libavcodec/avcodec.h
>>>>> +++ b/libavcodec/avcodec.h
>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>    */
>>>>>   #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>> +/**
>>>>> + * The encoder will keep a reference to the packet and may reuse it later.
>>>>> + */
>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>> +
>>>>>   struct AVCodecInternal;
>>>>>   /**
>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>        * - encoding: set by user
>>>>>        */
>>>>>       int export_side_data;
>>>>> +
>>>>> +    /**
>>>>> +     * This callback is called at the beginning of each packet to get a data
>>>>> +     * buffer for it.
>>>>> +     *
>>>>> +     * The following field will be set in the packet before this callback is
>>>>> +     * called:
>>>>> +     * - size
>>>>> +     * This callback must use the above value to calculate the required buffer size,
>>>>> +     * which must padded by at least AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>> +     *
>>>>> +     * This callback must fill the following fields in the packet:
>>>>> +     * - data
>>>>>
>>>>
>>>> Is the data pointer allowed to be in write-only memory?
>>>>
>>>
>>> I'm not sure what the use case for this would be, so probably no?
>>>
>>
>> The two use-cases I see for this API are:
>>
>> * You want to avoid a copy when combining the output with something else.  E.g. you pass a pointer to the block of memory following where you are going to put your header data (for something you are going to send over the network, say).
>>
>> * You want to avoid a copy when passing the output directly to something external.  E.g. you pass a pointer to a memory-mapped device buffer (such as a V4L2 buffer, say).
>>
>> In the second case, write-only memory on an external device seems possible, as does memory which is, say, readable but uncached, so reading it is a really bad idea.
>>
>
> Allowing the second case would depend on how encoders behave. Some may attempt to read data already written to the output packet. It's not like all of them allocate the packet, do a memcpy from an internal buffer, then return.
> There is also the flag meant to signal that the encoder will keep a reference to the packet around, which more or less implies it will be read later in the encoding process.
>

+1. That was one reason I wanted the flag kept.


>>>> Does it have any alignment requirements?
>>>>
>>>
>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>>
>>
>> I think say that explicitly.  avcodec_default_get_encoder_buffer() does give you aligned memory, even though it isn't needed.
>>
>
> Would saying "There's no alignment requirement for the data pointer" add anything of value to the doxy? If i don't mention any kind of alignment requirement, it's because there isn't any, and it's implicit.
> I listed the requirements the user needs to keep in mind, like the padding and the need for an AVBufferRef. But if you think it's worth adding, then sure.
>

I definitely think packet buffer alignment should be mandated. v210 and various
packing/unpacking codecs' SIMD depend on this.
Should have the same rules as the alignment on AVFrames.


>>>>> +     * - buf must contain a pointer to an AVBufferRef structure. The packet's
>>>>> +     *   data pointer must be contained in it.
>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(), and av_buffer_ref().
>>>>> +     *
>>>>> +     * If AV_CODEC_CAP_DR1 is not set then get_encoder_buffer() must call
>>>>> +     * avcodec_default_get_encoder_buffer() instead of providing a buffer allocated by
>>>>> +     * some other means.
>>>>> +     *
>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in flags then the packet may be reused
>>>>> +     * (read and/or written to if it is writable) later by libavcodec.
>>>>> +     *
>>>>> +     * This callback must be thread-safe, as when frame multithreading is used, it may
>>>>> +     * be called from multiple threads simultaneously.
>>>>>
>>>>
>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is it really necessary?
>>>>
>>>
>>> This was a suggestion by Lynne, i personally don't know. We support frame threading encoding (For intra-only codecs), but currently ff_alloc_packet2() does not seem to be thread safe, seeing it calls av_fast_padded_malloc(), yet it's called by frame threaded encoders.
>>> Should i remove this?
>>>
>>
>> I don't know, I was asking only because it sounds tricky.  For cases with a limited number of buffers available (like memory-mapped devices) you are going to need locking anyway, so maybe rentrancy adds no additional inconvenience.
>>

Are there any actual devices or APIs which put a limit on the number of packets a user
can have at one point? I'm not aware of any.
I think users should handle locking themselves if they need to.

On 2/21/2021 6:04 PM, James Almer wrote:
> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>> On 21/02/2021 20:00, James Almer wrote:
>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>> This callback is functionally the same as get_buffer2() is for 
>>>>> decoders, and
>>>>> implements for the new encode API the functionality of the old 
>>>>> encode API had
>>>>> where the user could provide their own buffers.
>>>>>
>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>> ---
>>>>> Used the names Lynne suggested this time, plus a line about how the 
>>>>> callback
>>>>> must be thread safe.
>>>>>
>>>>>   libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>   libavcodec/codec.h   |  8 ++++---
>>>>>   libavcodec/encode.c  | 54 
>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>   libavcodec/encode.h  |  8 +++++++
>>>>>   libavcodec/options.c |  1 +
>>>>>   5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>> --- a/libavcodec/avcodec.h
>>>>> +++ b/libavcodec/avcodec.h
>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>    */
>>>>>   #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>> +/**
>>>>> + * The encoder will keep a reference to the packet and may reuse 
>>>>> it later.
>>>>> + */
>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>> +
>>>>>   struct AVCodecInternal;
>>>>>   /**
>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>        * - encoding: set by user
>>>>>        */
>>>>>       int export_side_data;
>>>>> +
>>>>> +    /**
>>>>> +     * This callback is called at the beginning of each packet to 
>>>>> get a data
>>>>> +     * buffer for it.
>>>>> +     *
>>>>> +     * The following field will be set in the packet before this 
>>>>> callback is
>>>>> +     * called:
>>>>> +     * - size
>>>>> +     * This callback must use the above value to calculate the 
>>>>> required buffer size,
>>>>> +     * which must padded by at least AV_INPUT_BUFFER_PADDING_SIZE 
>>>>> bytes.
>>>>> +     *
>>>>> +     * This callback must fill the following fields in the packet:
>>>>> +     * - data
>>>>
>>>> Is the data pointer allowed to be in write-only memory?
>>>
>>> I'm not sure what the use case for this would be, so probably no?
>>
>> The two use-cases I see for this API are:
>>
>> * You want to avoid a copy when combining the output with something 
>> else.  E.g. you pass a pointer to the block of memory following where 
>> you are going to put your header data (for something you are going to 
>> send over the network, say).
>>
>> * You want to avoid a copy when passing the output directly to 
>> something external.  E.g. you pass a pointer to a memory-mapped device 
>> buffer (such as a V4L2 buffer, say).
>>
>> In the second case, write-only memory on an external device seems 
>> possible, as does memory which is, say, readable but uncached, so 
>> reading it is a really bad idea.
> 
> Allowing the second case would depend on how encoders behave. Some may 
> attempt to read data already written to the output packet. It's not like 
> all of them allocate the packet, do a memcpy from an internal buffer, 
> then return.
> There is also the flag meant to signal that the encoder will keep a 
> reference to the packet around, which more or less implies it will be 
> read later in the encoding process.
> 
> The doxy for avcodec_encode_video2(), which allowed the user to provide 
> their own buffers in the output packet, does not mention any kind of 
> requirement for the data pointer, so I don't think we can say it's an 
> allowed scenario here either.
> 
>>
>>>> Does it have any alignment requirements?
>>>
>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>
>> I think say that explicitly.  avcodec_default_get_encoder_buffer() 
>> does give you aligned memory, even though it isn't needed.
> 
> Would saying "There's no alignment requirement for the data pointer" add 
> anything of value to the doxy? If i don't mention any kind of alignment 
> requirement, it's because there isn't any, and it's implicit.
> I listed the requirements the user needs to keep in mind, like the 
> padding and the need for an AVBufferRef. But if you think it's worth 
> adding, then sure.
> 
>>
>>>>> +     * - buf must contain a pointer to an AVBufferRef structure. 
>>>>> The packet's
>>>>> +     *   data pointer must be contained in it.
>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(), and 
>>>>> av_buffer_ref().
>>>>> +     *
>>>>> +     * If AV_CODEC_CAP_DR1 is not set then get_encoder_buffer() 
>>>>> must call
>>>>> +     * avcodec_default_get_encoder_buffer() instead of providing a 
>>>>> buffer allocated by
>>>>> +     * some other means.
>>>>> +     *
>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in flags then the 
>>>>> packet may be reused
>>>>> +     * (read and/or written to if it is writable) later by 
>>>>> libavcodec.
>>>>> +     *
>>>>> +     * This callback must be thread-safe, as when frame 
>>>>> multithreading is used, it may
>>>>> +     * be called from multiple threads simultaneously.
>>>>
>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is it really 
>>>> necessary?
>>>
>>> This was a suggestion by Lynne, i personally don't know. We support 
>>> frame threading encoding (For intra-only codecs), but currently 
>>> ff_alloc_packet2() does not seem to be thread safe, seeing it calls 
>>> av_fast_padded_malloc(), yet it's called by frame threaded encoders.
>>> Should i remove this?
>>
>> I don't know, I was asking only because it sounds tricky.  For cases 
>> with a limited number of buffers available (like memory-mapped 
>> devices) you are going to need locking anyway, so maybe rentrancy adds 
>> no additional inconvenience.
>>
>>>>> +     *
>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>> +     *
>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>> +     * - decoding: unused
>>>>> +     */
>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s, AVPacket 
>>>>> *pkt, int flags);
>>>>
>>>> Can the encoder ask for arbitrarily many packets?
>>>>
>>>> Can the user return "not yet" somehow to this if they have a fixed 
>>>> output buffer pool but no buffer is currently available?
>>>
>>> No, as is it can't. Return values < 0 are considered errors.
>>>
>>>>
>>>> I don't much like the idea of the user suspending the thread in the 
>>>> callback until they have some available, which might work in some 
>>>> cases but might also deadlock if an avcodec_receive_packet() call is 
>>>> blocked by it.
>>>
>>> Can we make what's in essence a malloc() call return something like 
>>> EAGAIN, and this in turn be propagated back to 
>>> encode_receive_packet_internal()?
>>
>> Maybe, or if it has many threads maybe it could wait for something 
>> else to finish first.
>>
>>> Couldn't this potentially end up in the forbidden scenario of 
>>> avcodec_send_frame() and avcodec_receive_packet() both returning EAGAIN?
>>
>> Yes.  If the forbidden case happens then the encoder is stuck anyway 
>> and can't make any forward progress so we need to error out properly, 
>> but the EAGAIN return isn't needed if there is something else to do on 
>> another thread.
> 
> Ok, but I'm not familiar or knowledgeable enough with the frame thread 
> encoder code to implement this.

Looked at bit into this. AVCodec->encode2() based encoders don't support 
returning EAGAIN at all, as it completely breaks the frame threading 
logic. It would require a considerable rewrite in order to re-add a task 
that didn't fail but also didn't succeed.

Non frame threading encoders could probably support it with some minimal 
changes, but i don't think suddenly letting an scenario that was until 
now guaranteed to never happen start happening (avcodec_send_frame() and 
avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an 
API break.
Letting the user's custom get_encode_buffer() callback suspend the 
thread is IMO acceptable. In frame threading scenarios, the other 
threads are still working on their own packets (afaics none depends on 
the others, since it's intra only encoders only).

On 2/22/2021 7:27 PM, James Almer wrote:
> On 2/21/2021 6:04 PM, James Almer wrote:
>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>> On 21/02/2021 20:00, James Almer wrote:
>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>> This callback is functionally the same as get_buffer2() is for 
>>>>>> decoders, and
>>>>>> implements for the new encode API the functionality of the old 
>>>>>> encode API had
>>>>>> where the user could provide their own buffers.
>>>>>>
>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>> ---
>>>>>> Used the names Lynne suggested this time, plus a line about how 
>>>>>> the callback
>>>>>> must be thread safe.
>>>>>>
>>>>>>   libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>>   libavcodec/codec.h   |  8 ++++---
>>>>>>   libavcodec/encode.c  | 54 
>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>   libavcodec/encode.h  |  8 +++++++
>>>>>>   libavcodec/options.c |  1 +
>>>>>>   5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>
>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>> --- a/libavcodec/avcodec.h
>>>>>> +++ b/libavcodec/avcodec.h
>>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>>    */
>>>>>>   #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>> +/**
>>>>>> + * The encoder will keep a reference to the packet and may reuse 
>>>>>> it later.
>>>>>> + */
>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>> +
>>>>>>   struct AVCodecInternal;
>>>>>>   /**
>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>        * - encoding: set by user
>>>>>>        */
>>>>>>       int export_side_data;
>>>>>> +
>>>>>> +    /**
>>>>>> +     * This callback is called at the beginning of each packet to 
>>>>>> get a data
>>>>>> +     * buffer for it.
>>>>>> +     *
>>>>>> +     * The following field will be set in the packet before this 
>>>>>> callback is
>>>>>> +     * called:
>>>>>> +     * - size
>>>>>> +     * This callback must use the above value to calculate the 
>>>>>> required buffer size,
>>>>>> +     * which must padded by at least AV_INPUT_BUFFER_PADDING_SIZE 
>>>>>> bytes.
>>>>>> +     *
>>>>>> +     * This callback must fill the following fields in the packet:
>>>>>> +     * - data
>>>>>
>>>>> Is the data pointer allowed to be in write-only memory?
>>>>
>>>> I'm not sure what the use case for this would be, so probably no?
>>>
>>> The two use-cases I see for this API are:
>>>
>>> * You want to avoid a copy when combining the output with something 
>>> else.  E.g. you pass a pointer to the block of memory following where 
>>> you are going to put your header data (for something you are going to 
>>> send over the network, say).
>>>
>>> * You want to avoid a copy when passing the output directly to 
>>> something external.  E.g. you pass a pointer to a memory-mapped 
>>> device buffer (such as a V4L2 buffer, say).
>>>
>>> In the second case, write-only memory on an external device seems 
>>> possible, as does memory which is, say, readable but uncached, so 
>>> reading it is a really bad idea.
>>
>> Allowing the second case would depend on how encoders behave. Some may 
>> attempt to read data already written to the output packet. It's not 
>> like all of them allocate the packet, do a memcpy from an internal 
>> buffer, then return.
>> There is also the flag meant to signal that the encoder will keep a 
>> reference to the packet around, which more or less implies it will be 
>> read later in the encoding process.
>>
>> The doxy for avcodec_encode_video2(), which allowed the user to 
>> provide their own buffers in the output packet, does not mention any 
>> kind of requirement for the data pointer, so I don't think we can say 
>> it's an allowed scenario here either.
>>
>>>
>>>>> Does it have any alignment requirements?
>>>>
>>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>>
>>> I think say that explicitly.  avcodec_default_get_encoder_buffer() 
>>> does give you aligned memory, even though it isn't needed.
>>
>> Would saying "There's no alignment requirement for the data pointer" 
>> add anything of value to the doxy? If i don't mention any kind of 
>> alignment requirement, it's because there isn't any, and it's implicit.
>> I listed the requirements the user needs to keep in mind, like the 
>> padding and the need for an AVBufferRef. But if you think it's worth 
>> adding, then sure.
>>
>>>
>>>>>> +     * - buf must contain a pointer to an AVBufferRef structure. 
>>>>>> The packet's
>>>>>> +     *   data pointer must be contained in it.
>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(), and 
>>>>>> av_buffer_ref().
>>>>>> +     *
>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then get_encoder_buffer() 
>>>>>> must call
>>>>>> +     * avcodec_default_get_encoder_buffer() instead of providing 
>>>>>> a buffer allocated by
>>>>>> +     * some other means.
>>>>>> +     *
>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in flags then the 
>>>>>> packet may be reused
>>>>>> +     * (read and/or written to if it is writable) later by 
>>>>>> libavcodec.
>>>>>> +     *
>>>>>> +     * This callback must be thread-safe, as when frame 
>>>>>> multithreading is used, it may
>>>>>> +     * be called from multiple threads simultaneously.
>>>>>
>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is it 
>>>>> really necessary?
>>>>
>>>> This was a suggestion by Lynne, i personally don't know. We support 
>>>> frame threading encoding (For intra-only codecs), but currently 
>>>> ff_alloc_packet2() does not seem to be thread safe, seeing it calls 
>>>> av_fast_padded_malloc(), yet it's called by frame threaded encoders.
>>>> Should i remove this?
>>>
>>> I don't know, I was asking only because it sounds tricky.  For cases 
>>> with a limited number of buffers available (like memory-mapped 
>>> devices) you are going to need locking anyway, so maybe rentrancy 
>>> adds no additional inconvenience.
>>>
>>>>>> +     *
>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>> +     *
>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>> +     * - decoding: unused
>>>>>> +     */
>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s, AVPacket 
>>>>>> *pkt, int flags);
>>>>>
>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>
>>>>> Can the user return "not yet" somehow to this if they have a fixed 
>>>>> output buffer pool but no buffer is currently available?
>>>>
>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>
>>>>>
>>>>> I don't much like the idea of the user suspending the thread in the 
>>>>> callback until they have some available, which might work in some 
>>>>> cases but might also deadlock if an avcodec_receive_packet() call 
>>>>> is blocked by it.
>>>>
>>>> Can we make what's in essence a malloc() call return something like 
>>>> EAGAIN, and this in turn be propagated back to 
>>>> encode_receive_packet_internal()?
>>>
>>> Maybe, or if it has many threads maybe it could wait for something 
>>> else to finish first.
>>>
>>>> Couldn't this potentially end up in the forbidden scenario of 
>>>> avcodec_send_frame() and avcodec_receive_packet() both returning 
>>>> EAGAIN?
>>>
>>> Yes.  If the forbidden case happens then the encoder is stuck anyway 
>>> and can't make any forward progress so we need to error out properly, 
>>> but the EAGAIN return isn't needed if there is something else to do 
>>> on another thread.
>>
>> Ok, but I'm not familiar or knowledgeable enough with the frame thread 
>> encoder code to implement this.
> 
> Looked at bit into this. AVCodec->encode2() based encoders don't support 
> returning EAGAIN at all, as it completely breaks the frame threading 
> logic. It would require a considerable rewrite in order to re-add a task 
> that didn't fail but also didn't succeed.
> 
> Non frame threading encoders could probably support it with some minimal 
> changes, but i don't think suddenly letting an scenario that was until 
> now guaranteed to never happen start happening (avcodec_send_frame() and 
> avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an 
> API break.
> Letting the user's custom get_encode_buffer() callback suspend the 
> thread is IMO acceptable. In frame threading scenarios, the other 
> threads are still working on their own packets (afaics none depends on 
> the others, since it's intra only encoders only).

Ping. I'd like to get this in.

Mar 8, 2021, 21:31 by jamrial@gmail.com:

> On 2/22/2021 7:27 PM, James Almer wrote:
>
>>
>> Looked at bit into this. AVCodec->encode2() based encoders don't support returning EAGAIN at all, as it completely breaks the frame threading logic. It would require a considerable rewrite in order to re-add a task that didn't fail but also didn't succeed.
>>
>> Non frame threading encoders could probably support it with some minimal changes, but i don't think suddenly letting an scenario that was until now guaranteed to never happen start happening (avcodec_send_frame() and avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API break.
>> Letting the user's custom get_encode_buffer() callback suspend the thread is IMO acceptable. In frame threading scenarios, the other threads are still working on their own packets (afaics none depends on the others, since it's intra only encoders only).
>>
>
> Ping. I'd like to get this in.
>

https://ffmpeg.org/pipermail/ffmpeg-devel/2021-February/276679.html

On 3/8/2021 5:48 PM, Lynne wrote:
> Mar 8, 2021, 21:31 by jamrial@gmail.com:
> 
>> On 2/22/2021 7:27 PM, James Almer wrote:
>>
>>>
>>> Looked at bit into this. AVCodec->encode2() based encoders don't support returning EAGAIN at all, as it completely breaks the frame threading logic. It would require a considerable rewrite in order to re-add a task that didn't fail but also didn't succeed.
>>>
>>> Non frame threading encoders could probably support it with some minimal changes, but i don't think suddenly letting an scenario that was until now guaranteed to never happen start happening (avcodec_send_frame() and avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API break.
>>> Letting the user's custom get_encode_buffer() callback suspend the thread is IMO acceptable. In frame threading scenarios, the other threads are still working on their own packets (afaics none depends on the others, since it's intra only encoders only).
>>>
>>
>> Ping. I'd like to get this in.
>>
> 
> https://ffmpeg.org/pipermail/ffmpeg-devel/2021-February/276679.html

You agreed with me, so i didn't think i needed to reply to that email. 
It's Mark who hasn't yet said if he's ok with it.

And regarding to the AVPacket data alignment, that's a separate 
discussion for an API change unrelated to this.

On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
> On 2/21/2021 6:04 PM, James Almer wrote:
> > On 2/21/2021 5:29 PM, Mark Thompson wrote:
> > > On 21/02/2021 20:00, James Almer wrote:
> > > > On 2/21/2021 4:13 PM, Mark Thompson wrote:
> > > > > On 21/02/2021 17:35, James Almer wrote:
> > > > > > This callback is functionally the same as get_buffer2()
> > > > > > is for decoders, and
> > > > > > implements for the new encode API the functionality of
> > > > > > the old encode API had
> > > > > > where the user could provide their own buffers.
> > > > > > 
> > > > > > Signed-off-by: James Almer <jamrial@gmail.com>
> > > > > > ---
> > > > > > Used the names Lynne suggested this time, plus a line
> > > > > > about how the callback
> > > > > > must be thread safe.
> > > > > > 
> > > > > >   libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
> > > > > >   libavcodec/codec.h   |  8 ++++---
> > > > > >   libavcodec/encode.c  | 54
> > > > > > +++++++++++++++++++++++++++++++++++++++++++-
> > > > > >   libavcodec/encode.h  |  8 +++++++
> > > > > >   libavcodec/options.c |  1 +
> > > > > >   5 files changed, 112 insertions(+), 4 deletions(-)
> > > > > > 
> > > > > > diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
> > > > > > index 7dbf083a24..e60eb16ce1 100644
> > > > > > --- a/libavcodec/avcodec.h
> > > > > > +++ b/libavcodec/avcodec.h
> > > > > > @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
> > > > > >    */
> > > > > >   #define AV_GET_BUFFER_FLAG_REF (1 << 0)
> > > > > > +/**
> > > > > > + * The encoder will keep a reference to the packet and
> > > > > > may reuse it later.
> > > > > > + */
> > > > > > +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
> > > > > > +
> > > > > >   struct AVCodecInternal;
> > > > > >   /**
> > > > > > @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
> > > > > >        * - encoding: set by user
> > > > > >        */
> > > > > >       int export_side_data;
> > > > > > +
> > > > > > +    /**
> > > > > > +     * This callback is called at the beginning of each
> > > > > > packet to get a data
> > > > > > +     * buffer for it.
> > > > > > +     *
> > > > > > +     * The following field will be set in the packet
> > > > > > before this callback is
> > > > > > +     * called:
> > > > > > +     * - size
> > > > > > +     * This callback must use the above value to
> > > > > > calculate the required buffer size,
> > > > > > +     * which must padded by at least
> > > > > > AV_INPUT_BUFFER_PADDING_SIZE bytes.
> > > > > > +     *
> > > > > > +     * This callback must fill the following fields in the packet:
> > > > > > +     * - data
> > > > > 
> > > > > Is the data pointer allowed to be in write-only memory?
> > > > 
> > > > I'm not sure what the use case for this would be, so probably no?
> > > 
> > > The two use-cases I see for this API are:
> > > 
> > > * You want to avoid a copy when combining the output with something
> > > else.  E.g. you pass a pointer to the block of memory following
> > > where you are going to put your header data (for something you are
> > > going to send over the network, say).
> > > 
> > > * You want to avoid a copy when passing the output directly to
> > > something external.  E.g. you pass a pointer to a memory-mapped
> > > device buffer (such as a V4L2 buffer, say).
> > > 
> > > In the second case, write-only memory on an external device seems
> > > possible, as does memory which is, say, readable but uncached, so
> > > reading it is a really bad idea.
> > 
> > Allowing the second case would depend on how encoders behave. Some may
> > attempt to read data already written to the output packet. It's not like
> > all of them allocate the packet, do a memcpy from an internal buffer,
> > then return.
> > There is also the flag meant to signal that the encoder will keep a
> > reference to the packet around, which more or less implies it will be
> > read later in the encoding process.
> > 
> > The doxy for avcodec_encode_video2(), which allowed the user to provide
> > their own buffers in the output packet, does not mention any kind of
> > requirement for the data pointer, so I don't think we can say it's an
> > allowed scenario here either.
> > 
> > > 
> > > > > Does it have any alignment requirements?
> > > > 
> > > > No, just padding. AVPacket doesn't require alignment for the payload.
> > > 
> > > I think say that explicitly.  avcodec_default_get_encoder_buffer()
> > > does give you aligned memory, even though it isn't needed.
> > 
> > Would saying "There's no alignment requirement for the data pointer" add
> > anything of value to the doxy? If i don't mention any kind of alignment
> > requirement, it's because there isn't any, and it's implicit.
> > I listed the requirements the user needs to keep in mind, like the
> > padding and the need for an AVBufferRef. But if you think it's worth
> > adding, then sure.
> > 
> > > 
> > > > > > +     * - buf must contain a pointer to an AVBufferRef
> > > > > > structure. The packet's
> > > > > > +     *   data pointer must be contained in it.
> > > > > > +     *   See: av_buffer_create(), av_buffer_alloc(),
> > > > > > and av_buffer_ref().
> > > > > > +     *
> > > > > > +     * If AV_CODEC_CAP_DR1 is not set then
> > > > > > get_encoder_buffer() must call
> > > > > > +     * avcodec_default_get_encoder_buffer() instead of
> > > > > > providing a buffer allocated by
> > > > > > +     * some other means.
> > > > > > +     *
> > > > > > +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
> > > > > > flags then the packet may be reused
> > > > > > +     * (read and/or written to if it is writable) later
> > > > > > by libavcodec.
> > > > > > +     *
> > > > > > +     * This callback must be thread-safe, as when frame
> > > > > > multithreading is used, it may
> > > > > > +     * be called from multiple threads simultaneously.
> > > > > 
> > > > > Allowing simulatenous calls feels unexpectedly tricky.  Is
> > > > > it really necessary?
> > > > 
> > > > This was a suggestion by Lynne, i personally don't know. We
> > > > support frame threading encoding (For intra-only codecs), but
> > > > currently ff_alloc_packet2() does not seem to be thread safe,
> > > > seeing it calls av_fast_padded_malloc(), yet it's called by
> > > > frame threaded encoders.
> > > > Should i remove this?
> > > 
> > > I don't know, I was asking only because it sounds tricky.  For cases
> > > with a limited number of buffers available (like memory-mapped
> > > devices) you are going to need locking anyway, so maybe rentrancy
> > > adds no additional inconvenience.
> > > 
> > > > > > +     *
> > > > > > +     * @see avcodec_default_get_encoder_buffer()
> > > > > > +     *
> > > > > > +     * - encoding: Set by libavcodec, user can override.
> > > > > > +     * - decoding: unused
> > > > > > +     */
> > > > > > +    int (*get_encoder_buffer)(struct AVCodecContext *s,
> > > > > > AVPacket *pkt, int flags);
> > > > > 
> > > > > Can the encoder ask for arbitrarily many packets?
> > > > > 
> > > > > Can the user return "not yet" somehow to this if they have a
> > > > > fixed output buffer pool but no buffer is currently
> > > > > available?
> > > > 
> > > > No, as is it can't. Return values < 0 are considered errors.
> > > > 
> > > > > 
> > > > > I don't much like the idea of the user suspending the thread
> > > > > in the callback until they have some available, which might
> > > > > work in some cases but might also deadlock if an
> > > > > avcodec_receive_packet() call is blocked by it.
> > > > 
> > > > Can we make what's in essence a malloc() call return something
> > > > like EAGAIN, and this in turn be propagated back to
> > > > encode_receive_packet_internal()?
> > > 
> > > Maybe, or if it has many threads maybe it could wait for something
> > > else to finish first.
> > > 
> > > > Couldn't this potentially end up in the forbidden scenario of
> > > > avcodec_send_frame() and avcodec_receive_packet() both returning
> > > > EAGAIN?
> > > 
> > > Yes.  If the forbidden case happens then the encoder is stuck anyway
> > > and can't make any forward progress so we need to error out
> > > properly, but the EAGAIN return isn't needed if there is something
> > > else to do on another thread.
> > 
> > Ok, but I'm not familiar or knowledgeable enough with the frame thread
> > encoder code to implement this.
> 
> Looked at bit into this. AVCodec->encode2() based encoders don't support
> returning EAGAIN at all, as it completely breaks the frame threading logic.
> It would require a considerable rewrite in order to re-add a task that
> didn't fail but also didn't succeed.
> 
> Non frame threading encoders could probably support it with some minimal
> changes, but i don't think suddenly letting an scenario that was until now
> guaranteed to never happen start happening (avcodec_send_frame() and
> avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
> break.
> Letting the user's custom get_encode_buffer() callback suspend the thread is
> IMO acceptable. In frame threading scenarios, the other threads are still
> working on their own packets (afaics none depends on the others, since it's
> intra only encoders only).

I think it was not suggested in the thread so:
if the users allocation fails the code can fallback to the default allocator
That would lead to the relation:
If a users allocator can fail (out of buffers) it must be able to handle
that only some of the returned packets are from its own allocator

About alignment, we should at least recommand that allocated packets are
aligned not less than what out av_malloc() would align to.
Is there a reason to align less ?

About multithreading, if there are 2 single thread encoders, they could of
course call the callback at the same time. These would then have seperate
AVCodecContexts though.
Maybe it should be documented more precissely what needs locking / when
AVCodecContexts / opaques

Thanks

On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
> On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
>> On 2/21/2021 6:04 PM, James Almer wrote:
>>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>>> On 21/02/2021 20:00, James Almer wrote:
>>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>>> This callback is functionally the same as get_buffer2()
>>>>>>> is for decoders, and
>>>>>>> implements for the new encode API the functionality of
>>>>>>> the old encode API had
>>>>>>> where the user could provide their own buffers.
>>>>>>>
>>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>>> ---
>>>>>>> Used the names Lynne suggested this time, plus a line
>>>>>>> about how the callback
>>>>>>> must be thread safe.
>>>>>>>
>>>>>>>    libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>>>    libavcodec/codec.h   |  8 ++++---
>>>>>>>    libavcodec/encode.c  | 54
>>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>    libavcodec/encode.h  |  8 +++++++
>>>>>>>    libavcodec/options.c |  1 +
>>>>>>>    5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>>
>>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>>> --- a/libavcodec/avcodec.h
>>>>>>> +++ b/libavcodec/avcodec.h
>>>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>>>     */
>>>>>>>    #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>>> +/**
>>>>>>> + * The encoder will keep a reference to the packet and
>>>>>>> may reuse it later.
>>>>>>> + */
>>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>>> +
>>>>>>>    struct AVCodecInternal;
>>>>>>>    /**
>>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>>         * - encoding: set by user
>>>>>>>         */
>>>>>>>        int export_side_data;
>>>>>>> +
>>>>>>> +    /**
>>>>>>> +     * This callback is called at the beginning of each
>>>>>>> packet to get a data
>>>>>>> +     * buffer for it.
>>>>>>> +     *
>>>>>>> +     * The following field will be set in the packet
>>>>>>> before this callback is
>>>>>>> +     * called:
>>>>>>> +     * - size
>>>>>>> +     * This callback must use the above value to
>>>>>>> calculate the required buffer size,
>>>>>>> +     * which must padded by at least
>>>>>>> AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>>> +     *
>>>>>>> +     * This callback must fill the following fields in the packet:
>>>>>>> +     * - data
>>>>>>
>>>>>> Is the data pointer allowed to be in write-only memory?
>>>>>
>>>>> I'm not sure what the use case for this would be, so probably no?
>>>>
>>>> The two use-cases I see for this API are:
>>>>
>>>> * You want to avoid a copy when combining the output with something
>>>> else.  E.g. you pass a pointer to the block of memory following
>>>> where you are going to put your header data (for something you are
>>>> going to send over the network, say).
>>>>
>>>> * You want to avoid a copy when passing the output directly to
>>>> something external.  E.g. you pass a pointer to a memory-mapped
>>>> device buffer (such as a V4L2 buffer, say).
>>>>
>>>> In the second case, write-only memory on an external device seems
>>>> possible, as does memory which is, say, readable but uncached, so
>>>> reading it is a really bad idea.
>>>
>>> Allowing the second case would depend on how encoders behave. Some may
>>> attempt to read data already written to the output packet. It's not like
>>> all of them allocate the packet, do a memcpy from an internal buffer,
>>> then return.
>>> There is also the flag meant to signal that the encoder will keep a
>>> reference to the packet around, which more or less implies it will be
>>> read later in the encoding process.
>>>
>>> The doxy for avcodec_encode_video2(), which allowed the user to provide
>>> their own buffers in the output packet, does not mention any kind of
>>> requirement for the data pointer, so I don't think we can say it's an
>>> allowed scenario here either.
>>>
>>>>
>>>>>> Does it have any alignment requirements?
>>>>>
>>>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>>>
>>>> I think say that explicitly.  avcodec_default_get_encoder_buffer()
>>>> does give you aligned memory, even though it isn't needed.
>>>
>>> Would saying "There's no alignment requirement for the data pointer" add
>>> anything of value to the doxy? If i don't mention any kind of alignment
>>> requirement, it's because there isn't any, and it's implicit.
>>> I listed the requirements the user needs to keep in mind, like the
>>> padding and the need for an AVBufferRef. But if you think it's worth
>>> adding, then sure.
>>>
>>>>
>>>>>>> +     * - buf must contain a pointer to an AVBufferRef
>>>>>>> structure. The packet's
>>>>>>> +     *   data pointer must be contained in it.
>>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(),
>>>>>>> and av_buffer_ref().
>>>>>>> +     *
>>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then
>>>>>>> get_encoder_buffer() must call
>>>>>>> +     * avcodec_default_get_encoder_buffer() instead of
>>>>>>> providing a buffer allocated by
>>>>>>> +     * some other means.
>>>>>>> +     *
>>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
>>>>>>> flags then the packet may be reused
>>>>>>> +     * (read and/or written to if it is writable) later
>>>>>>> by libavcodec.
>>>>>>> +     *
>>>>>>> +     * This callback must be thread-safe, as when frame
>>>>>>> multithreading is used, it may
>>>>>>> +     * be called from multiple threads simultaneously.
>>>>>>
>>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is
>>>>>> it really necessary?
>>>>>
>>>>> This was a suggestion by Lynne, i personally don't know. We
>>>>> support frame threading encoding (For intra-only codecs), but
>>>>> currently ff_alloc_packet2() does not seem to be thread safe,
>>>>> seeing it calls av_fast_padded_malloc(), yet it's called by
>>>>> frame threaded encoders.
>>>>> Should i remove this?
>>>>
>>>> I don't know, I was asking only because it sounds tricky.  For cases
>>>> with a limited number of buffers available (like memory-mapped
>>>> devices) you are going to need locking anyway, so maybe rentrancy
>>>> adds no additional inconvenience.
>>>>
>>>>>>> +     *
>>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>>> +     *
>>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>>> +     * - decoding: unused
>>>>>>> +     */
>>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s,
>>>>>>> AVPacket *pkt, int flags);
>>>>>>
>>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>>
>>>>>> Can the user return "not yet" somehow to this if they have a
>>>>>> fixed output buffer pool but no buffer is currently
>>>>>> available?
>>>>>
>>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>>
>>>>>>
>>>>>> I don't much like the idea of the user suspending the thread
>>>>>> in the callback until they have some available, which might
>>>>>> work in some cases but might also deadlock if an
>>>>>> avcodec_receive_packet() call is blocked by it.
>>>>>
>>>>> Can we make what's in essence a malloc() call return something
>>>>> like EAGAIN, and this in turn be propagated back to
>>>>> encode_receive_packet_internal()?
>>>>
>>>> Maybe, or if it has many threads maybe it could wait for something
>>>> else to finish first.
>>>>
>>>>> Couldn't this potentially end up in the forbidden scenario of
>>>>> avcodec_send_frame() and avcodec_receive_packet() both returning
>>>>> EAGAIN?
>>>>
>>>> Yes.  If the forbidden case happens then the encoder is stuck anyway
>>>> and can't make any forward progress so we need to error out
>>>> properly, but the EAGAIN return isn't needed if there is something
>>>> else to do on another thread.
>>>
>>> Ok, but I'm not familiar or knowledgeable enough with the frame thread
>>> encoder code to implement this.
>>
>> Looked at bit into this. AVCodec->encode2() based encoders don't support
>> returning EAGAIN at all, as it completely breaks the frame threading logic.
>> It would require a considerable rewrite in order to re-add a task that
>> didn't fail but also didn't succeed.
>>
>> Non frame threading encoders could probably support it with some minimal
>> changes, but i don't think suddenly letting an scenario that was until now
>> guaranteed to never happen start happening (avcodec_send_frame() and
>> avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
>> break.
>> Letting the user's custom get_encode_buffer() callback suspend the thread is
>> IMO acceptable. In frame threading scenarios, the other threads are still
>> working on their own packets (afaics none depends on the others, since it's
>> intra only encoders only).
> 
> I think it was not suggested in the thread so:
> if the users allocation fails the code can fallback to the default allocator
> That would lead to the relation:
> If a users allocator can fail (out of buffers) it must be able to handle
> that only some of the returned packets are from its own allocator

In general, custom allocators are used when the caller doesn't want to 
use the default one. But yes, they could use 
avcodec_default_get_encoder_buffer() as fallback, which is why it was 
added to begin with. Same applies to get_buffer2() custom 
implementations, and so far i don't think anybody had issues identifying 
what allocated a packet buffer.

One of the additions to AVPacket people were talking about was a user 
opaque field that libav* would never touch or look at beyond propagating 
them around all the way to the output AVFrame, if any. This opaque field 
could perhaps store such allocator specific information the caller could 
use to identify packets allocated by their own allocator, or those by 
avcodec_default_get_encoder_buffer().

> 
> About alignment, we should at least recommand that allocated packets are
> aligned not less than what out av_malloc() would align to.
> Is there a reason to align less ?

There's no alignment requirement for AVPacket->data, and av_new_packet() 
uses av_buffer_realloc(), which does not guarantee any alignment 
whatsoever on platforms other than Windows. So basically, packet payload 
buffers allocated by our own helpers never had any alignment.

> 
> About multithreading, if there are 2 single thread encoders, they could of
> course call the callback at the same time. These would then have seperate
> AVCodecContexts though.
> Maybe it should be documented more precissely what needs locking / when
> AVCodecContexts / opaques

Each call/thread will have their own separate AVCodecContext, yes. The 
callback should ensure it can be called from multiple threads at the 
same time not to access one or another avctx or avpkt fields, but in 
case whatever they use to provide buffers needs such considerations 
(Like when using a buffer pool).
Not sure what extra comment could be added beyond what i already added 
as suggested by Lynne. The user is already told to only look at 
avpkt->size and to fill avpkt->{data,buf}.

> 
> Thanks
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>

On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
> On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
> > On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
> > > On 2/21/2021 6:04 PM, James Almer wrote:
> > > > On 2/21/2021 5:29 PM, Mark Thompson wrote:
> > > > > On 21/02/2021 20:00, James Almer wrote:
> > > > > > On 2/21/2021 4:13 PM, Mark Thompson wrote:
> > > > > > > On 21/02/2021 17:35, James Almer wrote:
> > > > > > > > This callback is functionally the same as get_buffer2()
> > > > > > > > is for decoders, and
> > > > > > > > implements for the new encode API the functionality of
> > > > > > > > the old encode API had
> > > > > > > > where the user could provide their own buffers.
> > > > > > > > 
> > > > > > > > Signed-off-by: James Almer <jamrial@gmail.com>
> > > > > > > > ---
> > > > > > > > Used the names Lynne suggested this time, plus a line
> > > > > > > > about how the callback
> > > > > > > > must be thread safe.
> > > > > > > > 
> > > > > > > >    libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
> > > > > > > >    libavcodec/codec.h   |  8 ++++---
> > > > > > > >    libavcodec/encode.c  | 54
> > > > > > > > +++++++++++++++++++++++++++++++++++++++++++-
> > > > > > > >    libavcodec/encode.h  |  8 +++++++
> > > > > > > >    libavcodec/options.c |  1 +
> > > > > > > >    5 files changed, 112 insertions(+), 4 deletions(-)
> > > > > > > > 
> > > > > > > > diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
> > > > > > > > index 7dbf083a24..e60eb16ce1 100644
> > > > > > > > --- a/libavcodec/avcodec.h
> > > > > > > > +++ b/libavcodec/avcodec.h
> > > > > > > > @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
> > > > > > > >     */
> > > > > > > >    #define AV_GET_BUFFER_FLAG_REF (1 << 0)
> > > > > > > > +/**
> > > > > > > > + * The encoder will keep a reference to the packet and
> > > > > > > > may reuse it later.
> > > > > > > > + */
> > > > > > > > +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
> > > > > > > > +
> > > > > > > >    struct AVCodecInternal;
> > > > > > > >    /**
> > > > > > > > @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
> > > > > > > >         * - encoding: set by user
> > > > > > > >         */
> > > > > > > >        int export_side_data;
> > > > > > > > +
> > > > > > > > +    /**
> > > > > > > > +     * This callback is called at the beginning of each
> > > > > > > > packet to get a data
> > > > > > > > +     * buffer for it.
> > > > > > > > +     *
> > > > > > > > +     * The following field will be set in the packet
> > > > > > > > before this callback is
> > > > > > > > +     * called:
> > > > > > > > +     * - size
> > > > > > > > +     * This callback must use the above value to
> > > > > > > > calculate the required buffer size,
> > > > > > > > +     * which must padded by at least
> > > > > > > > AV_INPUT_BUFFER_PADDING_SIZE bytes.
> > > > > > > > +     *
> > > > > > > > +     * This callback must fill the following fields in the packet:
> > > > > > > > +     * - data
> > > > > > > 
> > > > > > > Is the data pointer allowed to be in write-only memory?
> > > > > > 
> > > > > > I'm not sure what the use case for this would be, so probably no?
> > > > > 
> > > > > The two use-cases I see for this API are:
> > > > > 
> > > > > * You want to avoid a copy when combining the output with something
> > > > > else.  E.g. you pass a pointer to the block of memory following
> > > > > where you are going to put your header data (for something you are
> > > > > going to send over the network, say).
> > > > > 
> > > > > * You want to avoid a copy when passing the output directly to
> > > > > something external.  E.g. you pass a pointer to a memory-mapped
> > > > > device buffer (such as a V4L2 buffer, say).
> > > > > 
> > > > > In the second case, write-only memory on an external device seems
> > > > > possible, as does memory which is, say, readable but uncached, so
> > > > > reading it is a really bad idea.
> > > > 
> > > > Allowing the second case would depend on how encoders behave. Some may
> > > > attempt to read data already written to the output packet. It's not like
> > > > all of them allocate the packet, do a memcpy from an internal buffer,
> > > > then return.
> > > > There is also the flag meant to signal that the encoder will keep a
> > > > reference to the packet around, which more or less implies it will be
> > > > read later in the encoding process.
> > > > 
> > > > The doxy for avcodec_encode_video2(), which allowed the user to provide
> > > > their own buffers in the output packet, does not mention any kind of
> > > > requirement for the data pointer, so I don't think we can say it's an
> > > > allowed scenario here either.
> > > > 
> > > > > 
> > > > > > > Does it have any alignment requirements?
> > > > > > 
> > > > > > No, just padding. AVPacket doesn't require alignment for the payload.
> > > > > 
> > > > > I think say that explicitly.  avcodec_default_get_encoder_buffer()
> > > > > does give you aligned memory, even though it isn't needed.
> > > > 
> > > > Would saying "There's no alignment requirement for the data pointer" add
> > > > anything of value to the doxy? If i don't mention any kind of alignment
> > > > requirement, it's because there isn't any, and it's implicit.
> > > > I listed the requirements the user needs to keep in mind, like the
> > > > padding and the need for an AVBufferRef. But if you think it's worth
> > > > adding, then sure.
> > > > 
> > > > > 
> > > > > > > > +     * - buf must contain a pointer to an AVBufferRef
> > > > > > > > structure. The packet's
> > > > > > > > +     *   data pointer must be contained in it.
> > > > > > > > +     *   See: av_buffer_create(), av_buffer_alloc(),
> > > > > > > > and av_buffer_ref().
> > > > > > > > +     *
> > > > > > > > +     * If AV_CODEC_CAP_DR1 is not set then
> > > > > > > > get_encoder_buffer() must call
> > > > > > > > +     * avcodec_default_get_encoder_buffer() instead of
> > > > > > > > providing a buffer allocated by
> > > > > > > > +     * some other means.
> > > > > > > > +     *
> > > > > > > > +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
> > > > > > > > flags then the packet may be reused
> > > > > > > > +     * (read and/or written to if it is writable) later
> > > > > > > > by libavcodec.
> > > > > > > > +     *
> > > > > > > > +     * This callback must be thread-safe, as when frame
> > > > > > > > multithreading is used, it may
> > > > > > > > +     * be called from multiple threads simultaneously.
> > > > > > > 
> > > > > > > Allowing simulatenous calls feels unexpectedly tricky.  Is
> > > > > > > it really necessary?
> > > > > > 
> > > > > > This was a suggestion by Lynne, i personally don't know. We
> > > > > > support frame threading encoding (For intra-only codecs), but
> > > > > > currently ff_alloc_packet2() does not seem to be thread safe,
> > > > > > seeing it calls av_fast_padded_malloc(), yet it's called by
> > > > > > frame threaded encoders.
> > > > > > Should i remove this?
> > > > > 
> > > > > I don't know, I was asking only because it sounds tricky.  For cases
> > > > > with a limited number of buffers available (like memory-mapped
> > > > > devices) you are going to need locking anyway, so maybe rentrancy
> > > > > adds no additional inconvenience.
> > > > > 
> > > > > > > > +     *
> > > > > > > > +     * @see avcodec_default_get_encoder_buffer()
> > > > > > > > +     *
> > > > > > > > +     * - encoding: Set by libavcodec, user can override.
> > > > > > > > +     * - decoding: unused
> > > > > > > > +     */
> > > > > > > > +    int (*get_encoder_buffer)(struct AVCodecContext *s,
> > > > > > > > AVPacket *pkt, int flags);
> > > > > > > 
> > > > > > > Can the encoder ask for arbitrarily many packets?
> > > > > > > 
> > > > > > > Can the user return "not yet" somehow to this if they have a
> > > > > > > fixed output buffer pool but no buffer is currently
> > > > > > > available?
> > > > > > 
> > > > > > No, as is it can't. Return values < 0 are considered errors.
> > > > > > 
> > > > > > > 
> > > > > > > I don't much like the idea of the user suspending the thread
> > > > > > > in the callback until they have some available, which might
> > > > > > > work in some cases but might also deadlock if an
> > > > > > > avcodec_receive_packet() call is blocked by it.
> > > > > > 
> > > > > > Can we make what's in essence a malloc() call return something
> > > > > > like EAGAIN, and this in turn be propagated back to
> > > > > > encode_receive_packet_internal()?
> > > > > 
> > > > > Maybe, or if it has many threads maybe it could wait for something
> > > > > else to finish first.
> > > > > 
> > > > > > Couldn't this potentially end up in the forbidden scenario of
> > > > > > avcodec_send_frame() and avcodec_receive_packet() both returning
> > > > > > EAGAIN?
> > > > > 
> > > > > Yes.  If the forbidden case happens then the encoder is stuck anyway
> > > > > and can't make any forward progress so we need to error out
> > > > > properly, but the EAGAIN return isn't needed if there is something
> > > > > else to do on another thread.
> > > > 
> > > > Ok, but I'm not familiar or knowledgeable enough with the frame thread
> > > > encoder code to implement this.
> > > 
> > > Looked at bit into this. AVCodec->encode2() based encoders don't support
> > > returning EAGAIN at all, as it completely breaks the frame threading logic.
> > > It would require a considerable rewrite in order to re-add a task that
> > > didn't fail but also didn't succeed.
> > > 
> > > Non frame threading encoders could probably support it with some minimal
> > > changes, but i don't think suddenly letting an scenario that was until now
> > > guaranteed to never happen start happening (avcodec_send_frame() and
> > > avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
> > > break.
> > > Letting the user's custom get_encode_buffer() callback suspend the thread is
> > > IMO acceptable. In frame threading scenarios, the other threads are still
> > > working on their own packets (afaics none depends on the others, since it's
> > > intra only encoders only).
> > 
> > I think it was not suggested in the thread so:
> > if the users allocation fails the code can fallback to the default allocator
> > That would lead to the relation:
> > If a users allocator can fail (out of buffers) it must be able to handle
> > that only some of the returned packets are from its own allocator
> 
> In general, custom allocators are used when the caller doesn't want to use
> the default one. But yes, they could use
> avcodec_default_get_encoder_buffer() as fallback, which is why it was added
> to begin with. Same applies to get_buffer2() custom implementations, and so
> far i don't think anybody had issues identifying what allocated a packet
> buffer.
> 
> One of the additions to AVPacket people were talking about was a user opaque
> field that libav* would never touch or look at beyond propagating them
> around all the way to the output AVFrame, if any. This opaque field could
> perhaps store such allocator specific information the caller could use to
> identify packets allocated by their own allocator, or those by
> avcodec_default_get_encoder_buffer().
> 
> > 
> > About alignment, we should at least recommand that allocated packets are
> > aligned not less than what out av_malloc() would align to.
> > Is there a reason to align less ?
> 
> There's no alignment requirement for AVPacket->data, and av_new_packet()
> uses av_buffer_realloc(), which does not guarantee any alignment whatsoever
> on platforms other than Windows. So basically, packet payload buffers
> allocated by our own helpers never had any alignment.

for the purpose of exporting raw images, alignment would be "nice to have"
because later filters may need it or need to memcpy

not really achieving this but, realloc() provides buffers with alignment
sufficient for any C type. So for example if on a ARM int requires 4 byte
alignment then realloc() will at least return 4 byte aligned buffers


> 
> > 
> > About multithreading, if there are 2 single thread encoders, they could of
> > course call the callback at the same time. These would then have seperate
> > AVCodecContexts though.
> > Maybe it should be documented more precissely what needs locking / when
> > AVCodecContexts / opaques
> 
> Each call/thread will have their own separate AVCodecContext, yes. The
> callback should ensure it can be called from multiple threads at the same
> time not to access one or another avctx or avpkt fields, but in case
> whatever they use to provide buffers needs such considerations (Like when
> using a buffer pool).
> Not sure what extra comment could be added beyond what i already added as
> suggested by Lynne. The user is already told to only look at avpkt->size and
> to fill avpkt->{data,buf}.

ok

[...]

On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
> On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
>> On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
>>> On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
>>>> On 2/21/2021 6:04 PM, James Almer wrote:
>>>>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>>>>> On 21/02/2021 20:00, James Almer wrote:
>>>>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>>>>> This callback is functionally the same as get_buffer2()
>>>>>>>>> is for decoders, and
>>>>>>>>> implements for the new encode API the functionality of
>>>>>>>>> the old encode API had
>>>>>>>>> where the user could provide their own buffers.
>>>>>>>>>
>>>>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>>>>> ---
>>>>>>>>> Used the names Lynne suggested this time, plus a line
>>>>>>>>> about how the callback
>>>>>>>>> must be thread safe.
>>>>>>>>>
>>>>>>>>>     libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>>>>>     libavcodec/codec.h   |  8 ++++---
>>>>>>>>>     libavcodec/encode.c  | 54
>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>     libavcodec/encode.h  |  8 +++++++
>>>>>>>>>     libavcodec/options.c |  1 +
>>>>>>>>>     5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>>>>> --- a/libavcodec/avcodec.h
>>>>>>>>> +++ b/libavcodec/avcodec.h
>>>>>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>>>>>      */
>>>>>>>>>     #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>> +/**
>>>>>>>>> + * The encoder will keep a reference to the packet and
>>>>>>>>> may reuse it later.
>>>>>>>>> + */
>>>>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>> +
>>>>>>>>>     struct AVCodecInternal;
>>>>>>>>>     /**
>>>>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>>>>          * - encoding: set by user
>>>>>>>>>          */
>>>>>>>>>         int export_side_data;
>>>>>>>>> +
>>>>>>>>> +    /**
>>>>>>>>> +     * This callback is called at the beginning of each
>>>>>>>>> packet to get a data
>>>>>>>>> +     * buffer for it.
>>>>>>>>> +     *
>>>>>>>>> +     * The following field will be set in the packet
>>>>>>>>> before this callback is
>>>>>>>>> +     * called:
>>>>>>>>> +     * - size
>>>>>>>>> +     * This callback must use the above value to
>>>>>>>>> calculate the required buffer size,
>>>>>>>>> +     * which must padded by at least
>>>>>>>>> AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>>>>> +     *
>>>>>>>>> +     * This callback must fill the following fields in the packet:
>>>>>>>>> +     * - data
>>>>>>>>
>>>>>>>> Is the data pointer allowed to be in write-only memory?
>>>>>>>
>>>>>>> I'm not sure what the use case for this would be, so probably no?
>>>>>>
>>>>>> The two use-cases I see for this API are:
>>>>>>
>>>>>> * You want to avoid a copy when combining the output with something
>>>>>> else.  E.g. you pass a pointer to the block of memory following
>>>>>> where you are going to put your header data (for something you are
>>>>>> going to send over the network, say).
>>>>>>
>>>>>> * You want to avoid a copy when passing the output directly to
>>>>>> something external.  E.g. you pass a pointer to a memory-mapped
>>>>>> device buffer (such as a V4L2 buffer, say).
>>>>>>
>>>>>> In the second case, write-only memory on an external device seems
>>>>>> possible, as does memory which is, say, readable but uncached, so
>>>>>> reading it is a really bad idea.
>>>>>
>>>>> Allowing the second case would depend on how encoders behave. Some may
>>>>> attempt to read data already written to the output packet. It's not like
>>>>> all of them allocate the packet, do a memcpy from an internal buffer,
>>>>> then return.
>>>>> There is also the flag meant to signal that the encoder will keep a
>>>>> reference to the packet around, which more or less implies it will be
>>>>> read later in the encoding process.
>>>>>
>>>>> The doxy for avcodec_encode_video2(), which allowed the user to provide
>>>>> their own buffers in the output packet, does not mention any kind of
>>>>> requirement for the data pointer, so I don't think we can say it's an
>>>>> allowed scenario here either.
>>>>>
>>>>>>
>>>>>>>> Does it have any alignment requirements?
>>>>>>>
>>>>>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>>>>>
>>>>>> I think say that explicitly.  avcodec_default_get_encoder_buffer()
>>>>>> does give you aligned memory, even though it isn't needed.
>>>>>
>>>>> Would saying "There's no alignment requirement for the data pointer" add
>>>>> anything of value to the doxy? If i don't mention any kind of alignment
>>>>> requirement, it's because there isn't any, and it's implicit.
>>>>> I listed the requirements the user needs to keep in mind, like the
>>>>> padding and the need for an AVBufferRef. But if you think it's worth
>>>>> adding, then sure.
>>>>>
>>>>>>
>>>>>>>>> +     * - buf must contain a pointer to an AVBufferRef
>>>>>>>>> structure. The packet's
>>>>>>>>> +     *   data pointer must be contained in it.
>>>>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(),
>>>>>>>>> and av_buffer_ref().
>>>>>>>>> +     *
>>>>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then
>>>>>>>>> get_encoder_buffer() must call
>>>>>>>>> +     * avcodec_default_get_encoder_buffer() instead of
>>>>>>>>> providing a buffer allocated by
>>>>>>>>> +     * some other means.
>>>>>>>>> +     *
>>>>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
>>>>>>>>> flags then the packet may be reused
>>>>>>>>> +     * (read and/or written to if it is writable) later
>>>>>>>>> by libavcodec.
>>>>>>>>> +     *
>>>>>>>>> +     * This callback must be thread-safe, as when frame
>>>>>>>>> multithreading is used, it may
>>>>>>>>> +     * be called from multiple threads simultaneously.
>>>>>>>>
>>>>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is
>>>>>>>> it really necessary?
>>>>>>>
>>>>>>> This was a suggestion by Lynne, i personally don't know. We
>>>>>>> support frame threading encoding (For intra-only codecs), but
>>>>>>> currently ff_alloc_packet2() does not seem to be thread safe,
>>>>>>> seeing it calls av_fast_padded_malloc(), yet it's called by
>>>>>>> frame threaded encoders.
>>>>>>> Should i remove this?
>>>>>>
>>>>>> I don't know, I was asking only because it sounds tricky.  For cases
>>>>>> with a limited number of buffers available (like memory-mapped
>>>>>> devices) you are going to need locking anyway, so maybe rentrancy
>>>>>> adds no additional inconvenience.
>>>>>>
>>>>>>>>> +     *
>>>>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>>>>> +     *
>>>>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>>>>> +     * - decoding: unused
>>>>>>>>> +     */
>>>>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s,
>>>>>>>>> AVPacket *pkt, int flags);
>>>>>>>>
>>>>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>>>>
>>>>>>>> Can the user return "not yet" somehow to this if they have a
>>>>>>>> fixed output buffer pool but no buffer is currently
>>>>>>>> available?
>>>>>>>
>>>>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>>>>
>>>>>>>>
>>>>>>>> I don't much like the idea of the user suspending the thread
>>>>>>>> in the callback until they have some available, which might
>>>>>>>> work in some cases but might also deadlock if an
>>>>>>>> avcodec_receive_packet() call is blocked by it.
>>>>>>>
>>>>>>> Can we make what's in essence a malloc() call return something
>>>>>>> like EAGAIN, and this in turn be propagated back to
>>>>>>> encode_receive_packet_internal()?
>>>>>>
>>>>>> Maybe, or if it has many threads maybe it could wait for something
>>>>>> else to finish first.
>>>>>>
>>>>>>> Couldn't this potentially end up in the forbidden scenario of
>>>>>>> avcodec_send_frame() and avcodec_receive_packet() both returning
>>>>>>> EAGAIN?
>>>>>>
>>>>>> Yes.  If the forbidden case happens then the encoder is stuck anyway
>>>>>> and can't make any forward progress so we need to error out
>>>>>> properly, but the EAGAIN return isn't needed if there is something
>>>>>> else to do on another thread.
>>>>>
>>>>> Ok, but I'm not familiar or knowledgeable enough with the frame thread
>>>>> encoder code to implement this.
>>>>
>>>> Looked at bit into this. AVCodec->encode2() based encoders don't support
>>>> returning EAGAIN at all, as it completely breaks the frame threading logic.
>>>> It would require a considerable rewrite in order to re-add a task that
>>>> didn't fail but also didn't succeed.
>>>>
>>>> Non frame threading encoders could probably support it with some minimal
>>>> changes, but i don't think suddenly letting an scenario that was until now
>>>> guaranteed to never happen start happening (avcodec_send_frame() and
>>>> avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
>>>> break.
>>>> Letting the user's custom get_encode_buffer() callback suspend the thread is
>>>> IMO acceptable. In frame threading scenarios, the other threads are still
>>>> working on their own packets (afaics none depends on the others, since it's
>>>> intra only encoders only).
>>>
>>> I think it was not suggested in the thread so:
>>> if the users allocation fails the code can fallback to the default allocator
>>> That would lead to the relation:
>>> If a users allocator can fail (out of buffers) it must be able to handle
>>> that only some of the returned packets are from its own allocator
>>
>> In general, custom allocators are used when the caller doesn't want to use
>> the default one. But yes, they could use
>> avcodec_default_get_encoder_buffer() as fallback, which is why it was added
>> to begin with. Same applies to get_buffer2() custom implementations, and so
>> far i don't think anybody had issues identifying what allocated a packet
>> buffer.
>>
>> One of the additions to AVPacket people were talking about was a user opaque
>> field that libav* would never touch or look at beyond propagating them
>> around all the way to the output AVFrame, if any. This opaque field could
>> perhaps store such allocator specific information the caller could use to
>> identify packets allocated by their own allocator, or those by
>> avcodec_default_get_encoder_buffer().
>>
>>>
>>> About alignment, we should at least recommand that allocated packets are
>>> aligned not less than what out av_malloc() would align to.
>>> Is there a reason to align less ?
>>
>> There's no alignment requirement for AVPacket->data, and av_new_packet()
>> uses av_buffer_realloc(), which does not guarantee any alignment whatsoever
>> on platforms other than Windows. So basically, packet payload buffers
>> allocated by our own helpers never had any alignment.
> 
> for the purpose of exporting raw images, alignment would be "nice to have"
> because later filters may need it or need to memcpy

Filters don't use AVPackets, they use AVFrames.

> 
> not really achieving this but, realloc() provides buffers with alignment
> sufficient for any C type. So for example if on a ARM int requires 4 byte
> alignment then realloc() will at least return 4 byte aligned buffers

I'm not sure where you're going with this. This is about allocating 
AVPacket data buffers, not AVFrame data buffers. Alignment is not 
requested or required. I could use av_buffer_create() to wrap a pointer 
to some non aligned stack buffer and put it in an AVPacket and no 
decoder or muxer will have issues with it, because they were all written 
with such expectations in mind. This was in fact commonplace before 
reference counting with the old encode/decode API, and exactly the 
functionality this callback is trying to retain for the new.

Most of out helpers use av_realloc() to allocate a packet's payload, 
some demuxer helpers use av_malloc() instead. Some encoders could even 
wrap buffers allocated by some external library to achieve zero-copy 
encoding (the librav1e wrapper was going to do it, but ultimately didn't 
because the external buffers did not satisfy the padding requirements).
It makes no difference, and none of them are preferable over the rest.

> 
> 
>>
>>>
>>> About multithreading, if there are 2 single thread encoders, they could of
>>> course call the callback at the same time. These would then have seperate
>>> AVCodecContexts though.
>>> Maybe it should be documented more precissely what needs locking / when
>>> AVCodecContexts / opaques
>>
>> Each call/thread will have their own separate AVCodecContext, yes. The
>> callback should ensure it can be called from multiple threads at the same
>> time not to access one or another avctx or avpkt fields, but in case
>> whatever they use to provide buffers needs such considerations (Like when
>> using a buffer pool).
>> Not sure what extra comment could be added beyond what i already added as
>> suggested by Lynne. The user is already told to only look at avpkt->size and
>> to fill avpkt->{data,buf}.
> 
> ok
> 
> [...]
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>

On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
> On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
> > On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
> > > On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
> > > > On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
> > > > > On 2/21/2021 6:04 PM, James Almer wrote:
> > > > > > On 2/21/2021 5:29 PM, Mark Thompson wrote:
> > > > > > > On 21/02/2021 20:00, James Almer wrote:
> > > > > > > > On 2/21/2021 4:13 PM, Mark Thompson wrote:
> > > > > > > > > On 21/02/2021 17:35, James Almer wrote:
> > > > > > > > > > This callback is functionally the same as get_buffer2()
> > > > > > > > > > is for decoders, and
> > > > > > > > > > implements for the new encode API the functionality of
> > > > > > > > > > the old encode API had
> > > > > > > > > > where the user could provide their own buffers.
> > > > > > > > > > 
> > > > > > > > > > Signed-off-by: James Almer <jamrial@gmail.com>
> > > > > > > > > > ---
> > > > > > > > > > Used the names Lynne suggested this time, plus a line
> > > > > > > > > > about how the callback
> > > > > > > > > > must be thread safe.
> > > > > > > > > > 
> > > > > > > > > >     libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > >     libavcodec/codec.h   |  8 ++++---
> > > > > > > > > >     libavcodec/encode.c  | 54
> > > > > > > > > > +++++++++++++++++++++++++++++++++++++++++++-
> > > > > > > > > >     libavcodec/encode.h  |  8 +++++++
> > > > > > > > > >     libavcodec/options.c |  1 +
> > > > > > > > > >     5 files changed, 112 insertions(+), 4 deletions(-)
> > > > > > > > > > 
> > > > > > > > > > diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
> > > > > > > > > > index 7dbf083a24..e60eb16ce1 100644
> > > > > > > > > > --- a/libavcodec/avcodec.h
> > > > > > > > > > +++ b/libavcodec/avcodec.h
> > > > > > > > > > @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
> > > > > > > > > >      */
> > > > > > > > > >     #define AV_GET_BUFFER_FLAG_REF (1 << 0)
> > > > > > > > > > +/**
> > > > > > > > > > + * The encoder will keep a reference to the packet and
> > > > > > > > > > may reuse it later.
> > > > > > > > > > + */
> > > > > > > > > > +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
> > > > > > > > > > +
> > > > > > > > > >     struct AVCodecInternal;
> > > > > > > > > >     /**
> > > > > > > > > > @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
> > > > > > > > > >          * - encoding: set by user
> > > > > > > > > >          */
> > > > > > > > > >         int export_side_data;
> > > > > > > > > > +
> > > > > > > > > > +    /**
> > > > > > > > > > +     * This callback is called at the beginning of each
> > > > > > > > > > packet to get a data
> > > > > > > > > > +     * buffer for it.
> > > > > > > > > > +     *
> > > > > > > > > > +     * The following field will be set in the packet
> > > > > > > > > > before this callback is
> > > > > > > > > > +     * called:
> > > > > > > > > > +     * - size
> > > > > > > > > > +     * This callback must use the above value to
> > > > > > > > > > calculate the required buffer size,
> > > > > > > > > > +     * which must padded by at least
> > > > > > > > > > AV_INPUT_BUFFER_PADDING_SIZE bytes.
> > > > > > > > > > +     *
> > > > > > > > > > +     * This callback must fill the following fields in the packet:
> > > > > > > > > > +     * - data
> > > > > > > > > 
> > > > > > > > > Is the data pointer allowed to be in write-only memory?
> > > > > > > > 
> > > > > > > > I'm not sure what the use case for this would be, so probably no?
> > > > > > > 
> > > > > > > The two use-cases I see for this API are:
> > > > > > > 
> > > > > > > * You want to avoid a copy when combining the output with something
> > > > > > > else.  E.g. you pass a pointer to the block of memory following
> > > > > > > where you are going to put your header data (for something you are
> > > > > > > going to send over the network, say).
> > > > > > > 
> > > > > > > * You want to avoid a copy when passing the output directly to
> > > > > > > something external.  E.g. you pass a pointer to a memory-mapped
> > > > > > > device buffer (such as a V4L2 buffer, say).
> > > > > > > 
> > > > > > > In the second case, write-only memory on an external device seems
> > > > > > > possible, as does memory which is, say, readable but uncached, so
> > > > > > > reading it is a really bad idea.
> > > > > > 
> > > > > > Allowing the second case would depend on how encoders behave. Some may
> > > > > > attempt to read data already written to the output packet. It's not like
> > > > > > all of them allocate the packet, do a memcpy from an internal buffer,
> > > > > > then return.
> > > > > > There is also the flag meant to signal that the encoder will keep a
> > > > > > reference to the packet around, which more or less implies it will be
> > > > > > read later in the encoding process.
> > > > > > 
> > > > > > The doxy for avcodec_encode_video2(), which allowed the user to provide
> > > > > > their own buffers in the output packet, does not mention any kind of
> > > > > > requirement for the data pointer, so I don't think we can say it's an
> > > > > > allowed scenario here either.
> > > > > > 
> > > > > > > 
> > > > > > > > > Does it have any alignment requirements?
> > > > > > > > 
> > > > > > > > No, just padding. AVPacket doesn't require alignment for the payload.
> > > > > > > 
> > > > > > > I think say that explicitly.  avcodec_default_get_encoder_buffer()
> > > > > > > does give you aligned memory, even though it isn't needed.
> > > > > > 
> > > > > > Would saying "There's no alignment requirement for the data pointer" add
> > > > > > anything of value to the doxy? If i don't mention any kind of alignment
> > > > > > requirement, it's because there isn't any, and it's implicit.
> > > > > > I listed the requirements the user needs to keep in mind, like the
> > > > > > padding and the need for an AVBufferRef. But if you think it's worth
> > > > > > adding, then sure.
> > > > > > 
> > > > > > > 
> > > > > > > > > > +     * - buf must contain a pointer to an AVBufferRef
> > > > > > > > > > structure. The packet's
> > > > > > > > > > +     *   data pointer must be contained in it.
> > > > > > > > > > +     *   See: av_buffer_create(), av_buffer_alloc(),
> > > > > > > > > > and av_buffer_ref().
> > > > > > > > > > +     *
> > > > > > > > > > +     * If AV_CODEC_CAP_DR1 is not set then
> > > > > > > > > > get_encoder_buffer() must call
> > > > > > > > > > +     * avcodec_default_get_encoder_buffer() instead of
> > > > > > > > > > providing a buffer allocated by
> > > > > > > > > > +     * some other means.
> > > > > > > > > > +     *
> > > > > > > > > > +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
> > > > > > > > > > flags then the packet may be reused
> > > > > > > > > > +     * (read and/or written to if it is writable) later
> > > > > > > > > > by libavcodec.
> > > > > > > > > > +     *
> > > > > > > > > > +     * This callback must be thread-safe, as when frame
> > > > > > > > > > multithreading is used, it may
> > > > > > > > > > +     * be called from multiple threads simultaneously.
> > > > > > > > > 
> > > > > > > > > Allowing simulatenous calls feels unexpectedly tricky.  Is
> > > > > > > > > it really necessary?
> > > > > > > > 
> > > > > > > > This was a suggestion by Lynne, i personally don't know. We
> > > > > > > > support frame threading encoding (For intra-only codecs), but
> > > > > > > > currently ff_alloc_packet2() does not seem to be thread safe,
> > > > > > > > seeing it calls av_fast_padded_malloc(), yet it's called by
> > > > > > > > frame threaded encoders.
> > > > > > > > Should i remove this?
> > > > > > > 
> > > > > > > I don't know, I was asking only because it sounds tricky.  For cases
> > > > > > > with a limited number of buffers available (like memory-mapped
> > > > > > > devices) you are going to need locking anyway, so maybe rentrancy
> > > > > > > adds no additional inconvenience.
> > > > > > > 
> > > > > > > > > > +     *
> > > > > > > > > > +     * @see avcodec_default_get_encoder_buffer()
> > > > > > > > > > +     *
> > > > > > > > > > +     * - encoding: Set by libavcodec, user can override.
> > > > > > > > > > +     * - decoding: unused
> > > > > > > > > > +     */
> > > > > > > > > > +    int (*get_encoder_buffer)(struct AVCodecContext *s,
> > > > > > > > > > AVPacket *pkt, int flags);
> > > > > > > > > 
> > > > > > > > > Can the encoder ask for arbitrarily many packets?
> > > > > > > > > 
> > > > > > > > > Can the user return "not yet" somehow to this if they have a
> > > > > > > > > fixed output buffer pool but no buffer is currently
> > > > > > > > > available?
> > > > > > > > 
> > > > > > > > No, as is it can't. Return values < 0 are considered errors.
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > I don't much like the idea of the user suspending the thread
> > > > > > > > > in the callback until they have some available, which might
> > > > > > > > > work in some cases but might also deadlock if an
> > > > > > > > > avcodec_receive_packet() call is blocked by it.
> > > > > > > > 
> > > > > > > > Can we make what's in essence a malloc() call return something
> > > > > > > > like EAGAIN, and this in turn be propagated back to
> > > > > > > > encode_receive_packet_internal()?
> > > > > > > 
> > > > > > > Maybe, or if it has many threads maybe it could wait for something
> > > > > > > else to finish first.
> > > > > > > 
> > > > > > > > Couldn't this potentially end up in the forbidden scenario of
> > > > > > > > avcodec_send_frame() and avcodec_receive_packet() both returning
> > > > > > > > EAGAIN?
> > > > > > > 
> > > > > > > Yes.  If the forbidden case happens then the encoder is stuck anyway
> > > > > > > and can't make any forward progress so we need to error out
> > > > > > > properly, but the EAGAIN return isn't needed if there is something
> > > > > > > else to do on another thread.
> > > > > > 
> > > > > > Ok, but I'm not familiar or knowledgeable enough with the frame thread
> > > > > > encoder code to implement this.
> > > > > 
> > > > > Looked at bit into this. AVCodec->encode2() based encoders don't support
> > > > > returning EAGAIN at all, as it completely breaks the frame threading logic.
> > > > > It would require a considerable rewrite in order to re-add a task that
> > > > > didn't fail but also didn't succeed.
> > > > > 
> > > > > Non frame threading encoders could probably support it with some minimal
> > > > > changes, but i don't think suddenly letting an scenario that was until now
> > > > > guaranteed to never happen start happening (avcodec_send_frame() and
> > > > > avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
> > > > > break.
> > > > > Letting the user's custom get_encode_buffer() callback suspend the thread is
> > > > > IMO acceptable. In frame threading scenarios, the other threads are still
> > > > > working on their own packets (afaics none depends on the others, since it's
> > > > > intra only encoders only).
> > > > 
> > > > I think it was not suggested in the thread so:
> > > > if the users allocation fails the code can fallback to the default allocator
> > > > That would lead to the relation:
> > > > If a users allocator can fail (out of buffers) it must be able to handle
> > > > that only some of the returned packets are from its own allocator
> > > 
> > > In general, custom allocators are used when the caller doesn't want to use
> > > the default one. But yes, they could use
> > > avcodec_default_get_encoder_buffer() as fallback, which is why it was added
> > > to begin with. Same applies to get_buffer2() custom implementations, and so
> > > far i don't think anybody had issues identifying what allocated a packet
> > > buffer.
> > > 
> > > One of the additions to AVPacket people were talking about was a user opaque
> > > field that libav* would never touch or look at beyond propagating them
> > > around all the way to the output AVFrame, if any. This opaque field could
> > > perhaps store such allocator specific information the caller could use to
> > > identify packets allocated by their own allocator, or those by
> > > avcodec_default_get_encoder_buffer().
> > > 
> > > > 
> > > > About alignment, we should at least recommand that allocated packets are
> > > > aligned not less than what out av_malloc() would align to.
> > > > Is there a reason to align less ?
> > > 
> > > There's no alignment requirement for AVPacket->data, and av_new_packet()
> > > uses av_buffer_realloc(), which does not guarantee any alignment whatsoever
> > > on platforms other than Windows. So basically, packet payload buffers
> > > allocated by our own helpers never had any alignment.
> > 
> > for the purpose of exporting raw images, alignment would be "nice to have"
> > because later filters may need it or need to memcpy
> 
> Filters don't use AVPackets, they use AVFrames.

demuxers return AVPackets, so do encoders.
These can contain raw frames.

also i see for example in rawdec:
frame->buf[0] = av_buffer_ref(avpkt->buf);

Thanks

[...]

Michael Niedermayer:
> On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
>> On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
>>> On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
>>>> On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
>>>>> On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
>>>>>> On 2/21/2021 6:04 PM, James Almer wrote:
>>>>>>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>>>>>>> On 21/02/2021 20:00, James Almer wrote:
>>>>>>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>>>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>>>>>>> This callback is functionally the same as get_buffer2()
>>>>>>>>>>> is for decoders, and
>>>>>>>>>>> implements for the new encode API the functionality of
>>>>>>>>>>> the old encode API had
>>>>>>>>>>> where the user could provide their own buffers.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>>>>>>> ---
>>>>>>>>>>> Used the names Lynne suggested this time, plus a line
>>>>>>>>>>> about how the callback
>>>>>>>>>>> must be thread safe.
>>>>>>>>>>>
>>>>>>>>>>>     libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>     libavcodec/codec.h   |  8 ++++---
>>>>>>>>>>>     libavcodec/encode.c  | 54
>>>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>     libavcodec/encode.h  |  8 +++++++
>>>>>>>>>>>     libavcodec/options.c |  1 +
>>>>>>>>>>>     5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>>>>>>> --- a/libavcodec/avcodec.h
>>>>>>>>>>> +++ b/libavcodec/avcodec.h
>>>>>>>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>>>>>>>      */
>>>>>>>>>>>     #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>> +/**
>>>>>>>>>>> + * The encoder will keep a reference to the packet and
>>>>>>>>>>> may reuse it later.
>>>>>>>>>>> + */
>>>>>>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>> +
>>>>>>>>>>>     struct AVCodecInternal;
>>>>>>>>>>>     /**
>>>>>>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>>>>>>          * - encoding: set by user
>>>>>>>>>>>          */
>>>>>>>>>>>         int export_side_data;
>>>>>>>>>>> +
>>>>>>>>>>> +    /**
>>>>>>>>>>> +     * This callback is called at the beginning of each
>>>>>>>>>>> packet to get a data
>>>>>>>>>>> +     * buffer for it.
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * The following field will be set in the packet
>>>>>>>>>>> before this callback is
>>>>>>>>>>> +     * called:
>>>>>>>>>>> +     * - size
>>>>>>>>>>> +     * This callback must use the above value to
>>>>>>>>>>> calculate the required buffer size,
>>>>>>>>>>> +     * which must padded by at least
>>>>>>>>>>> AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * This callback must fill the following fields in the packet:
>>>>>>>>>>> +     * - data
>>>>>>>>>>
>>>>>>>>>> Is the data pointer allowed to be in write-only memory?
>>>>>>>>>
>>>>>>>>> I'm not sure what the use case for this would be, so probably no?
>>>>>>>>
>>>>>>>> The two use-cases I see for this API are:
>>>>>>>>
>>>>>>>> * You want to avoid a copy when combining the output with something
>>>>>>>> else.  E.g. you pass a pointer to the block of memory following
>>>>>>>> where you are going to put your header data (for something you are
>>>>>>>> going to send over the network, say).
>>>>>>>>
>>>>>>>> * You want to avoid a copy when passing the output directly to
>>>>>>>> something external.  E.g. you pass a pointer to a memory-mapped
>>>>>>>> device buffer (such as a V4L2 buffer, say).
>>>>>>>>
>>>>>>>> In the second case, write-only memory on an external device seems
>>>>>>>> possible, as does memory which is, say, readable but uncached, so
>>>>>>>> reading it is a really bad idea.
>>>>>>>
>>>>>>> Allowing the second case would depend on how encoders behave. Some may
>>>>>>> attempt to read data already written to the output packet. It's not like
>>>>>>> all of them allocate the packet, do a memcpy from an internal buffer,
>>>>>>> then return.
>>>>>>> There is also the flag meant to signal that the encoder will keep a
>>>>>>> reference to the packet around, which more or less implies it will be
>>>>>>> read later in the encoding process.
>>>>>>>
>>>>>>> The doxy for avcodec_encode_video2(), which allowed the user to provide
>>>>>>> their own buffers in the output packet, does not mention any kind of
>>>>>>> requirement for the data pointer, so I don't think we can say it's an
>>>>>>> allowed scenario here either.
>>>>>>>
>>>>>>>>
>>>>>>>>>> Does it have any alignment requirements?
>>>>>>>>>
>>>>>>>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>>>>>>>
>>>>>>>> I think say that explicitly.  avcodec_default_get_encoder_buffer()
>>>>>>>> does give you aligned memory, even though it isn't needed.
>>>>>>>
>>>>>>> Would saying "There's no alignment requirement for the data pointer" add
>>>>>>> anything of value to the doxy? If i don't mention any kind of alignment
>>>>>>> requirement, it's because there isn't any, and it's implicit.
>>>>>>> I listed the requirements the user needs to keep in mind, like the
>>>>>>> padding and the need for an AVBufferRef. But if you think it's worth
>>>>>>> adding, then sure.
>>>>>>>
>>>>>>>>
>>>>>>>>>>> +     * - buf must contain a pointer to an AVBufferRef
>>>>>>>>>>> structure. The packet's
>>>>>>>>>>> +     *   data pointer must be contained in it.
>>>>>>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(),
>>>>>>>>>>> and av_buffer_ref().
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then
>>>>>>>>>>> get_encoder_buffer() must call
>>>>>>>>>>> +     * avcodec_default_get_encoder_buffer() instead of
>>>>>>>>>>> providing a buffer allocated by
>>>>>>>>>>> +     * some other means.
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
>>>>>>>>>>> flags then the packet may be reused
>>>>>>>>>>> +     * (read and/or written to if it is writable) later
>>>>>>>>>>> by libavcodec.
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * This callback must be thread-safe, as when frame
>>>>>>>>>>> multithreading is used, it may
>>>>>>>>>>> +     * be called from multiple threads simultaneously.
>>>>>>>>>>
>>>>>>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is
>>>>>>>>>> it really necessary?
>>>>>>>>>
>>>>>>>>> This was a suggestion by Lynne, i personally don't know. We
>>>>>>>>> support frame threading encoding (For intra-only codecs), but
>>>>>>>>> currently ff_alloc_packet2() does not seem to be thread safe,
>>>>>>>>> seeing it calls av_fast_padded_malloc(), yet it's called by
>>>>>>>>> frame threaded encoders.
>>>>>>>>> Should i remove this?
>>>>>>>>
>>>>>>>> I don't know, I was asking only because it sounds tricky.  For cases
>>>>>>>> with a limited number of buffers available (like memory-mapped
>>>>>>>> devices) you are going to need locking anyway, so maybe rentrancy
>>>>>>>> adds no additional inconvenience.
>>>>>>>>
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>>>>>>> +     * - decoding: unused
>>>>>>>>>>> +     */
>>>>>>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s,
>>>>>>>>>>> AVPacket *pkt, int flags);
>>>>>>>>>>
>>>>>>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>>>>>>
>>>>>>>>>> Can the user return "not yet" somehow to this if they have a
>>>>>>>>>> fixed output buffer pool but no buffer is currently
>>>>>>>>>> available?
>>>>>>>>>
>>>>>>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I don't much like the idea of the user suspending the thread
>>>>>>>>>> in the callback until they have some available, which might
>>>>>>>>>> work in some cases but might also deadlock if an
>>>>>>>>>> avcodec_receive_packet() call is blocked by it.
>>>>>>>>>
>>>>>>>>> Can we make what's in essence a malloc() call return something
>>>>>>>>> like EAGAIN, and this in turn be propagated back to
>>>>>>>>> encode_receive_packet_internal()?
>>>>>>>>
>>>>>>>> Maybe, or if it has many threads maybe it could wait for something
>>>>>>>> else to finish first.
>>>>>>>>
>>>>>>>>> Couldn't this potentially end up in the forbidden scenario of
>>>>>>>>> avcodec_send_frame() and avcodec_receive_packet() both returning
>>>>>>>>> EAGAIN?
>>>>>>>>
>>>>>>>> Yes.  If the forbidden case happens then the encoder is stuck anyway
>>>>>>>> and can't make any forward progress so we need to error out
>>>>>>>> properly, but the EAGAIN return isn't needed if there is something
>>>>>>>> else to do on another thread.
>>>>>>>
>>>>>>> Ok, but I'm not familiar or knowledgeable enough with the frame thread
>>>>>>> encoder code to implement this.
>>>>>>
>>>>>> Looked at bit into this. AVCodec->encode2() based encoders don't support
>>>>>> returning EAGAIN at all, as it completely breaks the frame threading logic.
>>>>>> It would require a considerable rewrite in order to re-add a task that
>>>>>> didn't fail but also didn't succeed.
>>>>>>
>>>>>> Non frame threading encoders could probably support it with some minimal
>>>>>> changes, but i don't think suddenly letting an scenario that was until now
>>>>>> guaranteed to never happen start happening (avcodec_send_frame() and
>>>>>> avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
>>>>>> break.
>>>>>> Letting the user's custom get_encode_buffer() callback suspend the thread is
>>>>>> IMO acceptable. In frame threading scenarios, the other threads are still
>>>>>> working on their own packets (afaics none depends on the others, since it's
>>>>>> intra only encoders only).
>>>>>
>>>>> I think it was not suggested in the thread so:
>>>>> if the users allocation fails the code can fallback to the default allocator
>>>>> That would lead to the relation:
>>>>> If a users allocator can fail (out of buffers) it must be able to handle
>>>>> that only some of the returned packets are from its own allocator
>>>>
>>>> In general, custom allocators are used when the caller doesn't want to use
>>>> the default one. But yes, they could use
>>>> avcodec_default_get_encoder_buffer() as fallback, which is why it was added
>>>> to begin with. Same applies to get_buffer2() custom implementations, and so
>>>> far i don't think anybody had issues identifying what allocated a packet
>>>> buffer.
>>>>
>>>> One of the additions to AVPacket people were talking about was a user opaque
>>>> field that libav* would never touch or look at beyond propagating them
>>>> around all the way to the output AVFrame, if any. This opaque field could
>>>> perhaps store such allocator specific information the caller could use to
>>>> identify packets allocated by their own allocator, or those by
>>>> avcodec_default_get_encoder_buffer().
>>>>
>>>>>
>>>>> About alignment, we should at least recommand that allocated packets are
>>>>> aligned not less than what out av_malloc() would align to.
>>>>> Is there a reason to align less ?
>>>>
>>>> There's no alignment requirement for AVPacket->data, and av_new_packet()
>>>> uses av_buffer_realloc(), which does not guarantee any alignment whatsoever
>>>> on platforms other than Windows. So basically, packet payload buffers
>>>> allocated by our own helpers never had any alignment.
>>>
>>> for the purpose of exporting raw images, alignment would be "nice to have"
>>> because later filters may need it or need to memcpy
>>
>> Filters don't use AVPackets, they use AVFrames.
> 
> demuxers return AVPackets, so do encoders.
> These can contain raw frames.
> 
> also i see for example in rawdec:
> frame->buf[0] = av_buffer_ref(avpkt->buf);
> 

That seems to be completely broken: I see no check for whether the
packet is writable at all. Will investigate.

- Andreas

On 3/12/2021 1:32 PM, Michael Niedermayer wrote:
> On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
>> On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
>>> On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
>>>> On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
>>>>> On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
>>>>>> On 2/21/2021 6:04 PM, James Almer wrote:
>>>>>>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>>>>>>> On 21/02/2021 20:00, James Almer wrote:
>>>>>>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>>>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>>>>>>> This callback is functionally the same as get_buffer2()
>>>>>>>>>>> is for decoders, and
>>>>>>>>>>> implements for the new encode API the functionality of
>>>>>>>>>>> the old encode API had
>>>>>>>>>>> where the user could provide their own buffers.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>>>>>>> ---
>>>>>>>>>>> Used the names Lynne suggested this time, plus a line
>>>>>>>>>>> about how the callback
>>>>>>>>>>> must be thread safe.
>>>>>>>>>>>
>>>>>>>>>>>      libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>      libavcodec/codec.h   |  8 ++++---
>>>>>>>>>>>      libavcodec/encode.c  | 54
>>>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>      libavcodec/encode.h  |  8 +++++++
>>>>>>>>>>>      libavcodec/options.c |  1 +
>>>>>>>>>>>      5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>>>>>>> --- a/libavcodec/avcodec.h
>>>>>>>>>>> +++ b/libavcodec/avcodec.h
>>>>>>>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>>>>>>>       */
>>>>>>>>>>>      #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>> +/**
>>>>>>>>>>> + * The encoder will keep a reference to the packet and
>>>>>>>>>>> may reuse it later.
>>>>>>>>>>> + */
>>>>>>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>> +
>>>>>>>>>>>      struct AVCodecInternal;
>>>>>>>>>>>      /**
>>>>>>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>>>>>>           * - encoding: set by user
>>>>>>>>>>>           */
>>>>>>>>>>>          int export_side_data;
>>>>>>>>>>> +
>>>>>>>>>>> +    /**
>>>>>>>>>>> +     * This callback is called at the beginning of each
>>>>>>>>>>> packet to get a data
>>>>>>>>>>> +     * buffer for it.
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * The following field will be set in the packet
>>>>>>>>>>> before this callback is
>>>>>>>>>>> +     * called:
>>>>>>>>>>> +     * - size
>>>>>>>>>>> +     * This callback must use the above value to
>>>>>>>>>>> calculate the required buffer size,
>>>>>>>>>>> +     * which must padded by at least
>>>>>>>>>>> AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * This callback must fill the following fields in the packet:
>>>>>>>>>>> +     * - data
>>>>>>>>>>
>>>>>>>>>> Is the data pointer allowed to be in write-only memory?
>>>>>>>>>
>>>>>>>>> I'm not sure what the use case for this would be, so probably no?
>>>>>>>>
>>>>>>>> The two use-cases I see for this API are:
>>>>>>>>
>>>>>>>> * You want to avoid a copy when combining the output with something
>>>>>>>> else.  E.g. you pass a pointer to the block of memory following
>>>>>>>> where you are going to put your header data (for something you are
>>>>>>>> going to send over the network, say).
>>>>>>>>
>>>>>>>> * You want to avoid a copy when passing the output directly to
>>>>>>>> something external.  E.g. you pass a pointer to a memory-mapped
>>>>>>>> device buffer (such as a V4L2 buffer, say).
>>>>>>>>
>>>>>>>> In the second case, write-only memory on an external device seems
>>>>>>>> possible, as does memory which is, say, readable but uncached, so
>>>>>>>> reading it is a really bad idea.
>>>>>>>
>>>>>>> Allowing the second case would depend on how encoders behave. Some may
>>>>>>> attempt to read data already written to the output packet. It's not like
>>>>>>> all of them allocate the packet, do a memcpy from an internal buffer,
>>>>>>> then return.
>>>>>>> There is also the flag meant to signal that the encoder will keep a
>>>>>>> reference to the packet around, which more or less implies it will be
>>>>>>> read later in the encoding process.
>>>>>>>
>>>>>>> The doxy for avcodec_encode_video2(), which allowed the user to provide
>>>>>>> their own buffers in the output packet, does not mention any kind of
>>>>>>> requirement for the data pointer, so I don't think we can say it's an
>>>>>>> allowed scenario here either.
>>>>>>>
>>>>>>>>
>>>>>>>>>> Does it have any alignment requirements?
>>>>>>>>>
>>>>>>>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>>>>>>>
>>>>>>>> I think say that explicitly.  avcodec_default_get_encoder_buffer()
>>>>>>>> does give you aligned memory, even though it isn't needed.
>>>>>>>
>>>>>>> Would saying "There's no alignment requirement for the data pointer" add
>>>>>>> anything of value to the doxy? If i don't mention any kind of alignment
>>>>>>> requirement, it's because there isn't any, and it's implicit.
>>>>>>> I listed the requirements the user needs to keep in mind, like the
>>>>>>> padding and the need for an AVBufferRef. But if you think it's worth
>>>>>>> adding, then sure.
>>>>>>>
>>>>>>>>
>>>>>>>>>>> +     * - buf must contain a pointer to an AVBufferRef
>>>>>>>>>>> structure. The packet's
>>>>>>>>>>> +     *   data pointer must be contained in it.
>>>>>>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(),
>>>>>>>>>>> and av_buffer_ref().
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then
>>>>>>>>>>> get_encoder_buffer() must call
>>>>>>>>>>> +     * avcodec_default_get_encoder_buffer() instead of
>>>>>>>>>>> providing a buffer allocated by
>>>>>>>>>>> +     * some other means.
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
>>>>>>>>>>> flags then the packet may be reused
>>>>>>>>>>> +     * (read and/or written to if it is writable) later
>>>>>>>>>>> by libavcodec.
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * This callback must be thread-safe, as when frame
>>>>>>>>>>> multithreading is used, it may
>>>>>>>>>>> +     * be called from multiple threads simultaneously.
>>>>>>>>>>
>>>>>>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is
>>>>>>>>>> it really necessary?
>>>>>>>>>
>>>>>>>>> This was a suggestion by Lynne, i personally don't know. We
>>>>>>>>> support frame threading encoding (For intra-only codecs), but
>>>>>>>>> currently ff_alloc_packet2() does not seem to be thread safe,
>>>>>>>>> seeing it calls av_fast_padded_malloc(), yet it's called by
>>>>>>>>> frame threaded encoders.
>>>>>>>>> Should i remove this?
>>>>>>>>
>>>>>>>> I don't know, I was asking only because it sounds tricky.  For cases
>>>>>>>> with a limited number of buffers available (like memory-mapped
>>>>>>>> devices) you are going to need locking anyway, so maybe rentrancy
>>>>>>>> adds no additional inconvenience.
>>>>>>>>
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>>>>>>> +     *
>>>>>>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>>>>>>> +     * - decoding: unused
>>>>>>>>>>> +     */
>>>>>>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s,
>>>>>>>>>>> AVPacket *pkt, int flags);
>>>>>>>>>>
>>>>>>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>>>>>>
>>>>>>>>>> Can the user return "not yet" somehow to this if they have a
>>>>>>>>>> fixed output buffer pool but no buffer is currently
>>>>>>>>>> available?
>>>>>>>>>
>>>>>>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I don't much like the idea of the user suspending the thread
>>>>>>>>>> in the callback until they have some available, which might
>>>>>>>>>> work in some cases but might also deadlock if an
>>>>>>>>>> avcodec_receive_packet() call is blocked by it.
>>>>>>>>>
>>>>>>>>> Can we make what's in essence a malloc() call return something
>>>>>>>>> like EAGAIN, and this in turn be propagated back to
>>>>>>>>> encode_receive_packet_internal()?
>>>>>>>>
>>>>>>>> Maybe, or if it has many threads maybe it could wait for something
>>>>>>>> else to finish first.
>>>>>>>>
>>>>>>>>> Couldn't this potentially end up in the forbidden scenario of
>>>>>>>>> avcodec_send_frame() and avcodec_receive_packet() both returning
>>>>>>>>> EAGAIN?
>>>>>>>>
>>>>>>>> Yes.  If the forbidden case happens then the encoder is stuck anyway
>>>>>>>> and can't make any forward progress so we need to error out
>>>>>>>> properly, but the EAGAIN return isn't needed if there is something
>>>>>>>> else to do on another thread.
>>>>>>>
>>>>>>> Ok, but I'm not familiar or knowledgeable enough with the frame thread
>>>>>>> encoder code to implement this.
>>>>>>
>>>>>> Looked at bit into this. AVCodec->encode2() based encoders don't support
>>>>>> returning EAGAIN at all, as it completely breaks the frame threading logic.
>>>>>> It would require a considerable rewrite in order to re-add a task that
>>>>>> didn't fail but also didn't succeed.
>>>>>>
>>>>>> Non frame threading encoders could probably support it with some minimal
>>>>>> changes, but i don't think suddenly letting an scenario that was until now
>>>>>> guaranteed to never happen start happening (avcodec_send_frame() and
>>>>>> avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
>>>>>> break.
>>>>>> Letting the user's custom get_encode_buffer() callback suspend the thread is
>>>>>> IMO acceptable. In frame threading scenarios, the other threads are still
>>>>>> working on their own packets (afaics none depends on the others, since it's
>>>>>> intra only encoders only).
>>>>>
>>>>> I think it was not suggested in the thread so:
>>>>> if the users allocation fails the code can fallback to the default allocator
>>>>> That would lead to the relation:
>>>>> If a users allocator can fail (out of buffers) it must be able to handle
>>>>> that only some of the returned packets are from its own allocator
>>>>
>>>> In general, custom allocators are used when the caller doesn't want to use
>>>> the default one. But yes, they could use
>>>> avcodec_default_get_encoder_buffer() as fallback, which is why it was added
>>>> to begin with. Same applies to get_buffer2() custom implementations, and so
>>>> far i don't think anybody had issues identifying what allocated a packet
>>>> buffer.
>>>>
>>>> One of the additions to AVPacket people were talking about was a user opaque
>>>> field that libav* would never touch or look at beyond propagating them
>>>> around all the way to the output AVFrame, if any. This opaque field could
>>>> perhaps store such allocator specific information the caller could use to
>>>> identify packets allocated by their own allocator, or those by
>>>> avcodec_default_get_encoder_buffer().
>>>>
>>>>>
>>>>> About alignment, we should at least recommand that allocated packets are
>>>>> aligned not less than what out av_malloc() would align to.
>>>>> Is there a reason to align less ?
>>>>
>>>> There's no alignment requirement for AVPacket->data, and av_new_packet()
>>>> uses av_buffer_realloc(), which does not guarantee any alignment whatsoever
>>>> on platforms other than Windows. So basically, packet payload buffers
>>>> allocated by our own helpers never had any alignment.
>>>
>>> for the purpose of exporting raw images, alignment would be "nice to have"
>>> because later filters may need it or need to memcpy
>>
>> Filters don't use AVPackets, they use AVFrames.
> 
> demuxers return AVPackets, so do encoders.
> These can contain raw frames.
> 
> also i see for example in rawdec:
> frame->buf[0] = av_buffer_ref(avpkt->buf);

I ask again, where are you going with this? The alignment for AVPacket 
data buffers is defined: There is *none*.
It never mattered, with the old or the new API. With the old, users 
could pass an AVPacket to avcodec_encode_video2() with data pointing to 
whatever they wanted. With the new, I'm trying to retain that exact 
possibility with the addition of this callback.

In my local copy of this patch, where i addressed the request by Lynne 
to rename the callback to get_encode_buffer(), i also added a mention to 
the section where it mentions how to fill pkt->data that says "alignment 
requirements for AVPacket apply, if any".
Any unnecessary constrain to what users can make pkt->data point to in 
this callback over what an AVPacket itself requires will mean the 
callback will no longer be a direct replacement for the user provided 
buffer feature from the old encode API. It will go from users simply 
porting their code from the old API to the new, to users porting their 
code *and* being forced to redefine their buffers for no reason.

And don't start talking about how packets contain frames, because I'm 
going to remind people of the travesty that is the wrapped_avframe 
encoder and the vapoursynth demuxer using sizeof(AVFrame), and nobody 
but me calling the latter out when it was introduced, and it being 
committed anyway.

On Fri, Mar 12, 2021 at 02:03:52PM -0300, James Almer wrote:
> On 3/12/2021 1:32 PM, Michael Niedermayer wrote:
> > On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
> > > On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
> > > > On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
> > > > > On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
> > > > > > On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
> > > > > > > On 2/21/2021 6:04 PM, James Almer wrote:
> > > > > > > > On 2/21/2021 5:29 PM, Mark Thompson wrote:
> > > > > > > > > On 21/02/2021 20:00, James Almer wrote:
> > > > > > > > > > On 2/21/2021 4:13 PM, Mark Thompson wrote:
> > > > > > > > > > > On 21/02/2021 17:35, James Almer wrote:
> > > > > > > > > > > > This callback is functionally the same as get_buffer2()
> > > > > > > > > > > > is for decoders, and
> > > > > > > > > > > > implements for the new encode API the functionality of
> > > > > > > > > > > > the old encode API had
> > > > > > > > > > > > where the user could provide their own buffers.
> > > > > > > > > > > > 
> > > > > > > > > > > > Signed-off-by: James Almer <jamrial@gmail.com>
> > > > > > > > > > > > ---
> > > > > > > > > > > > Used the names Lynne suggested this time, plus a line
> > > > > > > > > > > > about how the callback
> > > > > > > > > > > > must be thread safe.
> > > > > > > > > > > > 
> > > > > > > > > > > >      libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > >      libavcodec/codec.h   |  8 ++++---
> > > > > > > > > > > >      libavcodec/encode.c  | 54
> > > > > > > > > > > > +++++++++++++++++++++++++++++++++++++++++++-
> > > > > > > > > > > >      libavcodec/encode.h  |  8 +++++++
> > > > > > > > > > > >      libavcodec/options.c |  1 +
> > > > > > > > > > > >      5 files changed, 112 insertions(+), 4 deletions(-)
> > > > > > > > > > > > 
> > > > > > > > > > > > diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
> > > > > > > > > > > > index 7dbf083a24..e60eb16ce1 100644
> > > > > > > > > > > > --- a/libavcodec/avcodec.h
> > > > > > > > > > > > +++ b/libavcodec/avcodec.h
> > > > > > > > > > > > @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
> > > > > > > > > > > >       */
> > > > > > > > > > > >      #define AV_GET_BUFFER_FLAG_REF (1 << 0)
> > > > > > > > > > > > +/**
> > > > > > > > > > > > + * The encoder will keep a reference to the packet and
> > > > > > > > > > > > may reuse it later.
> > > > > > > > > > > > + */
> > > > > > > > > > > > +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
> > > > > > > > > > > > +
> > > > > > > > > > > >      struct AVCodecInternal;
> > > > > > > > > > > >      /**
> > > > > > > > > > > > @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
> > > > > > > > > > > >           * - encoding: set by user
> > > > > > > > > > > >           */
> > > > > > > > > > > >          int export_side_data;
> > > > > > > > > > > > +
> > > > > > > > > > > > +    /**
> > > > > > > > > > > > +     * This callback is called at the beginning of each
> > > > > > > > > > > > packet to get a data
> > > > > > > > > > > > +     * buffer for it.
> > > > > > > > > > > > +     *
> > > > > > > > > > > > +     * The following field will be set in the packet
> > > > > > > > > > > > before this callback is
> > > > > > > > > > > > +     * called:
> > > > > > > > > > > > +     * - size
> > > > > > > > > > > > +     * This callback must use the above value to
> > > > > > > > > > > > calculate the required buffer size,
> > > > > > > > > > > > +     * which must padded by at least
> > > > > > > > > > > > AV_INPUT_BUFFER_PADDING_SIZE bytes.
> > > > > > > > > > > > +     *
> > > > > > > > > > > > +     * This callback must fill the following fields in the packet:
> > > > > > > > > > > > +     * - data
> > > > > > > > > > > 
> > > > > > > > > > > Is the data pointer allowed to be in write-only memory?
> > > > > > > > > > 
> > > > > > > > > > I'm not sure what the use case for this would be, so probably no?
> > > > > > > > > 
> > > > > > > > > The two use-cases I see for this API are:
> > > > > > > > > 
> > > > > > > > > * You want to avoid a copy when combining the output with something
> > > > > > > > > else.  E.g. you pass a pointer to the block of memory following
> > > > > > > > > where you are going to put your header data (for something you are
> > > > > > > > > going to send over the network, say).
> > > > > > > > > 
> > > > > > > > > * You want to avoid a copy when passing the output directly to
> > > > > > > > > something external.  E.g. you pass a pointer to a memory-mapped
> > > > > > > > > device buffer (such as a V4L2 buffer, say).
> > > > > > > > > 
> > > > > > > > > In the second case, write-only memory on an external device seems
> > > > > > > > > possible, as does memory which is, say, readable but uncached, so
> > > > > > > > > reading it is a really bad idea.
> > > > > > > > 
> > > > > > > > Allowing the second case would depend on how encoders behave. Some may
> > > > > > > > attempt to read data already written to the output packet. It's not like
> > > > > > > > all of them allocate the packet, do a memcpy from an internal buffer,
> > > > > > > > then return.
> > > > > > > > There is also the flag meant to signal that the encoder will keep a
> > > > > > > > reference to the packet around, which more or less implies it will be
> > > > > > > > read later in the encoding process.
> > > > > > > > 
> > > > > > > > The doxy for avcodec_encode_video2(), which allowed the user to provide
> > > > > > > > their own buffers in the output packet, does not mention any kind of
> > > > > > > > requirement for the data pointer, so I don't think we can say it's an
> > > > > > > > allowed scenario here either.
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > > > Does it have any alignment requirements?
> > > > > > > > > > 
> > > > > > > > > > No, just padding. AVPacket doesn't require alignment for the payload.
> > > > > > > > > 
> > > > > > > > > I think say that explicitly.  avcodec_default_get_encoder_buffer()
> > > > > > > > > does give you aligned memory, even though it isn't needed.
> > > > > > > > 
> > > > > > > > Would saying "There's no alignment requirement for the data pointer" add
> > > > > > > > anything of value to the doxy? If i don't mention any kind of alignment
> > > > > > > > requirement, it's because there isn't any, and it's implicit.
> > > > > > > > I listed the requirements the user needs to keep in mind, like the
> > > > > > > > padding and the need for an AVBufferRef. But if you think it's worth
> > > > > > > > adding, then sure.
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > > > > +     * - buf must contain a pointer to an AVBufferRef
> > > > > > > > > > > > structure. The packet's
> > > > > > > > > > > > +     *   data pointer must be contained in it.
> > > > > > > > > > > > +     *   See: av_buffer_create(), av_buffer_alloc(),
> > > > > > > > > > > > and av_buffer_ref().
> > > > > > > > > > > > +     *
> > > > > > > > > > > > +     * If AV_CODEC_CAP_DR1 is not set then
> > > > > > > > > > > > get_encoder_buffer() must call
> > > > > > > > > > > > +     * avcodec_default_get_encoder_buffer() instead of
> > > > > > > > > > > > providing a buffer allocated by
> > > > > > > > > > > > +     * some other means.
> > > > > > > > > > > > +     *
> > > > > > > > > > > > +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
> > > > > > > > > > > > flags then the packet may be reused
> > > > > > > > > > > > +     * (read and/or written to if it is writable) later
> > > > > > > > > > > > by libavcodec.
> > > > > > > > > > > > +     *
> > > > > > > > > > > > +     * This callback must be thread-safe, as when frame
> > > > > > > > > > > > multithreading is used, it may
> > > > > > > > > > > > +     * be called from multiple threads simultaneously.
> > > > > > > > > > > 
> > > > > > > > > > > Allowing simulatenous calls feels unexpectedly tricky.  Is
> > > > > > > > > > > it really necessary?
> > > > > > > > > > 
> > > > > > > > > > This was a suggestion by Lynne, i personally don't know. We
> > > > > > > > > > support frame threading encoding (For intra-only codecs), but
> > > > > > > > > > currently ff_alloc_packet2() does not seem to be thread safe,
> > > > > > > > > > seeing it calls av_fast_padded_malloc(), yet it's called by
> > > > > > > > > > frame threaded encoders.
> > > > > > > > > > Should i remove this?
> > > > > > > > > 
> > > > > > > > > I don't know, I was asking only because it sounds tricky.  For cases
> > > > > > > > > with a limited number of buffers available (like memory-mapped
> > > > > > > > > devices) you are going to need locking anyway, so maybe rentrancy
> > > > > > > > > adds no additional inconvenience.
> > > > > > > > > 
> > > > > > > > > > > > +     *
> > > > > > > > > > > > +     * @see avcodec_default_get_encoder_buffer()
> > > > > > > > > > > > +     *
> > > > > > > > > > > > +     * - encoding: Set by libavcodec, user can override.
> > > > > > > > > > > > +     * - decoding: unused
> > > > > > > > > > > > +     */
> > > > > > > > > > > > +    int (*get_encoder_buffer)(struct AVCodecContext *s,
> > > > > > > > > > > > AVPacket *pkt, int flags);
> > > > > > > > > > > 
> > > > > > > > > > > Can the encoder ask for arbitrarily many packets?
> > > > > > > > > > > 
> > > > > > > > > > > Can the user return "not yet" somehow to this if they have a
> > > > > > > > > > > fixed output buffer pool but no buffer is currently
> > > > > > > > > > > available?
> > > > > > > > > > 
> > > > > > > > > > No, as is it can't. Return values < 0 are considered errors.
> > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > I don't much like the idea of the user suspending the thread
> > > > > > > > > > > in the callback until they have some available, which might
> > > > > > > > > > > work in some cases but might also deadlock if an
> > > > > > > > > > > avcodec_receive_packet() call is blocked by it.
> > > > > > > > > > 
> > > > > > > > > > Can we make what's in essence a malloc() call return something
> > > > > > > > > > like EAGAIN, and this in turn be propagated back to
> > > > > > > > > > encode_receive_packet_internal()?
> > > > > > > > > 
> > > > > > > > > Maybe, or if it has many threads maybe it could wait for something
> > > > > > > > > else to finish first.
> > > > > > > > > 
> > > > > > > > > > Couldn't this potentially end up in the forbidden scenario of
> > > > > > > > > > avcodec_send_frame() and avcodec_receive_packet() both returning
> > > > > > > > > > EAGAIN?
> > > > > > > > > 
> > > > > > > > > Yes.  If the forbidden case happens then the encoder is stuck anyway
> > > > > > > > > and can't make any forward progress so we need to error out
> > > > > > > > > properly, but the EAGAIN return isn't needed if there is something
> > > > > > > > > else to do on another thread.
> > > > > > > > 
> > > > > > > > Ok, but I'm not familiar or knowledgeable enough with the frame thread
> > > > > > > > encoder code to implement this.
> > > > > > > 
> > > > > > > Looked at bit into this. AVCodec->encode2() based encoders don't support
> > > > > > > returning EAGAIN at all, as it completely breaks the frame threading logic.
> > > > > > > It would require a considerable rewrite in order to re-add a task that
> > > > > > > didn't fail but also didn't succeed.
> > > > > > > 
> > > > > > > Non frame threading encoders could probably support it with some minimal
> > > > > > > changes, but i don't think suddenly letting an scenario that was until now
> > > > > > > guaranteed to never happen start happening (avcodec_send_frame() and
> > > > > > > avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
> > > > > > > break.
> > > > > > > Letting the user's custom get_encode_buffer() callback suspend the thread is
> > > > > > > IMO acceptable. In frame threading scenarios, the other threads are still
> > > > > > > working on their own packets (afaics none depends on the others, since it's
> > > > > > > intra only encoders only).
> > > > > > 
> > > > > > I think it was not suggested in the thread so:
> > > > > > if the users allocation fails the code can fallback to the default allocator
> > > > > > That would lead to the relation:
> > > > > > If a users allocator can fail (out of buffers) it must be able to handle
> > > > > > that only some of the returned packets are from its own allocator
> > > > > 
> > > > > In general, custom allocators are used when the caller doesn't want to use
> > > > > the default one. But yes, they could use
> > > > > avcodec_default_get_encoder_buffer() as fallback, which is why it was added
> > > > > to begin with. Same applies to get_buffer2() custom implementations, and so
> > > > > far i don't think anybody had issues identifying what allocated a packet
> > > > > buffer.
> > > > > 
> > > > > One of the additions to AVPacket people were talking about was a user opaque
> > > > > field that libav* would never touch or look at beyond propagating them
> > > > > around all the way to the output AVFrame, if any. This opaque field could
> > > > > perhaps store such allocator specific information the caller could use to
> > > > > identify packets allocated by their own allocator, or those by
> > > > > avcodec_default_get_encoder_buffer().
> > > > > 
> > > > > > 
> > > > > > About alignment, we should at least recommand that allocated packets are
> > > > > > aligned not less than what out av_malloc() would align to.
> > > > > > Is there a reason to align less ?
> > > > > 
> > > > > There's no alignment requirement for AVPacket->data, and av_new_packet()
> > > > > uses av_buffer_realloc(), which does not guarantee any alignment whatsoever
> > > > > on platforms other than Windows. So basically, packet payload buffers
> > > > > allocated by our own helpers never had any alignment.
> > > > 
> > > > for the purpose of exporting raw images, alignment would be "nice to have"
> > > > because later filters may need it or need to memcpy
> > > 
> > > Filters don't use AVPackets, they use AVFrames.
> > 
> > demuxers return AVPackets, so do encoders.
> > These can contain raw frames.
> > 
> > also i see for example in rawdec:
> > frame->buf[0] = av_buffer_ref(avpkt->buf);
> 
> I ask again, where are you going with this? The alignment for AVPacket data
> buffers is defined: There is *none*.

I simply stated that 'alignment would be "nice to have"'.
and then showed some cases where it would be usefull.

I guess where iam going with this is, is the API you add extensible?
That is if something is not supported now, can it be added later without
adding a new API.

Thanks

[...]

On Fri, Mar 12, 2021 at 08:30:23PM +0100, Michael Niedermayer wrote:
> On Fri, Mar 12, 2021 at 02:03:52PM -0300, James Almer wrote:
> > On 3/12/2021 1:32 PM, Michael Niedermayer wrote:
> > > On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
> > > > On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
> > > > > On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
> > > > > > On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
> > > > > > > On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
> > > > > > > > On 2/21/2021 6:04 PM, James Almer wrote:
> > > > > > > > > On 2/21/2021 5:29 PM, Mark Thompson wrote:
> > > > > > > > > > On 21/02/2021 20:00, James Almer wrote:
> > > > > > > > > > > On 2/21/2021 4:13 PM, Mark Thompson wrote:
> > > > > > > > > > > > On 21/02/2021 17:35, James Almer wrote:
> > > > > > > > > > > > > This callback is functionally the same as get_buffer2()
> > > > > > > > > > > > > is for decoders, and
> > > > > > > > > > > > > implements for the new encode API the functionality of
> > > > > > > > > > > > > the old encode API had
> > > > > > > > > > > > > where the user could provide their own buffers.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Signed-off-by: James Almer <jamrial@gmail.com>
> > > > > > > > > > > > > ---
> > > > > > > > > > > > > Used the names Lynne suggested this time, plus a line
> > > > > > > > > > > > > about how the callback
> > > > > > > > > > > > > must be thread safe.
> > > > > > > > > > > > > 
> > > > > > > > > > > > >      libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > >      libavcodec/codec.h   |  8 ++++---
> > > > > > > > > > > > >      libavcodec/encode.c  | 54
> > > > > > > > > > > > > +++++++++++++++++++++++++++++++++++++++++++-
> > > > > > > > > > > > >      libavcodec/encode.h  |  8 +++++++
> > > > > > > > > > > > >      libavcodec/options.c |  1 +
> > > > > > > > > > > > >      5 files changed, 112 insertions(+), 4 deletions(-)
> > > > > > > > > > > > > 
> > > > > > > > > > > > > diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
> > > > > > > > > > > > > index 7dbf083a24..e60eb16ce1 100644
> > > > > > > > > > > > > --- a/libavcodec/avcodec.h
> > > > > > > > > > > > > +++ b/libavcodec/avcodec.h
> > > > > > > > > > > > > @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
> > > > > > > > > > > > >       */
> > > > > > > > > > > > >      #define AV_GET_BUFFER_FLAG_REF (1 << 0)
> > > > > > > > > > > > > +/**
> > > > > > > > > > > > > + * The encoder will keep a reference to the packet and
> > > > > > > > > > > > > may reuse it later.
> > > > > > > > > > > > > + */
> > > > > > > > > > > > > +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
> > > > > > > > > > > > > +
> > > > > > > > > > > > >      struct AVCodecInternal;
> > > > > > > > > > > > >      /**
> > > > > > > > > > > > > @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
> > > > > > > > > > > > >           * - encoding: set by user
> > > > > > > > > > > > >           */
> > > > > > > > > > > > >          int export_side_data;
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +    /**
> > > > > > > > > > > > > +     * This callback is called at the beginning of each
> > > > > > > > > > > > > packet to get a data
> > > > > > > > > > > > > +     * buffer for it.
> > > > > > > > > > > > > +     *
> > > > > > > > > > > > > +     * The following field will be set in the packet
> > > > > > > > > > > > > before this callback is
> > > > > > > > > > > > > +     * called:
> > > > > > > > > > > > > +     * - size
> > > > > > > > > > > > > +     * This callback must use the above value to
> > > > > > > > > > > > > calculate the required buffer size,
> > > > > > > > > > > > > +     * which must padded by at least
> > > > > > > > > > > > > AV_INPUT_BUFFER_PADDING_SIZE bytes.
> > > > > > > > > > > > > +     *
> > > > > > > > > > > > > +     * This callback must fill the following fields in the packet:
> > > > > > > > > > > > > +     * - data
> > > > > > > > > > > > 
> > > > > > > > > > > > Is the data pointer allowed to be in write-only memory?
> > > > > > > > > > > 
> > > > > > > > > > > I'm not sure what the use case for this would be, so probably no?
> > > > > > > > > > 
> > > > > > > > > > The two use-cases I see for this API are:
> > > > > > > > > > 
> > > > > > > > > > * You want to avoid a copy when combining the output with something
> > > > > > > > > > else.  E.g. you pass a pointer to the block of memory following
> > > > > > > > > > where you are going to put your header data (for something you are
> > > > > > > > > > going to send over the network, say).
> > > > > > > > > > 
> > > > > > > > > > * You want to avoid a copy when passing the output directly to
> > > > > > > > > > something external.  E.g. you pass a pointer to a memory-mapped
> > > > > > > > > > device buffer (such as a V4L2 buffer, say).
> > > > > > > > > > 
> > > > > > > > > > In the second case, write-only memory on an external device seems
> > > > > > > > > > possible, as does memory which is, say, readable but uncached, so
> > > > > > > > > > reading it is a really bad idea.
> > > > > > > > > 
> > > > > > > > > Allowing the second case would depend on how encoders behave. Some may
> > > > > > > > > attempt to read data already written to the output packet. It's not like
> > > > > > > > > all of them allocate the packet, do a memcpy from an internal buffer,
> > > > > > > > > then return.
> > > > > > > > > There is also the flag meant to signal that the encoder will keep a
> > > > > > > > > reference to the packet around, which more or less implies it will be
> > > > > > > > > read later in the encoding process.
> > > > > > > > > 
> > > > > > > > > The doxy for avcodec_encode_video2(), which allowed the user to provide
> > > > > > > > > their own buffers in the output packet, does not mention any kind of
> > > > > > > > > requirement for the data pointer, so I don't think we can say it's an
> > > > > > > > > allowed scenario here either.
> > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > > > Does it have any alignment requirements?
> > > > > > > > > > > 
> > > > > > > > > > > No, just padding. AVPacket doesn't require alignment for the payload.
> > > > > > > > > > 
> > > > > > > > > > I think say that explicitly.  avcodec_default_get_encoder_buffer()
> > > > > > > > > > does give you aligned memory, even though it isn't needed.
> > > > > > > > > 
> > > > > > > > > Would saying "There's no alignment requirement for the data pointer" add
> > > > > > > > > anything of value to the doxy? If i don't mention any kind of alignment
> > > > > > > > > requirement, it's because there isn't any, and it's implicit.
> > > > > > > > > I listed the requirements the user needs to keep in mind, like the
> > > > > > > > > padding and the need for an AVBufferRef. But if you think it's worth
> > > > > > > > > adding, then sure.
> > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > > > > +     * - buf must contain a pointer to an AVBufferRef
> > > > > > > > > > > > > structure. The packet's
> > > > > > > > > > > > > +     *   data pointer must be contained in it.
> > > > > > > > > > > > > +     *   See: av_buffer_create(), av_buffer_alloc(),
> > > > > > > > > > > > > and av_buffer_ref().
> > > > > > > > > > > > > +     *
> > > > > > > > > > > > > +     * If AV_CODEC_CAP_DR1 is not set then
> > > > > > > > > > > > > get_encoder_buffer() must call
> > > > > > > > > > > > > +     * avcodec_default_get_encoder_buffer() instead of
> > > > > > > > > > > > > providing a buffer allocated by
> > > > > > > > > > > > > +     * some other means.
> > > > > > > > > > > > > +     *
> > > > > > > > > > > > > +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
> > > > > > > > > > > > > flags then the packet may be reused
> > > > > > > > > > > > > +     * (read and/or written to if it is writable) later
> > > > > > > > > > > > > by libavcodec.
> > > > > > > > > > > > > +     *
> > > > > > > > > > > > > +     * This callback must be thread-safe, as when frame
> > > > > > > > > > > > > multithreading is used, it may
> > > > > > > > > > > > > +     * be called from multiple threads simultaneously.
> > > > > > > > > > > > 
> > > > > > > > > > > > Allowing simulatenous calls feels unexpectedly tricky.  Is
> > > > > > > > > > > > it really necessary?
> > > > > > > > > > > 
> > > > > > > > > > > This was a suggestion by Lynne, i personally don't know. We
> > > > > > > > > > > support frame threading encoding (For intra-only codecs), but
> > > > > > > > > > > currently ff_alloc_packet2() does not seem to be thread safe,
> > > > > > > > > > > seeing it calls av_fast_padded_malloc(), yet it's called by
> > > > > > > > > > > frame threaded encoders.
> > > > > > > > > > > Should i remove this?
> > > > > > > > > > 
> > > > > > > > > > I don't know, I was asking only because it sounds tricky.  For cases
> > > > > > > > > > with a limited number of buffers available (like memory-mapped
> > > > > > > > > > devices) you are going to need locking anyway, so maybe rentrancy
> > > > > > > > > > adds no additional inconvenience.
> > > > > > > > > > 
> > > > > > > > > > > > > +     *
> > > > > > > > > > > > > +     * @see avcodec_default_get_encoder_buffer()
> > > > > > > > > > > > > +     *
> > > > > > > > > > > > > +     * - encoding: Set by libavcodec, user can override.
> > > > > > > > > > > > > +     * - decoding: unused
> > > > > > > > > > > > > +     */
> > > > > > > > > > > > > +    int (*get_encoder_buffer)(struct AVCodecContext *s,
> > > > > > > > > > > > > AVPacket *pkt, int flags);
> > > > > > > > > > > > 
> > > > > > > > > > > > Can the encoder ask for arbitrarily many packets?
> > > > > > > > > > > > 
> > > > > > > > > > > > Can the user return "not yet" somehow to this if they have a
> > > > > > > > > > > > fixed output buffer pool but no buffer is currently
> > > > > > > > > > > > available?
> > > > > > > > > > > 
> > > > > > > > > > > No, as is it can't. Return values < 0 are considered errors.
> > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > I don't much like the idea of the user suspending the thread
> > > > > > > > > > > > in the callback until they have some available, which might
> > > > > > > > > > > > work in some cases but might also deadlock if an
> > > > > > > > > > > > avcodec_receive_packet() call is blocked by it.
> > > > > > > > > > > 
> > > > > > > > > > > Can we make what's in essence a malloc() call return something
> > > > > > > > > > > like EAGAIN, and this in turn be propagated back to
> > > > > > > > > > > encode_receive_packet_internal()?
> > > > > > > > > > 
> > > > > > > > > > Maybe, or if it has many threads maybe it could wait for something
> > > > > > > > > > else to finish first.
> > > > > > > > > > 
> > > > > > > > > > > Couldn't this potentially end up in the forbidden scenario of
> > > > > > > > > > > avcodec_send_frame() and avcodec_receive_packet() both returning
> > > > > > > > > > > EAGAIN?
> > > > > > > > > > 
> > > > > > > > > > Yes.  If the forbidden case happens then the encoder is stuck anyway
> > > > > > > > > > and can't make any forward progress so we need to error out
> > > > > > > > > > properly, but the EAGAIN return isn't needed if there is something
> > > > > > > > > > else to do on another thread.
> > > > > > > > > 
> > > > > > > > > Ok, but I'm not familiar or knowledgeable enough with the frame thread
> > > > > > > > > encoder code to implement this.
> > > > > > > > 
> > > > > > > > Looked at bit into this. AVCodec->encode2() based encoders don't support
> > > > > > > > returning EAGAIN at all, as it completely breaks the frame threading logic.
> > > > > > > > It would require a considerable rewrite in order to re-add a task that
> > > > > > > > didn't fail but also didn't succeed.
> > > > > > > > 
> > > > > > > > Non frame threading encoders could probably support it with some minimal
> > > > > > > > changes, but i don't think suddenly letting an scenario that was until now
> > > > > > > > guaranteed to never happen start happening (avcodec_send_frame() and
> > > > > > > > avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
> > > > > > > > break.
> > > > > > > > Letting the user's custom get_encode_buffer() callback suspend the thread is
> > > > > > > > IMO acceptable. In frame threading scenarios, the other threads are still
> > > > > > > > working on their own packets (afaics none depends on the others, since it's
> > > > > > > > intra only encoders only).
> > > > > > > 
> > > > > > > I think it was not suggested in the thread so:
> > > > > > > if the users allocation fails the code can fallback to the default allocator
> > > > > > > That would lead to the relation:
> > > > > > > If a users allocator can fail (out of buffers) it must be able to handle
> > > > > > > that only some of the returned packets are from its own allocator
> > > > > > 
> > > > > > In general, custom allocators are used when the caller doesn't want to use
> > > > > > the default one. But yes, they could use
> > > > > > avcodec_default_get_encoder_buffer() as fallback, which is why it was added
> > > > > > to begin with. Same applies to get_buffer2() custom implementations, and so
> > > > > > far i don't think anybody had issues identifying what allocated a packet
> > > > > > buffer.
> > > > > > 
> > > > > > One of the additions to AVPacket people were talking about was a user opaque
> > > > > > field that libav* would never touch or look at beyond propagating them
> > > > > > around all the way to the output AVFrame, if any. This opaque field could
> > > > > > perhaps store such allocator specific information the caller could use to
> > > > > > identify packets allocated by their own allocator, or those by
> > > > > > avcodec_default_get_encoder_buffer().
> > > > > > 
> > > > > > > 
> > > > > > > About alignment, we should at least recommand that allocated packets are
> > > > > > > aligned not less than what out av_malloc() would align to.
> > > > > > > Is there a reason to align less ?
> > > > > > 
> > > > > > There's no alignment requirement for AVPacket->data, and av_new_packet()
> > > > > > uses av_buffer_realloc(), which does not guarantee any alignment whatsoever
> > > > > > on platforms other than Windows. So basically, packet payload buffers
> > > > > > allocated by our own helpers never had any alignment.
> > > > > 
> > > > > for the purpose of exporting raw images, alignment would be "nice to have"
> > > > > because later filters may need it or need to memcpy
> > > > 
> > > > Filters don't use AVPackets, they use AVFrames.
> > > 
> > > demuxers return AVPackets, so do encoders.
> > > These can contain raw frames.
> > > 
> > > also i see for example in rawdec:
> > > frame->buf[0] = av_buffer_ref(avpkt->buf);
> > 
> > I ask again, where are you going with this? The alignment for AVPacket data
> > buffers is defined: There is *none*.
> 
> I simply stated that 'alignment would be "nice to have"'.
> and then showed some cases where it would be usefull.
> 
> I guess where iam going with this is, is the API you add extensible?
> That is if something is not supported now, can it be added later without
> adding a new API.

Related to this, what do the flags passed into get_encoder_buffer() mean
exactly ?
are they "hints" so that a new/unknown flag can be ignored by an implementation
or what else should an application do with a unknown flag, this should
be clarified in the documentation

thx


[...]

On 3/12/2021 4:30 PM, Michael Niedermayer wrote:
> On Fri, Mar 12, 2021 at 02:03:52PM -0300, James Almer wrote:
>> On 3/12/2021 1:32 PM, Michael Niedermayer wrote:
>>> On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
>>>> On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
>>>>> On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
>>>>>> On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
>>>>>>> On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
>>>>>>>> On 2/21/2021 6:04 PM, James Almer wrote:
>>>>>>>>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>>>>>>>>> On 21/02/2021 20:00, James Almer wrote:
>>>>>>>>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>>>>>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>>>>>>>>> This callback is functionally the same as get_buffer2()
>>>>>>>>>>>>> is for decoders, and
>>>>>>>>>>>>> implements for the new encode API the functionality of
>>>>>>>>>>>>> the old encode API had
>>>>>>>>>>>>> where the user could provide their own buffers.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> Used the names Lynne suggested this time, plus a line
>>>>>>>>>>>>> about how the callback
>>>>>>>>>>>>> must be thread safe.
>>>>>>>>>>>>>
>>>>>>>>>>>>>       libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>       libavcodec/codec.h   |  8 ++++---
>>>>>>>>>>>>>       libavcodec/encode.c  | 54
>>>>>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>       libavcodec/encode.h  |  8 +++++++
>>>>>>>>>>>>>       libavcodec/options.c |  1 +
>>>>>>>>>>>>>       5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>>>>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>>>>>>>>> --- a/libavcodec/avcodec.h
>>>>>>>>>>>>> +++ b/libavcodec/avcodec.h
>>>>>>>>>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>>>>>>>>>        */
>>>>>>>>>>>>>       #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>>> +/**
>>>>>>>>>>>>> + * The encoder will keep a reference to the packet and
>>>>>>>>>>>>> may reuse it later.
>>>>>>>>>>>>> + */
>>>>>>>>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>>> +
>>>>>>>>>>>>>       struct AVCodecInternal;
>>>>>>>>>>>>>       /**
>>>>>>>>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>>>>>>>>            * - encoding: set by user
>>>>>>>>>>>>>            */
>>>>>>>>>>>>>           int export_side_data;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    /**
>>>>>>>>>>>>> +     * This callback is called at the beginning of each
>>>>>>>>>>>>> packet to get a data
>>>>>>>>>>>>> +     * buffer for it.
>>>>>>>>>>>>> +     *
>>>>>>>>>>>>> +     * The following field will be set in the packet
>>>>>>>>>>>>> before this callback is
>>>>>>>>>>>>> +     * called:
>>>>>>>>>>>>> +     * - size
>>>>>>>>>>>>> +     * This callback must use the above value to
>>>>>>>>>>>>> calculate the required buffer size,
>>>>>>>>>>>>> +     * which must padded by at least
>>>>>>>>>>>>> AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>>>>>>>>> +     *
>>>>>>>>>>>>> +     * This callback must fill the following fields in the packet:
>>>>>>>>>>>>> +     * - data
>>>>>>>>>>>>
>>>>>>>>>>>> Is the data pointer allowed to be in write-only memory?
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure what the use case for this would be, so probably no?
>>>>>>>>>>
>>>>>>>>>> The two use-cases I see for this API are:
>>>>>>>>>>
>>>>>>>>>> * You want to avoid a copy when combining the output with something
>>>>>>>>>> else.  E.g. you pass a pointer to the block of memory following
>>>>>>>>>> where you are going to put your header data (for something you are
>>>>>>>>>> going to send over the network, say).
>>>>>>>>>>
>>>>>>>>>> * You want to avoid a copy when passing the output directly to
>>>>>>>>>> something external.  E.g. you pass a pointer to a memory-mapped
>>>>>>>>>> device buffer (such as a V4L2 buffer, say).
>>>>>>>>>>
>>>>>>>>>> In the second case, write-only memory on an external device seems
>>>>>>>>>> possible, as does memory which is, say, readable but uncached, so
>>>>>>>>>> reading it is a really bad idea.
>>>>>>>>>
>>>>>>>>> Allowing the second case would depend on how encoders behave. Some may
>>>>>>>>> attempt to read data already written to the output packet. It's not like
>>>>>>>>> all of them allocate the packet, do a memcpy from an internal buffer,
>>>>>>>>> then return.
>>>>>>>>> There is also the flag meant to signal that the encoder will keep a
>>>>>>>>> reference to the packet around, which more or less implies it will be
>>>>>>>>> read later in the encoding process.
>>>>>>>>>
>>>>>>>>> The doxy for avcodec_encode_video2(), which allowed the user to provide
>>>>>>>>> their own buffers in the output packet, does not mention any kind of
>>>>>>>>> requirement for the data pointer, so I don't think we can say it's an
>>>>>>>>> allowed scenario here either.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> Does it have any alignment requirements?
>>>>>>>>>>>
>>>>>>>>>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>>>>>>>>>
>>>>>>>>>> I think say that explicitly.  avcodec_default_get_encoder_buffer()
>>>>>>>>>> does give you aligned memory, even though it isn't needed.
>>>>>>>>>
>>>>>>>>> Would saying "There's no alignment requirement for the data pointer" add
>>>>>>>>> anything of value to the doxy? If i don't mention any kind of alignment
>>>>>>>>> requirement, it's because there isn't any, and it's implicit.
>>>>>>>>> I listed the requirements the user needs to keep in mind, like the
>>>>>>>>> padding and the need for an AVBufferRef. But if you think it's worth
>>>>>>>>> adding, then sure.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> +     * - buf must contain a pointer to an AVBufferRef
>>>>>>>>>>>>> structure. The packet's
>>>>>>>>>>>>> +     *   data pointer must be contained in it.
>>>>>>>>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(),
>>>>>>>>>>>>> and av_buffer_ref().
>>>>>>>>>>>>> +     *
>>>>>>>>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then
>>>>>>>>>>>>> get_encoder_buffer() must call
>>>>>>>>>>>>> +     * avcodec_default_get_encoder_buffer() instead of
>>>>>>>>>>>>> providing a buffer allocated by
>>>>>>>>>>>>> +     * some other means.
>>>>>>>>>>>>> +     *
>>>>>>>>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
>>>>>>>>>>>>> flags then the packet may be reused
>>>>>>>>>>>>> +     * (read and/or written to if it is writable) later
>>>>>>>>>>>>> by libavcodec.
>>>>>>>>>>>>> +     *
>>>>>>>>>>>>> +     * This callback must be thread-safe, as when frame
>>>>>>>>>>>>> multithreading is used, it may
>>>>>>>>>>>>> +     * be called from multiple threads simultaneously.
>>>>>>>>>>>>
>>>>>>>>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is
>>>>>>>>>>>> it really necessary?
>>>>>>>>>>>
>>>>>>>>>>> This was a suggestion by Lynne, i personally don't know. We
>>>>>>>>>>> support frame threading encoding (For intra-only codecs), but
>>>>>>>>>>> currently ff_alloc_packet2() does not seem to be thread safe,
>>>>>>>>>>> seeing it calls av_fast_padded_malloc(), yet it's called by
>>>>>>>>>>> frame threaded encoders.
>>>>>>>>>>> Should i remove this?
>>>>>>>>>>
>>>>>>>>>> I don't know, I was asking only because it sounds tricky.  For cases
>>>>>>>>>> with a limited number of buffers available (like memory-mapped
>>>>>>>>>> devices) you are going to need locking anyway, so maybe rentrancy
>>>>>>>>>> adds no additional inconvenience.
>>>>>>>>>>
>>>>>>>>>>>>> +     *
>>>>>>>>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>>>>>>>>> +     *
>>>>>>>>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>>>>>>>>> +     * - decoding: unused
>>>>>>>>>>>>> +     */
>>>>>>>>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s,
>>>>>>>>>>>>> AVPacket *pkt, int flags);
>>>>>>>>>>>>
>>>>>>>>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>>>>>>>>
>>>>>>>>>>>> Can the user return "not yet" somehow to this if they have a
>>>>>>>>>>>> fixed output buffer pool but no buffer is currently
>>>>>>>>>>>> available?
>>>>>>>>>>>
>>>>>>>>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I don't much like the idea of the user suspending the thread
>>>>>>>>>>>> in the callback until they have some available, which might
>>>>>>>>>>>> work in some cases but might also deadlock if an
>>>>>>>>>>>> avcodec_receive_packet() call is blocked by it.
>>>>>>>>>>>
>>>>>>>>>>> Can we make what's in essence a malloc() call return something
>>>>>>>>>>> like EAGAIN, and this in turn be propagated back to
>>>>>>>>>>> encode_receive_packet_internal()?
>>>>>>>>>>
>>>>>>>>>> Maybe, or if it has many threads maybe it could wait for something
>>>>>>>>>> else to finish first.
>>>>>>>>>>
>>>>>>>>>>> Couldn't this potentially end up in the forbidden scenario of
>>>>>>>>>>> avcodec_send_frame() and avcodec_receive_packet() both returning
>>>>>>>>>>> EAGAIN?
>>>>>>>>>>
>>>>>>>>>> Yes.  If the forbidden case happens then the encoder is stuck anyway
>>>>>>>>>> and can't make any forward progress so we need to error out
>>>>>>>>>> properly, but the EAGAIN return isn't needed if there is something
>>>>>>>>>> else to do on another thread.
>>>>>>>>>
>>>>>>>>> Ok, but I'm not familiar or knowledgeable enough with the frame thread
>>>>>>>>> encoder code to implement this.
>>>>>>>>
>>>>>>>> Looked at bit into this. AVCodec->encode2() based encoders don't support
>>>>>>>> returning EAGAIN at all, as it completely breaks the frame threading logic.
>>>>>>>> It would require a considerable rewrite in order to re-add a task that
>>>>>>>> didn't fail but also didn't succeed.
>>>>>>>>
>>>>>>>> Non frame threading encoders could probably support it with some minimal
>>>>>>>> changes, but i don't think suddenly letting an scenario that was until now
>>>>>>>> guaranteed to never happen start happening (avcodec_send_frame() and
>>>>>>>> avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
>>>>>>>> break.
>>>>>>>> Letting the user's custom get_encode_buffer() callback suspend the thread is
>>>>>>>> IMO acceptable. In frame threading scenarios, the other threads are still
>>>>>>>> working on their own packets (afaics none depends on the others, since it's
>>>>>>>> intra only encoders only).
>>>>>>>
>>>>>>> I think it was not suggested in the thread so:
>>>>>>> if the users allocation fails the code can fallback to the default allocator
>>>>>>> That would lead to the relation:
>>>>>>> If a users allocator can fail (out of buffers) it must be able to handle
>>>>>>> that only some of the returned packets are from its own allocator
>>>>>>
>>>>>> In general, custom allocators are used when the caller doesn't want to use
>>>>>> the default one. But yes, they could use
>>>>>> avcodec_default_get_encoder_buffer() as fallback, which is why it was added
>>>>>> to begin with. Same applies to get_buffer2() custom implementations, and so
>>>>>> far i don't think anybody had issues identifying what allocated a packet
>>>>>> buffer.
>>>>>>
>>>>>> One of the additions to AVPacket people were talking about was a user opaque
>>>>>> field that libav* would never touch or look at beyond propagating them
>>>>>> around all the way to the output AVFrame, if any. This opaque field could
>>>>>> perhaps store such allocator specific information the caller could use to
>>>>>> identify packets allocated by their own allocator, or those by
>>>>>> avcodec_default_get_encoder_buffer().
>>>>>>
>>>>>>>
>>>>>>> About alignment, we should at least recommand that allocated packets are
>>>>>>> aligned not less than what out av_malloc() would align to.
>>>>>>> Is there a reason to align less ?
>>>>>>
>>>>>> There's no alignment requirement for AVPacket->data, and av_new_packet()
>>>>>> uses av_buffer_realloc(), which does not guarantee any alignment whatsoever
>>>>>> on platforms other than Windows. So basically, packet payload buffers
>>>>>> allocated by our own helpers never had any alignment.
>>>>>
>>>>> for the purpose of exporting raw images, alignment would be "nice to have"
>>>>> because later filters may need it or need to memcpy
>>>>
>>>> Filters don't use AVPackets, they use AVFrames.
>>>
>>> demuxers return AVPackets, so do encoders.
>>> These can contain raw frames.
>>>
>>> also i see for example in rawdec:
>>> frame->buf[0] = av_buffer_ref(avpkt->buf);
>>
>> I ask again, where are you going with this? The alignment for AVPacket data
>> buffers is defined: There is *none*.
> 
> I simply stated that 'alignment would be "nice to have"'.
> and then showed some cases where it would be usefull.

But don't those cases already happen, and without required or guaranteed 
alignment?

> 
> I guess where iam going with this is, is the API you add extensible?
> That is if something is not supported now, can it be added later without
> adding a new API.

I should, it shares a signature with get_buffer2(). That means the 
packet to fill (Which fields can be read from it and set can be easily 
redefined), avctx so the user can have access to avctx->opaque and so we 
can eventually use something like a buffer pool in the default allocator 
callback, and a flags parameter to tell the callback there are requirements.

Which makes me realize, maybe a flag to tell the callback "Alignment is 
required" could solve your concerns?

> 
> Thanks
> 
> [...]
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>

On 3/12/2021 4:46 PM, James Almer wrote:
> On 3/12/2021 4:30 PM, Michael Niedermayer wrote:
>> On Fri, Mar 12, 2021 at 02:03:52PM -0300, James Almer wrote:
>>> On 3/12/2021 1:32 PM, Michael Niedermayer wrote:
>>>> On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
>>>>> On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
>>>>>> On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
>>>>>>> On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
>>>>>>>> On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
>>>>>>>>> On 2/21/2021 6:04 PM, James Almer wrote:
>>>>>>>>>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>>>>>>>>>> On 21/02/2021 20:00, James Almer wrote:
>>>>>>>>>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>>>>>>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>>>>>>>>>> This callback is functionally the same as get_buffer2()
>>>>>>>>>>>>>> is for decoders, and
>>>>>>>>>>>>>> implements for the new encode API the functionality of
>>>>>>>>>>>>>> the old encode API had
>>>>>>>>>>>>>> where the user could provide their own buffers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>> Used the names Lynne suggested this time, plus a line
>>>>>>>>>>>>>> about how the callback
>>>>>>>>>>>>>> must be thread safe.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>       libavcodec/avcodec.h | 45 
>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>       libavcodec/codec.h   |  8 ++++---
>>>>>>>>>>>>>>       libavcodec/encode.c  | 54
>>>>>>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>       libavcodec/encode.h  |  8 +++++++
>>>>>>>>>>>>>>       libavcodec/options.c |  1 +
>>>>>>>>>>>>>>       5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>>>>>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>>>>>>>>>> --- a/libavcodec/avcodec.h
>>>>>>>>>>>>>> +++ b/libavcodec/avcodec.h
>>>>>>>>>>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>>>>>>>>>>        */
>>>>>>>>>>>>>>       #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>> + * The encoder will keep a reference to the packet and
>>>>>>>>>>>>>> may reuse it later.
>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>       struct AVCodecInternal;
>>>>>>>>>>>>>>       /**
>>>>>>>>>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>>>>>>>>>            * - encoding: set by user
>>>>>>>>>>>>>>            */
>>>>>>>>>>>>>>           int export_side_data;
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +    /**
>>>>>>>>>>>>>> +     * This callback is called at the beginning of each
>>>>>>>>>>>>>> packet to get a data
>>>>>>>>>>>>>> +     * buffer for it.
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * The following field will be set in the packet
>>>>>>>>>>>>>> before this callback is
>>>>>>>>>>>>>> +     * called:
>>>>>>>>>>>>>> +     * - size
>>>>>>>>>>>>>> +     * This callback must use the above value to
>>>>>>>>>>>>>> calculate the required buffer size,
>>>>>>>>>>>>>> +     * which must padded by at least
>>>>>>>>>>>>>> AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * This callback must fill the following fields in 
>>>>>>>>>>>>>> the packet:
>>>>>>>>>>>>>> +     * - data
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is the data pointer allowed to be in write-only memory?
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure what the use case for this would be, so 
>>>>>>>>>>>> probably no?
>>>>>>>>>>>
>>>>>>>>>>> The two use-cases I see for this API are:
>>>>>>>>>>>
>>>>>>>>>>> * You want to avoid a copy when combining the output with 
>>>>>>>>>>> something
>>>>>>>>>>> else.  E.g. you pass a pointer to the block of memory following
>>>>>>>>>>> where you are going to put your header data (for something 
>>>>>>>>>>> you are
>>>>>>>>>>> going to send over the network, say).
>>>>>>>>>>>
>>>>>>>>>>> * You want to avoid a copy when passing the output directly to
>>>>>>>>>>> something external.  E.g. you pass a pointer to a memory-mapped
>>>>>>>>>>> device buffer (such as a V4L2 buffer, say).
>>>>>>>>>>>
>>>>>>>>>>> In the second case, write-only memory on an external device 
>>>>>>>>>>> seems
>>>>>>>>>>> possible, as does memory which is, say, readable but 
>>>>>>>>>>> uncached, so
>>>>>>>>>>> reading it is a really bad idea.
>>>>>>>>>>
>>>>>>>>>> Allowing the second case would depend on how encoders behave. 
>>>>>>>>>> Some may
>>>>>>>>>> attempt to read data already written to the output packet. 
>>>>>>>>>> It's not like
>>>>>>>>>> all of them allocate the packet, do a memcpy from an internal 
>>>>>>>>>> buffer,
>>>>>>>>>> then return.
>>>>>>>>>> There is also the flag meant to signal that the encoder will 
>>>>>>>>>> keep a
>>>>>>>>>> reference to the packet around, which more or less implies it 
>>>>>>>>>> will be
>>>>>>>>>> read later in the encoding process.
>>>>>>>>>>
>>>>>>>>>> The doxy for avcodec_encode_video2(), which allowed the user 
>>>>>>>>>> to provide
>>>>>>>>>> their own buffers in the output packet, does not mention any 
>>>>>>>>>> kind of
>>>>>>>>>> requirement for the data pointer, so I don't think we can say 
>>>>>>>>>> it's an
>>>>>>>>>> allowed scenario here either.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> Does it have any alignment requirements?
>>>>>>>>>>>>
>>>>>>>>>>>> No, just padding. AVPacket doesn't require alignment for the 
>>>>>>>>>>>> payload.
>>>>>>>>>>>
>>>>>>>>>>> I think say that explicitly.  
>>>>>>>>>>> avcodec_default_get_encoder_buffer()
>>>>>>>>>>> does give you aligned memory, even though it isn't needed.
>>>>>>>>>>
>>>>>>>>>> Would saying "There's no alignment requirement for the data 
>>>>>>>>>> pointer" add
>>>>>>>>>> anything of value to the doxy? If i don't mention any kind of 
>>>>>>>>>> alignment
>>>>>>>>>> requirement, it's because there isn't any, and it's implicit.
>>>>>>>>>> I listed the requirements the user needs to keep in mind, like 
>>>>>>>>>> the
>>>>>>>>>> padding and the need for an AVBufferRef. But if you think it's 
>>>>>>>>>> worth
>>>>>>>>>> adding, then sure.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> +     * - buf must contain a pointer to an AVBufferRef
>>>>>>>>>>>>>> structure. The packet's
>>>>>>>>>>>>>> +     *   data pointer must be contained in it.
>>>>>>>>>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(),
>>>>>>>>>>>>>> and av_buffer_ref().
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then
>>>>>>>>>>>>>> get_encoder_buffer() must call
>>>>>>>>>>>>>> +     * avcodec_default_get_encoder_buffer() instead of
>>>>>>>>>>>>>> providing a buffer allocated by
>>>>>>>>>>>>>> +     * some other means.
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
>>>>>>>>>>>>>> flags then the packet may be reused
>>>>>>>>>>>>>> +     * (read and/or written to if it is writable) later
>>>>>>>>>>>>>> by libavcodec.
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * This callback must be thread-safe, as when frame
>>>>>>>>>>>>>> multithreading is used, it may
>>>>>>>>>>>>>> +     * be called from multiple threads simultaneously.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is
>>>>>>>>>>>>> it really necessary?
>>>>>>>>>>>>
>>>>>>>>>>>> This was a suggestion by Lynne, i personally don't know. We
>>>>>>>>>>>> support frame threading encoding (For intra-only codecs), but
>>>>>>>>>>>> currently ff_alloc_packet2() does not seem to be thread safe,
>>>>>>>>>>>> seeing it calls av_fast_padded_malloc(), yet it's called by
>>>>>>>>>>>> frame threaded encoders.
>>>>>>>>>>>> Should i remove this?
>>>>>>>>>>>
>>>>>>>>>>> I don't know, I was asking only because it sounds tricky.  
>>>>>>>>>>> For cases
>>>>>>>>>>> with a limited number of buffers available (like memory-mapped
>>>>>>>>>>> devices) you are going to need locking anyway, so maybe 
>>>>>>>>>>> rentrancy
>>>>>>>>>>> adds no additional inconvenience.
>>>>>>>>>>>
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>>>>>>>>>> +     * - decoding: unused
>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s,
>>>>>>>>>>>>>> AVPacket *pkt, int flags);
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can the user return "not yet" somehow to this if they have a
>>>>>>>>>>>>> fixed output buffer pool but no buffer is currently
>>>>>>>>>>>>> available?
>>>>>>>>>>>>
>>>>>>>>>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't much like the idea of the user suspending the thread
>>>>>>>>>>>>> in the callback until they have some available, which might
>>>>>>>>>>>>> work in some cases but might also deadlock if an
>>>>>>>>>>>>> avcodec_receive_packet() call is blocked by it.
>>>>>>>>>>>>
>>>>>>>>>>>> Can we make what's in essence a malloc() call return something
>>>>>>>>>>>> like EAGAIN, and this in turn be propagated back to
>>>>>>>>>>>> encode_receive_packet_internal()?
>>>>>>>>>>>
>>>>>>>>>>> Maybe, or if it has many threads maybe it could wait for 
>>>>>>>>>>> something
>>>>>>>>>>> else to finish first.
>>>>>>>>>>>
>>>>>>>>>>>> Couldn't this potentially end up in the forbidden scenario of
>>>>>>>>>>>> avcodec_send_frame() and avcodec_receive_packet() both 
>>>>>>>>>>>> returning
>>>>>>>>>>>> EAGAIN?
>>>>>>>>>>>
>>>>>>>>>>> Yes.  If the forbidden case happens then the encoder is stuck 
>>>>>>>>>>> anyway
>>>>>>>>>>> and can't make any forward progress so we need to error out
>>>>>>>>>>> properly, but the EAGAIN return isn't needed if there is 
>>>>>>>>>>> something
>>>>>>>>>>> else to do on another thread.
>>>>>>>>>>
>>>>>>>>>> Ok, but I'm not familiar or knowledgeable enough with the 
>>>>>>>>>> frame thread
>>>>>>>>>> encoder code to implement this.
>>>>>>>>>
>>>>>>>>> Looked at bit into this. AVCodec->encode2() based encoders 
>>>>>>>>> don't support
>>>>>>>>> returning EAGAIN at all, as it completely breaks the frame 
>>>>>>>>> threading logic.
>>>>>>>>> It would require a considerable rewrite in order to re-add a 
>>>>>>>>> task that
>>>>>>>>> didn't fail but also didn't succeed.
>>>>>>>>>
>>>>>>>>> Non frame threading encoders could probably support it with 
>>>>>>>>> some minimal
>>>>>>>>> changes, but i don't think suddenly letting an scenario that 
>>>>>>>>> was until now
>>>>>>>>> guaranteed to never happen start happening 
>>>>>>>>> (avcodec_send_frame() and
>>>>>>>>> avcodec_receive_packet() both returning EAGAIN) is a good idea. 
>>>>>>>>> It's an API
>>>>>>>>> break.
>>>>>>>>> Letting the user's custom get_encode_buffer() callback suspend 
>>>>>>>>> the thread is
>>>>>>>>> IMO acceptable. In frame threading scenarios, the other threads 
>>>>>>>>> are still
>>>>>>>>> working on their own packets (afaics none depends on the 
>>>>>>>>> others, since it's
>>>>>>>>> intra only encoders only).
>>>>>>>>
>>>>>>>> I think it was not suggested in the thread so:
>>>>>>>> if the users allocation fails the code can fallback to the 
>>>>>>>> default allocator
>>>>>>>> That would lead to the relation:
>>>>>>>> If a users allocator can fail (out of buffers) it must be able 
>>>>>>>> to handle
>>>>>>>> that only some of the returned packets are from its own allocator
>>>>>>>
>>>>>>> In general, custom allocators are used when the caller doesn't 
>>>>>>> want to use
>>>>>>> the default one. But yes, they could use
>>>>>>> avcodec_default_get_encoder_buffer() as fallback, which is why it 
>>>>>>> was added
>>>>>>> to begin with. Same applies to get_buffer2() custom 
>>>>>>> implementations, and so
>>>>>>> far i don't think anybody had issues identifying what allocated a 
>>>>>>> packet
>>>>>>> buffer.
>>>>>>>
>>>>>>> One of the additions to AVPacket people were talking about was a 
>>>>>>> user opaque
>>>>>>> field that libav* would never touch or look at beyond propagating 
>>>>>>> them
>>>>>>> around all the way to the output AVFrame, if any. This opaque 
>>>>>>> field could
>>>>>>> perhaps store such allocator specific information the caller 
>>>>>>> could use to
>>>>>>> identify packets allocated by their own allocator, or those by
>>>>>>> avcodec_default_get_encoder_buffer().
>>>>>>>
>>>>>>>>
>>>>>>>> About alignment, we should at least recommand that allocated 
>>>>>>>> packets are
>>>>>>>> aligned not less than what out av_malloc() would align to.
>>>>>>>> Is there a reason to align less ?
>>>>>>>
>>>>>>> There's no alignment requirement for AVPacket->data, and 
>>>>>>> av_new_packet()
>>>>>>> uses av_buffer_realloc(), which does not guarantee any alignment 
>>>>>>> whatsoever
>>>>>>> on platforms other than Windows. So basically, packet payload 
>>>>>>> buffers
>>>>>>> allocated by our own helpers never had any alignment.
>>>>>>
>>>>>> for the purpose of exporting raw images, alignment would be "nice 
>>>>>> to have"
>>>>>> because later filters may need it or need to memcpy
>>>>>
>>>>> Filters don't use AVPackets, they use AVFrames.
>>>>
>>>> demuxers return AVPackets, so do encoders.
>>>> These can contain raw frames.
>>>>
>>>> also i see for example in rawdec:
>>>> frame->buf[0] = av_buffer_ref(avpkt->buf);
>>>
>>> I ask again, where are you going with this? The alignment for 
>>> AVPacket data
>>> buffers is defined: There is *none*.
>>
>> I simply stated that 'alignment would be "nice to have"'.
>> and then showed some cases where it would be usefull.
> 
> But don't those cases already happen, and without required or guaranteed 
> alignment?
> 
>>
>> I guess where iam going with this is, is the API you add extensible?
>> That is if something is not supported now, can it be added later without
>> adding a new API.
> 
> I should, it shares a signature with get_buffer2(). That means the 
> packet to fill (Which fields can be read from it and set can be easily 
> redefined), avctx so the user can have access to avctx->opaque and so we 
> can eventually use something like a buffer pool in the default allocator 
> callback, and a flags parameter to tell the callback there are 
> requirements.
> 
> Which makes me realize, maybe a flag to tell the callback "Alignment is 
> required" could solve your concerns?

Actually, thinking about it, it's the same situation as always requiring 
it. The mere existence of such a flag would require users of the old API 
moving onto the new to redefine their buffers, since now they *may* need 
to align them, when before they didn't. So not really an option.

> 
>>
>> Thanks
>>
>> [...]
>>
>>
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>>
>

On 3/12/2021 4:42 PM, Michael Niedermayer wrote:
> On Fri, Mar 12, 2021 at 08:30:23PM +0100, Michael Niedermayer wrote:
>> On Fri, Mar 12, 2021 at 02:03:52PM -0300, James Almer wrote:
>>> On 3/12/2021 1:32 PM, Michael Niedermayer wrote:
>>>> On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
>>>>> On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
>>>>>> On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
>>>>>>> On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
>>>>>>>> On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
>>>>>>>>> On 2/21/2021 6:04 PM, James Almer wrote:
>>>>>>>>>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>>>>>>>>>> On 21/02/2021 20:00, James Almer wrote:
>>>>>>>>>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>>>>>>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>>>>>>>>>> This callback is functionally the same as get_buffer2()
>>>>>>>>>>>>>> is for decoders, and
>>>>>>>>>>>>>> implements for the new encode API the functionality of
>>>>>>>>>>>>>> the old encode API had
>>>>>>>>>>>>>> where the user could provide their own buffers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>> Used the names Lynne suggested this time, plus a line
>>>>>>>>>>>>>> about how the callback
>>>>>>>>>>>>>> must be thread safe.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>       libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>       libavcodec/codec.h   |  8 ++++---
>>>>>>>>>>>>>>       libavcodec/encode.c  | 54
>>>>>>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>       libavcodec/encode.h  |  8 +++++++
>>>>>>>>>>>>>>       libavcodec/options.c |  1 +
>>>>>>>>>>>>>>       5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>>>>>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>>>>>>>>>> --- a/libavcodec/avcodec.h
>>>>>>>>>>>>>> +++ b/libavcodec/avcodec.h
>>>>>>>>>>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>>>>>>>>>>        */
>>>>>>>>>>>>>>       #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>> + * The encoder will keep a reference to the packet and
>>>>>>>>>>>>>> may reuse it later.
>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>       struct AVCodecInternal;
>>>>>>>>>>>>>>       /**
>>>>>>>>>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>>>>>>>>>            * - encoding: set by user
>>>>>>>>>>>>>>            */
>>>>>>>>>>>>>>           int export_side_data;
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +    /**
>>>>>>>>>>>>>> +     * This callback is called at the beginning of each
>>>>>>>>>>>>>> packet to get a data
>>>>>>>>>>>>>> +     * buffer for it.
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * The following field will be set in the packet
>>>>>>>>>>>>>> before this callback is
>>>>>>>>>>>>>> +     * called:
>>>>>>>>>>>>>> +     * - size
>>>>>>>>>>>>>> +     * This callback must use the above value to
>>>>>>>>>>>>>> calculate the required buffer size,
>>>>>>>>>>>>>> +     * which must padded by at least
>>>>>>>>>>>>>> AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * This callback must fill the following fields in the packet:
>>>>>>>>>>>>>> +     * - data
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is the data pointer allowed to be in write-only memory?
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure what the use case for this would be, so probably no?
>>>>>>>>>>>
>>>>>>>>>>> The two use-cases I see for this API are:
>>>>>>>>>>>
>>>>>>>>>>> * You want to avoid a copy when combining the output with something
>>>>>>>>>>> else.  E.g. you pass a pointer to the block of memory following
>>>>>>>>>>> where you are going to put your header data (for something you are
>>>>>>>>>>> going to send over the network, say).
>>>>>>>>>>>
>>>>>>>>>>> * You want to avoid a copy when passing the output directly to
>>>>>>>>>>> something external.  E.g. you pass a pointer to a memory-mapped
>>>>>>>>>>> device buffer (such as a V4L2 buffer, say).
>>>>>>>>>>>
>>>>>>>>>>> In the second case, write-only memory on an external device seems
>>>>>>>>>>> possible, as does memory which is, say, readable but uncached, so
>>>>>>>>>>> reading it is a really bad idea.
>>>>>>>>>>
>>>>>>>>>> Allowing the second case would depend on how encoders behave. Some may
>>>>>>>>>> attempt to read data already written to the output packet. It's not like
>>>>>>>>>> all of them allocate the packet, do a memcpy from an internal buffer,
>>>>>>>>>> then return.
>>>>>>>>>> There is also the flag meant to signal that the encoder will keep a
>>>>>>>>>> reference to the packet around, which more or less implies it will be
>>>>>>>>>> read later in the encoding process.
>>>>>>>>>>
>>>>>>>>>> The doxy for avcodec_encode_video2(), which allowed the user to provide
>>>>>>>>>> their own buffers in the output packet, does not mention any kind of
>>>>>>>>>> requirement for the data pointer, so I don't think we can say it's an
>>>>>>>>>> allowed scenario here either.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> Does it have any alignment requirements?
>>>>>>>>>>>>
>>>>>>>>>>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>>>>>>>>>>
>>>>>>>>>>> I think say that explicitly.  avcodec_default_get_encoder_buffer()
>>>>>>>>>>> does give you aligned memory, even though it isn't needed.
>>>>>>>>>>
>>>>>>>>>> Would saying "There's no alignment requirement for the data pointer" add
>>>>>>>>>> anything of value to the doxy? If i don't mention any kind of alignment
>>>>>>>>>> requirement, it's because there isn't any, and it's implicit.
>>>>>>>>>> I listed the requirements the user needs to keep in mind, like the
>>>>>>>>>> padding and the need for an AVBufferRef. But if you think it's worth
>>>>>>>>>> adding, then sure.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> +     * - buf must contain a pointer to an AVBufferRef
>>>>>>>>>>>>>> structure. The packet's
>>>>>>>>>>>>>> +     *   data pointer must be contained in it.
>>>>>>>>>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(),
>>>>>>>>>>>>>> and av_buffer_ref().
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then
>>>>>>>>>>>>>> get_encoder_buffer() must call
>>>>>>>>>>>>>> +     * avcodec_default_get_encoder_buffer() instead of
>>>>>>>>>>>>>> providing a buffer allocated by
>>>>>>>>>>>>>> +     * some other means.
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
>>>>>>>>>>>>>> flags then the packet may be reused
>>>>>>>>>>>>>> +     * (read and/or written to if it is writable) later
>>>>>>>>>>>>>> by libavcodec.
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * This callback must be thread-safe, as when frame
>>>>>>>>>>>>>> multithreading is used, it may
>>>>>>>>>>>>>> +     * be called from multiple threads simultaneously.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is
>>>>>>>>>>>>> it really necessary?
>>>>>>>>>>>>
>>>>>>>>>>>> This was a suggestion by Lynne, i personally don't know. We
>>>>>>>>>>>> support frame threading encoding (For intra-only codecs), but
>>>>>>>>>>>> currently ff_alloc_packet2() does not seem to be thread safe,
>>>>>>>>>>>> seeing it calls av_fast_padded_malloc(), yet it's called by
>>>>>>>>>>>> frame threaded encoders.
>>>>>>>>>>>> Should i remove this?
>>>>>>>>>>>
>>>>>>>>>>> I don't know, I was asking only because it sounds tricky.  For cases
>>>>>>>>>>> with a limited number of buffers available (like memory-mapped
>>>>>>>>>>> devices) you are going to need locking anyway, so maybe rentrancy
>>>>>>>>>>> adds no additional inconvenience.
>>>>>>>>>>>
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>>>>>>>>>> +     * - decoding: unused
>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s,
>>>>>>>>>>>>>> AVPacket *pkt, int flags);
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can the user return "not yet" somehow to this if they have a
>>>>>>>>>>>>> fixed output buffer pool but no buffer is currently
>>>>>>>>>>>>> available?
>>>>>>>>>>>>
>>>>>>>>>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't much like the idea of the user suspending the thread
>>>>>>>>>>>>> in the callback until they have some available, which might
>>>>>>>>>>>>> work in some cases but might also deadlock if an
>>>>>>>>>>>>> avcodec_receive_packet() call is blocked by it.
>>>>>>>>>>>>
>>>>>>>>>>>> Can we make what's in essence a malloc() call return something
>>>>>>>>>>>> like EAGAIN, and this in turn be propagated back to
>>>>>>>>>>>> encode_receive_packet_internal()?
>>>>>>>>>>>
>>>>>>>>>>> Maybe, or if it has many threads maybe it could wait for something
>>>>>>>>>>> else to finish first.
>>>>>>>>>>>
>>>>>>>>>>>> Couldn't this potentially end up in the forbidden scenario of
>>>>>>>>>>>> avcodec_send_frame() and avcodec_receive_packet() both returning
>>>>>>>>>>>> EAGAIN?
>>>>>>>>>>>
>>>>>>>>>>> Yes.  If the forbidden case happens then the encoder is stuck anyway
>>>>>>>>>>> and can't make any forward progress so we need to error out
>>>>>>>>>>> properly, but the EAGAIN return isn't needed if there is something
>>>>>>>>>>> else to do on another thread.
>>>>>>>>>>
>>>>>>>>>> Ok, but I'm not familiar or knowledgeable enough with the frame thread
>>>>>>>>>> encoder code to implement this.
>>>>>>>>>
>>>>>>>>> Looked at bit into this. AVCodec->encode2() based encoders don't support
>>>>>>>>> returning EAGAIN at all, as it completely breaks the frame threading logic.
>>>>>>>>> It would require a considerable rewrite in order to re-add a task that
>>>>>>>>> didn't fail but also didn't succeed.
>>>>>>>>>
>>>>>>>>> Non frame threading encoders could probably support it with some minimal
>>>>>>>>> changes, but i don't think suddenly letting an scenario that was until now
>>>>>>>>> guaranteed to never happen start happening (avcodec_send_frame() and
>>>>>>>>> avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
>>>>>>>>> break.
>>>>>>>>> Letting the user's custom get_encode_buffer() callback suspend the thread is
>>>>>>>>> IMO acceptable. In frame threading scenarios, the other threads are still
>>>>>>>>> working on their own packets (afaics none depends on the others, since it's
>>>>>>>>> intra only encoders only).
>>>>>>>>
>>>>>>>> I think it was not suggested in the thread so:
>>>>>>>> if the users allocation fails the code can fallback to the default allocator
>>>>>>>> That would lead to the relation:
>>>>>>>> If a users allocator can fail (out of buffers) it must be able to handle
>>>>>>>> that only some of the returned packets are from its own allocator
>>>>>>>
>>>>>>> In general, custom allocators are used when the caller doesn't want to use
>>>>>>> the default one. But yes, they could use
>>>>>>> avcodec_default_get_encoder_buffer() as fallback, which is why it was added
>>>>>>> to begin with. Same applies to get_buffer2() custom implementations, and so
>>>>>>> far i don't think anybody had issues identifying what allocated a packet
>>>>>>> buffer.
>>>>>>>
>>>>>>> One of the additions to AVPacket people were talking about was a user opaque
>>>>>>> field that libav* would never touch or look at beyond propagating them
>>>>>>> around all the way to the output AVFrame, if any. This opaque field could
>>>>>>> perhaps store such allocator specific information the caller could use to
>>>>>>> identify packets allocated by their own allocator, or those by
>>>>>>> avcodec_default_get_encoder_buffer().
>>>>>>>
>>>>>>>>
>>>>>>>> About alignment, we should at least recommand that allocated packets are
>>>>>>>> aligned not less than what out av_malloc() would align to.
>>>>>>>> Is there a reason to align less ?
>>>>>>>
>>>>>>> There's no alignment requirement for AVPacket->data, and av_new_packet()
>>>>>>> uses av_buffer_realloc(), which does not guarantee any alignment whatsoever
>>>>>>> on platforms other than Windows. So basically, packet payload buffers
>>>>>>> allocated by our own helpers never had any alignment.
>>>>>>
>>>>>> for the purpose of exporting raw images, alignment would be "nice to have"
>>>>>> because later filters may need it or need to memcpy
>>>>>
>>>>> Filters don't use AVPackets, they use AVFrames.
>>>>
>>>> demuxers return AVPackets, so do encoders.
>>>> These can contain raw frames.
>>>>
>>>> also i see for example in rawdec:
>>>> frame->buf[0] = av_buffer_ref(avpkt->buf);
>>>
>>> I ask again, where are you going with this? The alignment for AVPacket data
>>> buffers is defined: There is *none*.
>>
>> I simply stated that 'alignment would be "nice to have"'.
>> and then showed some cases where it would be usefull.
>>
>> I guess where iam going with this is, is the API you add extensible?
>> That is if something is not supported now, can it be added later without
>> adding a new API.
> 
> Related to this, what do the flags passed into get_encoder_buffer() mean
> exactly ?
> are they "hints" so that a new/unknown flag can be ignored by an implementation
> or what else should an application do with a unknown flag, this should
> be clarified in the documentation

They are hints, same as in get_buffer2(), like with the _REF flag. They 
can't really be requirements since the callback should be able to ignore 
new flags it doesn't understand.

> 
> thx
> 
> 
> [...]
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>

James Almer:
> On 3/12/2021 4:46 PM, James Almer wrote:
>> On 3/12/2021 4:30 PM, Michael Niedermayer wrote:
>>> On Fri, Mar 12, 2021 at 02:03:52PM -0300, James Almer wrote:
>>>> On 3/12/2021 1:32 PM, Michael Niedermayer wrote:
>>>>> On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
>>>>>> On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
>>>>>>> On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
>>>>>>>> On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
>>>>>>>>> On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
>>>>>>>>>> On 2/21/2021 6:04 PM, James Almer wrote:
>>>>>>>>>>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>>>>>>>>>>> On 21/02/2021 20:00, James Almer wrote:
>>>>>>>>>>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>>>>>>>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>>>>>>>>>>> This callback is functionally the same as get_buffer2()
>>>>>>>>>>>>>>> is for decoders, and
>>>>>>>>>>>>>>> implements for the new encode API the functionality of
>>>>>>>>>>>>>>> the old encode API had
>>>>>>>>>>>>>>> where the user could provide their own buffers.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>> Used the names Lynne suggested this time, plus a line
>>>>>>>>>>>>>>> about how the callback
>>>>>>>>>>>>>>> must be thread safe.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>       libavcodec/avcodec.h | 45
>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>       libavcodec/codec.h   |  8 ++++---
>>>>>>>>>>>>>>>       libavcodec/encode.c  | 54
>>>>>>>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>       libavcodec/encode.h  |  8 +++++++
>>>>>>>>>>>>>>>       libavcodec/options.c |  1 +
>>>>>>>>>>>>>>>       5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>>>>>>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>>>>>>>>>>> --- a/libavcodec/avcodec.h
>>>>>>>>>>>>>>> +++ b/libavcodec/avcodec.h
>>>>>>>>>>>>>>> @@ -513,6 +513,11 @@ typedef struct
>>>>>>>>>>>>>>> AVProducerReferenceTime {
>>>>>>>>>>>>>>>        */
>>>>>>>>>>>>>>>       #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>>> + * The encoder will keep a reference to the packet and
>>>>>>>>>>>>>>> may reuse it later.
>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>       struct AVCodecInternal;
>>>>>>>>>>>>>>>       /**
>>>>>>>>>>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>>>>>>>>>>            * - encoding: set by user
>>>>>>>>>>>>>>>            */
>>>>>>>>>>>>>>>           int export_side_data;
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +    /**
>>>>>>>>>>>>>>> +     * This callback is called at the beginning of each
>>>>>>>>>>>>>>> packet to get a data
>>>>>>>>>>>>>>> +     * buffer for it.
>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>> +     * The following field will be set in the packet
>>>>>>>>>>>>>>> before this callback is
>>>>>>>>>>>>>>> +     * called:
>>>>>>>>>>>>>>> +     * - size
>>>>>>>>>>>>>>> +     * This callback must use the above value to
>>>>>>>>>>>>>>> calculate the required buffer size,
>>>>>>>>>>>>>>> +     * which must padded by at least
>>>>>>>>>>>>>>> AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>> +     * This callback must fill the following fields in
>>>>>>>>>>>>>>> the packet:
>>>>>>>>>>>>>>> +     * - data
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is the data pointer allowed to be in write-only memory?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure what the use case for this would be, so
>>>>>>>>>>>>> probably no?
>>>>>>>>>>>>
>>>>>>>>>>>> The two use-cases I see for this API are:
>>>>>>>>>>>>
>>>>>>>>>>>> * You want to avoid a copy when combining the output with
>>>>>>>>>>>> something
>>>>>>>>>>>> else.  E.g. you pass a pointer to the block of memory following
>>>>>>>>>>>> where you are going to put your header data (for something
>>>>>>>>>>>> you are
>>>>>>>>>>>> going to send over the network, say).
>>>>>>>>>>>>
>>>>>>>>>>>> * You want to avoid a copy when passing the output directly to
>>>>>>>>>>>> something external.  E.g. you pass a pointer to a memory-mapped
>>>>>>>>>>>> device buffer (such as a V4L2 buffer, say).
>>>>>>>>>>>>
>>>>>>>>>>>> In the second case, write-only memory on an external device
>>>>>>>>>>>> seems
>>>>>>>>>>>> possible, as does memory which is, say, readable but
>>>>>>>>>>>> uncached, so
>>>>>>>>>>>> reading it is a really bad idea.
>>>>>>>>>>>
>>>>>>>>>>> Allowing the second case would depend on how encoders behave.
>>>>>>>>>>> Some may
>>>>>>>>>>> attempt to read data already written to the output packet.
>>>>>>>>>>> It's not like
>>>>>>>>>>> all of them allocate the packet, do a memcpy from an internal
>>>>>>>>>>> buffer,
>>>>>>>>>>> then return.
>>>>>>>>>>> There is also the flag meant to signal that the encoder will
>>>>>>>>>>> keep a
>>>>>>>>>>> reference to the packet around, which more or less implies it
>>>>>>>>>>> will be
>>>>>>>>>>> read later in the encoding process.
>>>>>>>>>>>
>>>>>>>>>>> The doxy for avcodec_encode_video2(), which allowed the user
>>>>>>>>>>> to provide
>>>>>>>>>>> their own buffers in the output packet, does not mention any
>>>>>>>>>>> kind of
>>>>>>>>>>> requirement for the data pointer, so I don't think we can say
>>>>>>>>>>> it's an
>>>>>>>>>>> allowed scenario here either.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>> Does it have any alignment requirements?
>>>>>>>>>>>>>
>>>>>>>>>>>>> No, just padding. AVPacket doesn't require alignment for
>>>>>>>>>>>>> the payload.
>>>>>>>>>>>>
>>>>>>>>>>>> I think say that explicitly. 
>>>>>>>>>>>> avcodec_default_get_encoder_buffer()
>>>>>>>>>>>> does give you aligned memory, even though it isn't needed.
>>>>>>>>>>>
>>>>>>>>>>> Would saying "There's no alignment requirement for the data
>>>>>>>>>>> pointer" add
>>>>>>>>>>> anything of value to the doxy? If i don't mention any kind of
>>>>>>>>>>> alignment
>>>>>>>>>>> requirement, it's because there isn't any, and it's implicit.
>>>>>>>>>>> I listed the requirements the user needs to keep in mind,
>>>>>>>>>>> like the
>>>>>>>>>>> padding and the need for an AVBufferRef. But if you think
>>>>>>>>>>> it's worth
>>>>>>>>>>> adding, then sure.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>> +     * - buf must contain a pointer to an AVBufferRef
>>>>>>>>>>>>>>> structure. The packet's
>>>>>>>>>>>>>>> +     *   data pointer must be contained in it.
>>>>>>>>>>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(),
>>>>>>>>>>>>>>> and av_buffer_ref().
>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then
>>>>>>>>>>>>>>> get_encoder_buffer() must call
>>>>>>>>>>>>>>> +     * avcodec_default_get_encoder_buffer() instead of
>>>>>>>>>>>>>>> providing a buffer allocated by
>>>>>>>>>>>>>>> +     * some other means.
>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
>>>>>>>>>>>>>>> flags then the packet may be reused
>>>>>>>>>>>>>>> +     * (read and/or written to if it is writable) later
>>>>>>>>>>>>>>> by libavcodec.
>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>> +     * This callback must be thread-safe, as when frame
>>>>>>>>>>>>>>> multithreading is used, it may
>>>>>>>>>>>>>>> +     * be called from multiple threads simultaneously.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is
>>>>>>>>>>>>>> it really necessary?
>>>>>>>>>>>>>
>>>>>>>>>>>>> This was a suggestion by Lynne, i personally don't know. We
>>>>>>>>>>>>> support frame threading encoding (For intra-only codecs), but
>>>>>>>>>>>>> currently ff_alloc_packet2() does not seem to be thread safe,
>>>>>>>>>>>>> seeing it calls av_fast_padded_malloc(), yet it's called by
>>>>>>>>>>>>> frame threaded encoders.
>>>>>>>>>>>>> Should i remove this?
>>>>>>>>>>>>
>>>>>>>>>>>> I don't know, I was asking only because it sounds tricky. 
>>>>>>>>>>>> For cases
>>>>>>>>>>>> with a limited number of buffers available (like memory-mapped
>>>>>>>>>>>> devices) you are going to need locking anyway, so maybe
>>>>>>>>>>>> rentrancy
>>>>>>>>>>>> adds no additional inconvenience.
>>>>>>>>>>>>
>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>>>>>>>>>>> +     * - decoding: unused
>>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s,
>>>>>>>>>>>>>>> AVPacket *pkt, int flags);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can the user return "not yet" somehow to this if they have a
>>>>>>>>>>>>>> fixed output buffer pool but no buffer is currently
>>>>>>>>>>>>>> available?
>>>>>>>>>>>>>
>>>>>>>>>>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't much like the idea of the user suspending the thread
>>>>>>>>>>>>>> in the callback until they have some available, which might
>>>>>>>>>>>>>> work in some cases but might also deadlock if an
>>>>>>>>>>>>>> avcodec_receive_packet() call is blocked by it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can we make what's in essence a malloc() call return something
>>>>>>>>>>>>> like EAGAIN, and this in turn be propagated back to
>>>>>>>>>>>>> encode_receive_packet_internal()?
>>>>>>>>>>>>
>>>>>>>>>>>> Maybe, or if it has many threads maybe it could wait for
>>>>>>>>>>>> something
>>>>>>>>>>>> else to finish first.
>>>>>>>>>>>>
>>>>>>>>>>>>> Couldn't this potentially end up in the forbidden scenario of
>>>>>>>>>>>>> avcodec_send_frame() and avcodec_receive_packet() both
>>>>>>>>>>>>> returning
>>>>>>>>>>>>> EAGAIN?
>>>>>>>>>>>>
>>>>>>>>>>>> Yes.  If the forbidden case happens then the encoder is
>>>>>>>>>>>> stuck anyway
>>>>>>>>>>>> and can't make any forward progress so we need to error out
>>>>>>>>>>>> properly, but the EAGAIN return isn't needed if there is
>>>>>>>>>>>> something
>>>>>>>>>>>> else to do on another thread.
>>>>>>>>>>>
>>>>>>>>>>> Ok, but I'm not familiar or knowledgeable enough with the
>>>>>>>>>>> frame thread
>>>>>>>>>>> encoder code to implement this.
>>>>>>>>>>
>>>>>>>>>> Looked at bit into this. AVCodec->encode2() based encoders
>>>>>>>>>> don't support
>>>>>>>>>> returning EAGAIN at all, as it completely breaks the frame
>>>>>>>>>> threading logic.
>>>>>>>>>> It would require a considerable rewrite in order to re-add a
>>>>>>>>>> task that
>>>>>>>>>> didn't fail but also didn't succeed.
>>>>>>>>>>
>>>>>>>>>> Non frame threading encoders could probably support it with
>>>>>>>>>> some minimal
>>>>>>>>>> changes, but i don't think suddenly letting an scenario that
>>>>>>>>>> was until now
>>>>>>>>>> guaranteed to never happen start happening
>>>>>>>>>> (avcodec_send_frame() and
>>>>>>>>>> avcodec_receive_packet() both returning EAGAIN) is a good
>>>>>>>>>> idea. It's an API
>>>>>>>>>> break.
>>>>>>>>>> Letting the user's custom get_encode_buffer() callback suspend
>>>>>>>>>> the thread is
>>>>>>>>>> IMO acceptable. In frame threading scenarios, the other
>>>>>>>>>> threads are still
>>>>>>>>>> working on their own packets (afaics none depends on the
>>>>>>>>>> others, since it's
>>>>>>>>>> intra only encoders only).
>>>>>>>>>
>>>>>>>>> I think it was not suggested in the thread so:
>>>>>>>>> if the users allocation fails the code can fallback to the
>>>>>>>>> default allocator
>>>>>>>>> That would lead to the relation:
>>>>>>>>> If a users allocator can fail (out of buffers) it must be able
>>>>>>>>> to handle
>>>>>>>>> that only some of the returned packets are from its own allocator
>>>>>>>>
>>>>>>>> In general, custom allocators are used when the caller doesn't
>>>>>>>> want to use
>>>>>>>> the default one. But yes, they could use
>>>>>>>> avcodec_default_get_encoder_buffer() as fallback, which is why
>>>>>>>> it was added
>>>>>>>> to begin with. Same applies to get_buffer2() custom
>>>>>>>> implementations, and so
>>>>>>>> far i don't think anybody had issues identifying what allocated
>>>>>>>> a packet
>>>>>>>> buffer.
>>>>>>>>
>>>>>>>> One of the additions to AVPacket people were talking about was a
>>>>>>>> user opaque
>>>>>>>> field that libav* would never touch or look at beyond
>>>>>>>> propagating them
>>>>>>>> around all the way to the output AVFrame, if any. This opaque
>>>>>>>> field could
>>>>>>>> perhaps store such allocator specific information the caller
>>>>>>>> could use to
>>>>>>>> identify packets allocated by their own allocator, or those by
>>>>>>>> avcodec_default_get_encoder_buffer().
>>>>>>>>
>>>>>>>>>
>>>>>>>>> About alignment, we should at least recommand that allocated
>>>>>>>>> packets are
>>>>>>>>> aligned not less than what out av_malloc() would align to.
>>>>>>>>> Is there a reason to align less ?
>>>>>>>>
>>>>>>>> There's no alignment requirement for AVPacket->data, and
>>>>>>>> av_new_packet()
>>>>>>>> uses av_buffer_realloc(), which does not guarantee any alignment
>>>>>>>> whatsoever
>>>>>>>> on platforms other than Windows. So basically, packet payload
>>>>>>>> buffers
>>>>>>>> allocated by our own helpers never had any alignment.
>>>>>>>
>>>>>>> for the purpose of exporting raw images, alignment would be "nice
>>>>>>> to have"
>>>>>>> because later filters may need it or need to memcpy
>>>>>>
>>>>>> Filters don't use AVPackets, they use AVFrames.
>>>>>
>>>>> demuxers return AVPackets, so do encoders.
>>>>> These can contain raw frames.
>>>>>
>>>>> also i see for example in rawdec:
>>>>> frame->buf[0] = av_buffer_ref(avpkt->buf);
>>>>
>>>> I ask again, where are you going with this? The alignment for
>>>> AVPacket data
>>>> buffers is defined: There is *none*.
>>>
>>> I simply stated that 'alignment would be "nice to have"'.
>>> and then showed some cases where it would be usefull.
>>
>> But don't those cases already happen, and without required or
>> guaranteed alignment?
>>
>>>
>>> I guess where iam going with this is, is the API you add extensible?
>>> That is if something is not supported now, can it be added later without
>>> adding a new API.
>>
>> I should, it shares a signature with get_buffer2(). That means the
>> packet to fill (Which fields can be read from it and set can be easily
>> redefined), avctx so the user can have access to avctx->opaque and so
>> we can eventually use something like a buffer pool in the default
>> allocator callback, and a flags parameter to tell the callback there
>> are requirements.
>>
>> Which makes me realize, maybe a flag to tell the callback "Alignment
>> is required" could solve your concerns?
> 
> Actually, thinking about it, it's the same situation as always requiring
> it. The mere existence of such a flag would require users of the old API
> moving onto the new to redefine their buffers, since now they *may* need
> to align them, when before they didn't. So not really an option.
> 

One could say that currently (i.e. up until the next + 1 bump),
alignment is advisable as it may improve performance, but that from next
+ 1 bump onwards the desired alignment is required.
Furthermore one can add public fields to each encoder containing the
maximal alignment that buffers for these encoders may required. This way
users may choose to use the default buffers instead of their own for the
(few) encoders that need bigger alignment. Said flag would be a bit like
the max_lowres value for decoders.
Notice that I am of course only speaking about requirements for packet
buffers for encoders, not for AVPackets in general.

- Andreas

On 3/12/2021 5:11 PM, Andreas Rheinhardt wrote:
> James Almer:
>> On 3/12/2021 4:46 PM, James Almer wrote:
>>> On 3/12/2021 4:30 PM, Michael Niedermayer wrote:
>>>> On Fri, Mar 12, 2021 at 02:03:52PM -0300, James Almer wrote:
>>>>> On 3/12/2021 1:32 PM, Michael Niedermayer wrote:
>>>>>> On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
>>>>>>> On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
>>>>>>>> On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
>>>>>>>>> On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
>>>>>>>>>> On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
>>>>>>>>>>> On 2/21/2021 6:04 PM, James Almer wrote:
>>>>>>>>>>>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>>>>>>>>>>>> On 21/02/2021 20:00, James Almer wrote:
>>>>>>>>>>>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>>>>>>>>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>>>>>>>>>>>> This callback is functionally the same as get_buffer2()
>>>>>>>>>>>>>>>> is for decoders, and
>>>>>>>>>>>>>>>> implements for the new encode API the functionality of
>>>>>>>>>>>>>>>> the old encode API had
>>>>>>>>>>>>>>>> where the user could provide their own buffers.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>> Used the names Lynne suggested this time, plus a line
>>>>>>>>>>>>>>>> about how the callback
>>>>>>>>>>>>>>>> must be thread safe.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>        libavcodec/avcodec.h | 45
>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>>        libavcodec/codec.h   |  8 ++++---
>>>>>>>>>>>>>>>>        libavcodec/encode.c  | 54
>>>>>>>>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>>        libavcodec/encode.h  |  8 +++++++
>>>>>>>>>>>>>>>>        libavcodec/options.c |  1 +
>>>>>>>>>>>>>>>>        5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>>>>>>>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>>>>>>>>>>>> --- a/libavcodec/avcodec.h
>>>>>>>>>>>>>>>> +++ b/libavcodec/avcodec.h
>>>>>>>>>>>>>>>> @@ -513,6 +513,11 @@ typedef struct
>>>>>>>>>>>>>>>> AVProducerReferenceTime {
>>>>>>>>>>>>>>>>         */
>>>>>>>>>>>>>>>>        #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>>>> + * The encoder will keep a reference to the packet and
>>>>>>>>>>>>>>>> may reuse it later.
>>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>        struct AVCodecInternal;
>>>>>>>>>>>>>>>>        /**
>>>>>>>>>>>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>>>>>>>>>>>             * - encoding: set by user
>>>>>>>>>>>>>>>>             */
>>>>>>>>>>>>>>>>            int export_side_data;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    /**
>>>>>>>>>>>>>>>> +     * This callback is called at the beginning of each
>>>>>>>>>>>>>>>> packet to get a data
>>>>>>>>>>>>>>>> +     * buffer for it.
>>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>>> +     * The following field will be set in the packet
>>>>>>>>>>>>>>>> before this callback is
>>>>>>>>>>>>>>>> +     * called:
>>>>>>>>>>>>>>>> +     * - size
>>>>>>>>>>>>>>>> +     * This callback must use the above value to
>>>>>>>>>>>>>>>> calculate the required buffer size,
>>>>>>>>>>>>>>>> +     * which must padded by at least
>>>>>>>>>>>>>>>> AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>>> +     * This callback must fill the following fields in
>>>>>>>>>>>>>>>> the packet:
>>>>>>>>>>>>>>>> +     * - data
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is the data pointer allowed to be in write-only memory?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not sure what the use case for this would be, so
>>>>>>>>>>>>>> probably no?
>>>>>>>>>>>>>
>>>>>>>>>>>>> The two use-cases I see for this API are:
>>>>>>>>>>>>>
>>>>>>>>>>>>> * You want to avoid a copy when combining the output with
>>>>>>>>>>>>> something
>>>>>>>>>>>>> else.  E.g. you pass a pointer to the block of memory following
>>>>>>>>>>>>> where you are going to put your header data (for something
>>>>>>>>>>>>> you are
>>>>>>>>>>>>> going to send over the network, say).
>>>>>>>>>>>>>
>>>>>>>>>>>>> * You want to avoid a copy when passing the output directly to
>>>>>>>>>>>>> something external.  E.g. you pass a pointer to a memory-mapped
>>>>>>>>>>>>> device buffer (such as a V4L2 buffer, say).
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the second case, write-only memory on an external device
>>>>>>>>>>>>> seems
>>>>>>>>>>>>> possible, as does memory which is, say, readable but
>>>>>>>>>>>>> uncached, so
>>>>>>>>>>>>> reading it is a really bad idea.
>>>>>>>>>>>>
>>>>>>>>>>>> Allowing the second case would depend on how encoders behave.
>>>>>>>>>>>> Some may
>>>>>>>>>>>> attempt to read data already written to the output packet.
>>>>>>>>>>>> It's not like
>>>>>>>>>>>> all of them allocate the packet, do a memcpy from an internal
>>>>>>>>>>>> buffer,
>>>>>>>>>>>> then return.
>>>>>>>>>>>> There is also the flag meant to signal that the encoder will
>>>>>>>>>>>> keep a
>>>>>>>>>>>> reference to the packet around, which more or less implies it
>>>>>>>>>>>> will be
>>>>>>>>>>>> read later in the encoding process.
>>>>>>>>>>>>
>>>>>>>>>>>> The doxy for avcodec_encode_video2(), which allowed the user
>>>>>>>>>>>> to provide
>>>>>>>>>>>> their own buffers in the output packet, does not mention any
>>>>>>>>>>>> kind of
>>>>>>>>>>>> requirement for the data pointer, so I don't think we can say
>>>>>>>>>>>> it's an
>>>>>>>>>>>> allowed scenario here either.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Does it have any alignment requirements?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> No, just padding. AVPacket doesn't require alignment for
>>>>>>>>>>>>>> the payload.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think say that explicitly.
>>>>>>>>>>>>> avcodec_default_get_encoder_buffer()
>>>>>>>>>>>>> does give you aligned memory, even though it isn't needed.
>>>>>>>>>>>>
>>>>>>>>>>>> Would saying "There's no alignment requirement for the data
>>>>>>>>>>>> pointer" add
>>>>>>>>>>>> anything of value to the doxy? If i don't mention any kind of
>>>>>>>>>>>> alignment
>>>>>>>>>>>> requirement, it's because there isn't any, and it's implicit.
>>>>>>>>>>>> I listed the requirements the user needs to keep in mind,
>>>>>>>>>>>> like the
>>>>>>>>>>>> padding and the need for an AVBufferRef. But if you think
>>>>>>>>>>>> it's worth
>>>>>>>>>>>> adding, then sure.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +     * - buf must contain a pointer to an AVBufferRef
>>>>>>>>>>>>>>>> structure. The packet's
>>>>>>>>>>>>>>>> +     *   data pointer must be contained in it.
>>>>>>>>>>>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(),
>>>>>>>>>>>>>>>> and av_buffer_ref().
>>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then
>>>>>>>>>>>>>>>> get_encoder_buffer() must call
>>>>>>>>>>>>>>>> +     * avcodec_default_get_encoder_buffer() instead of
>>>>>>>>>>>>>>>> providing a buffer allocated by
>>>>>>>>>>>>>>>> +     * some other means.
>>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
>>>>>>>>>>>>>>>> flags then the packet may be reused
>>>>>>>>>>>>>>>> +     * (read and/or written to if it is writable) later
>>>>>>>>>>>>>>>> by libavcodec.
>>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>>> +     * This callback must be thread-safe, as when frame
>>>>>>>>>>>>>>>> multithreading is used, it may
>>>>>>>>>>>>>>>> +     * be called from multiple threads simultaneously.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is
>>>>>>>>>>>>>>> it really necessary?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This was a suggestion by Lynne, i personally don't know. We
>>>>>>>>>>>>>> support frame threading encoding (For intra-only codecs), but
>>>>>>>>>>>>>> currently ff_alloc_packet2() does not seem to be thread safe,
>>>>>>>>>>>>>> seeing it calls av_fast_padded_malloc(), yet it's called by
>>>>>>>>>>>>>> frame threaded encoders.
>>>>>>>>>>>>>> Should i remove this?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't know, I was asking only because it sounds tricky.
>>>>>>>>>>>>> For cases
>>>>>>>>>>>>> with a limited number of buffers available (like memory-mapped
>>>>>>>>>>>>> devices) you are going to need locking anyway, so maybe
>>>>>>>>>>>>> rentrancy
>>>>>>>>>>>>> adds no additional inconvenience.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>>>>>>>>>>>> +     *
>>>>>>>>>>>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>>>>>>>>>>>> +     * - decoding: unused
>>>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s,
>>>>>>>>>>>>>>>> AVPacket *pkt, int flags);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can the user return "not yet" somehow to this if they have a
>>>>>>>>>>>>>>> fixed output buffer pool but no buffer is currently
>>>>>>>>>>>>>>> available?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't much like the idea of the user suspending the thread
>>>>>>>>>>>>>>> in the callback until they have some available, which might
>>>>>>>>>>>>>>> work in some cases but might also deadlock if an
>>>>>>>>>>>>>>> avcodec_receive_packet() call is blocked by it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can we make what's in essence a malloc() call return something
>>>>>>>>>>>>>> like EAGAIN, and this in turn be propagated back to
>>>>>>>>>>>>>> encode_receive_packet_internal()?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe, or if it has many threads maybe it could wait for
>>>>>>>>>>>>> something
>>>>>>>>>>>>> else to finish first.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Couldn't this potentially end up in the forbidden scenario of
>>>>>>>>>>>>>> avcodec_send_frame() and avcodec_receive_packet() both
>>>>>>>>>>>>>> returning
>>>>>>>>>>>>>> EAGAIN?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes.  If the forbidden case happens then the encoder is
>>>>>>>>>>>>> stuck anyway
>>>>>>>>>>>>> and can't make any forward progress so we need to error out
>>>>>>>>>>>>> properly, but the EAGAIN return isn't needed if there is
>>>>>>>>>>>>> something
>>>>>>>>>>>>> else to do on another thread.
>>>>>>>>>>>>
>>>>>>>>>>>> Ok, but I'm not familiar or knowledgeable enough with the
>>>>>>>>>>>> frame thread
>>>>>>>>>>>> encoder code to implement this.
>>>>>>>>>>>
>>>>>>>>>>> Looked at bit into this. AVCodec->encode2() based encoders
>>>>>>>>>>> don't support
>>>>>>>>>>> returning EAGAIN at all, as it completely breaks the frame
>>>>>>>>>>> threading logic.
>>>>>>>>>>> It would require a considerable rewrite in order to re-add a
>>>>>>>>>>> task that
>>>>>>>>>>> didn't fail but also didn't succeed.
>>>>>>>>>>>
>>>>>>>>>>> Non frame threading encoders could probably support it with
>>>>>>>>>>> some minimal
>>>>>>>>>>> changes, but i don't think suddenly letting an scenario that
>>>>>>>>>>> was until now
>>>>>>>>>>> guaranteed to never happen start happening
>>>>>>>>>>> (avcodec_send_frame() and
>>>>>>>>>>> avcodec_receive_packet() both returning EAGAIN) is a good
>>>>>>>>>>> idea. It's an API
>>>>>>>>>>> break.
>>>>>>>>>>> Letting the user's custom get_encode_buffer() callback suspend
>>>>>>>>>>> the thread is
>>>>>>>>>>> IMO acceptable. In frame threading scenarios, the other
>>>>>>>>>>> threads are still
>>>>>>>>>>> working on their own packets (afaics none depends on the
>>>>>>>>>>> others, since it's
>>>>>>>>>>> intra only encoders only).
>>>>>>>>>>
>>>>>>>>>> I think it was not suggested in the thread so:
>>>>>>>>>> if the users allocation fails the code can fallback to the
>>>>>>>>>> default allocator
>>>>>>>>>> That would lead to the relation:
>>>>>>>>>> If a users allocator can fail (out of buffers) it must be able
>>>>>>>>>> to handle
>>>>>>>>>> that only some of the returned packets are from its own allocator
>>>>>>>>>
>>>>>>>>> In general, custom allocators are used when the caller doesn't
>>>>>>>>> want to use
>>>>>>>>> the default one. But yes, they could use
>>>>>>>>> avcodec_default_get_encoder_buffer() as fallback, which is why
>>>>>>>>> it was added
>>>>>>>>> to begin with. Same applies to get_buffer2() custom
>>>>>>>>> implementations, and so
>>>>>>>>> far i don't think anybody had issues identifying what allocated
>>>>>>>>> a packet
>>>>>>>>> buffer.
>>>>>>>>>
>>>>>>>>> One of the additions to AVPacket people were talking about was a
>>>>>>>>> user opaque
>>>>>>>>> field that libav* would never touch or look at beyond
>>>>>>>>> propagating them
>>>>>>>>> around all the way to the output AVFrame, if any. This opaque
>>>>>>>>> field could
>>>>>>>>> perhaps store such allocator specific information the caller
>>>>>>>>> could use to
>>>>>>>>> identify packets allocated by their own allocator, or those by
>>>>>>>>> avcodec_default_get_encoder_buffer().
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> About alignment, we should at least recommand that allocated
>>>>>>>>>> packets are
>>>>>>>>>> aligned not less than what out av_malloc() would align to.
>>>>>>>>>> Is there a reason to align less ?
>>>>>>>>>
>>>>>>>>> There's no alignment requirement for AVPacket->data, and
>>>>>>>>> av_new_packet()
>>>>>>>>> uses av_buffer_realloc(), which does not guarantee any alignment
>>>>>>>>> whatsoever
>>>>>>>>> on platforms other than Windows. So basically, packet payload
>>>>>>>>> buffers
>>>>>>>>> allocated by our own helpers never had any alignment.
>>>>>>>>
>>>>>>>> for the purpose of exporting raw images, alignment would be "nice
>>>>>>>> to have"
>>>>>>>> because later filters may need it or need to memcpy
>>>>>>>
>>>>>>> Filters don't use AVPackets, they use AVFrames.
>>>>>>
>>>>>> demuxers return AVPackets, so do encoders.
>>>>>> These can contain raw frames.
>>>>>>
>>>>>> also i see for example in rawdec:
>>>>>> frame->buf[0] = av_buffer_ref(avpkt->buf);
>>>>>
>>>>> I ask again, where are you going with this? The alignment for
>>>>> AVPacket data
>>>>> buffers is defined: There is *none*.
>>>>
>>>> I simply stated that 'alignment would be "nice to have"'.
>>>> and then showed some cases where it would be usefull.
>>>
>>> But don't those cases already happen, and without required or
>>> guaranteed alignment?
>>>
>>>>
>>>> I guess where iam going with this is, is the API you add extensible?
>>>> That is if something is not supported now, can it be added later without
>>>> adding a new API.
>>>
>>> I should, it shares a signature with get_buffer2(). That means the
>>> packet to fill (Which fields can be read from it and set can be easily
>>> redefined), avctx so the user can have access to avctx->opaque and so
>>> we can eventually use something like a buffer pool in the default
>>> allocator callback, and a flags parameter to tell the callback there
>>> are requirements.
>>>
>>> Which makes me realize, maybe a flag to tell the callback "Alignment
>>> is required" could solve your concerns?
>>
>> Actually, thinking about it, it's the same situation as always requiring
>> it. The mere existence of such a flag would require users of the old API
>> moving onto the new to redefine their buffers, since now they *may* need
>> to align them, when before they didn't. So not really an option.
>>
> 
> One could say that currently (i.e. up until the next + 1 bump),
> alignment is advisable as it may improve performance, but that from next
> + 1 bump onwards the desired alignment is required.

It really makes no difference. The end result is that the library user 
will eventually not be able to use their buffers as they have until now.

Since the encoder needs to set the DR1 codec cap in order for the 
callback to be something other the default one, any encoder that 
actually requires alignment outside of the one defined for AVPacket 
itself should simply not set it. Such an encoder couldn't even use 
av_new_packet(), since it doesn't use av_malloc().

It's why decoders like the libdav1d wrapper don't set DR1. get_buffer2() 
can't guarantee the alignment or padding required by the external library.

> Furthermore one can add public fields to each encoder containing the
> maximal alignment that buffers for these encoders may required. This way
> users may choose to use the default buffers instead of their own for the
> (few) encoders that need bigger alignment. Said flag would be a bit like
> the max_lowres value for decoders.
> Notice that I am of course only speaking about requirements for packet
> buffers for encoders, not for AVPackets in general.
> 
> - Andreas
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>

Andreas Rheinhardt:
> Michael Niedermayer:
>> On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
>>> On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
>>>> On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
>>>>> On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
>>>>>> On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
>>>>>>> On 2/21/2021 6:04 PM, James Almer wrote:
>>>>>>>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>>>>>>>> On 21/02/2021 20:00, James Almer wrote:
>>>>>>>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>>>>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>>>>>>>> This callback is functionally the same as get_buffer2()
>>>>>>>>>>>> is for decoders, and
>>>>>>>>>>>> implements for the new encode API the functionality of
>>>>>>>>>>>> the old encode API had
>>>>>>>>>>>> where the user could provide their own buffers.
>>>>>>>>>>>>
>>>>>>>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>>>>>>>> ---
>>>>>>>>>>>> Used the names Lynne suggested this time, plus a line
>>>>>>>>>>>> about how the callback
>>>>>>>>>>>> must be thread safe.
>>>>>>>>>>>>
>>>>>>>>>>>>     libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>     libavcodec/codec.h   |  8 ++++---
>>>>>>>>>>>>     libavcodec/encode.c  | 54
>>>>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>     libavcodec/encode.h  |  8 +++++++
>>>>>>>>>>>>     libavcodec/options.c |  1 +
>>>>>>>>>>>>     5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>>>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>>>>>>>> --- a/libavcodec/avcodec.h
>>>>>>>>>>>> +++ b/libavcodec/avcodec.h
>>>>>>>>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>>>>>>>>      */
>>>>>>>>>>>>     #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>> +/**
>>>>>>>>>>>> + * The encoder will keep a reference to the packet and
>>>>>>>>>>>> may reuse it later.
>>>>>>>>>>>> + */
>>>>>>>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>>>>>>>> +
>>>>>>>>>>>>     struct AVCodecInternal;
>>>>>>>>>>>>     /**
>>>>>>>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>>>>>>>          * - encoding: set by user
>>>>>>>>>>>>          */
>>>>>>>>>>>>         int export_side_data;
>>>>>>>>>>>> +
>>>>>>>>>>>> +    /**
>>>>>>>>>>>> +     * This callback is called at the beginning of each
>>>>>>>>>>>> packet to get a data
>>>>>>>>>>>> +     * buffer for it.
>>>>>>>>>>>> +     *
>>>>>>>>>>>> +     * The following field will be set in the packet
>>>>>>>>>>>> before this callback is
>>>>>>>>>>>> +     * called:
>>>>>>>>>>>> +     * - size
>>>>>>>>>>>> +     * This callback must use the above value to
>>>>>>>>>>>> calculate the required buffer size,
>>>>>>>>>>>> +     * which must padded by at least
>>>>>>>>>>>> AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>>>>>>>> +     *
>>>>>>>>>>>> +     * This callback must fill the following fields in the packet:
>>>>>>>>>>>> +     * - data
>>>>>>>>>>>
>>>>>>>>>>> Is the data pointer allowed to be in write-only memory?
>>>>>>>>>>
>>>>>>>>>> I'm not sure what the use case for this would be, so probably no?
>>>>>>>>>
>>>>>>>>> The two use-cases I see for this API are:
>>>>>>>>>
>>>>>>>>> * You want to avoid a copy when combining the output with something
>>>>>>>>> else.  E.g. you pass a pointer to the block of memory following
>>>>>>>>> where you are going to put your header data (for something you are
>>>>>>>>> going to send over the network, say).
>>>>>>>>>
>>>>>>>>> * You want to avoid a copy when passing the output directly to
>>>>>>>>> something external.  E.g. you pass a pointer to a memory-mapped
>>>>>>>>> device buffer (such as a V4L2 buffer, say).
>>>>>>>>>
>>>>>>>>> In the second case, write-only memory on an external device seems
>>>>>>>>> possible, as does memory which is, say, readable but uncached, so
>>>>>>>>> reading it is a really bad idea.
>>>>>>>>
>>>>>>>> Allowing the second case would depend on how encoders behave. Some may
>>>>>>>> attempt to read data already written to the output packet. It's not like
>>>>>>>> all of them allocate the packet, do a memcpy from an internal buffer,
>>>>>>>> then return.
>>>>>>>> There is also the flag meant to signal that the encoder will keep a
>>>>>>>> reference to the packet around, which more or less implies it will be
>>>>>>>> read later in the encoding process.
>>>>>>>>
>>>>>>>> The doxy for avcodec_encode_video2(), which allowed the user to provide
>>>>>>>> their own buffers in the output packet, does not mention any kind of
>>>>>>>> requirement for the data pointer, so I don't think we can say it's an
>>>>>>>> allowed scenario here either.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> Does it have any alignment requirements?
>>>>>>>>>>
>>>>>>>>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>>>>>>>>
>>>>>>>>> I think say that explicitly.  avcodec_default_get_encoder_buffer()
>>>>>>>>> does give you aligned memory, even though it isn't needed.
>>>>>>>>
>>>>>>>> Would saying "There's no alignment requirement for the data pointer" add
>>>>>>>> anything of value to the doxy? If i don't mention any kind of alignment
>>>>>>>> requirement, it's because there isn't any, and it's implicit.
>>>>>>>> I listed the requirements the user needs to keep in mind, like the
>>>>>>>> padding and the need for an AVBufferRef. But if you think it's worth
>>>>>>>> adding, then sure.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> +     * - buf must contain a pointer to an AVBufferRef
>>>>>>>>>>>> structure. The packet's
>>>>>>>>>>>> +     *   data pointer must be contained in it.
>>>>>>>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(),
>>>>>>>>>>>> and av_buffer_ref().
>>>>>>>>>>>> +     *
>>>>>>>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then
>>>>>>>>>>>> get_encoder_buffer() must call
>>>>>>>>>>>> +     * avcodec_default_get_encoder_buffer() instead of
>>>>>>>>>>>> providing a buffer allocated by
>>>>>>>>>>>> +     * some other means.
>>>>>>>>>>>> +     *
>>>>>>>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
>>>>>>>>>>>> flags then the packet may be reused
>>>>>>>>>>>> +     * (read and/or written to if it is writable) later
>>>>>>>>>>>> by libavcodec.
>>>>>>>>>>>> +     *
>>>>>>>>>>>> +     * This callback must be thread-safe, as when frame
>>>>>>>>>>>> multithreading is used, it may
>>>>>>>>>>>> +     * be called from multiple threads simultaneously.
>>>>>>>>>>>
>>>>>>>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is
>>>>>>>>>>> it really necessary?
>>>>>>>>>>
>>>>>>>>>> This was a suggestion by Lynne, i personally don't know. We
>>>>>>>>>> support frame threading encoding (For intra-only codecs), but
>>>>>>>>>> currently ff_alloc_packet2() does not seem to be thread safe,
>>>>>>>>>> seeing it calls av_fast_padded_malloc(), yet it's called by
>>>>>>>>>> frame threaded encoders.
>>>>>>>>>> Should i remove this?
>>>>>>>>>
>>>>>>>>> I don't know, I was asking only because it sounds tricky.  For cases
>>>>>>>>> with a limited number of buffers available (like memory-mapped
>>>>>>>>> devices) you are going to need locking anyway, so maybe rentrancy
>>>>>>>>> adds no additional inconvenience.
>>>>>>>>>
>>>>>>>>>>>> +     *
>>>>>>>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>>>>>>>> +     *
>>>>>>>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>>>>>>>> +     * - decoding: unused
>>>>>>>>>>>> +     */
>>>>>>>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s,
>>>>>>>>>>>> AVPacket *pkt, int flags);
>>>>>>>>>>>
>>>>>>>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>>>>>>>
>>>>>>>>>>> Can the user return "not yet" somehow to this if they have a
>>>>>>>>>>> fixed output buffer pool but no buffer is currently
>>>>>>>>>>> available?
>>>>>>>>>>
>>>>>>>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I don't much like the idea of the user suspending the thread
>>>>>>>>>>> in the callback until they have some available, which might
>>>>>>>>>>> work in some cases but might also deadlock if an
>>>>>>>>>>> avcodec_receive_packet() call is blocked by it.
>>>>>>>>>>
>>>>>>>>>> Can we make what's in essence a malloc() call return something
>>>>>>>>>> like EAGAIN, and this in turn be propagated back to
>>>>>>>>>> encode_receive_packet_internal()?
>>>>>>>>>
>>>>>>>>> Maybe, or if it has many threads maybe it could wait for something
>>>>>>>>> else to finish first.
>>>>>>>>>
>>>>>>>>>> Couldn't this potentially end up in the forbidden scenario of
>>>>>>>>>> avcodec_send_frame() and avcodec_receive_packet() both returning
>>>>>>>>>> EAGAIN?
>>>>>>>>>
>>>>>>>>> Yes.  If the forbidden case happens then the encoder is stuck anyway
>>>>>>>>> and can't make any forward progress so we need to error out
>>>>>>>>> properly, but the EAGAIN return isn't needed if there is something
>>>>>>>>> else to do on another thread.
>>>>>>>>
>>>>>>>> Ok, but I'm not familiar or knowledgeable enough with the frame thread
>>>>>>>> encoder code to implement this.
>>>>>>>
>>>>>>> Looked at bit into this. AVCodec->encode2() based encoders don't support
>>>>>>> returning EAGAIN at all, as it completely breaks the frame threading logic.
>>>>>>> It would require a considerable rewrite in order to re-add a task that
>>>>>>> didn't fail but also didn't succeed.
>>>>>>>
>>>>>>> Non frame threading encoders could probably support it with some minimal
>>>>>>> changes, but i don't think suddenly letting an scenario that was until now
>>>>>>> guaranteed to never happen start happening (avcodec_send_frame() and
>>>>>>> avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API
>>>>>>> break.
>>>>>>> Letting the user's custom get_encode_buffer() callback suspend the thread is
>>>>>>> IMO acceptable. In frame threading scenarios, the other threads are still
>>>>>>> working on their own packets (afaics none depends on the others, since it's
>>>>>>> intra only encoders only).
>>>>>>
>>>>>> I think it was not suggested in the thread so:
>>>>>> if the users allocation fails the code can fallback to the default allocator
>>>>>> That would lead to the relation:
>>>>>> If a users allocator can fail (out of buffers) it must be able to handle
>>>>>> that only some of the returned packets are from its own allocator
>>>>>
>>>>> In general, custom allocators are used when the caller doesn't want to use
>>>>> the default one. But yes, they could use
>>>>> avcodec_default_get_encoder_buffer() as fallback, which is why it was added
>>>>> to begin with. Same applies to get_buffer2() custom implementations, and so
>>>>> far i don't think anybody had issues identifying what allocated a packet
>>>>> buffer.
>>>>>
>>>>> One of the additions to AVPacket people were talking about was a user opaque
>>>>> field that libav* would never touch or look at beyond propagating them
>>>>> around all the way to the output AVFrame, if any. This opaque field could
>>>>> perhaps store such allocator specific information the caller could use to
>>>>> identify packets allocated by their own allocator, or those by
>>>>> avcodec_default_get_encoder_buffer().
>>>>>
>>>>>>
>>>>>> About alignment, we should at least recommand that allocated packets are
>>>>>> aligned not less than what out av_malloc() would align to.
>>>>>> Is there a reason to align less ?
>>>>>
>>>>> There's no alignment requirement for AVPacket->data, and av_new_packet()
>>>>> uses av_buffer_realloc(), which does not guarantee any alignment whatsoever
>>>>> on platforms other than Windows. So basically, packet payload buffers
>>>>> allocated by our own helpers never had any alignment.
>>>>
>>>> for the purpose of exporting raw images, alignment would be "nice to have"
>>>> because later filters may need it or need to memcpy
>>>
>>> Filters don't use AVPackets, they use AVFrames.
>>
>> demuxers return AVPackets, so do encoders.
>> These can contain raw frames.
>>
>> also i see for example in rawdec:
>> frame->buf[0] = av_buffer_ref(avpkt->buf);
>>
> 
> That seems to be completely broken: I see no check for whether the
> packet is writable at all. Will investigate.
> 
> - Andreas
> 
It is indeed possible for rawdec to write to non-writable buffers here.
It at least "works" with rgba64be. The checksums of a streamcopied track
change when one also decodes the track.

- Andreas

On Fri, Mar 12, 2021 at 04:59:16PM -0300, James Almer wrote:
> On 3/12/2021 4:46 PM, James Almer wrote:
> > On 3/12/2021 4:30 PM, Michael Niedermayer wrote:
> > > On Fri, Mar 12, 2021 at 02:03:52PM -0300, James Almer wrote:
> > > > On 3/12/2021 1:32 PM, Michael Niedermayer wrote:
> > > > > On Thu, Mar 11, 2021 at 02:18:36PM -0300, James Almer wrote:
> > > > > > On 3/11/2021 1:35 PM, Michael Niedermayer wrote:
> > > > > > > On Wed, Mar 10, 2021 at 05:59:11PM -0300, James Almer wrote:
> > > > > > > > On 3/10/2021 5:18 PM, Michael Niedermayer wrote:
> > > > > > > > > On Mon, Feb 22, 2021 at 07:27:34PM -0300, James Almer wrote:
> > > > > > > > > > On 2/21/2021 6:04 PM, James Almer wrote:
> > > > > > > > > > > On 2/21/2021 5:29 PM, Mark Thompson wrote:
> > > > > > > > > > > > On 21/02/2021 20:00, James Almer wrote:
> > > > > > > > > > > > > On 2/21/2021 4:13 PM, Mark Thompson wrote:
> > > > > > > > > > > > > > On 21/02/2021 17:35, James Almer wrote:
> > > > > > > > > > > > > > > This callback is functionally the same as get_buffer2()
> > > > > > > > > > > > > > > is for decoders, and
> > > > > > > > > > > > > > > implements for the new encode API the functionality of
> > > > > > > > > > > > > > > the old encode API had
> > > > > > > > > > > > > > > where the user could provide their own buffers.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Signed-off-by: James Almer <jamrial@gmail.com>
> > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > Used the names Lynne suggested this time, plus a line
> > > > > > > > > > > > > > > about how the callback
> > > > > > > > > > > > > > > must be thread safe.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > >      
> > > > > > > > > > > > > > > libavcodec/avcodec.h
> > > > > > > > > > > > > > > | 45
> > > > > > > > > > > > > > > ++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > > > >       libavcodec/codec.h   |  8 ++++---
> > > > > > > > > > > > > > >       libavcodec/encode.c  | 54
> > > > > > > > > > > > > > > +++++++++++++++++++++++++++++++++++++++++++-
> > > > > > > > > > > > > > >       libavcodec/encode.h  |  8 +++++++
> > > > > > > > > > > > > > >       libavcodec/options.c |  1 +
> > > > > > > > > > > > > > >       5 files changed, 112 insertions(+), 4 deletions(-)
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
> > > > > > > > > > > > > > > index 7dbf083a24..e60eb16ce1 100644
> > > > > > > > > > > > > > > --- a/libavcodec/avcodec.h
> > > > > > > > > > > > > > > +++ b/libavcodec/avcodec.h
> > > > > > > > > > > > > > > @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
> > > > > > > > > > > > > > >        */
> > > > > > > > > > > > > > >       #define AV_GET_BUFFER_FLAG_REF (1 << 0)
> > > > > > > > > > > > > > > +/**
> > > > > > > > > > > > > > > + * The encoder will keep a reference to the packet and
> > > > > > > > > > > > > > > may reuse it later.
> > > > > > > > > > > > > > > + */
> > > > > > > > > > > > > > > +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > >       struct AVCodecInternal;
> > > > > > > > > > > > > > >       /**
> > > > > > > > > > > > > > > @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
> > > > > > > > > > > > > > >            * - encoding: set by user
> > > > > > > > > > > > > > >            */
> > > > > > > > > > > > > > >           int export_side_data;
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +    /**
> > > > > > > > > > > > > > > +     * This callback is called at the beginning of each
> > > > > > > > > > > > > > > packet to get a data
> > > > > > > > > > > > > > > +     * buffer for it.
> > > > > > > > > > > > > > > +     *
> > > > > > > > > > > > > > > +     * The following field will be set in the packet
> > > > > > > > > > > > > > > before this callback is
> > > > > > > > > > > > > > > +     * called:
> > > > > > > > > > > > > > > +     * - size
> > > > > > > > > > > > > > > +     * This callback must use the above value to
> > > > > > > > > > > > > > > calculate the required buffer size,
> > > > > > > > > > > > > > > +     * which must padded by at least
> > > > > > > > > > > > > > > AV_INPUT_BUFFER_PADDING_SIZE bytes.
> > > > > > > > > > > > > > > +     *
> > > > > > > > > > > > > > > +     * This
> > > > > > > > > > > > > > > callback must fill
> > > > > > > > > > > > > > > the following fields
> > > > > > > > > > > > > > > in the packet:
> > > > > > > > > > > > > > > +     * - data
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Is the data pointer allowed to be in write-only memory?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I'm not sure what the use
> > > > > > > > > > > > > case for this would be, so
> > > > > > > > > > > > > probably no?
> > > > > > > > > > > > 
> > > > > > > > > > > > The two use-cases I see for this API are:
> > > > > > > > > > > > 
> > > > > > > > > > > > * You want to avoid a copy when
> > > > > > > > > > > > combining the output with
> > > > > > > > > > > > something
> > > > > > > > > > > > else.  E.g. you pass a pointer to the block of memory following
> > > > > > > > > > > > where you are going to put your
> > > > > > > > > > > > header data (for something you
> > > > > > > > > > > > are
> > > > > > > > > > > > going to send over the network, say).
> > > > > > > > > > > > 
> > > > > > > > > > > > * You want to avoid a copy when passing the output directly to
> > > > > > > > > > > > something external.  E.g. you pass a pointer to a memory-mapped
> > > > > > > > > > > > device buffer (such as a V4L2 buffer, say).
> > > > > > > > > > > > 
> > > > > > > > > > > > In the second case, write-only
> > > > > > > > > > > > memory on an external device
> > > > > > > > > > > > seems
> > > > > > > > > > > > possible, as does memory which
> > > > > > > > > > > > is, say, readable but uncached,
> > > > > > > > > > > > so
> > > > > > > > > > > > reading it is a really bad idea.
> > > > > > > > > > > 
> > > > > > > > > > > Allowing the second case would
> > > > > > > > > > > depend on how encoders behave. Some
> > > > > > > > > > > may
> > > > > > > > > > > attempt to read data already written
> > > > > > > > > > > to the output packet. It's not like
> > > > > > > > > > > all of them allocate the packet, do
> > > > > > > > > > > a memcpy from an internal buffer,
> > > > > > > > > > > then return.
> > > > > > > > > > > There is also the flag meant to
> > > > > > > > > > > signal that the encoder will keep a
> > > > > > > > > > > reference to the packet around,
> > > > > > > > > > > which more or less implies it will
> > > > > > > > > > > be
> > > > > > > > > > > read later in the encoding process.
> > > > > > > > > > > 
> > > > > > > > > > > The doxy for
> > > > > > > > > > > avcodec_encode_video2(), which
> > > > > > > > > > > allowed the user to provide
> > > > > > > > > > > their own buffers in the output
> > > > > > > > > > > packet, does not mention any kind of
> > > > > > > > > > > requirement for the data pointer, so
> > > > > > > > > > > I don't think we can say it's an
> > > > > > > > > > > allowed scenario here either.
> > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > > > Does it have any alignment requirements?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > No, just padding. AVPacket
> > > > > > > > > > > > > doesn't require alignment
> > > > > > > > > > > > > for the payload.
> > > > > > > > > > > > 
> > > > > > > > > > > > I think say that explicitly.
> > > > > > > > > > > > avcodec_default_get_encoder_buffer()
> > > > > > > > > > > > does give you aligned memory, even though it isn't needed.
> > > > > > > > > > > 
> > > > > > > > > > > Would saying "There's no alignment
> > > > > > > > > > > requirement for the data pointer"
> > > > > > > > > > > add
> > > > > > > > > > > anything of value to the doxy? If i
> > > > > > > > > > > don't mention any kind of alignment
> > > > > > > > > > > requirement, it's because there isn't any, and it's implicit.
> > > > > > > > > > > I listed the requirements the user
> > > > > > > > > > > needs to keep in mind, like the
> > > > > > > > > > > padding and the need for an
> > > > > > > > > > > AVBufferRef. But if you think it's
> > > > > > > > > > > worth
> > > > > > > > > > > adding, then sure.
> > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > > > > +     * - buf must contain a pointer to an AVBufferRef
> > > > > > > > > > > > > > > structure. The packet's
> > > > > > > > > > > > > > > +     *   data pointer must be contained in it.
> > > > > > > > > > > > > > > +     *   See: av_buffer_create(), av_buffer_alloc(),
> > > > > > > > > > > > > > > and av_buffer_ref().
> > > > > > > > > > > > > > > +     *
> > > > > > > > > > > > > > > +     * If AV_CODEC_CAP_DR1 is not set then
> > > > > > > > > > > > > > > get_encoder_buffer() must call
> > > > > > > > > > > > > > > +     * avcodec_default_get_encoder_buffer() instead of
> > > > > > > > > > > > > > > providing a buffer allocated by
> > > > > > > > > > > > > > > +     * some other means.
> > > > > > > > > > > > > > > +     *
> > > > > > > > > > > > > > > +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in
> > > > > > > > > > > > > > > flags then the packet may be reused
> > > > > > > > > > > > > > > +     * (read and/or written to if it is writable) later
> > > > > > > > > > > > > > > by libavcodec.
> > > > > > > > > > > > > > > +     *
> > > > > > > > > > > > > > > +     * This callback must be thread-safe, as when frame
> > > > > > > > > > > > > > > multithreading is used, it may
> > > > > > > > > > > > > > > +     * be called from multiple threads simultaneously.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Allowing simulatenous calls feels unexpectedly tricky.  Is
> > > > > > > > > > > > > > it really necessary?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > This was a suggestion by Lynne, i personally don't know. We
> > > > > > > > > > > > > support frame threading encoding (For intra-only codecs), but
> > > > > > > > > > > > > currently ff_alloc_packet2() does not seem to be thread safe,
> > > > > > > > > > > > > seeing it calls av_fast_padded_malloc(), yet it's called by
> > > > > > > > > > > > > frame threaded encoders.
> > > > > > > > > > > > > Should i remove this?
> > > > > > > > > > > > 
> > > > > > > > > > > > I don't know, I was asking only
> > > > > > > > > > > > because it sounds tricky.  For
> > > > > > > > > > > > cases
> > > > > > > > > > > > with a limited number of buffers available (like memory-mapped
> > > > > > > > > > > > devices) you are going to need
> > > > > > > > > > > > locking anyway, so maybe
> > > > > > > > > > > > rentrancy
> > > > > > > > > > > > adds no additional inconvenience.
> > > > > > > > > > > > 
> > > > > > > > > > > > > > > +     *
> > > > > > > > > > > > > > > +     * @see avcodec_default_get_encoder_buffer()
> > > > > > > > > > > > > > > +     *
> > > > > > > > > > > > > > > +     * - encoding: Set by libavcodec, user can override.
> > > > > > > > > > > > > > > +     * - decoding: unused
> > > > > > > > > > > > > > > +     */
> > > > > > > > > > > > > > > +    int (*get_encoder_buffer)(struct AVCodecContext *s,
> > > > > > > > > > > > > > > AVPacket *pkt, int flags);
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Can the encoder ask for arbitrarily many packets?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Can the user return "not yet" somehow to this if they have a
> > > > > > > > > > > > > > fixed output buffer pool but no buffer is currently
> > > > > > > > > > > > > > available?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > No, as is it can't. Return values < 0 are considered errors.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I don't much like the idea of the user suspending the thread
> > > > > > > > > > > > > > in the callback until they have some available, which might
> > > > > > > > > > > > > > work in some cases but might also deadlock if an
> > > > > > > > > > > > > > avcodec_receive_packet() call is blocked by it.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Can we make what's in essence a malloc() call return something
> > > > > > > > > > > > > like EAGAIN, and this in turn be propagated back to
> > > > > > > > > > > > > encode_receive_packet_internal()?
> > > > > > > > > > > > 
> > > > > > > > > > > > Maybe, or if it has many threads
> > > > > > > > > > > > maybe it could wait for
> > > > > > > > > > > > something
> > > > > > > > > > > > else to finish first.
> > > > > > > > > > > > 
> > > > > > > > > > > > > Couldn't this potentially end up in the forbidden scenario of
> > > > > > > > > > > > > avcodec_send_frame() and
> > > > > > > > > > > > > avcodec_receive_packet()
> > > > > > > > > > > > > both returning
> > > > > > > > > > > > > EAGAIN?
> > > > > > > > > > > > 
> > > > > > > > > > > > Yes.  If the forbidden case
> > > > > > > > > > > > happens then the encoder is
> > > > > > > > > > > > stuck anyway
> > > > > > > > > > > > and can't make any forward progress so we need to error out
> > > > > > > > > > > > properly, but the EAGAIN return
> > > > > > > > > > > > isn't needed if there is
> > > > > > > > > > > > something
> > > > > > > > > > > > else to do on another thread.
> > > > > > > > > > > 
> > > > > > > > > > > Ok, but I'm not familiar or
> > > > > > > > > > > knowledgeable enough with the frame
> > > > > > > > > > > thread
> > > > > > > > > > > encoder code to implement this.
> > > > > > > > > > 
> > > > > > > > > > Looked at bit into this.
> > > > > > > > > > AVCodec->encode2() based encoders don't
> > > > > > > > > > support
> > > > > > > > > > returning EAGAIN at all, as it
> > > > > > > > > > completely breaks the frame threading
> > > > > > > > > > logic.
> > > > > > > > > > It would require a considerable rewrite
> > > > > > > > > > in order to re-add a task that
> > > > > > > > > > didn't fail but also didn't succeed.
> > > > > > > > > > 
> > > > > > > > > > Non frame threading encoders could
> > > > > > > > > > probably support it with some minimal
> > > > > > > > > > changes, but i don't think suddenly
> > > > > > > > > > letting an scenario that was until now
> > > > > > > > > > guaranteed to never happen start
> > > > > > > > > > happening (avcodec_send_frame() and
> > > > > > > > > > avcodec_receive_packet() both returning
> > > > > > > > > > EAGAIN) is a good idea. It's an API
> > > > > > > > > > break.
> > > > > > > > > > Letting the user's custom
> > > > > > > > > > get_encode_buffer() callback suspend the
> > > > > > > > > > thread is
> > > > > > > > > > IMO acceptable. In frame threading
> > > > > > > > > > scenarios, the other threads are still
> > > > > > > > > > working on their own packets (afaics
> > > > > > > > > > none depends on the others, since it's
> > > > > > > > > > intra only encoders only).
> > > > > > > > > 
> > > > > > > > > I think it was not suggested in the thread so:
> > > > > > > > > if the users allocation fails the code can
> > > > > > > > > fallback to the default allocator
> > > > > > > > > That would lead to the relation:
> > > > > > > > > If a users allocator can fail (out of
> > > > > > > > > buffers) it must be able to handle
> > > > > > > > > that only some of the returned packets are from its own allocator
> > > > > > > > 
> > > > > > > > In general, custom allocators are used when the
> > > > > > > > caller doesn't want to use
> > > > > > > > the default one. But yes, they could use
> > > > > > > > avcodec_default_get_encoder_buffer() as
> > > > > > > > fallback, which is why it was added
> > > > > > > > to begin with. Same applies to get_buffer2()
> > > > > > > > custom implementations, and so
> > > > > > > > far i don't think anybody had issues identifying
> > > > > > > > what allocated a packet
> > > > > > > > buffer.
> > > > > > > > 
> > > > > > > > One of the additions to AVPacket people were
> > > > > > > > talking about was a user opaque
> > > > > > > > field that libav* would never touch or look at
> > > > > > > > beyond propagating them
> > > > > > > > around all the way to the output AVFrame, if
> > > > > > > > any. This opaque field could
> > > > > > > > perhaps store such allocator specific
> > > > > > > > information the caller could use to
> > > > > > > > identify packets allocated by their own allocator, or those by
> > > > > > > > avcodec_default_get_encoder_buffer().
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > About alignment, we should at least
> > > > > > > > > recommand that allocated packets are
> > > > > > > > > aligned not less than what out av_malloc() would align to.
> > > > > > > > > Is there a reason to align less ?
> > > > > > > > 
> > > > > > > > There's no alignment requirement for
> > > > > > > > AVPacket->data, and av_new_packet()
> > > > > > > > uses av_buffer_realloc(), which does not
> > > > > > > > guarantee any alignment whatsoever
> > > > > > > > on platforms other than Windows. So basically,
> > > > > > > > packet payload buffers
> > > > > > > > allocated by our own helpers never had any alignment.
> > > > > > > 
> > > > > > > for the purpose of exporting raw images, alignment
> > > > > > > would be "nice to have"
> > > > > > > because later filters may need it or need to memcpy
> > > > > > 
> > > > > > Filters don't use AVPackets, they use AVFrames.
> > > > > 
> > > > > demuxers return AVPackets, so do encoders.
> > > > > These can contain raw frames.
> > > > > 
> > > > > also i see for example in rawdec:
> > > > > frame->buf[0] = av_buffer_ref(avpkt->buf);
> > > > 
> > > > I ask again, where are you going with this? The alignment for
> > > > AVPacket data
> > > > buffers is defined: There is *none*.
> > > 
> > > I simply stated that 'alignment would be "nice to have"'.
> > > and then showed some cases where it would be usefull.
> > 
> > But don't those cases already happen, and without required or guaranteed
> > alignment?
> > 
> > > 
> > > I guess where iam going with this is, is the API you add extensible?
> > > That is if something is not supported now, can it be added later without
> > > adding a new API.
> > 
> > I should, it shares a signature with get_buffer2(). That means the
> > packet to fill (Which fields can be read from it and set can be easily
> > redefined), avctx so the user can have access to avctx->opaque and so we
> > can eventually use something like a buffer pool in the default allocator
> > callback, and a flags parameter to tell the callback there are
> > requirements.
> > 
> > Which makes me realize, maybe a flag to tell the callback "Alignment is
> > required" could solve your concerns?
> 
> Actually, thinking about it, it's the same situation as always requiring it.
> The mere existence of such a flag would require users of the old API moving
> onto the new to redefine their buffers, since now they *may* need to align
> them, when before they didn't. So not really an option.

yes, maybe all this is overengeneering it.
so as said on irc, iam ok with the patch with a minor clarification on the
flags being hints

Thanks

[...]

On 22/02/2021 22:27, James Almer wrote:
> On 2/21/2021 6:04 PM, James Almer wrote:
>> On 2/21/2021 5:29 PM, Mark Thompson wrote:
>>> On 21/02/2021 20:00, James Almer wrote:
>>>> On 2/21/2021 4:13 PM, Mark Thompson wrote:
>>>>> On 21/02/2021 17:35, James Almer wrote:
>>>>>> This callback is functionally the same as get_buffer2() is for decoders, and
>>>>>> implements for the new encode API the functionality of the old encode API had
>>>>>> where the user could provide their own buffers.
>>>>>>
>>>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>>>> ---
>>>>>> Used the names Lynne suggested this time, plus a line about how the callback
>>>>>> must be thread safe.
>>>>>>
>>>>>>   libavcodec/avcodec.h | 45 ++++++++++++++++++++++++++++++++++++
>>>>>>   libavcodec/codec.h   |  8 ++++---
>>>>>>   libavcodec/encode.c  | 54 +++++++++++++++++++++++++++++++++++++++++++-
>>>>>>   libavcodec/encode.h  |  8 +++++++
>>>>>>   libavcodec/options.c |  1 +
>>>>>>   5 files changed, 112 insertions(+), 4 deletions(-)
>>>>>>
>>>>>> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
>>>>>> index 7dbf083a24..e60eb16ce1 100644
>>>>>> --- a/libavcodec/avcodec.h
>>>>>> +++ b/libavcodec/avcodec.h
>>>>>> @@ -513,6 +513,11 @@ typedef struct AVProducerReferenceTime {
>>>>>>    */
>>>>>>   #define AV_GET_BUFFER_FLAG_REF (1 << 0)
>>>>>> +/**
>>>>>> + * The encoder will keep a reference to the packet and may reuse it later.
>>>>>> + */
>>>>>> +#define AV_GET_ENCODER_BUFFER_FLAG_REF (1 << 0)
>>>>>> +
>>>>>>   struct AVCodecInternal;
>>>>>>   /**
>>>>>> @@ -2346,6 +2351,39 @@ typedef struct AVCodecContext {
>>>>>>        * - encoding: set by user
>>>>>>        */
>>>>>>       int export_side_data;
>>>>>> +
>>>>>> +    /**
>>>>>> +     * This callback is called at the beginning of each packet to get a data
>>>>>> +     * buffer for it.
>>>>>> +     *
>>>>>> +     * The following field will be set in the packet before this callback is
>>>>>> +     * called:
>>>>>> +     * - size
>>>>>> +     * This callback must use the above value to calculate the required buffer size,
>>>>>> +     * which must padded by at least AV_INPUT_BUFFER_PADDING_SIZE bytes.
>>>>>> +     *
>>>>>> +     * This callback must fill the following fields in the packet:
>>>>>> +     * - data
>>>>>
>>>>> Is the data pointer allowed to be in write-only memory?
>>>>
>>>> I'm not sure what the use case for this would be, so probably no?
>>>
>>> The two use-cases I see for this API are:
>>>
>>> * You want to avoid a copy when combining the output with something else.  E.g. you pass a pointer to the block of memory following where you are going to put your header data (for something you are going to send over the network, say).
>>>
>>> * You want to avoid a copy when passing the output directly to something external.  E.g. you pass a pointer to a memory-mapped device buffer (such as a V4L2 buffer, say).
>>>
>>> In the second case, write-only memory on an external device seems possible, as does memory which is, say, readable but uncached, so reading it is a really bad idea.
>>
>> Allowing the second case would depend on how encoders behave. Some may attempt to read data already written to the output packet. It's not like all of them allocate the packet, do a memcpy from an internal buffer, then return.
>> There is also the flag meant to signal that the encoder will keep a reference to the packet around, which more or less implies it will be read later in the encoding process.
>>
>> The doxy for avcodec_encode_video2(), which allowed the user to provide their own buffers in the output packet, does not mention any kind of requirement for the data pointer, so I don't think we can say it's an allowed scenario here either.

Fair enough.  If the tricky cases aren't allowed then there is no problem :)

>>>>> Does it have any alignment requirements?
>>>>
>>>> No, just padding. AVPacket doesn't require alignment for the payload.
>>>
>>> I think say that explicitly.  avcodec_default_get_encoder_buffer() does give you aligned memory, even though it isn't needed.
>>
>> Would saying "There's no alignment requirement for the data pointer" add anything of value to the doxy? If i don't mention any kind of alignment requirement, it's because there isn't any, and it's implicit.
>> I listed the requirements the user needs to keep in mind, like the padding and the need for an AVBufferRef. But if you think it's worth adding, then sure.
>>
>>>
>>>>>> +     * - buf must contain a pointer to an AVBufferRef structure. The packet's
>>>>>> +     *   data pointer must be contained in it.
>>>>>> +     *   See: av_buffer_create(), av_buffer_alloc(), and av_buffer_ref().
>>>>>> +     *
>>>>>> +     * If AV_CODEC_CAP_DR1 is not set then get_encoder_buffer() must call
>>>>>> +     * avcodec_default_get_encoder_buffer() instead of providing a buffer allocated by
>>>>>> +     * some other means.
>>>>>> +     *
>>>>>> +     * If AV_GET_ENCODER_BUFFER_FLAG_REF is set in flags then the packet may be reused
>>>>>> +     * (read and/or written to if it is writable) later by libavcodec.
>>>>>> +     *
>>>>>> +     * This callback must be thread-safe, as when frame multithreading is used, it may
>>>>>> +     * be called from multiple threads simultaneously.
>>>>>
>>>>> Allowing simulatenous calls feels unexpectedly tricky.  Is it really necessary?
>>>>
>>>> This was a suggestion by Lynne, i personally don't know. We support frame threading encoding (For intra-only codecs), but currently ff_alloc_packet2() does not seem to be thread safe, seeing it calls av_fast_padded_malloc(), yet it's called by frame threaded encoders.
>>>> Should i remove this?
>>>
>>> I don't know, I was asking only because it sounds tricky.  For cases with a limited number of buffers available (like memory-mapped devices) you are going to need locking anyway, so maybe rentrancy adds no additional inconvenience.
>>>
>>>>>> +     *
>>>>>> +     * @see avcodec_default_get_encoder_buffer()
>>>>>> +     *
>>>>>> +     * - encoding: Set by libavcodec, user can override.
>>>>>> +     * - decoding: unused
>>>>>> +     */
>>>>>> +    int (*get_encoder_buffer)(struct AVCodecContext *s, AVPacket *pkt, int flags);
>>>>>
>>>>> Can the encoder ask for arbitrarily many packets?
>>>>>
>>>>> Can the user return "not yet" somehow to this if they have a fixed output buffer pool but no buffer is currently available?
>>>>
>>>> No, as is it can't. Return values < 0 are considered errors.
>>>>
>>>>>
>>>>> I don't much like the idea of the user suspending the thread in the callback until they have some available, which might work in some cases but might also deadlock if an avcodec_receive_packet() call is blocked by it.
>>>>
>>>> Can we make what's in essence a malloc() call return something like EAGAIN, and this in turn be propagated back to encode_receive_packet_internal()?
>>>
>>> Maybe, or if it has many threads maybe it could wait for something else to finish first.
>>>
>>>> Couldn't this potentially end up in the forbidden scenario of avcodec_send_frame() and avcodec_receive_packet() both returning EAGAIN?
>>>
>>> Yes.  If the forbidden case happens then the encoder is stuck anyway and can't make any forward progress so we need to error out properly, but the EAGAIN return isn't needed if there is something else to do on another thread.
>>
>> Ok, but I'm not familiar or knowledgeable enough with the frame thread encoder code to implement this.
> 
> Looked at bit into this. AVCodec->encode2() based encoders don't support returning EAGAIN at all, as it completely breaks the frame threading logic. It would require a considerable rewrite in order to re-add a task that didn't fail but also didn't succeed.
> 
> Non frame threading encoders could probably support it with some minimal changes, but i don't think suddenly letting an scenario that was until now guaranteed to never happen start happening (avcodec_send_frame() and avcodec_receive_packet() both returning EAGAIN) is a good idea. It's an API break.
> Letting the user's custom get_encode_buffer() callback suspend the thread is IMO acceptable. In frame threading scenarios, the other threads are still working on their own packets (afaics none depends on the others, since it's intra only encoders only).

Ok, so the only consistent behaviour is that the callback failing is fatal (just like it is with get_buffer2).  That's fine, then.

No more from me, looks good.

Thanks,

- Mark

Mar 12, 2021, 21:00 by jamrial@gmail.com:

> On 3/12/2021 4:42 PM, Michael Niedermayer wrote:
>
>>
>> Related to this, what do the flags passed into get_encoder_buffer() mean
>> exactly ?
>> are they "hints" so that a new/unknown flag can be ignored by an implementation
>> or what else should an application do with a unknown flag, this should
>> be clarified in the documentation
>>
>
> They are hints, same as in get_buffer2(), like with the _REF flag. They can't really be requirements since the callback should be able to ignore new flags it doesn't understand.
>

If we have something, I'd rather not have a hint for alignment, but a comment
recommending "aligned data might be faster depending on the architecture and
encoder" or something along those lines.
That way in the future we can make AVPacket->data aligned without needing
to deprecate an alignment hint here.

On 3/12/2021 7:10 PM, Lynne wrote:
> 
> 
> 
> Mar 12, 2021, 21:00 by jamrial@gmail.com:
> 
>> On 3/12/2021 4:42 PM, Michael Niedermayer wrote:
>>
>>>
>>> Related to this, what do the flags passed into get_encoder_buffer() mean
>>> exactly ?
>>> are they "hints" so that a new/unknown flag can be ignored by an implementation
>>> or what else should an application do with a unknown flag, this should
>>> be clarified in the documentation
>>>
>>
>> They are hints, same as in get_buffer2(), like with the _REF flag. They can't really be requirements since the callback should be able to ignore new flags it doesn't understand.
>>
> 
> If we have something, I'd rather not have a hint for alignment, but a comment
> recommending "aligned data might be faster depending on the architecture and
> encoder" or something along those lines.
> That way in the future we can make AVPacket->data aligned without needing
> to deprecate an alignment hint here.

Added a comment like that, and pushed.

Thanks.