| Message ID | 20211216132151.8216-1-jamrial@gmail.com |
|---|---|
| Headers | show |
| Series | New channel layout API | expand |
On Thu, 16 Dec 2021, James Almer wrote: > Resending the first two patches only, since this is meant to > show the implementation of one of the several suggestions made > in the previous set that need to be discussed and hopefully > resolved in a call. Can you push the full branch somewhere? > > The proposals so far to extend the API to support either custom > labels for channels are, or some form of extra user information. > > - Fixed array of bytes to hold a label. Simple solution, but > the labels will have a hard limit that can only be extended > with a major bump. This is what i implemented in this version. > - "char *name" per channel that the user may allocate and the > API will manage, duplicate and free. Simple solution, and the > name can be arbitrarily long, but inefficient (av_strdup() per > channel with a custom label on layout copy). > - "const char *name" per channel for compile time constants, or > that the user may allocate and free. Very efficient, but for > non compile time strings ensuring they outlive the layout can > be tricky. > - Refcounted AVChannelCustom with a dictionary. This can't be > done with AVBufferRef, so it would require some other form > of reference counting. And a dictionary may add quite a bit of > complexity to the API, as you can set anything on them. Until we have proper refcounting API we can make the AVBufferRef in AVChannelLayout a void *, and only allow channel_layout functions to dereference it as an AVBufferRef. This would mean adding some extra helper functions to channel layout, but overall it is not unsolvable. The real question is that if you want to use refcounting and add helpers to query / replace per-channel metadata, or you find the idea too heavy weight and would like to stick to flat structs. > - Opaque id/s or pointer/s that the API will not touch beyond > passing them around (So unlike the above, the helpers would not > benefit from this). This can be combined with any of the above, > too, and i did as much in this version. > - Leave API as it was in v1. Maybe it is not said enough times, but thanks to everybody who worked on this. It certainly was huge work, and I know that it is a thankless effort to get such a big change merged. Any change based on my suggestions is appreciated, even if some of my ideas get rejected in the end. Thanks, Marton
On 12/16/2021 9:04 PM, Marton Balint wrote: > > > On Thu, 16 Dec 2021, James Almer wrote: > >> Resending the first two patches only, since this is meant to >> show the implementation of one of the several suggestions made >> in the previous set that need to be discussed and hopefully >> resolved in a call. > > Can you push the full branch somewhere? Just force pushed the latest version to the same repo as last time in https://github.com/jamrial/FFmpeg/commits/channel_layout > >> >> The proposals so far to extend the API to support either custom >> labels for channels are, or some form of extra user information. >> >> - Fixed array of bytes to hold a label. Simple solution, but >> the labels will have a hard limit that can only be extended >> with a major bump. This is what i implemented in this version. >> - "char *name" per channel that the user may allocate and the >> API will manage, duplicate and free. Simple solution, and the >> name can be arbitrarily long, but inefficient (av_strdup() per >> channel with a custom label on layout copy). >> - "const char *name" per channel for compile time constants, or >> that the user may allocate and free. Very efficient, but for >> non compile time strings ensuring they outlive the layout can >> be tricky. >> - Refcounted AVChannelCustom with a dictionary. This can't be >> done with AVBufferRef, so it would require some other form >> of reference counting. And a dictionary may add quite a bit of >> complexity to the API, as you can set anything on them. > > Until we have proper refcounting API we can make the AVBufferRef in > AVChannelLayout a void *, and only allow channel_layout functions to > dereference it as an AVBufferRef. This would mean adding some extra > helper functions to channel layout, but overall it is not unsolvable. > > The real question is that if you want to use refcounting and add helpers > to query / replace per-channel metadata, or you find the idea too heavy > weight and would like to stick to flat structs. > >> - Opaque id/s or pointer/s that the API will not touch beyond >> passing them around (So unlike the above, the helpers would not >> benefit from this). This can be combined with any of the above, >> too, and i did as much in this version. >> - Leave API as it was in v1. > > Maybe it is not said enough times, but thanks to everybody who worked on > this. It certainly was huge work, and I know that it is a thankless > effort to get such a big change merged. Any change based on my > suggestions is appreciated, even if some of my ideas get rejected in the > end. > > Thanks, > Marton > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
On Fri, Dec 17, 2021 at 01:04:19AM +0100, Marton Balint wrote: > > > On Thu, 16 Dec 2021, James Almer wrote: > > > Resending the first two patches only, since this is meant to > > show the implementation of one of the several suggestions made > > in the previous set that need to be discussed and hopefully > > resolved in a call. > > Can you push the full branch somewhere? > > > > > The proposals so far to extend the API to support either custom > > labels for channels are, or some form of extra user information. > > > > - Fixed array of bytes to hold a label. Simple solution, but > > the labels will have a hard limit that can only be extended > > with a major bump. This is what i implemented in this version. > > - "char *name" per channel that the user may allocate and the > > API will manage, duplicate and free. Simple solution, and the > > name can be arbitrarily long, but inefficient (av_strdup() per > > channel with a custom label on layout copy). > > - "const char *name" per channel for compile time constants, or > > that the user may allocate and free. Very efficient, but for > > non compile time strings ensuring they outlive the layout can > > be tricky. > > - Refcounted AVChannelCustom with a dictionary. This can't be > > done with AVBufferRef, so it would require some other form > > of reference counting. And a dictionary may add quite a bit of > > complexity to the API, as you can set anything on them. > > Until we have proper refcounting API we can make the AVBufferRef in > AVChannelLayout a void *, and only allow channel_layout functions to > dereference it as an AVBufferRef. This would mean adding some extra helper > functions to channel layout, but overall it is not unsolvable. > > The real question is that if you want to use refcounting and add helpers to > query / replace per-channel metadata, or you find the idea too heavy weight > and would like to stick to flat structs. what is the advantage of refcounting for channel metadata ? is it about the used memory, about the reduced need to copy ? what kind of metadata and what size do you expect ? bytes, kilobytes, megabytes, gigabytes per channel ? what is the overhead for dynamic allocation and ref counting? that is at which point does it even make sense ? thx [...]
On Fri, 17 Dec 2021, Michael Niedermayer wrote: > On Fri, Dec 17, 2021 at 01:04:19AM +0100, Marton Balint wrote: >> >> >> On Thu, 16 Dec 2021, James Almer wrote: >> >>> Resending the first two patches only, since this is meant to >>> show the implementation of one of the several suggestions made >>> in the previous set that need to be discussed and hopefully >>> resolved in a call. >> >> Can you push the full branch somewhere? >> >>> >>> The proposals so far to extend the API to support either custom >>> labels for channels are, or some form of extra user information. >>> >>> - Fixed array of bytes to hold a label. Simple solution, but >>> the labels will have a hard limit that can only be extended >>> with a major bump. This is what i implemented in this version. >>> - "char *name" per channel that the user may allocate and the >>> API will manage, duplicate and free. Simple solution, and the >>> name can be arbitrarily long, but inefficient (av_strdup() per >>> channel with a custom label on layout copy). >>> - "const char *name" per channel for compile time constants, or >>> that the user may allocate and free. Very efficient, but for >>> non compile time strings ensuring they outlive the layout can >>> be tricky. >>> - Refcounted AVChannelCustom with a dictionary. This can't be >>> done with AVBufferRef, so it would require some other form >>> of reference counting. And a dictionary may add quite a bit of >>> complexity to the API, as you can set anything on them. >> >> Until we have proper refcounting API we can make the AVBufferRef in >> AVChannelLayout a void *, and only allow channel_layout functions to >> dereference it as an AVBufferRef. This would mean adding some extra helper >> functions to channel layout, but overall it is not unsolvable. >> >> The real question is that if you want to use refcounting and add helpers to >> query / replace per-channel metadata, or you find the idea too heavy weight >> and would like to stick to flat structs. > > what is the advantage of refcounting for channel metadata ? > is it about the used memory, about the reduced need to copy ? Basicly it is the ability to store per-channel metadata in avdictionary, because otherwise it would have to be copyed, and avdictionary is very ineffective at copying because of many mallocs. > > what kind of metadata and what size do you expect ? > bytes, kilobytes, megabytes, gigabytes per channel ? Usually, nothing, because most format don't have support for per-channel metadata. In some cases it is going to be a couple of textual metadata key-value pairs, such as language, label, group, speaker, positon, so 4-5 dynamically allocated string pairs, plus the AVDictionary itself, multiplied by the number of channels in a layout. > > what is the overhead for dynamic allocation and ref counting? > that is at which point does it even make sense ? I don't have exact measurements. It is generally felt that copying AVDictionary per-channel is a huge overhead for something as lightweight as an audio frame which is a 2-4 kB per channel at most and only a couple of allocs usually not dependant on the number of channels. That's why refcounting was proposed. Also some people simply don't want to store extendable channel metadata in channel layout, and want to keep it simple. Regards, Marton
On Thu, 16 Dec 2021, James Almer wrote: > > > On 12/16/2021 9:04 PM, Marton Balint wrote: >> >> >> On Thu, 16 Dec 2021, James Almer wrote: >> >>> Resending the first two patches only, since this is meant to >>> show the implementation of one of the several suggestions made >>> in the previous set that need to be discussed and hopefully >>> resolved in a call. >> >> Can you push the full branch somewhere? > > Just force pushed the latest version to the same repo as last time in > https://github.com/jamrial/FFmpeg/commits/channel_layout Can you check the libfdk-aac patch? The decoder has some garbage text before the drc_boost context variable, the encoder does not compile with gcc because of the deprecation warning pragmas at the very end of the file. Thanks, Marton
On 12/17/2021 4:20 PM, Marton Balint wrote: > > > On Thu, 16 Dec 2021, James Almer wrote: > >> >> >> On 12/16/2021 9:04 PM, Marton Balint wrote: >>> >>> >>> On Thu, 16 Dec 2021, James Almer wrote: >>> >>>> Resending the first two patches only, since this is meant to >>>> show the implementation of one of the several suggestions made >>>> in the previous set that need to be discussed and hopefully >>>> resolved in a call. >>> >>> Can you push the full branch somewhere? >> >> Just force pushed the latest version to the same repo as last time in >> https://github.com/jamrial/FFmpeg/commits/channel_layout > > Can you check the libfdk-aac patch? The decoder has some garbage text > before the drc_boost context variable, the encoder does not compile with > gcc because of the deprecation warning pragmas at the very end of the file. > > Thanks, > Marton Should be fixed, sorry about that.
On Fri, Dec 17, 2021 at 07:04:08PM +0100, Marton Balint wrote: > > > On Fri, 17 Dec 2021, Michael Niedermayer wrote: > > > On Fri, Dec 17, 2021 at 01:04:19AM +0100, Marton Balint wrote: > > > > > > > > > On Thu, 16 Dec 2021, James Almer wrote: > > > > > > > Resending the first two patches only, since this is meant to > > > > show the implementation of one of the several suggestions made > > > > in the previous set that need to be discussed and hopefully > > > > resolved in a call. > > > > > > Can you push the full branch somewhere? > > > > > > > > > > > The proposals so far to extend the API to support either custom > > > > labels for channels are, or some form of extra user information. > > > > > > > > - Fixed array of bytes to hold a label. Simple solution, but > > > > the labels will have a hard limit that can only be extended > > > > with a major bump. This is what i implemented in this version. > > > > - "char *name" per channel that the user may allocate and the > > > > API will manage, duplicate and free. Simple solution, and the > > > > name can be arbitrarily long, but inefficient (av_strdup() per > > > > channel with a custom label on layout copy). > > > > - "const char *name" per channel for compile time constants, or > > > > that the user may allocate and free. Very efficient, but for > > > > non compile time strings ensuring they outlive the layout can > > > > be tricky. > > > > - Refcounted AVChannelCustom with a dictionary. This can't be > > > > done with AVBufferRef, so it would require some other form > > > > of reference counting. And a dictionary may add quite a bit of > > > > complexity to the API, as you can set anything on them. > > > > > > Until we have proper refcounting API we can make the AVBufferRef in > > > AVChannelLayout a void *, and only allow channel_layout functions to > > > dereference it as an AVBufferRef. This would mean adding some extra helper > > > functions to channel layout, but overall it is not unsolvable. > > > > > > The real question is that if you want to use refcounting and add helpers to > > > query / replace per-channel metadata, or you find the idea too heavy weight > > > and would like to stick to flat structs. > > > > what is the advantage of refcounting for channel metadata ? > > is it about the used memory, about the reduced need to copy ? > > Basicly it is the ability to store per-channel metadata in avdictionary, > because otherwise it would have to be copyed, and avdictionary is very > ineffective at copying because of many mallocs. > > > > > what kind of metadata and what size do you expect ? > > bytes, kilobytes, megabytes, gigabytes per channel ? > > Usually, nothing, because most format don't have support for per-channel > metadata. In some cases it is going to be a couple of textual metadata > key-value pairs, such as language, label, group, speaker, positon, so 4-5 > dynamically allocated string pairs, plus the AVDictionary itself, multiplied > by the number of channels in a layout. > > > > > what is the overhead for dynamic allocation and ref counting? > > that is at which point does it even make sense ? > > I don't have exact measurements. It is generally felt that copying > AVDictionary per-channel is a huge overhead for something as lightweight as > an audio frame which is a 2-4 kB per channel at most and only a couple of > allocs usually not dependant on the number of channels. That's why > refcounting was proposed. I was thinking more at a AVStream / AVCodecParameters level. How will a demuxer transport such metadata over a AVPacket into a decoder outputting metadata-filled AVFrames? thx [...]
On Sat, Dec 18, 2021 at 02:36:12PM +0100, Michael Niedermayer wrote: > On Fri, Dec 17, 2021 at 07:04:08PM +0100, Marton Balint wrote: > > > > > > On Fri, 17 Dec 2021, Michael Niedermayer wrote: > > > > > On Fri, Dec 17, 2021 at 01:04:19AM +0100, Marton Balint wrote: > > > > > > > > > > > > On Thu, 16 Dec 2021, James Almer wrote: > > > > > > > > > Resending the first two patches only, since this is meant to > > > > > show the implementation of one of the several suggestions made > > > > > in the previous set that need to be discussed and hopefully > > > > > resolved in a call. > > > > > > > > Can you push the full branch somewhere? > > > > > > > > > > > > > > The proposals so far to extend the API to support either custom > > > > > labels for channels are, or some form of extra user information. > > > > > > > > > > - Fixed array of bytes to hold a label. Simple solution, but > > > > > the labels will have a hard limit that can only be extended > > > > > with a major bump. This is what i implemented in this version. > > > > > - "char *name" per channel that the user may allocate and the > > > > > API will manage, duplicate and free. Simple solution, and the > > > > > name can be arbitrarily long, but inefficient (av_strdup() per > > > > > channel with a custom label on layout copy). > > > > > - "const char *name" per channel for compile time constants, or > > > > > that the user may allocate and free. Very efficient, but for > > > > > non compile time strings ensuring they outlive the layout can > > > > > be tricky. > > > > > - Refcounted AVChannelCustom with a dictionary. This can't be > > > > > done with AVBufferRef, so it would require some other form > > > > > of reference counting. And a dictionary may add quite a bit of > > > > > complexity to the API, as you can set anything on them. > > > > > > > > Until we have proper refcounting API we can make the AVBufferRef in > > > > AVChannelLayout a void *, and only allow channel_layout functions to > > > > dereference it as an AVBufferRef. This would mean adding some extra helper > > > > functions to channel layout, but overall it is not unsolvable. > > > > > > > > The real question is that if you want to use refcounting and add helpers to > > > > query / replace per-channel metadata, or you find the idea too heavy weight > > > > and would like to stick to flat structs. > > > > > > what is the advantage of refcounting for channel metadata ? > > > is it about the used memory, about the reduced need to copy ? > > > > Basicly it is the ability to store per-channel metadata in avdictionary, > > because otherwise it would have to be copyed, and avdictionary is very > > ineffective at copying because of many mallocs. > > > > > > > > what kind of metadata and what size do you expect ? > > > bytes, kilobytes, megabytes, gigabytes per channel ? > > > > Usually, nothing, because most format don't have support for per-channel > > metadata. In some cases it is going to be a couple of textual metadata > > key-value pairs, such as language, label, group, speaker, positon, so 4-5 > > dynamically allocated string pairs, plus the AVDictionary itself, multiplied > > by the number of channels in a layout. > > > > > > > > what is the overhead for dynamic allocation and ref counting? > > > that is at which point does it even make sense ? > > > > I don't have exact measurements. It is generally felt that copying > > AVDictionary per-channel is a huge overhead for something as lightweight as > > an audio frame which is a 2-4 kB per channel at most and only a couple of > > allocs usually not dependant on the number of channels. That's why > > refcounting was proposed. > > I was thinking more at a AVStream / AVCodecParameters level. > How will a demuxer transport such metadata over a AVPacket into a decoder > outputting metadata-filled AVFrames? or is this never needed ? thx [...]
On Sat, 18 Dec 2021, Michael Niedermayer wrote: > On Sat, Dec 18, 2021 at 02:36:12PM +0100, Michael Niedermayer wrote: >> On Fri, Dec 17, 2021 at 07:04:08PM +0100, Marton Balint wrote: >>> >>> >>> On Fri, 17 Dec 2021, Michael Niedermayer wrote: >>> >>>> On Fri, Dec 17, 2021 at 01:04:19AM +0100, Marton Balint wrote: >>>>> >>>>> >>>>> On Thu, 16 Dec 2021, James Almer wrote: >>>>> >>>>>> Resending the first two patches only, since this is meant to >>>>>> show the implementation of one of the several suggestions made >>>>>> in the previous set that need to be discussed and hopefully >>>>>> resolved in a call. >>>>> >>>>> Can you push the full branch somewhere? >>>>> >>>>>> >>>>>> The proposals so far to extend the API to support either custom >>>>>> labels for channels are, or some form of extra user information. >>>>>> >>>>>> - Fixed array of bytes to hold a label. Simple solution, but >>>>>> the labels will have a hard limit that can only be extended >>>>>> with a major bump. This is what i implemented in this version. >>>>>> - "char *name" per channel that the user may allocate and the >>>>>> API will manage, duplicate and free. Simple solution, and the >>>>>> name can be arbitrarily long, but inefficient (av_strdup() per >>>>>> channel with a custom label on layout copy). >>>>>> - "const char *name" per channel for compile time constants, or >>>>>> that the user may allocate and free. Very efficient, but for >>>>>> non compile time strings ensuring they outlive the layout can >>>>>> be tricky. >>>>>> - Refcounted AVChannelCustom with a dictionary. This can't be >>>>>> done with AVBufferRef, so it would require some other form >>>>>> of reference counting. And a dictionary may add quite a bit of >>>>>> complexity to the API, as you can set anything on them. >>>>> >>>>> Until we have proper refcounting API we can make the AVBufferRef in >>>>> AVChannelLayout a void *, and only allow channel_layout functions to >>>>> dereference it as an AVBufferRef. This would mean adding some extra helper >>>>> functions to channel layout, but overall it is not unsolvable. >>>>> >>>>> The real question is that if you want to use refcounting and add helpers to >>>>> query / replace per-channel metadata, or you find the idea too heavy weight >>>>> and would like to stick to flat structs. >>>> >>>> what is the advantage of refcounting for channel metadata ? >>>> is it about the used memory, about the reduced need to copy ? >>> >>> Basicly it is the ability to store per-channel metadata in avdictionary, >>> because otherwise it would have to be copyed, and avdictionary is very >>> ineffective at copying because of many mallocs. >>> >>>> >>>> what kind of metadata and what size do you expect ? >>>> bytes, kilobytes, megabytes, gigabytes per channel ? >>> >>> Usually, nothing, because most format don't have support for per-channel >>> metadata. In some cases it is going to be a couple of textual metadata >>> key-value pairs, such as language, label, group, speaker, positon, so 4-5 >>> dynamically allocated string pairs, plus the AVDictionary itself, multiplied >>> by the number of channels in a layout. >>> >>>> >>>> what is the overhead for dynamic allocation and ref counting? >>>> that is at which point does it even make sense ? >>> >>> I don't have exact measurements. It is generally felt that copying >>> AVDictionary per-channel is a huge overhead for something as lightweight as >>> an audio frame which is a 2-4 kB per channel at most and only a couple of >>> allocs usually not dependant on the number of channels. That's why >>> refcounting was proposed. >> >> I was thinking more at a AVStream / AVCodecParameters level. > >> How will a demuxer transport such metadata over a AVPacket into a decoder >> outputting metadata-filled AVFrames? > > or is this never needed ? I am not sure I understand. Usually metadata is passed from demuxer to decoder by avcodec_parameters_to_context(), this is used for all metadata which is in AVCodecParameters. For per-packet metadata ff_decode_frame_props() has some automatic packet side data -> frame side data transfer. AVStream side data may be transferred to AVPacket side data if av_format_inject_global_side_data() is used, but it is not enabled by default. Regards, Marton
On Sun, Dec 19, 2021 at 12:35:11PM +0100, Marton Balint wrote: > > > On Sat, 18 Dec 2021, Michael Niedermayer wrote: > > > On Sat, Dec 18, 2021 at 02:36:12PM +0100, Michael Niedermayer wrote: > > > On Fri, Dec 17, 2021 at 07:04:08PM +0100, Marton Balint wrote: > > > > > > > > > > > > On Fri, 17 Dec 2021, Michael Niedermayer wrote: > > > > > > > > > On Fri, Dec 17, 2021 at 01:04:19AM +0100, Marton Balint wrote: > > > > > > > > > > > > > > > > > > On Thu, 16 Dec 2021, James Almer wrote: > > > > > > > > > > > > > Resending the first two patches only, since this is meant to > > > > > > > show the implementation of one of the several suggestions made > > > > > > > in the previous set that need to be discussed and hopefully > > > > > > > resolved in a call. > > > > > > > > > > > > Can you push the full branch somewhere? > > > > > > > > > > > > > > > > > > > > The proposals so far to extend the API to support either custom > > > > > > > labels for channels are, or some form of extra user information. > > > > > > > > > > > > > > - Fixed array of bytes to hold a label. Simple solution, but > > > > > > > the labels will have a hard limit that can only be extended > > > > > > > with a major bump. This is what i implemented in this version. > > > > > > > - "char *name" per channel that the user may allocate and the > > > > > > > API will manage, duplicate and free. Simple solution, and the > > > > > > > name can be arbitrarily long, but inefficient (av_strdup() per > > > > > > > channel with a custom label on layout copy). > > > > > > > - "const char *name" per channel for compile time constants, or > > > > > > > that the user may allocate and free. Very efficient, but for > > > > > > > non compile time strings ensuring they outlive the layout can > > > > > > > be tricky. > > > > > > > - Refcounted AVChannelCustom with a dictionary. This can't be > > > > > > > done with AVBufferRef, so it would require some other form > > > > > > > of reference counting. And a dictionary may add quite a bit of > > > > > > > complexity to the API, as you can set anything on them. > > > > > > > > > > > > Until we have proper refcounting API we can make the AVBufferRef in > > > > > > AVChannelLayout a void *, and only allow channel_layout functions to > > > > > > dereference it as an AVBufferRef. This would mean adding some extra helper > > > > > > functions to channel layout, but overall it is not unsolvable. > > > > > > > > > > > > The real question is that if you want to use refcounting and add helpers to > > > > > > query / replace per-channel metadata, or you find the idea too heavy weight > > > > > > and would like to stick to flat structs. > > > > > > > > > > what is the advantage of refcounting for channel metadata ? > > > > > is it about the used memory, about the reduced need to copy ? > > > > > > > > Basicly it is the ability to store per-channel metadata in avdictionary, > > > > because otherwise it would have to be copyed, and avdictionary is very > > > > ineffective at copying because of many mallocs. > > > > > > > > > > > > > > what kind of metadata and what size do you expect ? > > > > > bytes, kilobytes, megabytes, gigabytes per channel ? > > > > > > > > Usually, nothing, because most format don't have support for per-channel > > > > metadata. In some cases it is going to be a couple of textual metadata > > > > key-value pairs, such as language, label, group, speaker, positon, so 4-5 > > > > dynamically allocated string pairs, plus the AVDictionary itself, multiplied > > > > by the number of channels in a layout. > > > > > > > > > > > > > > what is the overhead for dynamic allocation and ref counting? > > > > > that is at which point does it even make sense ? > > > > > > > > I don't have exact measurements. It is generally felt that copying > > > > AVDictionary per-channel is a huge overhead for something as lightweight as > > > > an audio frame which is a 2-4 kB per channel at most and only a couple of > > > > allocs usually not dependant on the number of channels. That's why > > > > refcounting was proposed. > > > > > > I was thinking more at a AVStream / AVCodecParameters level. > > > > > How will a demuxer transport such metadata over a AVPacket into a decoder > > > outputting metadata-filled AVFrames? > > > > or is this never needed ? > > I am not sure I understand. Usually metadata is passed from demuxer to > decoder by avcodec_parameters_to_context(), this is used for all metadata > which is in AVCodecParameters. > > For per-packet metadata ff_decode_frame_props() has some automatic packet > side data -> frame side data transfer. > > AVStream side data may be transferred to AVPacket side data if > av_format_inject_global_side_data() is used, but it is not enabled by > default. The sidedata in AVPacket is not channel specific, the data in AVFrames new channels is. The later is as you wrote expensive to copy/alloc so it needs ref counting what i was trying to point to was that if we need a way to transfer this data on a per packet base from demuxer forward then we need a flat format. At least with the current AVPackets And a flat format is much lighter to copy around so then one is pushed toward the question, "should that be used for AVFrames too?" a small <10kb audio frame with 8 audio channels and a ref counted dictionary or whatever per each of the 8 channels vs. a flat description. an extra is that a flat format is very simple to memcmp() to check if it changed. I have no position in this above, iam just wanting to make sure nothing is missed in this somewhat rushed design. also theres yet another orthogonal aspect which has been missed i think This per channel description whatever its form will be is usefull for another case. And that are the video/image channels. generally this is red/green/blue/(alpha) but there is material with other channels, IR/NIR/UV/... and i imagine probably depth from some lidar/radar system Ideally such a future image channel API can share parts with the audio one thx [...]