| Message ID | 20210819145342.104364-1-george@nsup.org |
|---|---|
| State | New |
| Headers | show |
| Series | [FFmpeg-devel] lavfi/formats: document the negotiation process. | expand |
| Context | Check | Description |
|---|---|---|
| andriy/x86_make | success | Make finished |
| andriy/x86_make_fate | success | Make fate finished |
| andriy/PPC64_make | success | Make finished |
| andriy/PPC64_make_fate | success | Make fate finished |
August 19, 2021 10:53 AM, "Nicolas George" <george@nsup.org> wrote: > Signed-off-by: Nicolas George <george@nsup.org> > --- > libavfilter/formats.h | 85 +++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 85 insertions(+) > > diff --git a/libavfilter/formats.h b/libavfilter/formats.h > index ed513c265a..b3e780a41d 100644 > --- a/libavfilter/formats.h > +++ b/libavfilter/formats.h > @@ -75,6 +75,91 @@ typedef struct AVFilterFormatMerger { > int (*can_merge)(const void *a, const void *b); > } AVFilterFormatsMerger; > > +/** > + * Callbacks and properties to describe the steps of a format negotiation. > + * > + * The steps are: > + * > + * 1. query_formats(): call the callbacks on all filter to set lists of > + * supported formats. > + * When links on a filter must eventually have the same > + * format, the lists of supported formats are the same > + * object in memory. > + * See: > + * http://www.normalesup.org/~george/articles/format_negotiation_in_libavfilter/#12 > + * > + * > + * 2. query_formats(): merge lists of supported formats or insert automatic > + * conversion filters. > + * Compute the intersection of the lists of supported > + * formats on the ends of links. If it succeeds, replace > + * both objects with the intersection everywhere they > + * are referenced. > + * If the intersection is empty, insert an automatic > + * conversion filter. > + * If several formats are negotiated at once (format, > + * rate, layout), only merge if all three can be, since > + * the conversion filter can convert all three at once. > + * This process goes on as long as progress is made. > + * See: > + * http://www.normalesup.org/~george/articles/format_negotiation_in_libavfilter/#14 > + * http://www.normalesup.org/~george/articles/format_negotiation_in_libavfilter/#29 > + * > + * 3. reduce_formats(): try to reduce format conversion within filters. > + * For each link where there is only one supported > + * formats on output, for each output of the connected > + * filter, if the media type is the same and said > + * format is supported, keep only this one. > + * This process goes on as long as progress is made. > + * Rationale: conversion filters will set a large list > + * of supported formats on outputs but users will > + * expect the output to be as close as possible as the > + * input (examples: scale without changing the pixel > + * format, resample without changint the layout). > + * FIXME: this can probably be done by merging the > + * input and output lists instead of re-implementing > + * the logic. > + * > + * 4. swap_sample_fmts(): > + * swap_samplerates(): > + * swap_channel_layouts(): For each filter with an input with only one > + * supported format, when outputs have several > + * supported formats, put the best one with > + * reference to the input at the beginning of the > + * list, to prepare it for being picked up by > + * pick_formats(). > + * The best format is the one that is most > + * similar to the input while not losing too much > + * information. > + * This process need to run only once. > + * FIXME: reduce_formats() operates on all inputs > + * with a single format, swap_*() operates on the > + * first one only: check if the difference makes > + * sense. > + * TODO: the swapping done for one filter can > + * override the swapping done for another filter > + * connected to the same list of formats, maybe > + * it would be better to compute a total score > + * for all connected filters and use the score to > + * pick the format instead of just swapping. > + * TODO: make the similarity logic available as > + * public functions in libavutil. > + * > + * 5. pick_formats(): Choose one format from the lists of supported formats, > + * use it for the link and reduce the list to a single > + * element to force other filters connected to the same > + * list to use it. > + * First process all links where there is a single format > + * and the output links of all filters with an input, > + * trying to preserve similarity between input and > + * outputs. > + * Repeat as long as process is made. > + * Then do a final run for the remaining filters. > + * FIXME: the similarity logic (the ref argument to > + * pick_format()) added in FFmpeg duplicates and > + * overrides the swapping logic added in libav. Better > + * merge them into a score system. > + */ > typedef struct AVFilterNegotiation { > unsigned nb; > const AVFilterFormatsMerger *mergers; > -- > 2.32.0 > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". lgtm
+ * The steps are: + * + * 1. query_formats(): call the callbacks on all filter to set lists of + * supported formats. + * When links on a filter must eventually have the same + * format, the lists of supported formats are the same + * object in memory. + * See: + *http://www.normalesup.org/~george/articles/format_negotiation_in_libavfilter/#12 <http://www.normalesup.org/%7Egeorge/articles/format_negotiation_in_libavfilter/#12> + * + * + * 2. query_formats(): merge lists of supported formats or insert automatic + * conversion filters. I know almost nothing about this stuff, but it doesn't sound right that the frist two steps use the same function. Isn't step 2 merge_formats()? Michael
Michael Koch (12021-08-20): > I know almost nothing about this stuff, but it doesn't sound right that the frist > two steps use the same function. Isn't step 2 merge_formats()? Unfortunately not. I wanted to separate them, but it is not quite possible because of the weird partial negotiation system that I regret introducing for amerge and is now used by about two and a half filters. This is one of the things that needs overhauling. Regards,
diff --git a/libavfilter/formats.h b/libavfilter/formats.h index ed513c265a..b3e780a41d 100644 --- a/libavfilter/formats.h +++ b/libavfilter/formats.h @@ -75,6 +75,91 @@ typedef struct AVFilterFormatMerger { int (*can_merge)(const void *a, const void *b); } AVFilterFormatsMerger; +/** + * Callbacks and properties to describe the steps of a format negotiation. + * + * The steps are: + * + * 1. query_formats(): call the callbacks on all filter to set lists of + * supported formats. + * When links on a filter must eventually have the same + * format, the lists of supported formats are the same + * object in memory. + * See: + * http://www.normalesup.org/~george/articles/format_negotiation_in_libavfilter/#12 + * + * + * 2. query_formats(): merge lists of supported formats or insert automatic + * conversion filters. + * Compute the intersection of the lists of supported + * formats on the ends of links. If it succeeds, replace + * both objects with the intersection everywhere they + * are referenced. + * If the intersection is empty, insert an automatic + * conversion filter. + * If several formats are negotiated at once (format, + * rate, layout), only merge if all three can be, since + * the conversion filter can convert all three at once. + * This process goes on as long as progress is made. + * See: + * http://www.normalesup.org/~george/articles/format_negotiation_in_libavfilter/#14 + * http://www.normalesup.org/~george/articles/format_negotiation_in_libavfilter/#29 + * + * 3. reduce_formats(): try to reduce format conversion within filters. + * For each link where there is only one supported + * formats on output, for each output of the connected + * filter, if the media type is the same and said + * format is supported, keep only this one. + * This process goes on as long as progress is made. + * Rationale: conversion filters will set a large list + * of supported formats on outputs but users will + * expect the output to be as close as possible as the + * input (examples: scale without changing the pixel + * format, resample without changint the layout). + * FIXME: this can probably be done by merging the + * input and output lists instead of re-implementing + * the logic. + * + * 4. swap_sample_fmts(): + * swap_samplerates(): + * swap_channel_layouts(): For each filter with an input with only one + * supported format, when outputs have several + * supported formats, put the best one with + * reference to the input at the beginning of the + * list, to prepare it for being picked up by + * pick_formats(). + * The best format is the one that is most + * similar to the input while not losing too much + * information. + * This process need to run only once. + * FIXME: reduce_formats() operates on all inputs + * with a single format, swap_*() operates on the + * first one only: check if the difference makes + * sense. + * TODO: the swapping done for one filter can + * override the swapping done for another filter + * connected to the same list of formats, maybe + * it would be better to compute a total score + * for all connected filters and use the score to + * pick the format instead of just swapping. + * TODO: make the similarity logic available as + * public functions in libavutil. + * + * 5. pick_formats(): Choose one format from the lists of supported formats, + * use it for the link and reduce the list to a single + * element to force other filters connected to the same + * list to use it. + * First process all links where there is a single format + * and the output links of all filters with an input, + * trying to preserve similarity between input and + * outputs. + * Repeat as long as process is made. + * Then do a final run for the remaining filters. + * FIXME: the similarity logic (the ref argument to + * pick_format()) added in FFmpeg duplicates and + * overrides the swapping logic added in libav. Better + * merge them into a score system. + */ typedef struct AVFilterNegotiation { unsigned nb; const AVFilterFormatsMerger *mergers;
Signed-off-by: Nicolas George <george@nsup.org> --- libavfilter/formats.h | 85 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+)