[FFmpeg-devel,3/5] lavfi: add a new filtergraph parsing API

Callers currently have two ways of adding filters to a graph - they can
either
- create, initialize, and link them manually
- use one of the avfilter_graph_parse*() functions, which take a
  (typically end-user-written) string, split it into individual filter
  definitions+options, then create filters, apply options, initialize
  filters, and finally link them - all based on information from this
  string.

A major problem with the second approach is that it performs many
actions as a single atomic unit, leaving the caller no space to
intervene in between. Such intervention would be useful e.g. to
- modify filter options;
- supply hardware device contexts;
both of which typically must be done before the filter is initialized.

Callers who need such intervention are then forced to invent their own
filtergraph parsing, which is clearly suboptimal.

This commit aims to address this problem by adding a new modular
filtergraph parsing API. It adds a new  avfilter_graph_segment_parse()
function to parse a string filtergraph description into an intermediate
tree-like representation (AVFilterGraphSegment and its children).

This intermediate form may then be applied step by step using further
new avfilter_graph_segment*() functions, with user intervention possible
between each step.
---
 doc/APIchanges            |  15 +
 libavfilter/avfilter.h    | 313 +++++++++++++++++
 libavfilter/graphparser.c | 689 +++++++++++++++++++++++++++++++++++++-
 3 files changed, 1010 insertions(+), 7 deletions(-)

Anton Khirnov (12023-01-20):
> Callers currently have two ways of adding filters to a graph - they can
> either
> - create, initialize, and link them manually
> - use one of the avfilter_graph_parse*() functions, which take a
>   (typically end-user-written) string, split it into individual filter
>   definitions+options, then create filters, apply options, initialize
>   filters, and finally link them - all based on information from this
>   string.
> 
> A major problem with the second approach is that it performs many
> actions as a single atomic unit, leaving the caller no space to
> intervene in between. Such intervention would be useful e.g. to
> - modify filter options;
> - supply hardware device contexts;
> both of which typically must be done before the filter is initialized.
> 
> Callers who need such intervention are then forced to invent their own
> filtergraph parsing, which is clearly suboptimal.
> 
> This commit aims to address this problem by adding a new modular
> filtergraph parsing API. It adds a new  avfilter_graph_segment_parse()
> function to parse a string filtergraph description into an intermediate
> tree-like representation (AVFilterGraphSegment and its children).
> 
> This intermediate form may then be applied step by step using further
> new avfilter_graph_segment*() functions, with user intervention possible
> between each step.
> ---
>  doc/APIchanges            |  15 +
>  libavfilter/avfilter.h    | 313 +++++++++++++++++
>  libavfilter/graphparser.c | 689 +++++++++++++++++++++++++++++++++++++-
>  3 files changed, 1010 insertions(+), 7 deletions(-)

The API seems ok on the whole, which was the biggest concern for a
change like that done in isolation.

> 
> diff --git a/doc/APIchanges b/doc/APIchanges
> index a11acadecd..f8e936ba9a 100644
> --- a/doc/APIchanges
> +++ b/doc/APIchanges
> @@ -14,6 +14,21 @@ libavutil:     2021-04-27
>  
>  API changes, most recent first:
>  
> +2023-xx-xx - xxxxxxxxxx - lavfi 8.x.100 - avfilter.h
> +  Add filtergraph segment parsing API.
> +  New structs:
> +    - AVFilterGraphSegment
> +    - AVFilterChain
> +    - AVFilterParams
> +    - AVFilterPadParams
> +  New functions:
> +    - avfilter_graph_segment_parse()
> +    - avfilter_graph_segment_create_filters()
> +    - avfilter_graph_segment_apply_opts()
> +    - avfilter_graph_segment_init()
> +    - avfilter_graph_segment_link()
> +    - avfilter_graph_segment_apply()
> +
>  2023-01-13 - xxxxxxxxxx - lavu 57.44.100 - ambient_viewing_environment.h frame.h
>    Adds a new structure for holding H.274 Ambient Viewing Environment metadata,
>    AVAmbientViewingEnvironment.
> diff --git a/libavfilter/avfilter.h b/libavfilter/avfilter.h
> index c2ec7a4b5f..037a8bc490 100644
> --- a/libavfilter/avfilter.h
> +++ b/libavfilter/avfilter.h
> @@ -1118,6 +1118,319 @@ int avfilter_graph_parse2(AVFilterGraph *graph, const char *filters,
>                            AVFilterInOut **inputs,
>                            AVFilterInOut **outputs);
>  
> +/**
> + * Parameters of a filter's input or output pad.
> + *
> + * Created as a child of AVFilterParams by avfilter_graph_segment_parse().
> + * Freed in avfilter_graph_segment_free().
> + */
> +typedef struct AVFilterPadParams {
> +    /**
> +     * An av_malloc()'ed string containing the pad label.
> +     *
> +     * May be av_free()'d and set to NULL by the caller, in which case this pad
> +     * will be treated as unlabeled for linking.
> +     * May also be replaced by another av_malloc()'ed string.
> +     */
> +    char *label;
> +} AVFilterPadParams;
> +
> +/**
> + * Parameters describing a filter to be created in a filtergraph.
> + *
> + * Created as a child of AVFilterGraphSegment by avfilter_graph_segment_parse().
> + * Freed in avfilter_graph_segment_free().
> + */
> +typedef struct AVFilterParams {
> +    /**
> +     * The filter context.
> +     *
> +     * Created by avfilter_graph_segment_create_filters() based on
> +     * AVFilterParams.filter_name and instance_name.
> +     *
> +     * Callers may also create the filter context manually, then they should
> +     * av_free() filter_name and set it to NULL. Such AVFilterParams instances
> +     * are then skipped by avfilter_graph_segment_create_filters().
> +     */
> +    AVFilterContext     *filter;
> +
> +    /**
> +     * Name of the AVFilter to be used.
> +     *
> +     * An av_malloc()'ed string, set by avfilter_graph_segment_parse(). Will be
> +     * passed to avfilter_get_by_name() by
> +     * avfilter_graph_segment_create_filters().
> +     *
> +     * Callers may av_free() this string and replace it with another one or
> +     * NULL. If the caller creates the filter instance manually, this string
> +     * MUST be set to NULL.
> +     *
> +     * When both AVFilterParams.filter an AVFilterParams.filter_name are NULL,
> +     * this AVFilterParams instance is skipped by avfilter_graph_segment_*()
> +     * functions.
> +     */
> +    char                *filter_name;
> +    /**
> +     * Name to be used for this filter instance.
> +     *
> +     * An av_malloc()'ed string, may be set by avfilter_graph_segment_parse() or
> +     * left NULL. The caller may av_free() this string and replace with another
> +     * one or NULL.
> +     *
> +     * Will be used by avfilter_graph_segment_create_filters() - passed as the
> +     * third argument to avfilter_graph_alloc_filter(), then freed and set to
> +     * NULL.
> +     */
> +    char                *instance_name;
> +
> +    /**
> +     * Options to be apllied to the filter.
> +     *
> +     * Filled by avfilter_graph_segment_parse(). Afterwards may be freely
> +     * modified by the caller.
> +     *
> +     * Will be applied to the filter by avfilter_graph_segment_apply_opts()
> +     * with an equivalent of av_opt_set_dict2(filter, &opts, * AV_OPT_SEARCH_CHILDREN),
> +     * i.e. any unapplied options will be left in this dictionary.
> +     */
> +    AVDictionary        *opts;
> +
> +    AVFilterPadParams  **inputs;
> +    unsigned          nb_inputs;
> +
> +    AVFilterPadParams  **outputs;
> +    unsigned          nb_outputs;
> +} AVFilterParams;
> +
> +/**
> + * A filterchain is a list of filter specifications.
> + *
> + * Created as a child of AVFilterGraphSegment by avfilter_graph_segment_parse().
> + * Freed in avfilter_graph_segment_free().
> + */
> +typedef struct AVFilterChain {
> +    AVFilterParams  **filters;
> +    size_t         nb_filters;
> +} AVFilterChain;
> +
> +/**
> + * A parsed representation of a filtergraph segment.
> + *
> + * A filtergraph segment is conceptually a list of filterchains, with some
> + * supplementary information (e.g. format conversion flags).
> + *
> + * Created by avfilter_graph_segment_parse(). Must be freed with
> + * avfilter_graph_segment_free().
> + */
> +typedef struct AVFilterGraphSegment {
> +    /**
> +     * The filtergraph this segment is associated with.
> +     * Set by avfilter_graph_segment_parse().
> +     */
> +    AVFilterGraph *graph;
> +
> +    /**
> +     * A list of filter chain contained in this segment..
> +     * Set in avfilter_graph_segment_parse().
> +     */
> +    AVFilterChain **chains;
> +    size_t       nb_chains;
> +
> +    /**
> +     * A string containing a colon-separated list of key=value options applied
> +     * to all scale filters in this segment.
> +     *
> +     * May be set by avfilter_graph_segment_parse().
> +     * The caller may free this string with av_free() and replace it with a
> +     * different av_malloc()'ed string.
> +     */
> +    char *scale_sws_opts;
> +} AVFilterGraphSegment;
> +
> +/**
> + * Parse a textual filtergraph description into an intermediate form.
> + *
> + * This intermediate representation is intended to be modified by the caller as
> + * described in the documentation of AVFilterGraphSegment and its children, and
> + * then applied to the graph either manually or with other
> + * avfilter_graph_segment_*() functions. See the documentation for
> + * avfilter_graph_segment_apply() for the canonical way to apply
> + * AVFilterGraphSegment.
> + *
> + * @param graph Filter graph the parsed segment is associated with. Will only be
> + *              used for logging and similar auxiliary purposes. The graph will
> + *              not be actually modified by this function - the parsing results
> + *              are instead stored in seg for further processing.
> + * @param graph_str a string describing the filtergraph segment
> + * @param flags reserved for future use, caller must set to 0 for now
> + * @param seg A pointer to the newly-created AVFilterGraphSegment is written
> + *            here on success. The graph segment is owned by the caller and must
> + *            be freed with avfilter_graph_segment_free() before graph itself is
> + *            freed.
> + *
> + * @return a non-negative number on success, a negative error code on failure
> + */
> +int avfilter_graph_segment_parse(AVFilterGraph *graph, const char *graph_str,
> +                                 int flags, AVFilterGraphSegment **seg);
> +
> +/**
> + * Create filters specified in a graph segment.
> + *
> + * Walk through the creation-pending AVFilterParams in the segment and create
> + * new filter instances for them.
> + * Creation-pending params are those where AVFilterParams.filter_name is
> + * non-NULL (and hence AVFilterParams.filter is NULL). All other AVFilterParams
> + * instances are ignored.
> + *
> + * For any filter created by this function, the corresponding
> + * AVFilterParams.filter is set to the newly-created filter context,
> + * AVFilterParams.filter_name and AVFilterParams.instance_name are freed and set
> + * to NULL.
> + *
> + * @param seg the filtergraph segment to process
> + * @param flags reserved for future use, caller must set to 0 for now
> + *
> + * @return
> + * - a non-negative number if all creation-pending filters were successfully
> + *   created
> + * - AVERROR_FILTER_NOT_FOUND if some filter's name did not correspond to a
> + *   known filter
> + * - another negative error code on other failures
> + *
> + * @note Calling this function multiple times is safe, as it is idempotent.
> + */
> +int avfilter_graph_segment_create_filters(AVFilterGraphSegment *seg, int flags);
> +
> +/**
> + * Apply parsed options to filter instances in a graph segment.
> + *
> + * Walk through all filter instances in the graph segment that have option
> + * dictionaries associated with them and apply those options with
> + * av_opt_set_dict2(..., AV_OPT_SEARCH_CHILDREN). AVFilterParams.opts is
> + * replaced by the dictionary output by av_opt_set_dict2(), which should be
> + * empty (NULL) if all options were successfully applied.
> + *
> + * If any options could not be found, this function will continue processing all
> + * other filters and finally return AVERROR_OPTION_NOT_FOUND (unless another
> + * error happens). The calling program may then deal with unapplied options as
> + * it wishes.
> + *
> + * Any creation-pending filters (see avfilter_graph_segment_create_filters())
> + * present in the segment will cause this function to fail. AVFilterParams with
> + * no associated filter context are simply skipped.
> + *
> + * @param seg the filtergraph segment to process
> + * @param flags reserved for future use, caller must set to 0 for now
> + *
> + * @return
> + * - a non-negative number if all options were successfully applied
> + * - AVERROR_OPTION_NOT_FOUND if some options were not found in a filter
> + * - another negative error code on other failures
> + *
> + * @note Calling this function multiple times is safe, as it is idempotent.
> + */
> +int avfilter_graph_segment_apply_opts(AVFilterGraphSegment *seg, int flags);
> +
> +/**
> + * Initialize all filter instances in a graph segment.
> + *
> + * Walk through all filter instances in the graph segment and call
> + * avfilter_init_dict(..., NULL) on those that have not been initialized yet.
> + *
> + * Any creation-pending filters (see avfilter_graph_segment_create_filters())
> + * present in the segment will cause this function to fail. AVFilterParams with
> + * no associated filter context or whose filter context is already initialized,
> + * are simply skipped.
> + *
> + * @param seg the filtergraph segment to process
> + * @param flags reserved for future use, caller must set to 0 for now
> + *
> + * @return
> + * - a non-negative number if all filter instances were successfully initialized
> + * - a negative error code on other failures
> + *
> + * @note Calling this function multiple times is safe, as it is idempotent.
> + */
> +int avfilter_graph_segment_init(AVFilterGraphSegment *seg, int flags);
> +
> +/**
> + * Link filters in a graph segment.
> + *
> + * Walk through all filter instances in the graph segment and try to link all
> + * unlinked input and output pads. Any creation-pending filters (see
> + * avfilter_graph_segment_create_filters()) present in the segment will cause
> + * this function to fail. Disabled filters and already linked pads are skipped.
> + *
> + * Every filter output pad that has a corresponding AVFilterPadParams with a
> + * non-NULL label is
> + * - linked to the input with the matching label, if one exists;
> + * - exported in the outputs linked list otherwise, with the label preserved.
> + * Unlabeled outputs are
> + * - linked to the first unlinked unlabeled input in the next non-disabled
> + *   filter in the chain, if one exists
> + * - exported in the ouputs linked list otherwise, with NULL label
> + *
> + * Similarly, unlinked input pads are exported in the inputs linked list.
> + *
> + * @param seg the filtergraph segment to process
> + * @param flags reserved for future use, caller must set to 0 for now
> + * @param[out] inputs  a linked list of all free (unlinked) inputs of the
> + *                     filters in this graph segment will be returned here. It
> + *                     is to be freed by the caller using avfilter_inout_free().
> + * @param[out] outputs a linked list of all free (unlinked) outputs of the
> + *                     filters in this graph segment will be returned here. It
> + *                     is to be freed by the caller using avfilter_inout_free().
> + *
> + * @return
> + * - a non-negative number on success
> + * - a negative error code on failure
> + *
> + * @note Calling this function multiple times is safe, as it is idempotent.
> + */
> +int avfilter_graph_segment_link(AVFilterGraphSegment *seg, int flags,
> +                                AVFilterInOut **inputs,
> +                                AVFilterInOut **outputs);
> +
> +/**
> + * Apply all filter/link descriptions from a graph segment to the associated filtergraph.
> + *
> + * This functions is currently equivalent to calling the following in sequence:
> + * - avfilter_graph_segment_create_filters();
> + * - avfilter_graph_segment_apply_opts();
> + * - avfilter_graph_segment_init();
> + * - avfilter_graph_segment_link();
> + * failing if any of them fails. This list may be extended in the future.
> + *
> + * Since the above functions are idempotent, the caller may call some of the
> + * manually, then do some custom processing on the filtergraph, then call this
> + * function to do the rest.
> + *
> + * @param seg the filtergraph segment to process
> + * @param flags reserved for future use, caller must set to 0 for now
> + * @param[out] inputs passed to avfilter_graph_segment_link()
> + * @param[out] outputs passed to avfilter_graph_segment_link()
> + * @return
> + * - a non-negative number on success
> + * - a negative error code on failure
> + *
> + * @note Calling this function multiple times is safe, as it is idempotent.
> + */
> +int avfilter_graph_segment_apply(AVFilterGraphSegment *seg, int flags,
> +                                 AVFilterInOut **inputs,
> +                                 AVFilterInOut **outputs);
> +
> +/**
> + * Free the provided AVFilterGraphSegment and everything associated with it.
> + *
> + * @param seg double pointer to the AVFilterGraphSegment to be freed. NULL will
> + * be written to this pointer on exit from this function.
> + *
> + * @note
> + * The filter contexts (AVFilterParams.filter) are owned by AVFilterGraph rather
> + * than AVFilterGraphSegment, so they are not freed.
> + */
> +void avfilter_graph_segment_free(AVFilterGraphSegment **seg);
> +
>  /**
>   * Send a command to one or more filter instances.
>   *
> diff --git a/libavfilter/graphparser.c b/libavfilter/graphparser.c
> index 0759c39014..ac6171e0c4 100644
> --- a/libavfilter/graphparser.c
> +++ b/libavfilter/graphparser.c
> @@ -24,10 +24,12 @@
>  #include <stdio.h>
>  
>  #include "libavutil/avstring.h"
> +#include "libavutil/dict.h"
>  #include "libavutil/mem.h"
>  #include "libavutil/opt.h"
>  
>  #include "avfilter.h"
> +#include "internal.h"
>  
>  #define WHITESPACES " \n\t\r"
>  
> @@ -386,7 +388,7 @@ static int parse_outputs(const char **buf, AVFilterInOut **curr_inputs,
>      return pad;
>  }
>  
> -static int parse_sws_flags(const char **buf, AVFilterGraph *graph)
> +static int parse_sws_flags(const char **buf, char **dst, void *log_ctx)
>  {
>      char *p = strchr(*buf, ';');
>  
> @@ -394,16 +396,16 @@ static int parse_sws_flags(const char **buf, AVFilterGraph *graph)
>          return 0;
>  
>      if (!p) {
> -        av_log(graph, AV_LOG_ERROR, "sws_flags not terminated with ';'.\n");
> +        av_log(log_ctx, AV_LOG_ERROR, "sws_flags not terminated with ';'.\n");
>          return AVERROR(EINVAL);
>      }
>  
>      *buf += 4;  // keep the 'flags=' part
>  
> -    av_freep(&graph->scale_sws_opts);
> -    if (!(graph->scale_sws_opts = av_mallocz(p - *buf + 1)))
> +    av_freep(dst);
> +    if (!(*dst = av_mallocz(p - *buf + 1)))
>          return AVERROR(ENOMEM);
> -    av_strlcpy(graph->scale_sws_opts, *buf, p - *buf + 1);
> +    av_strlcpy(*dst, *buf, p - *buf + 1);
>  
>      *buf = p + 1;
>      return 0;
> @@ -420,7 +422,7 @@ int avfilter_graph_parse2(AVFilterGraph *graph, const char *filters,
>  
>      filters += strspn(filters, WHITESPACES);
>  
> -    if ((ret = parse_sws_flags(&filters, graph)) < 0)
> +    if ((ret = parse_sws_flags(&filters, &graph->scale_sws_opts, graph)) < 0)
>          goto end;
>  
>      do {
> @@ -551,7 +553,7 @@ int avfilter_graph_parse_ptr(AVFilterGraph *graph, const char *filters,
>      AVFilterInOut *open_inputs  = open_inputs_ptr  ? *open_inputs_ptr  : NULL;
>      AVFilterInOut *open_outputs = open_outputs_ptr ? *open_outputs_ptr : NULL;
>  
> -    if ((ret = parse_sws_flags(&filters, graph)) < 0)
> +    if ((ret = parse_sws_flags(&filters, &graph->scale_sws_opts, graph)) < 0)
>          goto end;
>  
>      do {
> @@ -623,3 +625,676 @@ end:
>      }
>      return ret;
>  }
> +
> +static void pad_params_free(AVFilterPadParams **pfpp)
> +{
> +    AVFilterPadParams *fpp = *pfpp;
> +
> +    if (!fpp)
> +        return;
> +
> +    av_freep(&fpp->label);
> +
> +    av_freep(pfpp);
> +}
> +
> +static void filter_params_free(AVFilterParams **pp)
> +{
> +    AVFilterParams *p = *pp;
> +
> +    if (!p)
> +        return;
> +
> +    for (unsigned i = 0; i < p->nb_inputs; i++)
> +        pad_params_free(&p->inputs[i]);
> +    av_freep(&p->inputs);
> +
> +    for (unsigned i = 0; i < p->nb_outputs; i++)
> +        pad_params_free(&p->outputs[i]);
> +    av_freep(&p->outputs);
> +
> +    av_dict_free(&p->opts);
> +
> +    av_freep(&p->filter_name);
> +    av_freep(&p->instance_name);
> +
> +    av_freep(pp);
> +}
> +
> +static void chain_free(AVFilterChain **pch)
> +{
> +    AVFilterChain *ch = *pch;
> +
> +    if (!ch)
> +        return;
> +
> +    for (size_t i = 0; i < ch->nb_filters; i++)
> +        filter_params_free(&ch->filters[i]);
> +    av_freep(&ch->filters);
> +
> +    av_freep(pch);
> +}
> +
> +void avfilter_graph_segment_free(AVFilterGraphSegment **pseg)
> +{
> +    AVFilterGraphSegment *seg = *pseg;
> +
> +    if (!seg)
> +        return;
> +
> +    for (size_t i = 0; i < seg->nb_chains; i++)
> +        chain_free(&seg->chains[i]);
> +    av_freep(&seg->chains);
> +
> +    av_freep(&seg->scale_sws_opts);
> +
> +    av_freep(pseg);
> +}
> +
> +static int linklabels_parse(void *logctx, const char **linklabels,
> +                            AVFilterPadParams ***res, unsigned *nb_res)
> +{
> +    AVFilterPadParams **pp = NULL;
> +    unsigned nb = 0;
> +    int ret;
> +
> +    while (**linklabels == '[') {
> +        char *label;
> +        AVFilterPadParams *par, **tmp;
> +
> +        label = parse_link_name(linklabels, logctx);
> +        if (!label) {
> +            ret = AVERROR(EINVAL);
> +            goto fail;
> +        }
> +
> +        par = av_mallocz(sizeof(*par));
> +        if (!par) {
> +            av_freep(&label);
> +            ret = AVERROR(ENOMEM);
> +            goto fail;
> +        }
> +
> +        par->label = label;
> +

> +        tmp = av_realloc_array(pp, nb + 1, sizeof(*pp));
> +        if (!tmp) {
> +            pad_params_free(&par);
> +            ret = AVERROR(ENOMEM);
> +            goto fail;
> +        }

Use FF_DYNARRAY_ADD() or av_dynarray_add(). There are users with
stupidly large filter graphs, it should not be quadratic. Same at
several places below.

> +
> +        pp = tmp;
> +        pp[nb++] = par;
> +
> +        *linklabels += strspn(*linklabels, WHITESPACES);
> +    }
> +
> +    *res    = pp;
> +    *nb_res = nb;
> +
> +    return 0;
> +fail:
> +    for (unsigned i = 0; i < nb; i++)
> +        pad_params_free(&pp[i]);
> +    av_freep(&pp);
> +    return ret;
> +}
> +
> +static int filter_parse(void *logctx, const char **filter,
> +                        AVFilterParams **pp)
> +{
> +    AVFilterParams *p;
> +    char *inst_name;
> +    int ret;
> +
> +    p = av_mallocz(sizeof(*p));
> +    if (!p)
> +        return AVERROR(ENOMEM);
> +
> +    ret = linklabels_parse(logctx, filter, &p->inputs, &p->nb_inputs);
> +    if (ret < 0)
> +        goto fail;
> +
> +    p->filter_name = av_get_token(filter, "=,;[");
> +    if (!p->filter_name) {
> +        ret = AVERROR(ENOMEM);
> +        goto fail;
> +    }
> +
> +    inst_name = strchr(p->filter_name, '@');
> +    if (inst_name) {
> +        *inst_name++ = 0;
> +        p->instance_name = av_strdup(inst_name);
> +        if (!p->instance_name) {
> +            ret = AVERROR(ENOMEM);
> +            goto fail;
> +        }
> +    }
> +
> +    if (**filter == '=') {
> +        const AVFilter *f = avfilter_get_by_name(p->filter_name);
> +        char *opts;
> +
> +        (*filter)++;
> +
> +        opts = av_get_token(filter, "[],;");
> +        if (!opts) {
> +            ret = AVERROR(ENOMEM);
> +            goto fail;
> +        }
> +
> +        ret = ff_filter_opt_parse(logctx, f ? f->priv_class : NULL,
> +                                  &p->opts, opts);
> +        av_freep(&opts);
> +        if (ret < 0)
> +            goto fail;
> +    }
> +
> +    ret = linklabels_parse(logctx, filter, &p->outputs, &p->nb_outputs);
> +    if (ret < 0)
> +        goto fail;
> +
> +    *filter += strspn(*filter, WHITESPACES);
> +
> +    *pp = p;
> +    return 0;
> +fail:
> +    av_log(logctx, AV_LOG_ERROR,
> +           "Error parsing a filter description around: %s\n", *filter);
> +    filter_params_free(&p);
> +    return ret;
> +}
> +
> +static int chain_parse(void *logctx, const char **pchain,
> +                       AVFilterChain **pch)
> +{
> +    const char *chain = *pchain;
> +    AVFilterChain *ch;
> +    int ret;
> +
> +    *pch = NULL;
> +
> +    ch = av_mallocz(sizeof(*ch));
> +    if (!ch)
> +        return AVERROR(ENOMEM);
> +
> +    while (*chain) {
> +        AVFilterParams *p, **tmp;
> +        char chr;
> +
> +        ret = filter_parse(logctx, &chain, &p);
> +        if (ret < 0)
> +            goto fail;
> +
> +        tmp = av_realloc_array(ch->filters, ch->nb_filters + 1, sizeof(*ch->filters));
> +        if (!tmp) {
> +            filter_params_free(&p);
> +            ret = AVERROR(ENOMEM);
> +            goto fail;
> +        }
> +
> +        ch->filters = tmp;
> +        ch->filters[ch->nb_filters++] = p;
> +
> +        // a filter ends with one of: , ; end-of-string
> +        chr = *chain;
> +        if (chr && chr != ',' && chr != ';') {
> +            av_log(logctx, AV_LOG_ERROR,
> +                   "Trailing garbage after a filter: %s\n", chain);
> +            ret = AVERROR(EINVAL);
> +            goto fail;
> +        }
> +
> +        if (chr) {
> +            chain++;
> +            chain += strspn(chain, WHITESPACES);
> +
> +            if (chr == ';')
> +                break;
> +        }
> +    }
> +
> +    *pchain = chain;
> +    *pch    = ch;
> +
> +    return 0;
> +fail:
> +    av_log(logctx, AV_LOG_ERROR,
> +           "Error parsing filterchain '%s' around: %s\n", *pchain, chain);
> +    chain_free(&ch);
> +    return ret;
> +}
> +
> +int avfilter_graph_segment_parse(AVFilterGraph *graph, const char *graph_str,
> +                                 int flags, AVFilterGraphSegment **pseg)
> +{
> +    AVFilterGraphSegment *seg;
> +    int ret;
> +
> +    *pseg = NULL;
> +
> +    if (flags)
> +        return AVERROR(ENOSYS);
> +
> +    seg = av_mallocz(sizeof(*seg));
> +    if (!seg)
> +        return AVERROR(ENOMEM);
> +
> +    seg->graph = graph;
> +
> +    graph_str += strspn(graph_str, WHITESPACES);
> +
> +    ret = parse_sws_flags(&graph_str, &seg->scale_sws_opts, &graph);
> +    if (ret < 0)
> +        goto fail;
> +
> +    graph_str += strspn(graph_str, WHITESPACES);
> +
> +    while (*graph_str) {
> +        AVFilterChain *ch, **tmp;
> +
> +        ret = chain_parse(graph, &graph_str, &ch);
> +        if (ret < 0)
> +            goto fail;
> +
> +        tmp = av_realloc_array(seg->chains, seg->nb_chains + 1, sizeof(*seg->chains));
> +        if (!tmp) {
> +            chain_free(&ch);
> +            ret = AVERROR(ENOMEM);
> +            goto fail;
> +        }
> +
> +        seg->chains = tmp;
> +        seg->chains[seg->nb_chains++] = ch;
> +
> +        graph_str += strspn(graph_str, WHITESPACES);
> +    }
> +
> +    if (!seg->nb_chains) {
> +        av_log(graph, AV_LOG_ERROR, "No filters specified in the graph description\n");
> +        ret = AVERROR(EINVAL);
> +        goto fail;
> +    }
> +
> +    *pseg = seg;
> +
> +    return 0;
> +fail:
> +    avfilter_graph_segment_free(&seg);
> +    return ret;
> +}
> +
> +int avfilter_graph_segment_create_filters(AVFilterGraphSegment *seg, int flags)
> +{
> +    size_t idx = 0;
> +
> +    if (flags)
> +        return AVERROR(ENOSYS);
> +
> +    if (seg->scale_sws_opts) {
> +        av_freep(&seg->graph->scale_sws_opts);
> +        seg->graph->scale_sws_opts = av_strdup(seg->scale_sws_opts);
> +        if (!seg->graph->scale_sws_opts)
> +            return AVERROR(ENOMEM);
> +    }
> +
> +    for (size_t i = 0; i < seg->nb_chains; i++) {
> +        AVFilterChain *ch = seg->chains[i];
> +
> +        for (size_t j = 0; j < ch->nb_filters; j++) {
> +            AVFilterParams *p = ch->filters[j];
> +            const AVFilter *f = avfilter_get_by_name(p->filter_name);
> +            char inst_name[30], *name = p->instance_name ? p->instance_name :
> +                                                           inst_name;
> +
> +            // skip already processed filters
> +            if (p->filter || !p->filter_name)
> +                continue;
> +
> +            if (!f) {
> +                av_log(seg->graph, AV_LOG_ERROR,
> +                       "No such filter: '%s'\n", p->filter_name);
> +                return AVERROR_FILTER_NOT_FOUND;
> +            }
> +
> +            if (!p->instance_name)
> +                snprintf(inst_name, sizeof(inst_name), "Parsed_%s_%zu", f->name, idx);
> +
> +            p->filter = avfilter_graph_alloc_filter(seg->graph, f, name);
> +            if (!p->filter)
> +                return AVERROR(ENOMEM);
> +
> +            if (!strcmp(f->name, "scale") && seg->graph->scale_sws_opts) {
> +                int ret = av_set_options_string(p->filter, seg->graph->scale_sws_opts,
> +                                                "=", ":");
> +                if (ret < 0) {
> +                    avfilter_free(p->filter);
> +                    p->filter = NULL;
> +                    return ret;
> +                }
> +            }
> +
> +            av_freep(&p->filter_name);
> +            av_freep(&p->instance_name);
> +
> +            idx++;
> +        }
> +    }
> +
> +    return 0;
> +}
> +
> +static int fail_creation_pending(AVFilterGraphSegment *seg, const char *fn,
> +                                 const char *func)
> +{
> +    av_log(seg->graph, AV_LOG_ERROR,
> +           "A creation-pending filter '%s' present in the segment. All filters "
> +           "must be created or disabled before calling %s().\n", fn, func);
> +    return AVERROR(EINVAL);
> +}
> +
> +int avfilter_graph_segment_apply_opts(AVFilterGraphSegment *seg, int flags)
> +{
> +    int ret, leftover_opts = 0;
> +
> +    if (flags)
> +        return AVERROR(ENOSYS);
> +
> +    for (size_t i = 0; i < seg->nb_chains; i++) {
> +        AVFilterChain *ch = seg->chains[i];
> +
> +        for (size_t j = 0; j < ch->nb_filters; j++) {
> +            AVFilterParams *p = ch->filters[j];
> +
> +            if (p->filter_name)
> +                return fail_creation_pending(seg, p->filter_name, __func__);
> +            if (!p->filter || !p->opts)
> +                continue;
> +
> +            ret = av_opt_set_dict2(p->filter, &p->opts, AV_OPT_SEARCH_CHILDREN);
> +            if (ret < 0)
> +                return ret;
> +
> +            if (av_dict_count(p->opts))
> +                leftover_opts = 1;
> +        }
> +    }
> +
> +    return leftover_opts ? AVERROR_OPTION_NOT_FOUND : 0;
> +}
> +
> +int avfilter_graph_segment_init(AVFilterGraphSegment *seg, int flags)
> +{
> +    if (flags)
> +        return AVERROR(ENOSYS);
> +
> +    for (size_t i = 0; i < seg->nb_chains; i++) {
> +        AVFilterChain *ch = seg->chains[i];
> +
> +        for (size_t j = 0; j < ch->nb_filters; j++) {
> +            AVFilterParams *p = ch->filters[j];
> +            int ret;
> +
> +            if (p->filter_name)
> +                return fail_creation_pending(seg, p->filter_name, __func__);
> +            if (!p->filter || p->filter->internal->initialized)
> +                continue;
> +
> +            ret = avfilter_init_dict(p->filter, NULL);
> +            if (ret < 0)
> +                return ret;
> +        }
> +    }
> +
> +    return 0;
> +}
> +

> +static unsigned
> +find_linklabel(AVFilterGraphSegment *seg, const char *label,
> +               int output, size_t idx_chain, size_t idx_filter,
> +               AVFilterParams **pp)
> +{
> +    for (; idx_chain < seg->nb_chains; idx_chain++) {
> +        AVFilterChain *ch = seg->chains[idx_chain];
> +
> +        for (; idx_filter < ch->nb_filters; idx_filter++) {
> +            AVFilterParams *p = ch->filters[idx_filter];
> +            AVFilterPadParams **io = output ? p->outputs    : p->inputs;
> +            unsigned         nb_io = output ? p->nb_outputs : p->nb_inputs;
> +            AVFilterLink **l;
> +            unsigned nb_l;
> +
> +            if (!p->filter)
> +                continue;
> +
> +            l    = output ? p->filter->outputs    : p->filter->inputs;
> +            nb_l = output ? p->filter->nb_outputs : p->filter->nb_inputs;
> +
> +            for (unsigned i = 0; i < FFMIN(nb_io, nb_l); i++)
> +                if (!l[i] && io[i]->label && !strcmp(io[i]->label, label)) {
> +                    *pp = p;
> +                    return i;
> +                }
> +        }
> +
> +        idx_filter = 0;
> +    }
> +
> +    *pp = NULL;
> +    return 0;
> +}

If I read this code correctly, it will only find link labels that are
defined later in the chain or in a later chain. But referring to earlier
link labels is supported. Even cycles are meant to be supported at
parsing level (format negotiation will likely fail, though, but being
able to make cycles was a goal from the start).

> +
> +static int inout_add(AVFilterInOut **inouts, AVFilterContext *f, unsigned pad_idx,
> +                     const char *label)
> +{
> +    AVFilterInOut *io = av_mallocz(sizeof(*io));
> +
> +    if (!io)
> +        return AVERROR(ENOMEM);
> +
> +    io->filter_ctx = f;
> +    io->pad_idx    = pad_idx;
> +
> +    if (label) {
> +        io->name = av_strdup(label);
> +        if (!io->name) {
> +            avfilter_inout_free(&io);
> +            return AVERROR(ENOMEM);
> +        }
> +    }
> +
> +    append_inout(inouts, &io);
> +
> +    return 0;
> +}
> +
> +static int link_inputs(AVFilterGraphSegment *seg, size_t idx_chain,
> +                       size_t idx_filter, AVFilterInOut **inputs)
> +{
> +    AVFilterChain  *ch = seg->chains[idx_chain];
> +    AVFilterParams  *p = ch->filters[idx_filter];
> +    AVFilterContext *f = p->filter;
> +
> +    int ret;
> +
> +    if (f->nb_inputs < p->nb_inputs) {
> +        av_log(seg->graph, AV_LOG_ERROR,
> +               "More input link labels specified for filter '%s' than "
> +               "it has inputs: %u > %d\n", f->filter->name,
> +               p->nb_inputs, f->nb_inputs);
> +        return AVERROR(EINVAL);
> +    }
> +
> +    for (unsigned in = 0; in < f->nb_inputs; in++) {
> +        const char *label = (in < p->nb_inputs) ? p->inputs[in]->label : NULL;
> +
> +        // skip already linked inputs
> +        if (f->inputs[in])
> +            continue;
> +
> +        if (label) {
> +            AVFilterParams *po = NULL;
> +            unsigned idx = find_linklabel(seg, label, 1, idx_chain, idx_filter, &po);
> +
> +            if (po) {
> +                ret = avfilter_link(po->filter, idx, f, in);
> +                if (ret < 0)
> +                    return ret;
> +
> +                continue;
> +            }
> +        }
> +
> +        ret = inout_add(inputs, f, in, label);
> +        if (ret < 0)
> +            return ret;
> +    }
> +
> +    return 0;
> +}
> +
> +static int link_outputs(AVFilterGraphSegment *seg, size_t idx_chain,
> +                        size_t idx_filter, AVFilterInOut **outputs)
> +{
> +    AVFilterChain  *ch = seg->chains[idx_chain];
> +    AVFilterParams  *p = ch->filters[idx_filter];
> +    AVFilterContext *f = p->filter;
> +
> +    int ret;
> +
> +    if (f->nb_outputs < p->nb_outputs) {
> +        av_log(seg->graph, AV_LOG_ERROR,
> +               "More output link labels specified for filter '%s' than "
> +               "it has outputs: %u > %d\n", f->filter->name,
> +               p->nb_outputs, f->nb_outputs);
> +        return AVERROR(EINVAL);
> +    }
> +    for (unsigned out = 0; out < f->nb_outputs; out++) {
> +        char *label = (out < p->nb_outputs) ? p->outputs[out]->label : NULL;
> +
> +        // skip already linked outputs
> +        if (f->outputs[out])
> +            continue;
> +
> +        if (label) {
> +            AVFilterParams *po = NULL;
> +            unsigned idx = find_linklabel(seg, label, 0, idx_chain, idx_filter, &po);
> +
> +            if (po) {
> +                ret = avfilter_link(f, out, po->filter, idx);
> +                if (ret < 0)
> +                    return ret;
> +
> +                continue;
> +            }
> +        }
> +
> +        // if this output is unlabeled, try linking it to an unlabeled
> +        // input in the next non-disabled filter in the chain
> +        for (size_t i = idx_filter + 1; i < ch->nb_filters && !label; i++) {
> +            AVFilterParams *p_next = ch->filters[i];
> +
> +            if (!p_next->filter)
> +                continue;
> +
> +            for (unsigned in = 0; in < p_next->filter->nb_inputs; in++) {
> +                if (!p_next->filter->inputs[in] &&
> +                    (in >= p_next->nb_inputs || !p_next->inputs[in]->label)) {
> +                    ret = avfilter_link(f, out, p_next->filter, in);
> +                    if (ret < 0)
> +                        return ret;
> +
> +                    goto cont;
> +                }
> +            }
> +            break;
> +        }
> +
> +        ret = inout_add(outputs, f, out, label);
> +        if (ret < 0)
> +            return ret;
> +
> +cont:;
> +    }
> +
> +    return 0;
> +}
> +
> +int avfilter_graph_segment_link(AVFilterGraphSegment *seg, int flags,
> +                                AVFilterInOut **inputs,
> +                                AVFilterInOut **outputs)
> +{
> +    int ret;
> +
> +    *inputs  = NULL;
> +    *outputs = NULL;
> +
> +    if (flags)
> +        return AVERROR(ENOSYS);
> +
> +    for (size_t idx_chain = 0; idx_chain < seg->nb_chains; idx_chain++) {
> +        AVFilterChain *ch = seg->chains[idx_chain];
> +
> +        for (size_t idx_filter = 0; idx_filter < ch->nb_filters; idx_filter++) {
> +            AVFilterParams  *p = ch->filters[idx_filter];
> +
> +            if (p->filter_name) {
> +                ret = fail_creation_pending(seg, p->filter_name, __func__);
> +                goto fail;
> +            }
> +
> +            if (!p->filter)
> +                continue;
> +
> +            ret = link_inputs(seg, idx_chain, idx_filter, inputs);
> +            if (ret < 0)
> +                goto fail;
> +
> +            ret = link_outputs(seg, idx_chain, idx_filter, outputs);
> +            if (ret < 0)
> +                goto fail;
> +        }
> +    }
> +    return 0;
> +fail:
> +    avfilter_inout_free(inputs);
> +    avfilter_inout_free(outputs);
> +    return ret;
> +}
> +
> +int avfilter_graph_segment_apply(AVFilterGraphSegment *seg, int flags,
> +                                 AVFilterInOut **inputs,
> +                                 AVFilterInOut **outputs)
> +{
> +    int ret;
> +
> +    if (flags)
> +        return AVERROR(ENOSYS);
> +
> +    ret = avfilter_graph_segment_create_filters(seg, 0);
> +    if (ret < 0) {
> +        av_log(seg->graph, AV_LOG_ERROR, "Error creating filters\n");
> +        return ret;
> +    }
> +
> +    ret = avfilter_graph_segment_apply_opts(seg, 0);
> +    if (ret < 0) {
> +        av_log(seg->graph, AV_LOG_ERROR, "Error applying filter options\n");
> +        return ret;
> +    }
> +
> +    ret = avfilter_graph_segment_init(seg, 0);
> +    if (ret < 0) {
> +        av_log(seg->graph, AV_LOG_ERROR, "Error initializing filters\n");
> +        return ret;
> +    }
> +
> +    ret = avfilter_graph_segment_link(seg, 0, inputs, outputs);
> +    if (ret < 0) {
> +        av_log(seg->graph, AV_LOG_ERROR, "Error linking filters\n");
> +        return ret;
> +    }
> +
> +    return 0;
> +}

This is all I have found for now, but I still need to look at the tricky
parts of the code more carefully.

Quoting Nicolas George (2023-01-30 13:59:44)
> > +static unsigned
> > +find_linklabel(AVFilterGraphSegment *seg, const char *label,
> > +               int output, size_t idx_chain, size_t idx_filter,
> > +               AVFilterParams **pp)
> > +{
> > +    for (; idx_chain < seg->nb_chains; idx_chain++) {
> > +        AVFilterChain *ch = seg->chains[idx_chain];
> > +
> > +        for (; idx_filter < ch->nb_filters; idx_filter++) {
> > +            AVFilterParams *p = ch->filters[idx_filter];
> > +            AVFilterPadParams **io = output ? p->outputs    : p->inputs;
> > +            unsigned         nb_io = output ? p->nb_outputs : p->nb_inputs;
> > +            AVFilterLink **l;
> > +            unsigned nb_l;
> > +
> > +            if (!p->filter)
> > +                continue;
> > +
> > +            l    = output ? p->filter->outputs    : p->filter->inputs;
> > +            nb_l = output ? p->filter->nb_outputs : p->filter->nb_inputs;
> > +
> > +            for (unsigned i = 0; i < FFMIN(nb_io, nb_l); i++)
> > +                if (!l[i] && io[i]->label && !strcmp(io[i]->label, label)) {
> > +                    *pp = p;
> > +                    return i;
> > +                }
> > +        }
> > +
> > +        idx_filter = 0;
> > +    }
> > +
> > +    *pp = NULL;
> > +    return 0;
> > +}
> 
> If I read this code correctly, it will only find link labels that are
> defined later in the chain or in a later chain. But referring to earlier
> link labels is supported. Even cycles are meant to be supported at
> parsing level (format negotiation will likely fail, though, but being
> able to make cycles was a goal from the start).

The earlier filters have already been processed at this point, so if an
earlier filter had a link label that would match this one, then it's
already been linked. So there's no point in considering earlier filters.