mbox series

[FFmpeg-devel,RFC,00/25] YUVJ removal + filter negotiation

Message ID 20231109122534.124157-1-ffmpeg@haasn.xyz
Headers show
Series YUVJ removal + filter negotiation | expand

Message

Niklas Haas Nov. 9, 2023, 12:19 p.m. UTC
Following up on the previous attempts at YUVJ removal, this series
implements full filter negotiation. It is WIP because I still need to go
through the remaining AVCodec (which are not covered by FATE) and
properly tag their colorspace and color range support. (See patch 19/25)

The rest should be good, though, so please feel free to review them with
scrutiny.

I decided to combine YUV range negotiation with YUV colorspace
negotiation. Several important points:

1. We now explicitly ignore YUV metadata for non-YUV (RGB/XYZ/PAL)
   formats, as these have no hope of being useful. In particular, we
   force the colorspace to AVCOL_SPC_RGB, mostly for sanity/consistency.

2. We currently continue treating GRAY formats as forced full range,
   which is the swscale status quo. I have a separate series that adds
   full range negotiation to grayscale formats, but it is out of scope
   of this already ominous series.

It was a delicate balancing act to get the new filter negotiation API
integrated into all core filters (in particular vf_scale and
vf_buffersrc) while maintaining forward and backward compatibility and
not breaking FATE, but I managed to find an order that works, by first
adding support to vf_buffersrc to make sure accurate metadata is
available downstream _before_ we insert any conversion filters.

Comments

Anton Khirnov Dec. 6, 2023, 3:10 p.m. UTC | #1
Quoting Niklas Haas (2023-11-09 13:19:37)
> @@ -328,6 +341,30 @@ static const AVOption buffer_options[] = {
>      { "pixel_aspect",  "sample aspect ratio",    OFFSET(pixel_aspect),     AV_OPT_TYPE_RATIONAL, { .dbl = 0 }, 0, DBL_MAX, V },
>      { "time_base",     NULL,                     OFFSET(time_base),        AV_OPT_TYPE_RATIONAL, { .dbl = 0 }, 0, DBL_MAX, V },
>      { "frame_rate",    NULL,                     OFFSET(frame_rate),       AV_OPT_TYPE_RATIONAL, { .dbl = 0 }, 0, DBL_MAX, V },
> +    { "colorspace", "select colorspace", OFFSET(color_space), AV_OPT_TYPE_INT, {.i64=AVCOL_SPC_UNSPECIFIED}, 0, AVCOL_SPC_NB-1, V, "colorspace"},
> +    {   "gbr",         NULL,  0, AV_OPT_TYPE_CONST, {.i64=AVCOL_SPC_RGB},               INT_MIN, INT_MAX, V, "colorspace"},
            ^^^
        is that intentional?

> @@ -432,6 +471,15 @@ static int query_formats(AVFilterContext *ctx)
>          if ((ret = ff_add_format         (&formats, c->pix_fmt)) < 0 ||
>              (ret = ff_set_common_formats (ctx     , formats   )) < 0)
>              return ret;
> +        /* force specific colorspace/range downstream only for ordinary YUV */
> +        if (ff_fmt_is_regular_yuv(c->pix_fmt)) {

What will this do when the colorspace is not set, as it will not be with
existing callers.

> diff --git a/libavfilter/buffersrc.h b/libavfilter/buffersrc.h
> index 3b248b37cd..4357a7bbfb 100644
> --- a/libavfilter/buffersrc.h
> +++ b/libavfilter/buffersrc.h
> @@ -105,6 +105,12 @@ typedef struct AVBufferSrcParameters {
>       */
>      AVBufferRef *hw_frames_ctx;
>  
> +    /**
> +     * Video only, the YUV colorspace and range
> +     */
> +    enum AVColorSpace color_space;
> +    enum AVColorRange color_range;

This has to go at the end of the struct, otherwise you're breaking ABI.
Anton Khirnov Dec. 6, 2023, 3:13 p.m. UTC | #2
Quoting Niklas Haas (2023-11-09 13:19:38)
> From: Niklas Haas <git@haasn.dev>
> 
> Propagates input metadata to the input filter graph.
> ---
>  fftools/ffmpeg_filter.c | 24 +++++++++++++++++++++---
>  1 file changed, 21 insertions(+), 3 deletions(-)

Looks ok
Anton Khirnov Dec. 7, 2023, 4:02 p.m. UTC | #3
Quoting Niklas Haas (2023-11-09 13:19:39)
> From: Niklas Haas <git@haasn.dev>
> 
> This filter will always accept any input format, even if the user sets
> a specific in_range/in_color_matrix. This is to preserve status quo with
> current behavior, where passing a specific in_color_matrix merely
> overrides the incoming frames' attributes. (Use `vf_format` to force
> a specific input range)
> 
> Because changing colorspace and color_range now requires reconfiguring
> the link, we can lift sws_setColorspaceDetails out of scale_frame and
> into config_props. (This will also get re-called if the input frame
> properties change)
> ---
>  libavfilter/vf_scale.c | 188 ++++++++++++++++++++++-------------------
>  1 file changed, 103 insertions(+), 85 deletions(-)

Can't say I understand all the code in detail, but looks generally
reasonable.
Anton Khirnov Dec. 7, 2023, 4:04 p.m. UTC | #4
Quoting Niklas Haas (2023-11-09 13:19:40)
> From: Niklas Haas <git@haasn.dev>
> 
> This is never avertised as supported to upstream filters.

So what happens when such a frame is sent to lavfi?