[FFmpeg-devel] hwcontext_vaapi: remove duplicate formats from sw_format list

hwcontext_vaapi maps different VA fourcc to the same pix_fmt for U/V
plane swap cases, however duplicate formats are not expected in sw_format
list when merging formats.

For example:
ffmpeg -loglevel debug -init_hw_device vaapi -filter_hw_device vaapi0 \
-f lavfi -i smptebars -vf \
"hwupload=derive_device=vaapi,scale_vaapi,hwdownload,format=yuv420p" \
-vframes 1 -f null -

Without this fix, an auto scaler is required for the above command
   Duplicate formats in ff_merge_formats detected
   [auto_scaler_0 @ 0x560df58f4550] Setting 'flags' to value 'bicubic'
   [auto_scaler_0 @ 0x560df58f4550] w:iw h:ih flags:'bicubic' interl:0
   [Parsed_hwupload_0 @ 0x560df58f0ec0] auto-inserting filter
   'auto_scaler_0' between the filter 'graph 0 input from stream 0:0' and
   the filter 'Parsed_hwupload_0'

Signed-off-by: Haihao Xiang <haihao.xiang@intel.com>
---
 libavutil/hwcontext_vaapi.c | 30 +++++++++++++++++++++++++-----
 1 file changed, 25 insertions(+), 5 deletions(-)

On 24/07/2020 03:39, Haihao Xiang wrote:
> hwcontext_vaapi maps different VA fourcc to the same pix_fmt for U/V
> plane swap cases, however duplicate formats are not expected in sw_format
> list when merging formats.
> 
> For example:
> ffmpeg -loglevel debug -init_hw_device vaapi -filter_hw_device vaapi0 \
> -f lavfi -i smptebars -vf \
> "hwupload=derive_device=vaapi,scale_vaapi,hwdownload,format=yuv420p" \
> -vframes 1 -f null -
> 
> Without this fix, an auto scaler is required for the above command
>     Duplicate formats in ff_merge_formats detected
>     [auto_scaler_0 @ 0x560df58f4550] Setting 'flags' to value 'bicubic'
>     [auto_scaler_0 @ 0x560df58f4550] w:iw h:ih flags:'bicubic' interl:0
>     [Parsed_hwupload_0 @ 0x560df58f0ec0] auto-inserting filter
>     'auto_scaler_0' between the filter 'graph 0 input from stream 0:0' and
>     the filter 'Parsed_hwupload_0'
> 
> Signed-off-by: Haihao Xiang <haihao.xiang@intel.com>
> ---
>   libavutil/hwcontext_vaapi.c | 30 +++++++++++++++++++++++++-----
>   1 file changed, 25 insertions(+), 5 deletions(-)
> 
> diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
> index b31cf95850..fb9be19647 100644
> --- a/libavutil/hwcontext_vaapi.c
> +++ b/libavutil/hwcontext_vaapi.c
> @@ -268,14 +268,24 @@ static int vaapi_frames_get_constraints(AVHWDeviceContext *hwdev,
>               }
>   
>               for (i = j = 0; i < attr_count; i++) {
> +                int k;
> +
>                   if (attr_list[i].type != VASurfaceAttribPixelFormat)
>                       continue;
>                   fourcc = attr_list[i].value.value.i;
>                   pix_fmt = vaapi_pix_fmt_from_fourcc(fourcc);
> -                if (pix_fmt != AV_PIX_FMT_NONE)
> +
> +                if (pix_fmt == AV_PIX_FMT_NONE)
> +                    continue;
> +
> +                for (k = 0; k < j; k++) {
> +                    if (constraints->valid_sw_formats[k] == pix_fmt)
> +                        break;
> +                }
> +
> +                if (k == j)
>                       constraints->valid_sw_formats[j++] = pix_fmt;
>               }
> -            av_assert0(j == pix_fmt_count);
>               constraints->valid_sw_formats[j] = AV_PIX_FMT_NONE;
>           }
>       } else {
> @@ -287,9 +297,19 @@ static int vaapi_frames_get_constraints(AVHWDeviceContext *hwdev,
>               err = AVERROR(ENOMEM);
>               goto fail;
>           }
> -        for (i = 0; i < ctx->nb_formats; i++)
> -            constraints->valid_sw_formats[i] = ctx->formats[i].pix_fmt;
> -        constraints->valid_sw_formats[i] = AV_PIX_FMT_NONE;
> +        for (i = j = 0; i < ctx->nb_formats; i++) {
> +            int k;
> +
> +            for (k = 0; k < j; k++) {
> +                if (constraints->valid_sw_formats[k] == ctx->formats[i].pix_fmt)
> +                    break;
> +            }
> +
> +            if (k == j)
> +                constraints->valid_sw_formats[j++] = ctx->formats[i].pix_fmt;
> +        }
> +
> +        constraints->valid_sw_formats[j] = AV_PIX_FMT_NONE;
>       }
>   
>       constraints->valid_hw_formats = av_malloc_array(2, sizeof(pix_fmt));
> 

Not a very fun case.  Looks like a good way to handle it, though, so applied.

Thanks,

- Mark