[FFmpeg-devel,v2] hwcontext_vaapi: introduce AV_HWFRAME_MAP_{SEPARATE, COMBINED}_PLANES

This allows callers to ask for separate or combined plane formats.

Combined formats are desirable in many scenarios, for instance when
directly importing the DMA-BUFs into APIs such as GL (as a single
EGLImage), KMS or Wayland.

This patch was tested with the following Mesa MR:
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9015

v2:
- Add SEPARATE_PLANES flag
- Rename to AV_HWFRAME_COMBINED_PLANES

Signed-off-by: Simon Ser <contact@emersion.fr>
Cc: Mark Thompson <sw@jkqxz.net>
Cc: Haihao Xiang <haihao.xiang@intel.com>
Cc: Fei Wang <fei.w.wang@intel.com>
---
 libavutil/hwcontext.h       | 12 ++++++++++++
 libavutil/hwcontext_vaapi.c |  6 +++++-
 2 files changed, 17 insertions(+), 1 deletion(-)

Feb 16, 2021, 11:59 by contact@emersion.fr:

> This allows callers to ask for separate or combined plane formats.
>
> Combined formats are desirable in many scenarios, for instance when
> directly importing the DMA-BUFs into APIs such as GL (as a single
> EGLImage), KMS or Wayland.
>
> This patch was tested with the following Mesa MR:
> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9015
>
> v2:
> - Add SEPARATE_PLANES flag
> - Rename to AV_HWFRAME_COMBINED_PLANES
>
> Signed-off-by: Simon Ser <contact@emersion.fr>
> Cc: Mark Thompson <sw@jkqxz.net>
> Cc: Haihao Xiang <haihao.xiang@intel.com>
> Cc: Fei Wang <fei.w.wang@intel.com>
> ---
>  libavutil/hwcontext.h       | 12 ++++++++++++
>  libavutil/hwcontext_vaapi.c |  6 +++++-
>  2 files changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/libavutil/hwcontext.h b/libavutil/hwcontext.h
> index 04d19d89c2b8..e24cb1470595 100644
> --- a/libavutil/hwcontext.h
> +++ b/libavutil/hwcontext.h
> @@ -538,6 +538,18 @@ enum {
>  * be much lower than normal memory.
>  */
>  AV_HWFRAME_MAP_DIRECT    = 1 << 3,
> +    /**
> +     * For multi-planar frames, the mapping must break the frame into
> +     * single-plane formats.  For instance, an NV12 frame must be split into a
> +     * a 8-bit format for the Y plane and an interleaved 16-bit format for the
> +     * U/V plane with 2x2 subsampling.
> +     */
> +    AV_HWFRAME_MAP_SEPARATE_PLANES = 1 << 4,
> +    /**
> +     * For multi-planar frames, the mapping must result in a multi-planar
> +     * format.  This is the contrary of AV_HWFRAME_MAP_SEPARATE_PLANES.
> +     */
> +    AV_HWFRAME_MAP_COMBINED_PLANES = 1 << 5,
>  };
>  
>  /**
> diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
> index 2227d6ed6981..62eb1e2908b3 100644
> --- a/libavutil/hwcontext_vaapi.c
> +++ b/libavutil/hwcontext_vaapi.c
> @@ -1161,7 +1161,11 @@ static int vaapi_map_to_drm_esh(AVHWFramesContext *hwfc, AVFrame *dst,
>  
>  surface_id = (VASurfaceID)(uintptr_t)src->data[3];
>  
> -    export_flags = VA_EXPORT_SURFACE_SEPARATE_LAYERS;
> +    export_flags = 0;
> +    if (flags & AV_HWFRAME_MAP_COMBINED_PLANES)
> +        export_flags |= VA_EXPORT_SURFACE_COMPOSED_LAYERS;
> +    else
> +        export_flags |= VA_EXPORT_SURFACE_SEPARATE_LAYERS;
>  if (flags & AV_HWFRAME_MAP_READ)
>  export_flags |= VA_EXPORT_SURFACE_READ_ONLY;
>  if (flags & AV_HWFRAME_MAP_WRITE)
>

I think you should add AVERROR(ENOSYS) to all hwcontext mapping functions
which don't support setting this for now. Adding it to HWContextType.map_from/to
for all should be enough.
With this change LGTM (will need an APIchanges entry and a minor bump too though).