| Message ID | 1575470674-8257-1-git-send-email-linjie.fu@intel.com |
|---|---|
| State | New |
| Headers | show |
On 04/12/2019 14:44, Linjie Fu wrote: > VA_RT_FORMAT describes the desired sampling format for surface. > > When creating surface, VA_RT_FORMAT will be used firstly to choose > the expected fourcc/media_format for the surface. And the fourcc > will be revised by the value of VASurfaceAttribPixelFormat. > > Add vaapi_format_map support for new pixel_format. > This is fundamental for both VA-API and QSV. > > Signed-off-by: Linjie Fu <linjie.fu@intel.com> > --- > libavutil/hwcontext_vaapi.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c > index cf11764..296234b 100644 > --- a/libavutil/hwcontext_vaapi.c > +++ b/libavutil/hwcontext_vaapi.c > @@ -116,6 +116,13 @@ static const VAAPIFormatDescriptor vaapi_format_map[] = { > #endif > MAP(UYVY, YUV422, UYVY422, 0), > MAP(YUY2, YUV422, YUYV422, 0), > +#ifdef VA_FOURCC_Y210 > + MAP(Y210, YUV422_10, Y210, 0), > +#endif > + MAP(AYUV, YUV444, 0YUV, 0), > +#ifdef VA_FOURCC_Y410 > + MAP(Y410, YUV444_10, Y410, 0), > +#endif > MAP(411P, YUV411, YUV411P, 0), > MAP(422V, YUV422, YUV440P, 0), > MAP(444P, YUV444, YUV444P, 0), (I've now bought an Ice Lake machine so that I can actually investigate what's going on here properly.) The real problem is that the formats advertised by the driver for H.265/VP9 YUV 4:4:4 (AYUV and Y410) are incorrect. There isn't any alpha support anywhere in the hardware (none of the multilayer stuff), so using formats with alpha in them is wrong and makes all of the attempts to use it extremely confusing. To fix that we need to change the driver to use formats which actually reflect reality. This is mostly straightforward, because the inner code in the driver mainly cares about layout rather than what is actually in each channel. Some patches which kindof work (not particularly well tested): libva: <http://ixia.jkqxz.net/~mrt/4b8a1bb668ce23702f993f9ae7a8f96c/0001-Add-fourcc-values-for-YUV-4-4-4-formats.patch> iHD: <http://ixia.jkqxz.net/~mrt/4b8a1bb668ce23702f993f9ae7a8f96c/0001-Do-not-advertise-support-for-nonexistent-alpha-chann.patch> I've called the non-alpha formats XYUV and XV30 to match libdrm (see <https://cgit.freedesktop.org/mesa/drm/tree/include/drm/drm_fourcc.h#n164>). The 12-bit case also needs some correction - it's currently using Y216 and I think assuming that the user knows they need to ignore the alpha channel and the lower bits elsewhere, but we might need to have distinct formats to make it work properly outside the driver. - Mark PS: It might help to split out the 4:2:2 case (which doesn't have these problems because the formats there don't have fake alpha channels or unknown depth) to go forward, and look separately at fixing the driver for the others.
> -----Original Message----- > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > Mark Thompson > Sent: Saturday, December 28, 2019 06:41 > To: ffmpeg-devel@ffmpeg.org > Subject: Re: [FFmpeg-devel] [PATCH, 4/7] lavu/hwcontext_vaapi: add > vaapi_format_map support for 0YUV/Y210/Y410 > > On 04/12/2019 14:44, Linjie Fu wrote: > > VA_RT_FORMAT describes the desired sampling format for surface. > > > > When creating surface, VA_RT_FORMAT will be used firstly to choose > > the expected fourcc/media_format for the surface. And the fourcc > > will be revised by the value of VASurfaceAttribPixelFormat. > > > > Add vaapi_format_map support for new pixel_format. > > This is fundamental for both VA-API and QSV. > > > > Signed-off-by: Linjie Fu <linjie.fu@intel.com> > > --- > > libavutil/hwcontext_vaapi.c | 7 +++++++ > > 1 file changed, 7 insertions(+) > > > > diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c > > index cf11764..296234b 100644 > > --- a/libavutil/hwcontext_vaapi.c > > +++ b/libavutil/hwcontext_vaapi.c > > @@ -116,6 +116,13 @@ static const VAAPIFormatDescriptor > vaapi_format_map[] = { > > #endif > > MAP(UYVY, YUV422, UYVY422, 0), > > MAP(YUY2, YUV422, YUYV422, 0), > > +#ifdef VA_FOURCC_Y210 > > + MAP(Y210, YUV422_10, Y210, 0), > > +#endif > > + MAP(AYUV, YUV444, 0YUV, 0), > > +#ifdef VA_FOURCC_Y410 > > + MAP(Y410, YUV444_10, Y410, 0), > > +#endif > > MAP(411P, YUV411, YUV411P, 0), > > MAP(422V, YUV422, YUV440P, 0), > > MAP(444P, YUV444, YUV444P, 0), > > (I've now bought an Ice Lake machine so that I can actually investigate what's > going on here properly.) Appreciated it. > The real problem is that the formats advertised by the driver for H.265/VP9 > YUV 4:4:4 (AYUV and Y410) are incorrect. There isn't any alpha support > anywhere in the hardware (none of the multilayer stuff), so using formats > with alpha in them is wrong and makes all of the attempts to use it extremely > confusing. > > To fix that we need to change the driver to use formats which actually reflect > reality. This is mostly straightforward, because the inner code in the driver > mainly cares about layout rather than what is actually in each channel. > > Some patches which kindof work (not particularly well tested): > > libva: <http://ixia.jkqxz.net/~mrt/4b8a1bb668ce23702f993f9ae7a8f96c/0001- > Add-fourcc-values-for-YUV-4-4-4-formats.patch> > iHD: <http://ixia.jkqxz.net/~mrt/4b8a1bb668ce23702f993f9ae7a8f96c/0001- > Do-not-advertise-support-for-nonexistent-alpha-chann.patch> I got some concerns for the modifications: 1. RT FORMAT YUV444 would not be bind with VA_FOURCC_444P any more @@ -2136,10 +2151,13 @@ DdiMedia_CreateSurfaces2( expected_fourcc = VA_FOURCC_Y216; break; case VA_RT_FORMAT_YUV444: - expected_fourcc = VA_FOURCC_444P; + expected_fourcc = VA_FOURCC_XYUV; break; VAAPI decode of jpeg requires VA_RT_FORMAT_YUV444 with expected_fourcc = VA_FOURCC_444P. (Though it's not used frequently), changing the expected fourcc into XYUV may cause a change for media_format from Media_Format_444P to Media_Format_XYUV. 2. The changes in MediaLibvaCaps::QuerySurfaceAttributes which eliminate AYUV/Y410/Y416 may affect the pipelines already implemented in MSDK and GStreamer. (If so, there is a lot to be done/rework) > I've called the non-alpha formats XYUV and XV30 to match libdrm (see > <https://cgit.freedesktop.org/mesa/drm/tree/include/drm/drm_fourcc.h#n > 164>). The 12-bit case also needs some correction - it's currently using Y216 > and I think assuming that the user knows they need to ignore the alpha > channel and the lower bits elsewhere, but we might need to have distinct > formats to make it work properly outside the driver. > > - Mark > > PS: It might help to split out the 4:2:2 case (which doesn't have these > problems because the formats there don't have fake alpha channels or > unknown depth) to go forward, and look separately at fixing the driver for > the others. Thanks, I'll split and get 4:2:2 stuffs done firstly. - linjie BTW, what's the plan for submit these modifications in libva/media-driver?
diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c index cf11764..296234b 100644 --- a/libavutil/hwcontext_vaapi.c +++ b/libavutil/hwcontext_vaapi.c @@ -116,6 +116,13 @@ static const VAAPIFormatDescriptor vaapi_format_map[] = { #endif MAP(UYVY, YUV422, UYVY422, 0), MAP(YUY2, YUV422, YUYV422, 0), +#ifdef VA_FOURCC_Y210 + MAP(Y210, YUV422_10, Y210, 0), +#endif + MAP(AYUV, YUV444, 0YUV, 0), +#ifdef VA_FOURCC_Y410 + MAP(Y410, YUV444_10, Y410, 0), +#endif MAP(411P, YUV411, YUV411P, 0), MAP(422V, YUV422, YUV440P, 0), MAP(444P, YUV444, YUV444P, 0),
VA_RT_FORMAT describes the desired sampling format for surface. When creating surface, VA_RT_FORMAT will be used firstly to choose the expected fourcc/media_format for the surface. And the fourcc will be revised by the value of VASurfaceAttribPixelFormat. Add vaapi_format_map support for new pixel_format. This is fundamental for both VA-API and QSV. Signed-off-by: Linjie Fu <linjie.fu@intel.com> --- libavutil/hwcontext_vaapi.c | 7 +++++++ 1 file changed, 7 insertions(+)