[FFmpeg-devel,1/2] lavu/pixfmt: add AYUV pixel format

Submitted by Linjie Fu on Aug. 28, 2019, 6:22 a.m.

Details

Message ID 1566973334-19107-1-git-send-email-linjie.fu@intel.com
State New
Headers show

Commit Message

Linjie Fu Aug. 28, 2019, 6:22 a.m.
Add support for packed 4:4:4 pixel format AYUV.

It is the format that VAAPI/QSV uses when coping with 4:4:4
surfaces. Alpha channel will be set to default value for HEVC REXT
hw decode.

Signed-off-by: Linjie Fu <linjie.fu@intel.com>
---
 libavutil/pixdesc.c           | 13 +++++++++++++
 libavutil/pixfmt.h            |  2 ++
 libavutil/tests/pixfmt_best.c |  1 +
 libavutil/version.h           |  2 +-
 4 files changed, 17 insertions(+), 1 deletion(-)

Comments

Carl Eugen Hoyos Aug. 28, 2019, 6:50 a.m.
> Am 28.08.2019 um 08:22 schrieb Linjie Fu <linjie.fu@intel.com>:
> 
> Add support for packed 4:4:4 pixel format AYUV.
> 
> It is the format that VAAPI/QSV uses when coping with 4:4:4
> surfaces.

> Alpha channel will be set to default value for HEVC REXT
> hw decode.

What is the „default value“?

Carl Eugen
Linjie Fu Aug. 28, 2019, 7:33 a.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf

> Of Carl Eugen Hoyos

> Sent: Wednesday, August 28, 2019 14:51

> To: FFmpeg development discussions and patches <ffmpeg-

> devel@ffmpeg.org>

> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format

> 

> 

> 

> > Am 28.08.2019 um 08:22 schrieb Linjie Fu <linjie.fu@intel.com>:

> >

> > Add support for packed 4:4:4 pixel format AYUV.

> >

> > It is the format that VAAPI/QSV uses when coping with 4:4:4

> > surfaces.

> 

> > Alpha channel will be set to default value for HEVC REXT

> > hw decode.

> 

> What is the „default value“?

> 


Default zero. (and it should be 255 for display)
Carl Eugen Hoyos Aug. 28, 2019, 9:16 p.m.
Am Mi., 28. Aug. 2019 um 09:34 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:
>
> > -----Original Message-----
> > From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
> > Of Carl Eugen Hoyos
> > Sent: Wednesday, August 28, 2019 14:51
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel@ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format
> >
> >
> >
> > > Am 28.08.2019 um 08:22 schrieb Linjie Fu <linjie.fu@intel.com>:
> > >
> > > Add support for packed 4:4:4 pixel format AYUV.
> > >
> > > It is the format that VAAPI/QSV uses when coping with 4:4:4
> > > surfaces.
> >
> > > Alpha channel will be set to default value for HEVC REXT
> > > hw decode.
> >
> > What is the „default value“?
> >
>
> Default zero. (and it should be 255 for display)

If this is not a bug in the driver, there is no alpha channel that can be used.
Rename it to 0YUV and remove the alpha usage.

Carl Eugen
Linjie Fu Aug. 29, 2019, 1:34 a.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf

> Of Carl Eugen Hoyos

> Sent: Thursday, August 29, 2019 05:17

> To: FFmpeg development discussions and patches <ffmpeg-

> devel@ffmpeg.org>

> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format

> 

> Am Mi., 28. Aug. 2019 um 09:34 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:

> >

> > > -----Original Message-----

> > > From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On

> Behalf

> > > Of Carl Eugen Hoyos

> > > Sent: Wednesday, August 28, 2019 14:51

> > > To: FFmpeg development discussions and patches <ffmpeg-

> > > devel@ffmpeg.org>

> > > Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel

> format

> > >

> > >

> > >

> > > > Am 28.08.2019 um 08:22 schrieb Linjie Fu <linjie.fu@intel.com>:

> > > >

> > > > Add support for packed 4:4:4 pixel format AYUV.

> > > >

> > > > It is the format that VAAPI/QSV uses when coping with 4:4:4

> > > > surfaces.

> > >

> > > > Alpha channel will be set to default value for HEVC REXT

> > > > hw decode.

> > >

> > > What is the „default value“?

> > >

> >

> > Default zero. (and it should be 255 for display)

> 

> If this is not a bug in the driver, there is no alpha channel that can be used.

> Rename it to 0YUV and remove the alpha usage.

> 


From the pixel format level, adding AYUV format is for a more fundamental/common usage.
It is valid no matter the alpha data is 0 or 255 or specific value to indicate the transparency.

And the hardware decode result is an AYUV format with fixed alpha values, so it should still be
an AYUV format IMHO.

- linjie
Carl Eugen Hoyos Aug. 29, 2019, 11:47 a.m.
Am 29.08.2019 um 03:34 schrieb Fu, Linjie <linjie.fu@intel.com>:

>> -----Original Message-----
>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
>> Of Carl Eugen Hoyos
>> Sent: Thursday, August 29, 2019 05:17
>> To: FFmpeg development discussions and patches <ffmpeg-
>> devel@ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format
>> 
>>>> Am Mi., 28. Aug. 2019 um 09:34 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:
>>>> 
>>>> -----Original Message-----
>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On
>> Behalf
>>>> Of Carl Eugen Hoyos
>>>> Sent: Wednesday, August 28, 2019 14:51
>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>> devel@ffmpeg.org>
>>>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel
>> format
>>>> 
>>>> 
>>>> 
>>>>> Am 28.08.2019 um 08:22 schrieb Linjie Fu <linjie.fu@intel.com>:
>>>>> 
>>>>> Add support for packed 4:4:4 pixel format AYUV.
>>>>> 
>>>>> It is the format that VAAPI/QSV uses when coping with 4:4:4
>>>>> surfaces.
>>>> 
>>>>> Alpha channel will be set to default value for HEVC REXT
>>>>> hw decode.
>>>> 
>>>> What is the „default value“?
>>> 
>>> Default zero. (and it should be 255 for display)
>> 
>> If this is not a bug in the driver, there is no alpha channel that can be used.
>> Rename it to 0YUV and remove the alpha usage.
> 
> From the pixel format level, adding AYUV format is for a more fundamental/common usage.

Yes, but the new pixel format will only be accepted if it is needed for hardware compatibility.
If the hardware only supports 0YUV, it cannot be used to argue in favour of a new AYUV format.

Carl Eugen
Paul B Mahol Aug. 29, 2019, 11:54 a.m.
On 8/29/19, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>
>
> Am 29.08.2019 um 03:34 schrieb Fu, Linjie <linjie.fu@intel.com>:
>
>>> -----Original Message-----
>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
>>> Of Carl Eugen Hoyos
>>> Sent: Thursday, August 29, 2019 05:17
>>> To: FFmpeg development discussions and patches <ffmpeg-
>>> devel@ffmpeg.org>
>>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel
>>> format
>>>
>>>>> Am Mi., 28. Aug. 2019 um 09:34 Uhr schrieb Fu, Linjie
>>>>> <linjie.fu@intel.com>:
>>>>>
>>>>> -----Original Message-----
>>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On
>>> Behalf
>>>>> Of Carl Eugen Hoyos
>>>>> Sent: Wednesday, August 28, 2019 14:51
>>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>>> devel@ffmpeg.org>
>>>>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel
>>> format
>>>>>
>>>>>
>>>>>
>>>>>> Am 28.08.2019 um 08:22 schrieb Linjie Fu <linjie.fu@intel.com>:
>>>>>>
>>>>>> Add support for packed 4:4:4 pixel format AYUV.
>>>>>>
>>>>>> It is the format that VAAPI/QSV uses when coping with 4:4:4
>>>>>> surfaces.
>>>>>
>>>>>> Alpha channel will be set to default value for HEVC REXT
>>>>>> hw decode.
>>>>>
>>>>> What is the „default value“?
>>>>
>>>> Default zero. (and it should be 255 for display)
>>>
>>> If this is not a bug in the driver, there is no alpha channel that can be
>>> used.
>>> Rename it to 0YUV and remove the alpha usage.
>>
>> From the pixel format level, adding AYUV format is for a more
>> fundamental/common usage.
>
> Yes, but the new pixel format will only be accepted if it is needed for
> hardware compatibility.
> If the hardware only supports 0YUV, it cannot be used to argue in favour of
> a new AYUV format.

Nope.
>
> Carl Eugen
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Hendrik Leppkes Aug. 29, 2019, 12:30 p.m.
On Thu, Aug 29, 2019 at 1:47 PM Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>
>
>
> Am 29.08.2019 um 03:34 schrieb Fu, Linjie <linjie.fu@intel.com>:
>
> >> -----Original Message-----
> >> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
> >> Of Carl Eugen Hoyos
> >> Sent: Thursday, August 29, 2019 05:17
> >> To: FFmpeg development discussions and patches <ffmpeg-
> >> devel@ffmpeg.org>
> >> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format
> >>
> >>>> Am Mi., 28. Aug. 2019 um 09:34 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:
> >>>>
> >>>> -----Original Message-----
> >>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On
> >> Behalf
> >>>> Of Carl Eugen Hoyos
> >>>> Sent: Wednesday, August 28, 2019 14:51
> >>>> To: FFmpeg development discussions and patches <ffmpeg-
> >>>> devel@ffmpeg.org>
> >>>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel
> >> format
> >>>>
> >>>>
> >>>>
> >>>>> Am 28.08.2019 um 08:22 schrieb Linjie Fu <linjie.fu@intel.com>:
> >>>>>
> >>>>> Add support for packed 4:4:4 pixel format AYUV.
> >>>>>
> >>>>> It is the format that VAAPI/QSV uses when coping with 4:4:4
> >>>>> surfaces.
> >>>>
> >>>>> Alpha channel will be set to default value for HEVC REXT
> >>>>> hw decode.
> >>>>
> >>>> What is the „default value“?
> >>>
> >>> Default zero. (and it should be 255 for display)
> >>
> >> If this is not a bug in the driver, there is no alpha channel that can be used.
> >> Rename it to 0YUV and remove the alpha usage.
> >
> > From the pixel format level, adding AYUV format is for a more fundamental/common usage.
>
> Yes, but the new pixel format will only be accepted if it is needed for hardware compatibility.
> If the hardware only supports 0YUV, it cannot be used to argue in favour of a new AYUV format.
>

AYUV is a standard format used on Windows and documented by Microsoft
as the recommended 4:4:4 8-bit format for all purposes. Its hardly a
niche format. Its perfectly appropriate to support it.

- Hendrik
Carl Eugen Hoyos Aug. 29, 2019, 1:29 p.m.
> Am 29.08.2019 um 14:30 schrieb Hendrik Leppkes <h.leppkes@gmail.com>:
> 
>> On Thu, Aug 29, 2019 at 1:47 PM Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>> 
>> 
>> 
>> Am 29.08.2019 um 03:34 schrieb Fu, Linjie <linjie.fu@intel.com>:
>> 
>>>> -----Original Message-----
>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
>>>> Of Carl Eugen Hoyos
>>>> Sent: Thursday, August 29, 2019 05:17
>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>> devel@ffmpeg.org>
>>>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format
>>>> 
>>>>>> Am Mi., 28. Aug. 2019 um 09:34 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On
>>>> Behalf
>>>>>> Of Carl Eugen Hoyos
>>>>>> Sent: Wednesday, August 28, 2019 14:51
>>>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>>>> devel@ffmpeg.org>
>>>>>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel
>>>> format
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> Am 28.08.2019 um 08:22 schrieb Linjie Fu <linjie.fu@intel.com>:
>>>>>>> 
>>>>>>> Add support for packed 4:4:4 pixel format AYUV.
>>>>>>> 
>>>>>>> It is the format that VAAPI/QSV uses when coping with 4:4:4
>>>>>>> surfaces.
>>>>>> 
>>>>>>> Alpha channel will be set to default value for HEVC REXT
>>>>>>> hw decode.
>>>>>> 
>>>>>> What is the „default value“?
>>>>> 
>>>>> Default zero. (and it should be 255 for display)
>>>> 
>>>> If this is not a bug in the driver, there is no alpha channel that can be used.
>>>> Rename it to 0YUV and remove the alpha usage.
>>> 
>>> From the pixel format level, adding AYUV format is for a more fundamental/common usage.
>> 
>> Yes, but the new pixel format will only be accepted if it is needed for hardware compatibility.
>> If the hardware only supports 0YUV, it cannot be used to argue in favour of a new AYUV format.
>> 
> 
> AYUV is a standard format used on Windows and documented by Microsoft
> as the recommended 4:4:4 8-bit format for all purposes. Its hardly a
> niche format. Its perfectly appropriate to support it.

For which use case?

Carl Eugen
Carl Eugen Hoyos Aug. 29, 2019, 1:31 p.m.
> Am 29.08.2019 um 14:30 schrieb Hendrik Leppkes <h.leppkes@gmail.com>:
> 
>> On Thu, Aug 29, 2019 at 1:47 PM Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>> 
>> 
>> 
>> Am 29.08.2019 um 03:34 schrieb Fu, Linjie <linjie.fu@intel.com>:
>> 
>>>> -----Original Message-----
>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
>>>> Of Carl Eugen Hoyos
>>>> Sent: Thursday, August 29, 2019 05:17
>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>> devel@ffmpeg.org>
>>>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format
>>>> 
>>>>>> Am Mi., 28. Aug. 2019 um 09:34 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On
>>>> Behalf
>>>>>> Of Carl Eugen Hoyos
>>>>>> Sent: Wednesday, August 28, 2019 14:51
>>>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>>>> devel@ffmpeg.org>
>>>>>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel
>>>> format
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> Am 28.08.2019 um 08:22 schrieb Linjie Fu <linjie.fu@intel.com>:
>>>>>>> 
>>>>>>> Add support for packed 4:4:4 pixel format AYUV.
>>>>>>> 
>>>>>>> It is the format that VAAPI/QSV uses when coping with 4:4:4
>>>>>>> surfaces.
>>>>>> 
>>>>>>> Alpha channel will be set to default value for HEVC REXT
>>>>>>> hw decode.
>>>>>> 
>>>>>> What is the „default value“?
>>>>> 
>>>>> Default zero. (and it should be 255 for display)
>>>> 
>>>> If this is not a bug in the driver, there is no alpha channel that can be used.
>>>> Rename it to 0YUV and remove the alpha usage.
>>> 
>>> From the pixel format level, adding AYUV format is for a more fundamental/common usage.
>> 
>> Yes, but the new pixel format will only be accepted if it is needed for hardware compatibility.
>> If the hardware only supports 0YUV, it cannot be used to argue in favour of a new AYUV format.
>> 
> 
> AYUV is a standard format used on Windows and documented by Microsoft

> as the recommended 4:4:4 8-bit

Sorry, I missed this:
This is not the format suggested in this posted patch.
(It is the format I suggested.)

> format for all purposes. Its hardly a
> niche format. Its perfectly appropriate to support it.

Carl Eugen
Linjie Fu Aug. 29, 2019, 2:15 p.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf

> Of Carl Eugen Hoyos

> Sent: Thursday, August 29, 2019 21:31

> To: FFmpeg development discussions and patches <ffmpeg-

> devel@ffmpeg.org>

> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format

> 

> 

> 

> > Am 29.08.2019 um 14:30 schrieb Hendrik Leppkes <h.leppkes@gmail.com>:

> >

> >> On Thu, Aug 29, 2019 at 1:47 PM Carl Eugen Hoyos <ceffmpeg@gmail.com>

> wrote:

> >>

> >>

> >>

> >> Am 29.08.2019 um 03:34 schrieb Fu, Linjie <linjie.fu@intel.com>:

> >>

> >>>> -----Original Message-----

> >>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On

> Behalf

> >>>> Of Carl Eugen Hoyos

> >>>> Sent: Thursday, August 29, 2019 05:17

> >>>> To: FFmpeg development discussions and patches <ffmpeg-

> >>>> devel@ffmpeg.org>

> >>>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel

> format

> >>>>

> >>>>>> Am Mi., 28. Aug. 2019 um 09:34 Uhr schrieb Fu, Linjie

> <linjie.fu@intel.com>:

> >>>>>>

> >>>>>> -----Original Message-----

> >>>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org]

> On

> >>>> Behalf

> >>>>>> Of Carl Eugen Hoyos

> >>>>>> Sent: Wednesday, August 28, 2019 14:51

> >>>>>> To: FFmpeg development discussions and patches <ffmpeg-

> >>>>>> devel@ffmpeg.org>

> >>>>>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV

> pixel

> >>>> format

> >>>>>>

> >>>>>>

> >>>>>>

> >>>>>>> Am 28.08.2019 um 08:22 schrieb Linjie Fu <linjie.fu@intel.com>:

> >>>>>>>

> >>>>>>> Add support for packed 4:4:4 pixel format AYUV.

> >>>>>>>

> >>>>>>> It is the format that VAAPI/QSV uses when coping with 4:4:4

> >>>>>>> surfaces.

> >>>>>>

> >>>>>>> Alpha channel will be set to default value for HEVC REXT

> >>>>>>> hw decode.

> >>>>>>

> >>>>>> What is the „default value“?

> >>>>>

> >>>>> Default zero. (and it should be 255 for display)

> >>>>

> >>>> If this is not a bug in the driver, there is no alpha channel that can be

> used.

> >>>> Rename it to 0YUV and remove the alpha usage.

> >>>

> >>> From the pixel format level, adding AYUV format is for a more

> fundamental/common usage.

> >>

> >> Yes, but the new pixel format will only be accepted if it is needed for

> hardware compatibility.

> >> If the hardware only supports 0YUV, it cannot be used to argue in favour

> of a new AYUV format.

> >>

> >

> > AYUV is a standard format used on Windows and documented by Microsoft

> 

> > as the recommended 4:4:4 8-bit

> 

> Sorry, I missed this:

> This is not the format suggested in this posted patch.

> (It is the format I suggested.)

> 

> > format for all purposes. Its hardly a

> > niche format. Its perfectly appropriate to support it.


Got the concern.

If we need a reason to support this standard pixel format apart
from the hardware compatibility, it could be some requirements like CSC from
ARGB to YUV and don't want to lose the alpha channel information:
https://stackoverflow.com/questions/6023565/argb-to-yuv-color-spaces-in-net

"How to convert image beetween ARGB and YUV in c#?
I found formulas describing RGB to YUV conversion. So what should I do with alpha channel? "

Thus AYUV will make more sense.

- linjie
Carl Eugen Hoyos Aug. 29, 2019, 4:12 p.m.
Am Do., 29. Aug. 2019 um 16:16 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:

> Thus AYUV will make more sense.

If your hardware decoding does not produce valid alpha data (0xFF
for opaque, 0x00 for completely transparent), you cannot use an
FFmpeg pix_fmt that defines alpha for decoding.
If you define a pix_fmt that does not contain alpha information (as
needed for your hardware decoder), please do not use "A" in its
name.

Carl Eugen
Linjie Fu Aug. 31, 2019, 4:22 a.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf

> Of Carl Eugen Hoyos

> Sent: Friday, August 30, 2019 00:12

> To: FFmpeg development discussions and patches <ffmpeg-

> devel@ffmpeg.org>

> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format

> 

> Am Do., 29. Aug. 2019 um 16:16 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:

> 

> > Thus AYUV will make more sense.

> 

> If your hardware decoding does not produce valid alpha data (0xFF

> for opaque, 0x00 for completely transparent), you cannot use an

> FFmpeg pix_fmt that defines alpha for decoding.

> If you define a pix_fmt that does not contain alpha information (as

> needed for your hardware decoder), please do not use "A" in its

> name.


The hardware decode output is in packed 4:4:4:4 8 bit(32 bits in total). Apart
from Y/U/V channel, there is an additional need for a channel to store the
zero byte in surfaces(nb_components = 4). And it is designed in hardware/driver
for VA_FourCC_AYUV and AYUV format.

It's not quite proper to use something like "0YUV" directly which indicates that
there is no Alpha channel data like"RGB0" or "0RGB"(nb_components = 3).

"0YUV": YUV0/YUV0/...
"RGB0":RGB/RGB/...

It's true that we could only map Y/U/V data from surface to frame in ffmpeg and
use a 3-components pixel format like "0YUV", but IMHO it's kind of improper because
the output from hardware/driver is forced changed in Application level even if user
didn't call for a CSC:
	- hardware/driver output (in AYUV):  YUV0/YUV0/...
	- ffmpeg output (in 0YUV):                     YUV/YUV/...

Please feel free to offer your opinions on this question(AYUV or 0YUV) and let me
know how to push it step-further.

Thanks,
Linjie
Mark Thompson Aug. 31, 2019, 3:09 p.m.
On 31/08/2019 05:22, Fu, Linjie wrote:
>> -----Original Message-----
>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
>> Of Carl Eugen Hoyos
>> Sent: Friday, August 30, 2019 00:12
>> To: FFmpeg development discussions and patches <ffmpeg-
>> devel@ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format
>>
>> Am Do., 29. Aug. 2019 um 16:16 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:
>>
>>> Thus AYUV will make more sense.
>>
>> If your hardware decoding does not produce valid alpha data (0xFF
>> for opaque, 0x00 for completely transparent), you cannot use an
>> FFmpeg pix_fmt that defines alpha for decoding.
>> If you define a pix_fmt that does not contain alpha information (as
>> needed for your hardware decoder), please do not use "A" in its
>> name.
> 
> The hardware decode output is in packed 4:4:4:4 8 bit(32 bits in total). Apart> from Y/U/V channel, there is an additional need for a channel to store the
> zero byte in surfaces(nb_components = 4). And it is designed in hardware/driver
> for VA_FourCC_AYUV and AYUV format.
> 
> It's not quite proper to use something like "0YUV" directly which indicates that
> there is no Alpha channel data like"RGB0" or "0RGB"(nb_components = 3).
> 
> "0YUV": YUV0/YUV0/...
> "RGB0":RGB/RGB/...

I think you've misread what RGB0 is.  The component step is 4, not 3 (see libavutil/pixdesc.c) - the format with a step of 3 is RGB24.

RGBA:  R G B A R G B A ...
RGB0:  R G B 0 R G B 0 ...
RGB24: R G B R G B R G ...

Following the same pattern we would have:

AYUV:  A Y U V A Y U V ...
0YUV:  0 Y U V 0 Y U V ...

To me it looks like 0YUV is exactly what you want here.

- Mark
Linjie Fu Aug. 31, 2019, 5:25 p.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf

> Of Mark Thompson

> Sent: Saturday, August 31, 2019 23:10

> To: ffmpeg-devel@ffmpeg.org

> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format

> 

> On 31/08/2019 05:22, Fu, Linjie wrote:

> >> -----Original Message-----

> >> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On

> Behalf

> >> Of Carl Eugen Hoyos

> >> Sent: Friday, August 30, 2019 00:12

> >> To: FFmpeg development discussions and patches <ffmpeg-

> >> devel@ffmpeg.org>

> >> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel

> format

> >>

> >> Am Do., 29. Aug. 2019 um 16:16 Uhr schrieb Fu, Linjie

> <linjie.fu@intel.com>:

> >>

> >>> Thus AYUV will make more sense.

> >>

> >> If your hardware decoding does not produce valid alpha data (0xFF

> >> for opaque, 0x00 for completely transparent), you cannot use an

> >> FFmpeg pix_fmt that defines alpha for decoding.

> >> If you define a pix_fmt that does not contain alpha information (as

> >> needed for your hardware decoder), please do not use "A" in its

> >> name.

> >

> > The hardware decode output is in packed 4:4:4:4 8 bit(32 bits in total). Apart>

> from Y/U/V channel, there is an additional need for a channel to store the

> > zero byte in surfaces(nb_components = 4). And it is designed in

> hardware/driver

> > for VA_FourCC_AYUV and AYUV format.

> >

> > It's not quite proper to use something like "0YUV" directly which indicates

> that

> > there is no Alpha channel data like"RGB0" or "0RGB"(nb_components = 3).

> >

> > "0YUV": YUV0/YUV0/...

> > "RGB0":RGB/RGB/...

> 

> I think you've misread what RGB0 is.  The component step is 4, not 3 (see

> libavutil/pixdesc.c) - the format with a step of 3 is RGB24.

> 

> RGBA:  R G B A R G B A ...

> RGB0:  R G B 0 R G B 0 ...

> RGB24: R G B R G B R G ...


Thanks for pointing out this.
Just took the nb_components into account and misread the step info at first.
 
> Following the same pattern we would have:

> 

> AYUV:  A Y U V A Y U V ...

> 0YUV:  0 Y U V 0 Y U V ...

> 

> To me it looks like 0YUV is exactly what you want here.


Will update the patch if 0YUV is more acceptable.
And thanks Carl.

- linjie
Carl Eugen Hoyos Aug. 31, 2019, 5:57 p.m.
Am Sa., 31. Aug. 2019 um 19:25 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:
>
> > -----Original Message-----
> > From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
> > Of Mark Thompson
> > Sent: Saturday, August 31, 2019 23:10
> > To: ffmpeg-devel@ffmpeg.org
> > Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format
> >
> > On 31/08/2019 05:22, Fu, Linjie wrote:
> > >> -----Original Message-----
> > >> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On
> > Behalf
> > >> Of Carl Eugen Hoyos
> > >> Sent: Friday, August 30, 2019 00:12
> > >> To: FFmpeg development discussions and patches <ffmpeg-
> > >> devel@ffmpeg.org>
> > >> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel
> > format
> > >>
> > >> Am Do., 29. Aug. 2019 um 16:16 Uhr schrieb Fu, Linjie
> > <linjie.fu@intel.com>:
> > >>
> > >>> Thus AYUV will make more sense.
> > >>
> > >> If your hardware decoding does not produce valid alpha data (0xFF
> > >> for opaque, 0x00 for completely transparent), you cannot use an
> > >> FFmpeg pix_fmt that defines alpha for decoding.
> > >> If you define a pix_fmt that does not contain alpha information (as
> > >> needed for your hardware decoder), please do not use "A" in its
> > >> name.
> > >
> > > The hardware decode output is in packed 4:4:4:4 8 bit(32 bits in total). Apart>
> > from Y/U/V channel, there is an additional need for a channel to store the
> > > zero byte in surfaces(nb_components = 4). And it is designed in
> > hardware/driver
> > > for VA_FourCC_AYUV and AYUV format.
> > >
> > > It's not quite proper to use something like "0YUV" directly which indicates
> > that
> > > there is no Alpha channel data like"RGB0" or "0RGB"(nb_components = 3).
> > >
> > > "0YUV": YUV0/YUV0/...
> > > "RGB0":RGB/RGB/...
> >
> > I think you've misread what RGB0 is.  The component step is 4, not 3 (see
> > libavutil/pixdesc.c) - the format with a step of 3 is RGB24.
> >
> > RGBA:  R G B A R G B A ...
> > RGB0:  R G B 0 R G B 0 ...
> > RGB24: R G B R G B R G ...
>
> Thanks for pointing out this.
> Just took the nb_components into account and misread the step info at first.
>
> > Following the same pattern we would have:
> >
> > AYUV:  A Y U V A Y U V ...
> > 0YUV:  0 Y U V 0 Y U V ...
> >
> > To me it looks like 0YUV is exactly what you want here.
>
> Will update the patch if 0YUV is more acceptable.

Could you double-check if there is no bug in the decoder?
I ask because transparency is possible with hevc and the
relevant Windows format does support transparency from
how I read the specification.

Carl Eugen
Linjie Fu Sept. 3, 2019, 3 a.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf

> Of Carl Eugen Hoyos

> Sent: Sunday, September 1, 2019 01:57

> To: FFmpeg development discussions and patches <ffmpeg-

> devel@ffmpeg.org>

> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format

> 

> Am Sa., 31. Aug. 2019 um 19:25 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:

> >

> > > -----Original Message-----

> > > From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On

> Behalf

> > > Of Mark Thompson

> > > Sent: Saturday, August 31, 2019 23:10

> > > To: ffmpeg-devel@ffmpeg.org

> > > Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel

> format

> > >

> > > On 31/08/2019 05:22, Fu, Linjie wrote:

> > > >> -----Original Message-----

> > > >> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On

> > > Behalf

> > > >> Of Carl Eugen Hoyos

> > > >> Sent: Friday, August 30, 2019 00:12

> > > >> To: FFmpeg development discussions and patches <ffmpeg-

> > > >> devel@ffmpeg.org>

> > > >> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel

> > > format

> > > >>

> > > >> Am Do., 29. Aug. 2019 um 16:16 Uhr schrieb Fu, Linjie

> > > <linjie.fu@intel.com>:

> > > >>

> > > >>> Thus AYUV will make more sense.

> > > >>

> > > >> If your hardware decoding does not produce valid alpha data (0xFF

> > > >> for opaque, 0x00 for completely transparent), you cannot use an

> > > >> FFmpeg pix_fmt that defines alpha for decoding.

> > > >> If you define a pix_fmt that does not contain alpha information (as

> > > >> needed for your hardware decoder), please do not use "A" in its

> > > >> name.

> > > >

> > > > The hardware decode output is in packed 4:4:4:4 8 bit(32 bits in total).

> Apart>

> > > from Y/U/V channel, there is an additional need for a channel to store the

> > > > zero byte in surfaces(nb_components = 4). And it is designed in

> > > hardware/driver

> > > > for VA_FourCC_AYUV and AYUV format.

> > > >

> > > > It's not quite proper to use something like "0YUV" directly which

> indicates

> > > that

> > > > there is no Alpha channel data like"RGB0" or "0RGB"(nb_components =

> 3).

> > > >

> > > > "0YUV": YUV0/YUV0/...

> > > > "RGB0":RGB/RGB/...

> > >

> > > I think you've misread what RGB0 is.  The component step is 4, not 3 (see

> > > libavutil/pixdesc.c) - the format with a step of 3 is RGB24.

> > >

> > > RGBA:  R G B A R G B A ...

> > > RGB0:  R G B 0 R G B 0 ...

> > > RGB24: R G B R G B R G ...

> >

> > Thanks for pointing out this.

> > Just took the nb_components into account and misread the step info at

> first.

> >

> > > Following the same pattern we would have:

> > >

> > > AYUV:  A Y U V A Y U V ...

> > > 0YUV:  0 Y U V 0 Y U V ...

> > >

> > > To me it looks like 0YUV is exactly what you want here.

> >

> > Will update the patch if 0YUV is more acceptable.

> 

> Could you double-check if there is no bug in the decoder?

> I ask because transparency is possible with hevc and the

> relevant Windows format does support transparency from

> how I read the specification.

> 


Double confirmed.
The alpha channel is configurable on HCP_SURFACE_STATE command,
currently driver(iHD) set zero as default value.

"0YUV" is good enough, "AYUV" could be hold on.

- linjie
Linjie Fu Sept. 9, 2019, 3:19 p.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf

> Of Carl Eugen Hoyos

> Sent: Sunday, September 1, 2019 01:57

> To: FFmpeg development discussions and patches <ffmpeg-

> devel@ffmpeg.org>

> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel format

> 

> Am Sa., 31. Aug. 2019 um 19:25 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:

> >

> > > -----Original Message-----

> > > From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On

> Behalf

> > > Of Mark Thompson

> > > Sent: Saturday, August 31, 2019 23:10

> > > To: ffmpeg-devel@ffmpeg.org

> > > Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel

> format

> > >

> > > On 31/08/2019 05:22, Fu, Linjie wrote:

> > > >> -----Original Message-----

> > > >> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On

> > > Behalf

> > > >> Of Carl Eugen Hoyos

> > > >> Sent: Friday, August 30, 2019 00:12

> > > >> To: FFmpeg development discussions and patches <ffmpeg-

> > > >> devel@ffmpeg.org>

> > > >> Subject: Re: [FFmpeg-devel] [PATCH 1/2] lavu/pixfmt: add AYUV pixel

> > > format

> > > >>

> > > >> Am Do., 29. Aug. 2019 um 16:16 Uhr schrieb Fu, Linjie

> > > <linjie.fu@intel.com>:

> > > >>

> > > >>> Thus AYUV will make more sense.

> > > >>

> > > >> If your hardware decoding does not produce valid alpha data (0xFF

> > > >> for opaque, 0x00 for completely transparent), you cannot use an

> > > >> FFmpeg pix_fmt that defines alpha for decoding.

> > > >> If you define a pix_fmt that does not contain alpha information (as

> > > >> needed for your hardware decoder), please do not use "A" in its

> > > >> name.

> > > >

> > > > The hardware decode output is in packed 4:4:4:4 8 bit(32 bits in total).

> Apart>

> > > from Y/U/V channel, there is an additional need for a channel to store the

> > > > zero byte in surfaces(nb_components = 4). And it is designed in

> > > hardware/driver

> > > > for VA_FourCC_AYUV and AYUV format.

> > > >

> > > > It's not quite proper to use something like "0YUV" directly which

> indicates

> > > that

> > > > there is no Alpha channel data like"RGB0" or "0RGB"(nb_components =

> 3).

> > > >

> > > > "0YUV": YUV0/YUV0/...

> > > > "RGB0":RGB/RGB/...

> > >

> > > I think you've misread what RGB0 is.  The component step is 4, not 3 (see

> > > libavutil/pixdesc.c) - the format with a step of 3 is RGB24.

> > >

> > > RGBA:  R G B A R G B A ...

> > > RGB0:  R G B 0 R G B 0 ...

> > > RGB24: R G B R G B R G ...

> >

> > Thanks for pointing out this.

> > Just took the nb_components into account and misread the step info at

> first.

> >

> > > Following the same pattern we would have:

> > >

> > > AYUV:  A Y U V A Y U V ...

> > > 0YUV:  0 Y U V 0 Y U V ...

> > >

> > > To me it looks like 0YUV is exactly what you want here.

> >

> > Will update the patch if 0YUV is more acceptable.

> 

> Could you double-check if there is no bug in the decoder?

> I ask because transparency is possible with hevc and the

> relevant Windows format does support transparency from

> how I read the specification.

> 


Hi,

Rethink about this, hardware decoder should support a valid alpha data.
Default 0x00 or 0xFF  for alpha channel doesn't make sense in HEVC Rext
decoding.

Filed an issue in media driver:
https://github.com/intel/media-driver/issues/719

Will rebase and resend the patch set for QSV HEVC REXT decoding, basing on
AYUV,Y410, Y210.

Thanks,
Linjie

Patch hide | download patch | download mbox

diff --git a/libavutil/pixdesc.c b/libavutil/pixdesc.c
index b97b066..d217000 100644
--- a/libavutil/pixdesc.c
+++ b/libavutil/pixdesc.c
@@ -205,6 +205,19 @@  static const AVPixFmtDescriptor av_pix_fmt_descriptors[AV_PIX_FMT_NB] = {
             { 0, 4, 1, 0, 8, 3, 7, 2 },        /* V */
         },
     },
+    [AV_PIX_FMT_AYUV] = {
+        .name = "ayuv",
+        .nb_components = 4,
+        .log2_chroma_w = 0,
+        .log2_chroma_h = 0,
+        .comp = {
+            { 0, 4, 1, 0, 8, 3, 7, 2 },        /* Y */
+            { 0, 4, 2, 0, 8, 3, 7, 3 },        /* U */
+            { 0, 4, 3, 0, 8, 3, 7, 4 },        /* V */
+            { 0, 4, 0, 0, 8, 3, 7, 1 },        /* A */
+        },
+        .flags = AV_PIX_FMT_FLAG_ALPHA,
+    },
     [AV_PIX_FMT_RGB24] = {
         .name = "rgb24",
         .nb_components = 3,
diff --git a/libavutil/pixfmt.h b/libavutil/pixfmt.h
index 8b54c94..adbaec8 100644
--- a/libavutil/pixfmt.h
+++ b/libavutil/pixfmt.h
@@ -348,6 +348,8 @@  enum AVPixelFormat {
     AV_PIX_FMT_NV24,      ///< planar YUV 4:4:4, 24bpp, 1 plane for Y and 1 plane for the UV components, which are interleaved (first byte U and the following byte V)
     AV_PIX_FMT_NV42,      ///< as above, but U and V bytes are swapped
 
+    AV_PIX_FMT_AYUV,      ///< packed YUV 4:4:4, 32bpp,  A  Y Cb Cr,
+
     AV_PIX_FMT_NB         ///< number of pixel formats, DO NOT USE THIS if you want to link with shared libav* because the number of formats might differ between versions
 };
 
diff --git a/libavutil/tests/pixfmt_best.c b/libavutil/tests/pixfmt_best.c
index 53f7264..2939e48 100644
--- a/libavutil/tests/pixfmt_best.c
+++ b/libavutil/tests/pixfmt_best.c
@@ -91,6 +91,7 @@  int main(void)
     TEST(AV_PIX_FMT_YUVA420P,  AV_PIX_FMT_YUV420P);
     TEST(AV_PIX_FMT_YUVA422P,  AV_PIX_FMT_YUV422P);
     TEST(AV_PIX_FMT_YUVA444P,  AV_PIX_FMT_YUV444P);
+    TEST(AV_PIX_FMT_AYUV,      AV_PIX_FMT_YUV444P);
     TEST(AV_PIX_FMT_AYUV64,    AV_PIX_FMT_YUV444P16);
     TEST(AV_PIX_FMT_RGBA,      AV_PIX_FMT_RGB24);
     TEST(AV_PIX_FMT_ABGR,      AV_PIX_FMT_RGB24);
diff --git a/libavutil/version.h b/libavutil/version.h
index ecc6a7c..658a508 100644
--- a/libavutil/version.h
+++ b/libavutil/version.h
@@ -79,7 +79,7 @@ 
  */
 
 #define LIBAVUTIL_VERSION_MAJOR  56
-#define LIBAVUTIL_VERSION_MINOR  33
+#define LIBAVUTIL_VERSION_MINOR  34
 #define LIBAVUTIL_VERSION_MICRO 100
 
 #define LIBAVUTIL_VERSION_INT   AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \