[FFmpeg-devel,1/3] avutil: Add NV24 and NV42 pixel formats

Submitted by Philip Langdale on May 11, 2019, 6:31 p.m.

Details

Message ID 20190511183157.27909-2-philipl@overt.org
State New
Headers show

Commit Message

Philip Langdale May 11, 2019, 6:31 p.m.
These are the 4:4:4 variants of the semi-planar NV12/NV21 formats.

These formats are not used much, so we've never had a reason to add
them until now. VDPAU recently added support HEVC 4:4:4 content
and when you use the OpenGL interop, the returned surfaces are in
NV24 format, so we need the pixel format for media players, even
if there's no direct use within ffmpeg.

Separately, there are apparently webcams that use NV24, but I've
never seen one.

Signed-off-by: Philip Langdale <philipl@overt.org>
---
 libavutil/pixdesc.c           | 24 ++++++++++++++++++++++++
 libavutil/pixfmt.h            |  3 +++
 libavutil/tests/pixfmt_best.c |  1 +
 libavutil/version.h           |  2 +-
 4 files changed, 29 insertions(+), 1 deletion(-)

Comments

Carl Eugen Hoyos May 11, 2019, 10:25 p.m.
Am Sa., 11. Mai 2019 um 20:32 Uhr schrieb Philip Langdale <philipl@overt.org>:
>
> These are the 4:4:4 variants of the semi-planar NV12/NV21 formats.
>
> These formats are not used much, so we've never had a reason to add
> them until now. VDPAU recently added support HEVC 4:4:4 content
> and when you use the OpenGL interop, the returned surfaces are in

I wonder if there is an "only" missing somewhere.

> NV24 format, so we need the pixel format for media players, even
> if there's no direct use within ffmpeg.

Thank you!

> Separately, there are apparently webcams that use NV24, but I've
> never seen one.

[...]

> diff --git a/libavutil/version.h b/libavutil/version.h
> index c0968de621..4922e267cc 100644
> --- a/libavutil/version.h
> +++ b/libavutil/version.h
> @@ -80,7 +80,7 @@
>
>  #define LIBAVUTIL_VERSION_MAJOR  56
>  #define LIBAVUTIL_VERSION_MINOR  26
> -#define LIBAVUTIL_VERSION_MICRO 101
> +#define LIBAVUTIL_VERSION_MICRO 102

We typically raise minor for new pix_fmts, resetting
micro was forgotten the last times afaict.

Carl Eugen
Linjie Fu May 12, 2019, 3:38 a.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf

> Of Carl Eugen Hoyos

> Sent: Sunday, May 12, 2019 06:25

> To: FFmpeg development discussions and patches <ffmpeg-

> devel@ffmpeg.org>

> Subject: Re: [FFmpeg-devel] [PATCH 1/3] avutil: Add NV24 and NV42 pixel

> formats

> 

> Am Sa., 11. Mai 2019 um 20:32 Uhr schrieb Philip Langdale

> <philipl@overt.org>:

> >

> > These are the 4:4:4 variants of the semi-planar NV12/NV21 formats.

> >

> > These formats are not used much, so we've never had a reason to add

> > them until now. VDPAU recently added support HEVC 4:4:4 content

> > and when you use the OpenGL interop, the returned surfaces are in

> 

> I wonder if there is an "only" missing somewhere.

> 

> > NV24 format, so we need the pixel format for media players, even

> > if there's no direct use within ffmpeg.

> 

> Thank you!

> 

> > Separately, there are apparently webcams that use NV24, but I've

> > never seen one.

> 

> [...]

> 

> > diff --git a/libavutil/version.h b/libavutil/version.h

> > index c0968de621..4922e267cc 100644

> > --- a/libavutil/version.h

> > +++ b/libavutil/version.h

> > @@ -80,7 +80,7 @@

> >

> >  #define LIBAVUTIL_VERSION_MAJOR  56

> >  #define LIBAVUTIL_VERSION_MINOR  26

> > -#define LIBAVUTIL_VERSION_MICRO 101

> > +#define LIBAVUTIL_VERSION_MICRO 102

> 

> We typically raise minor for new pix_fmts, resetting

> micro was forgotten the last times afaict.

> 


Hi,

I'm working on adding support for HEVC 4:2:2/ 4:4:4, 8/10 bit, in vaapi decoding.
And I'm aslo thinking of add some new pix_fmt for packed FourCC:
444 8 bit :  AYUV
444 10 bit: Y410 
422 10 bit: Y210

So if I'd like to add these new pix_fmts, some modification is necessary too as far as I got from relevant threads:
1. add support in swscale for being able to convert to and from a format, including up and down scale;
2. add test.(maybe a query is needed too)
3. bump a new minor version.
Any other concerns should be followed?

I'm trying to make sure I understood this correctly.
Thanks in advance.

Linjie
Carl Eugen Hoyos May 12, 2019, 10:44 a.m.
Am So., 12. Mai 2019 um 05:38 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:

> I'm working on adding support for HEVC 4:2:2/ 4:4:4, 8/10 bit, in vaapi decoding.
> And I'm aslo thinking of add some new pix_fmt for packed FourCC:
> 444 8 bit :  AYUV
> 444 10 bit: Y410
> 422 10 bit: Y210

Do I understand correctly that for 420 8 bit, vaapi produces the
decoded image in a planar format but for 444 8 bit (and 10 bit),
you decided that a packed format makes more sense?
(Aren't luma and chroma stored independently in hevc?)

Note that no matter what you do with the decoded image, there
will be a (measurable) performance hit because of the needed
transformation into something (planar) that other filters and
encoders understand.

Carl Eugen
Linjie Fu May 13, 2019, 2:13 a.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf

> Of Carl Eugen Hoyos

> Sent: Sunday, May 12, 2019 18:45

> To: FFmpeg development discussions and patches <ffmpeg-

> devel@ffmpeg.org>

> Subject: Re: [FFmpeg-devel] [PATCH 1/3] avutil: Add NV24 and NV42 pixel

> formats

> 

> Am So., 12. Mai 2019 um 05:38 Uhr schrieb Fu, Linjie <linjie.fu@intel.com>:

> 

> > I'm working on adding support for HEVC 4:2:2/ 4:4:4, 8/10 bit, in vaapi

> decoding.

> > And I'm aslo thinking of add some new pix_fmt for packed FourCC:

> > 444 8 bit :  AYUV

> > 444 10 bit: Y410

> > 422 10 bit: Y210

> 

> Do I understand correctly that for 420 8 bit, vaapi produces the

> decoded image in a planar format but for 444 8 bit (and 10 bit),

> you decided that a packed format makes more sense?

> (Aren't luma and chroma stored independently in hevc?)

> 


It's true that for 420 8 bit, the decoded image is in planar format.
But as to 444 8 bit(and 10 bit), as far as I know in  media driver , the decoded image is produced in packed format.
See:
https://github.com/intel/media-driver/blob/8773ef5e82e8a151e1bcfef317f7808f319c91c2/media_driver/linux/gen11/ddi/media_libva_caps_g11.cpp#L842

        else if(profile == VAProfileHEVCMain444 || profile == VAProfileVP9Profile1)
        {
            attribs[i].type = VASurfaceAttribPixelFormat;
            attribs[i].value.type = VAGenericValueTypeInteger;
            attribs[i].flags = VA_SURFACE_ATTRIB_GETTABLE | VA_SURFACE_ATTRIB_SETTABLE;
            attribs[i].value.value.i = VA_FOURCC_AYUV;
            i++;
        }

> Note that no matter what you do with the decoded image, there

> will be a (measurable) performance hit because of the needed

> transformation into something (planar) that other filters and

> encoders understand.


Agree, the directly decoded format is better. 
And it's also one of the reasons I'd like to add these packed formats.

Linjie

Patch hide | download patch | download mbox

diff --git a/libavutil/pixdesc.c b/libavutil/pixdesc.c
index fe38344d73..b97b0665b0 100644
--- a/libavutil/pixdesc.c
+++ b/libavutil/pixdesc.c
@@ -2320,6 +2320,30 @@  static const AVPixFmtDescriptor av_pix_fmt_descriptors[AV_PIX_FMT_NB] = {
         },
         .flags = AV_PIX_FMT_FLAG_PLANAR | AV_PIX_FMT_FLAG_ALPHA,
     },
+    [AV_PIX_FMT_NV24] = {
+        .name = "nv24",
+        .nb_components = 3,
+        .log2_chroma_w = 0,
+        .log2_chroma_h = 0,
+        .comp = {
+            { 0, 1, 0, 0, 8, 0, 7, 1 },        /* Y */
+            { 1, 2, 0, 0, 8, 1, 7, 1 },        /* U */
+            { 1, 2, 1, 0, 8, 1, 7, 2 },        /* V */
+        },
+        .flags = AV_PIX_FMT_FLAG_PLANAR,
+    },
+    [AV_PIX_FMT_NV42] = {
+        .name = "nv42",
+        .nb_components = 3,
+        .log2_chroma_w = 0,
+        .log2_chroma_h = 0,
+        .comp = {
+            { 0, 1, 0, 0, 8, 0, 7, 1 },        /* Y */
+            { 1, 2, 1, 0, 8, 1, 7, 2 },        /* U */
+            { 1, 2, 0, 0, 8, 1, 7, 1 },        /* V */
+        },
+        .flags = AV_PIX_FMT_FLAG_PLANAR,
+    },
 };
 #if FF_API_PLUS1_MINUS1
 FF_ENABLE_DEPRECATION_WARNINGS
diff --git a/libavutil/pixfmt.h b/libavutil/pixfmt.h
index 24d1b7e415..8b54c9415b 100644
--- a/libavutil/pixfmt.h
+++ b/libavutil/pixfmt.h
@@ -345,6 +345,9 @@  enum AVPixelFormat {
     AV_PIX_FMT_YUVA444P12BE, ///< planar YUV 4:4:4,36bpp, (1 Cr & Cb sample per 1x1 Y samples), 12b alpha, big-endian
     AV_PIX_FMT_YUVA444P12LE, ///< planar YUV 4:4:4,36bpp, (1 Cr & Cb sample per 1x1 Y samples), 12b alpha, little-endian
 
+    AV_PIX_FMT_NV24,      ///< planar YUV 4:4:4, 24bpp, 1 plane for Y and 1 plane for the UV components, which are interleaved (first byte U and the following byte V)
+    AV_PIX_FMT_NV42,      ///< as above, but U and V bytes are swapped
+
     AV_PIX_FMT_NB         ///< number of pixel formats, DO NOT USE THIS if you want to link with shared libav* because the number of formats might differ between versions
 };
 
diff --git a/libavutil/tests/pixfmt_best.c b/libavutil/tests/pixfmt_best.c
index e98fcc19a5..53f7264207 100644
--- a/libavutil/tests/pixfmt_best.c
+++ b/libavutil/tests/pixfmt_best.c
@@ -76,6 +76,7 @@  int main(void)
     TEST(AV_PIX_FMT_P010,      AV_PIX_FMT_YUV420P10);
     TEST(AV_PIX_FMT_P016,      AV_PIX_FMT_YUV420P16);
     TEST(AV_PIX_FMT_NV16,      AV_PIX_FMT_YUV422P);
+    TEST(AV_PIX_FMT_NV24,      AV_PIX_FMT_YUV444P);
     TEST(AV_PIX_FMT_YUYV422,   AV_PIX_FMT_YUV422P);
     TEST(AV_PIX_FMT_UYVY422,   AV_PIX_FMT_YUV422P);
     TEST(AV_PIX_FMT_BGR565,    AV_PIX_FMT_RGB565);
diff --git a/libavutil/version.h b/libavutil/version.h
index c0968de621..4922e267cc 100644
--- a/libavutil/version.h
+++ b/libavutil/version.h
@@ -80,7 +80,7 @@ 
 
 #define LIBAVUTIL_VERSION_MAJOR  56
 #define LIBAVUTIL_VERSION_MINOR  26
-#define LIBAVUTIL_VERSION_MICRO 101
+#define LIBAVUTIL_VERSION_MICRO 102
 
 #define LIBAVUTIL_VERSION_INT   AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \
                                                LIBAVUTIL_VERSION_MINOR, \