[FFmpeg-devel,RFC,v2,2/3] libavcodec: Add thumbnail output to vaapi_h264 decoder

Submitted by Zachary Zhou on April 8, 2019, 8:53 a.m.

Details

Message ID 20190408085334.23947-2-zachary.zhou@intel.com
State New
Headers show

Commit Message

Zachary Zhou April 8, 2019, 8:53 a.m.
This is sample code for reference

HW support for decode+scaling in a single HW command (VDBOX+SFC).
The primary target usage is video analytics, but can be used playback,
transcoding, etc.

For VAAPI -
https://github.com/intel/libva
basically, it allows multiple outputs (in different resolutions) using the decode context in a single call (you can search for “additional_outputs” in va.h).

VAAPI sample code -
https://github.com/intel/libva-utils/commit/957a269f02b00760b7e807643c821ee26abc529b
---
 libavcodec/avcodec.h       |   8 +++
 libavcodec/decode.c        |  16 +++++
 libavcodec/options_table.h |   4 ++
 libavcodec/vaapi_decode.c  | 122 ++++++++++++++++++++++++++++++++++---
 libavcodec/vaapi_decode.h  |  30 +++++++++
 libavcodec/vaapi_h264.c    |  13 ++++
 6 files changed, 185 insertions(+), 8 deletions(-)

Comments

Hendrik Leppkes April 8, 2019, 9:27 a.m.
On Mon, Apr 8, 2019 at 10:54 AM Zachary Zhou <zachary.zhou@intel.com> wrote:
>
> This is sample code for reference
>
> HW support for decode+scaling in a single HW command (VDBOX+SFC).
> The primary target usage is video analytics, but can be used playback,
> transcoding, etc.
>
> For VAAPI -
> https://github.com/intel/libva
> basically, it allows multiple outputs (in different resolutions) using the decode context in a single call (you can search for “additional_outputs” in va.h).
>
> VAAPI sample code -
> https://github.com/intel/libva-utils/commit/957a269f02b00760b7e807643c821ee26abc529b
> ---
>  libavcodec/avcodec.h       |   8 +++
>  libavcodec/decode.c        |  16 +++++
>  libavcodec/options_table.h |   4 ++
>  libavcodec/vaapi_decode.c  | 122 ++++++++++++++++++++++++++++++++++---
>  libavcodec/vaapi_decode.h  |  30 +++++++++
>  libavcodec/vaapi_h264.c    |  13 ++++
>  6 files changed, 185 insertions(+), 8 deletions(-)
>
> diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
> index 0ce22ec4fa..36db21c0a5 100644
> --- a/libavcodec/avcodec.h
> +++ b/libavcodec/avcodec.h
> @@ -3357,6 +3357,14 @@ typedef struct AVCodecContext {
>       * - encoding: unused
>       */
>      int discard_damaged_percentage;
> +
> +    /*
> +     * Thumbnail options
> +     */
> +    int thumbnail_flags;
> +    int thumbnail_format;
> +    int thumbnail_width;
> +    int thumbnail_height;
>  } AVCodecContext;
>

Global fields for such a purpose seem not appropriate. We try to get
away from fields that only serve a single purpose with a single
decoder. Nevermind that they aren't even documented.

In general I must say I'm not sure I like this entire patchset. We
have component seperation for a reason, so trying to add magic
features like that is generally a bad idea.

- Hendrik
Mark Thompson April 9, 2019, 10:27 p.m.
On 08/04/2019 09:53, Zachary Zhou wrote:
> This is sample code for reference
> 
> HW support for decode+scaling in a single HW command (VDBOX+SFC).
> The primary target usage is video analytics, but can be used playback,
> transcoding, etc.
> 
> For VAAPI -
> https://github.com/intel/libva
> basically, it allows multiple outputs (in different resolutions) using the decode context in a single call (you can search for “additional_outputs” in va.h).
> 
> VAAPI sample code -
> https://github.com/intel/libva-utils/commit/957a269f02b00760b7e807643c821ee26abc529b
> ---
>  libavcodec/avcodec.h       |   8 +++
>  libavcodec/decode.c        |  16 +++++
>  libavcodec/options_table.h |   4 ++
>  libavcodec/vaapi_decode.c  | 122 ++++++++++++++++++++++++++++++++++---
>  libavcodec/vaapi_decode.h  |  30 +++++++++
>  libavcodec/vaapi_h264.c    |  13 ++++
>  6 files changed, 185 insertions(+), 8 deletions(-)

This doesn't fit into the libavcodec API at all.  If it is desirable to have some sort of generic multiple-output mechanism in libavcodec, please open a separate discussion considering that API specifically.

For the particular hardware instance you are talking about here, can you explain what this mechanism actually helps with?  My understanding was that the VDBOX / SFC chaining existed to save a small amount of power in restricted playback cases going directly to the display (that is, for maximising playback time on battery devices).  Since you are writing everything back to memory immediately here anyway, I don't see how it helps compared to just doing the scale as a filter operation.

- Mark
Zachary Zhou April 10, 2019, 1:22 a.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of

> Hendrik Leppkes

> Sent: Monday, April 8, 2019 5:27 PM

> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>

> Subject: Re: [FFmpeg-devel] [PATCH RFC v2 2/3] libavcodec: Add thumbnail

> output to vaapi_h264 decoder

> 

> On Mon, Apr 8, 2019 at 10:54 AM Zachary Zhou <zachary.zhou@intel.com>

> wrote:

> >

> > This is sample code for reference

> >

> > HW support for decode+scaling in a single HW command (VDBOX+SFC).

> > The primary target usage is video analytics, but can be used playback,

> > transcoding, etc.

> >

> > For VAAPI -

> > https://github.com/intel/libva

> > basically, it allows multiple outputs (in different resolutions) using the

> decode context in a single call (you can search for “additional_outputs” in

> va.h).

> >

> > VAAPI sample code -

> > https://github.com/intel/libva-utils/commit/957a269f02b00760b7e807643c

> > 821ee26abc529b

> > ---

> >  libavcodec/avcodec.h       |   8 +++

> >  libavcodec/decode.c        |  16 +++++

> >  libavcodec/options_table.h |   4 ++

> >  libavcodec/vaapi_decode.c  | 122

> > ++++++++++++++++++++++++++++++++++---

> >  libavcodec/vaapi_decode.h  |  30 +++++++++

> >  libavcodec/vaapi_h264.c    |  13 ++++

> >  6 files changed, 185 insertions(+), 8 deletions(-)

> >

> > diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h index

> > 0ce22ec4fa..36db21c0a5 100644

> > --- a/libavcodec/avcodec.h

> > +++ b/libavcodec/avcodec.h

> > @@ -3357,6 +3357,14 @@ typedef struct AVCodecContext {

> >       * - encoding: unused

> >       */

> >      int discard_damaged_percentage;

> > +

> > +    /*

> > +     * Thumbnail options

> > +     */

> > +    int thumbnail_flags;

> > +    int thumbnail_format;

> > +    int thumbnail_width;

> > +    int thumbnail_height;

> >  } AVCodecContext;

> >

> 

> Global fields for such a purpose seem not appropriate. We try to get away

> from fields that only serve a single purpose with a single decoder. Nevermind

> that they aren't even documented.

> 

> In general I must say I'm not sure I like this entire patchset. We have

> component seperation for a reason, so trying to add magic features like that is

> generally a bad idea.

> 

> - Hendrik


Thanks Hendrik for the comments. 
currently I didn't find any good places to set input parameters for decoder, in this feature I think the end user may want to set the scaling size and out format. 
And I don't know if ffmpeg will add better decoder input parameters control in future.
This feature will also save the scaling filter code writing for the end user.

-Zach 

> _______________________________________________

> ffmpeg-devel mailing list

> ffmpeg-devel@ffmpeg.org

> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

> 

> To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org

> with subject "unsubscribe".
Zachary Zhou April 10, 2019, 1:57 a.m.
> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of

> Mark Thompson

> Sent: Wednesday, April 10, 2019 6:27 AM

> To: ffmpeg-devel@ffmpeg.org

> Subject: Re: [FFmpeg-devel] [PATCH RFC v2 2/3] libavcodec: Add thumbnail

> output to vaapi_h264 decoder

> 

> On 08/04/2019 09:53, Zachary Zhou wrote:

> > This is sample code for reference

> >

> > HW support for decode+scaling in a single HW command (VDBOX+SFC).

> > The primary target usage is video analytics, but can be used playback,

> > transcoding, etc.

> >

> > For VAAPI -

> > https://github.com/intel/libva

> > basically, it allows multiple outputs (in different resolutions) using the

> decode context in a single call (you can search for “additional_outputs” in

> va.h).

> >

> > VAAPI sample code -

> > https://github.com/intel/libva-utils/commit/957a269f02b00760b7e807643c

> > 821ee26abc529b

> > ---

> >  libavcodec/avcodec.h       |   8 +++

> >  libavcodec/decode.c        |  16 +++++

> >  libavcodec/options_table.h |   4 ++

> >  libavcodec/vaapi_decode.c  | 122

> > ++++++++++++++++++++++++++++++++++---

> >  libavcodec/vaapi_decode.h  |  30 +++++++++

> >  libavcodec/vaapi_h264.c    |  13 ++++

> >  6 files changed, 185 insertions(+), 8 deletions(-)

> 

> This doesn't fit into the libavcodec API at all.  If it is desirable to have some

> sort of generic multiple-output mechanism in libavcodec, please open a

> separate discussion considering that API specifically.

> 

> For the particular hardware instance you are talking about here, can you

> explain what this mechanism actually helps with?  My understanding was that

> the VDBOX / SFC chaining existed to save a small amount of power in

> restricted playback cases going directly to the display (that is, for maximising

> playback time on battery devices).  Since you are writing everything back to

> memory immediately here anyway, I don't see how it helps compared to just

> doing the scale as a filter operation.

> 

> - Mark


Thanks Mark, I understand it doesn't fit the libavcodec API currently, and I will open a discussion regarding multiple-output mechanism in libavcodec.

you may already know following document -
https://01.org/sites/default/files/documentation/intel-gfx-prm-osrc-skl-vol15-sfc.pdf

As my investigation -
- This feature can save some power and memory bandwidth. 
- This feature can save the ffmpeg filters code writing for end user. it also support ARGB output for some AI use case, I think some AI cases may need ARGB input.

-Zach

> _______________________________________________

> ffmpeg-devel mailing list

> ffmpeg-devel@ffmpeg.org

> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

> 

> To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org

> with subject "unsubscribe".
Zhong Li April 10, 2019, 2:10 a.m.
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf

> Of Mark Thompson

> Sent: Wednesday, April 10, 2019 6:27 AM

> To: ffmpeg-devel@ffmpeg.org

> Subject: Re: [FFmpeg-devel] [PATCH RFC v2 2/3] libavcodec: Add thumbnail

> output to vaapi_h264 decoder

> 

> On 08/04/2019 09:53, Zachary Zhou wrote:

> > This is sample code for reference

> >

> > HW support for decode+scaling in a single HW command (VDBOX+SFC).

> > The primary target usage is video analytics, but can be used playback,

> > transcoding, etc.

> >

> > For VAAPI -

> > https://github.com/intel/libva

> > basically, it allows multiple outputs (in different resolutions) using the

> decode context in a single call (you can search for “additional_outputs” in

> va.h).

> >

> > VAAPI sample code -

> >

> https://github.com/intel/libva-utils/commit/957a269f02b00760b7e807643c

> > 821ee26abc529b

> > ---

> >  libavcodec/avcodec.h       |   8 +++

> >  libavcodec/decode.c        |  16 +++++

> >  libavcodec/options_table.h |   4 ++

> >  libavcodec/vaapi_decode.c  | 122

> > ++++++++++++++++++++++++++++++++++---

> >  libavcodec/vaapi_decode.h  |  30 +++++++++

> >  libavcodec/vaapi_h264.c    |  13 ++++

> >  6 files changed, 185 insertions(+), 8 deletions(-)

> 

> This doesn't fit into the libavcodec API at all.  If it is desirable to have some

> sort of generic multiple-output mechanism in libavcodec, please open a

> separate discussion considering that API specifically.

> 

> For the particular hardware instance you are talking about here, can you

> explain what this mechanism actually helps with?  My understanding was

> that the VDBOX / SFC chaining existed to save a small amount of power in

> restricted playback cases going directly to the display (that is, for maximising

> playback time on battery devices).  Since you are writing everything back to

> memory immediately here anyway, I don't see how it helps compared to just

> doing the scale as a filter operation.

> 

> - Mark


I would like to see some brief test data to show the benefit of this solution (e.g: power, performance, memory, ect) in the commit message. 
The comparison target should be "vaapi decoder in libavcodec + vaapi SFC scaling in libavfilter". 

"vaapi decoder in libavcodec + vaapi SFC scaling in libavfilter" path has been supported in FFmpeg, but an interesting thing is that I found that the "fast" mode is not SFC when working with iHD driver,
Instead VA_FILTER_SCALING_DEFAULT looks like SFC will take effect. More details please see: https://github.com/intel/media-driver/issues/582 
BTW, IMHO VA_FILTER_SCALING_FAST definition in vaapi, or "fast" mode named in FFmpeg is misleading, AVS mode is possible faster then SFC on high performance GPU. So would better to change it to be another name such as "low_power".
Zhong Li April 10, 2019, 2:14 a.m.
> As my investigation -

> - This feature can save some power and memory bandwidth.

> - This feature can save the ffmpeg filters code writing for end user. it also

> support ARGB output for some AI use case, I think some AI cases may need

> ARGB input.


IIUC, vaapi SFC scaling in libavfilter path should can support ARGB output too, am I wrong?

Patch hide | download patch | download mbox

diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
index 0ce22ec4fa..36db21c0a5 100644
--- a/libavcodec/avcodec.h
+++ b/libavcodec/avcodec.h
@@ -3357,6 +3357,14 @@  typedef struct AVCodecContext {
      * - encoding: unused
      */
     int discard_damaged_percentage;
+
+    /*
+     * Thumbnail options
+     */
+    int thumbnail_flags;
+    int thumbnail_format;
+    int thumbnail_width;
+    int thumbnail_height;
 } AVCodecContext;
 
 #if FF_API_CODEC_GET_SET
diff --git a/libavcodec/decode.c b/libavcodec/decode.c
index a32ff2fcd3..2107751197 100644
--- a/libavcodec/decode.c
+++ b/libavcodec/decode.c
@@ -1795,6 +1795,22 @@  FF_ENABLE_DEPRECATION_WARNINGS
         frame->channels = avctx->channels;
         break;
     }
+
+    //thumbnail side data
+    if (avctx->thumbnail_flags) {
+        AVFrameSideData *frame_thumbnail_sd = av_frame_new_side_data(frame,
+                                                                     AV_FRAME_DATA_THUMBNAIL_INFO,
+                                                                     sizeof(AVThumbnailInfo));
+        if (frame_thumbnail_sd) {
+            av_dict_set_int(&frame_thumbnail_sd->metadata, "thumbnail_flags", avctx->thumbnail_flags, 0);
+            av_dict_set_int(&frame_thumbnail_sd->metadata, "thumbnail_width", avctx->thumbnail_width, 0);
+            av_dict_set_int(&frame_thumbnail_sd->metadata, "thumbnail_height", avctx->thumbnail_height, 0);
+            av_dict_set_int(&frame_thumbnail_sd->metadata, "thumbnail_format", avctx->thumbnail_format, 0);
+
+            av_log(avctx, AV_LOG_DEBUG, "Thumbnail new side data\n");
+        }
+    }
+
     return 0;
 }
 
diff --git a/libavcodec/options_table.h b/libavcodec/options_table.h
index a3235bcd57..aa158f713c 100644
--- a/libavcodec/options_table.h
+++ b/libavcodec/options_table.h
@@ -480,6 +480,10 @@  static const AVOption avcodec_options[] = {
 {"allow_profile_mismatch", "attempt to decode anyway if HW accelerated decoder's supported profiles do not exactly match the stream", 0, AV_OPT_TYPE_CONST, {.i64 = AV_HWACCEL_FLAG_ALLOW_PROFILE_MISMATCH }, INT_MIN, INT_MAX, V | D, "hwaccel_flags"},
 {"extra_hw_frames", "Number of extra hardware frames to allocate for the user", OFFSET(extra_hw_frames), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, INT_MAX, V|D },
 {"discard_damaged_percentage", "Percentage of damaged samples to discard a frame", OFFSET(discard_damaged_percentage), AV_OPT_TYPE_INT, {.i64 = 95 }, 0, 100, V|D },
+{"thumbnail_flags", "set thumbnail flags", OFFSET(thumbnail_flags), AV_OPT_TYPE_INT, { .i64 = 1 }, 0, 1, V|D, "thumbnail"},
+{"thumbnail_format", "set thumbnail format", OFFSET(thumbnail_format), AV_OPT_TYPE_PIXEL_FMT, {.i64=AV_PIX_FMT_ARGB}, -1, INT_MAX, V|D, "thumbnail"},
+{"thumbnail_width", "set thumbnail width", OFFSET(thumbnail_width), AV_OPT_TYPE_INT, {.i64 = 480}, 0, INT_MAX, V|D, "thumbnail"},
+{"thumbnail_height", "set thumbnail height", OFFSET(thumbnail_height), AV_OPT_TYPE_INT, {.i64 = 360}, 0, INT_MAX, V|D, "thumbnail"},
 {NULL},
 };
 
diff --git a/libavcodec/vaapi_decode.c b/libavcodec/vaapi_decode.c
index 69512e1d45..2e9e762dc0 100644
--- a/libavcodec/vaapi_decode.c
+++ b/libavcodec/vaapi_decode.c
@@ -145,6 +145,17 @@  static void ff_vaapi_decode_destroy_buffers(AVCodecContext *avctx,
                    pic->slice_buffers[i], vas, vaErrorStr(vas));
         }
     }
+
+    //destory thumbnail buffer
+    if (avctx->thumbnail_flags) {
+        vas = vaDestroyBuffer(ctx->hwctx->display,
+                              pic->thumbnail_buffer);
+        if (vas != VA_STATUS_SUCCESS) {
+            av_log(avctx, AV_LOG_ERROR, "Failed to destroy thumbnail "
+                   "buffer %#x: %d (%s).\n",
+                   pic->thumbnail_buffer, vas, vaErrorStr(vas));
+        }
+    }
 }
 
 int ff_vaapi_decode_issue(AVCodecContext *avctx,
@@ -184,6 +195,56 @@  int ff_vaapi_decode_issue(AVCodecContext *avctx,
         goto fail_with_picture;
     }
 
+    //add thumbnail buffer
+    if (avctx->thumbnail_flags) {
+        VARectangle   rect_src;          /**< @brief Rectangle for source input */
+        VARectangle   rect_thumbnail;    /**< @brief Rectangle for thumbnail output */
+        VAProcPipelineParameterBuffer buffer;
+
+        memset(&rect_src, 0, sizeof(rect_src));
+        memset(&rect_thumbnail, 0, sizeof(rect_thumbnail));
+        memset(&buffer, 0, sizeof(buffer));
+
+        rect_src.x = rect_src.y = 0;
+        rect_src.width = (uint16_t)pic->thumbnail_src_width;
+        rect_src.height = (uint16_t)pic->thumbnail_src_height;
+
+        rect_thumbnail.x = rect_src.y = 0;
+        rect_thumbnail.width = (uint16_t)avctx->thumbnail_width;
+        rect_thumbnail.height = (uint16_t)avctx->thumbnail_height;
+
+        buffer.surface_region = &rect_src;
+        buffer.output_region = &rect_thumbnail;
+        buffer.additional_outputs = (VASurfaceID*)&(pic->thumbnail_output_surface);
+        buffer.num_additional_outputs = 1;
+
+        vas = vaCreateBuffer(ctx->hwctx->display, ctx->va_context,
+                                  VAProcPipelineParameterBufferType,
+                                  sizeof(buffer),
+                                  1,
+                                  (uint8_t*)&buffer,
+                                  &pic->thumbnail_buffer);
+        if (vas != VA_STATUS_SUCCESS) {
+            av_log(avctx, AV_LOG_ERROR, "Failed to create thumbnail buffer: "
+                   "%d (%s).\n", vas, vaErrorStr(vas));
+            err = AVERROR(EIO);
+            goto fail_with_picture;
+        }
+
+        av_log(avctx, AV_LOG_DEBUG, "H264 create thumbnail buffer.\n");
+
+        vas = vaRenderPicture(ctx->hwctx->display, ctx->va_context,
+                              &pic->thumbnail_buffer, 1);
+        if (vas != VA_STATUS_SUCCESS) {
+            av_log(avctx, AV_LOG_ERROR, "Failed to upload sfc: "
+                   "%d (%s).\n", vas, vaErrorStr(vas));
+            err = AVERROR(EIO);
+            goto fail_with_picture;
+        }
+
+        av_log(avctx, AV_LOG_DEBUG, "H264 render thumbnail buffer.\n");
+    }
+
     vas = vaEndPicture(ctx->hwctx->display, ctx->va_context);
     if (vas != VA_STATUS_SUCCESS) {
         av_log(avctx, AV_LOG_ERROR, "Failed to end picture decode "
@@ -493,14 +554,49 @@  static int vaapi_decode_make_config(AVCodecContext *avctx,
         }
     }
 
-    vas = vaCreateConfig(hwctx->display, matched_va_profile,
-                         VAEntrypointVLD, NULL, 0,
-                         va_config);
-    if (vas != VA_STATUS_SUCCESS) {
-        av_log(avctx, AV_LOG_ERROR, "Failed to create decode "
-               "configuration: %d (%s).\n", vas, vaErrorStr(vas));
-        err = AVERROR(EIO);
-        goto fail;
+    if (avctx->thumbnail_flags) {
+        VAConfigAttrib attrib;
+
+        memset(&attrib, 0, sizeof(attrib));
+        attrib.type = VAConfigAttribDecProcessing;
+        attrib.value = 0;
+
+        vaGetConfigAttributes(
+            hwctx->display,
+            matched_va_profile,
+            VAEntrypointVLD,
+            &attrib,
+            1);
+
+        if (attrib.value != VA_DEC_PROCESSING) {
+            err =  AVERROR(EINVAL);
+            goto fail;
+        }
+
+        av_log(avctx, AV_LOG_DEBUG, "vaapi driver supports VDSFC\n");
+
+        attrib.type = VAConfigAttribDecProcessing;
+        attrib.value = VA_DEC_PROCESSING;
+
+        vas = vaCreateConfig(hwctx->display, matched_va_profile,
+                             VAEntrypointVLD, &attrib, 1,
+                             va_config);
+        if (vas != VA_STATUS_SUCCESS) {
+            av_log(avctx, AV_LOG_ERROR, "Failed to create decode "
+                   "configuration: %d (%s).\n", vas, vaErrorStr(vas));
+            err = AVERROR(EIO);
+            goto fail;
+        }
+    } else {
+        vas = vaCreateConfig(hwctx->display, matched_va_profile,
+                             VAEntrypointVLD, NULL, 0,
+                             va_config);
+        if (vas != VA_STATUS_SUCCESS) {
+            av_log(avctx, AV_LOG_ERROR, "Failed to create decode "
+                   "configuration: %d (%s).\n", vas, vaErrorStr(vas));
+            err = AVERROR(EIO);
+            goto fail;
+        }
     }
 
     hwconfig = av_hwdevice_hwconfig_alloc(device_ref);
@@ -587,6 +683,7 @@  int ff_vaapi_common_frame_params(AVCodecContext *avctx,
                                  AVBufferRef *hw_frames_ctx)
 {
     AVHWFramesContext *hw_frames = (AVHWFramesContext *)hw_frames_ctx->data;
+    AVVAAPIFramesContext *avfc = hw_frames->hwctx;
     AVHWDeviceContext *device_ctx = hw_frames->device_ctx;
     AVVAAPIDeviceContext *hwctx;
     VAConfigID va_config = VA_INVALID_ID;
@@ -604,6 +701,15 @@  int ff_vaapi_common_frame_params(AVCodecContext *avctx,
     if (va_config != VA_INVALID_ID)
         vaDestroyConfig(hwctx->display, va_config);
 
+    if (avctx->thumbnail_flags) {
+        avfc->thumbnail_flags = avctx->thumbnail_flags;
+        avfc->thumbnail_format = avctx->thumbnail_format;
+        avfc->thumbnail_width = avctx->thumbnail_width;
+        avfc->thumbnail_height = avctx->thumbnail_height;
+
+        av_log(avctx, AV_LOG_DEBUG, "VDSFC save thumbnail info avfc\n");
+    }
+
     return 0;
 }
 
diff --git a/libavcodec/vaapi_decode.h b/libavcodec/vaapi_decode.h
index 6b415dd1d3..f70658153a 100644
--- a/libavcodec/vaapi_decode.h
+++ b/libavcodec/vaapi_decode.h
@@ -37,6 +37,29 @@  static inline VASurfaceID ff_vaapi_get_surface_id(AVFrame *pic)
     return (uintptr_t)pic->data[3];
 }
 
+static inline VASurfaceID ff_vaapi_get_thumbnail_surface_id(AVFrame *pic)
+{
+    VASurfaceID thumbnail_surface_id = VA_INVALID_ID;
+
+    AVFrameSideData *frame_thumbnail_sd = av_frame_get_side_data(pic,
+                                                           AV_FRAME_DATA_THUMBNAIL_INFO);
+    if (frame_thumbnail_sd) {
+        thumbnail_surface_id = (uintptr_t)frame_thumbnail_sd->buf->data;
+    }
+
+    return thumbnail_surface_id;
+}
+
+static inline int ff_vaapi_get_thumbnail_src_width(AVFrame *pic)
+{
+    return pic->width;
+}
+
+static inline int ff_vaapi_get_thumbnail_src_height(AVFrame *pic)
+{
+    return pic->height;
+}
+
 enum {
     MAX_PARAM_BUFFERS = 16,
 };
@@ -50,6 +73,13 @@  typedef struct VAAPIDecodePicture {
     int                nb_slices;
     VABufferID           *slice_buffers;
     int                   slices_allocated;
+
+    //thumbnail info
+    VASurfaceID           thumbnail_output_surface;
+    int                   thumbnail_src_width;
+    int                   thumbnail_src_height;
+    VABufferID            thumbnail_buffer;
+
 } VAAPIDecodePicture;
 
 typedef struct VAAPIDecodeContext {
diff --git a/libavcodec/vaapi_h264.c b/libavcodec/vaapi_h264.c
index 5854587a25..11919cafa4 100644
--- a/libavcodec/vaapi_h264.c
+++ b/libavcodec/vaapi_h264.c
@@ -302,6 +302,19 @@  static int vaapi_h264_start_frame(AVCodecContext          *avctx,
     if (err < 0)
         goto fail;
 
+    //get thumbnail surface id
+    if (avctx->thumbnail_flags) {
+        pic->thumbnail_output_surface = ff_vaapi_get_thumbnail_surface_id(h->cur_pic_ptr->f);
+
+        //get thumbnail src width and height
+        pic->thumbnail_src_width = ff_vaapi_get_thumbnail_src_width(h->cur_pic_ptr->f);
+        pic->thumbnail_src_height = ff_vaapi_get_thumbnail_src_height(h->cur_pic_ptr->f);
+
+        av_log(avctx, AV_LOG_DEBUG, "H264 get thumbnail surface id: %d\n", pic->thumbnail_output_surface);
+        av_log(avctx, AV_LOG_DEBUG, "H264 get thumbnail width: %d\n", pic->thumbnail_src_width);
+        av_log(avctx, AV_LOG_DEBUG, "H264 get thumbnail height: %d\n", pic->thumbnail_src_height);
+    }
+
     return 0;
 
 fail: