| Message ID | 1586752352-3030-1-git-send-email-linjie.fu@intel.com |
|---|---|
| State | New |
| Headers | show |
| Series | [FFmpeg-devel] lavc/vaapi_encode_h265: add low_delay_b option for HEVC | expand |
| Context | Check | Description |
|---|---|---|
| andriy/default | pending | |
| andriy/make | success | Make finished |
| andriy/make_fate | success | Make fate finished |
On Mon, Apr 13, 2020 at 12:39 PM Linjie Fu <linjie.fu@intel.com> wrote: > > Low delay B-frame is supported on ICL+ platform. > > For low power encoding, low_delay_b should be enabled by default. > > Low delay B: > <http://what-when-how.com/Tutorial/topic-397pct9eq3/High-Efficiency-Video-Coding-HEVC-288.html> > > There is an on-going work in libva and media-driver to add querys > support for low delay b, would add it once it's ready: > https://github.com/intel/libva/pull/220 > https://github.com/intel/libva/pull/364 > https://github.com/intel/media-driver/issues/721 > > Signed-off-by: Linjie Fu <linjie.fu@intel.com> > --- > doc/encoders.texi | 8 ++++++++ > libavcodec/vaapi_encode_h265.c | 19 +++++++++++++++++-- > 2 files changed, 25 insertions(+), 2 deletions(-) > > diff --git a/doc/encoders.texi b/doc/encoders.texi > index e23b6b3..b0812be 100644 > --- a/doc/encoders.texi > +++ b/doc/encoders.texi > @@ -3089,6 +3089,14 @@ Some combination of the following values: > Include HDR metadata if the input frames have it > (@emph{mastering_display_colour_volume} and @emph{content_light_level} > messages). > + > +@item low_delay_b > +Use low delay B-frames instead of P frames. Reordering of pictures is > +not allowed. The first picture is encoded as an I picture and subsequent > +pictures are encoded as B pictures. Moreover, since past B pictures are > +used for prediction, a low coding delay but with higher coding efficiency > +(because of bi-prediction) is achieved. > + > @end table > > @end table > diff --git a/libavcodec/vaapi_encode_h265.c b/libavcodec/vaapi_encode_h265.c > index 97dc5a7..cd48545 100644 > --- a/libavcodec/vaapi_encode_h265.c > +++ b/libavcodec/vaapi_encode_h265.c > @@ -62,6 +62,7 @@ typedef struct VAAPIEncodeH265Context { > int tier; > int level; > int sei; > + int low_delay_b; > > // Derived settings. > int fixed_qp_idr; > @@ -894,6 +895,9 @@ static int vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, > > sh->slice_type = hpic->slice_type; > > + if (sh->slice_type == HEVC_SLICE_P && priv->low_delay_b) > + sh->slice_type = HEVC_SLICE_B; Add a trace or info message in this, it's will help to reduce the sudden surprises, the other question is, if enable the low_delay_b in ICL- platform, what is the result of this? > + > sh->slice_pic_order_cnt_lsb = hpic->pic_order_cnt & > (1 << (sps->log2_max_pic_order_cnt_lsb_minus4 + 4)) - 1; > > @@ -1054,9 +1058,13 @@ static int vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, > vslice->ref_pic_list0[0] = vpic->reference_frames[0]; > } > if (pic->nb_refs >= 2) { > - // Forward reference for B-frame. > av_assert0(pic->type == PICTURE_TYPE_B); > - vslice->ref_pic_list1[0] = vpic->reference_frames[1]; > + if (priv->low_delay_b) > + // Reference for low delay B-frame > + vslice->ref_pic_list1[0] = vpic->reference_frames[0]; > + else > + // Forward reference for B-frame. > + vslice->ref_pic_list1[0] = vpic->reference_frames[1]; > } > > return 0; > @@ -1181,6 +1189,11 @@ static av_cold int vaapi_encode_h265_init(AVCodecContext *avctx) > if (priv->qp > 0) > ctx->explicit_qp = priv->qp; > > + /* low_delay_b is required for low power encoding */ > + priv->low_delay_b = ctx->low_power ? 1 : priv->low_delay_b; > + if (priv->low_delay_b) > + av_log(avctx, AV_LOG_VERBOSE, "Low delay B-frame enabled.\n"); > + > return ff_vaapi_encode_init(avctx); > } > > @@ -1256,6 +1269,8 @@ static const AVOption vaapi_encode_h265_options[] = { > 0, AV_OPT_TYPE_CONST, > { .i64 = SEI_MASTERING_DISPLAY | SEI_CONTENT_LIGHT_LEVEL }, > INT_MIN, INT_MAX, FLAGS, "sei" }, > + { "low_delay_b", "Use low delay B frames instead of P frames", > + OFFSET(low_delay_b), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, FLAGS }, > > { NULL }, > }; > -- > 2.7.4 > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> From: mypopy@gmail.com <mypopy@gmail.com> > Sent: Monday, April 13, 2020 15:43 > To: FFmpeg development discussions and patches <ffmpeg- > devel@ffmpeg.org> > Cc: Fu, Linjie <linjie.fu@intel.com> > Subject: Re: [FFmpeg-devel] [PATCH] lavc/vaapi_encode_h265: add > low_delay_b option for HEVC > > On Mon, Apr 13, 2020 at 12:39 PM Linjie Fu <linjie.fu@intel.com> wrote: > > > > Low delay B-frame is supported on ICL+ platform. > > > > For low power encoding, low_delay_b should be enabled by default. > > > > Low delay B: > > <http://what-when-how.com/Tutorial/topic-397pct9eq3/High-Efficiency- > Video-Coding-HEVC-288.html> > > > > There is an on-going work in libva and media-driver to add querys > > support for low delay b, would add it once it's ready: > > https://github.com/intel/libva/pull/220 > > https://github.com/intel/libva/pull/364 > > https://github.com/intel/media-driver/issues/721 > > > > Signed-off-by: Linjie Fu <linjie.fu@intel.com> > > --- > > doc/encoders.texi | 8 ++++++++ > > libavcodec/vaapi_encode_h265.c | 19 +++++++++++++++++-- > > 2 files changed, 25 insertions(+), 2 deletions(-) > > > > diff --git a/doc/encoders.texi b/doc/encoders.texi > > index e23b6b3..b0812be 100644 > > --- a/doc/encoders.texi > > +++ b/doc/encoders.texi > > @@ -3089,6 +3089,14 @@ Some combination of the following values: > > Include HDR metadata if the input frames have it > > (@emph{mastering_display_colour_volume} and > @emph{content_light_level} > > messages). > > + > > +@item low_delay_b > > +Use low delay B-frames instead of P frames. Reordering of pictures is > > +not allowed. The first picture is encoded as an I picture and subsequent > > +pictures are encoded as B pictures. Moreover, since past B pictures are > > +used for prediction, a low coding delay but with higher coding efficiency > > +(because of bi-prediction) is achieved. > > + > > @end table > > > > @end table > > diff --git a/libavcodec/vaapi_encode_h265.c > b/libavcodec/vaapi_encode_h265.c > > index 97dc5a7..cd48545 100644 > > --- a/libavcodec/vaapi_encode_h265.c > > +++ b/libavcodec/vaapi_encode_h265.c > > @@ -62,6 +62,7 @@ typedef struct VAAPIEncodeH265Context { > > int tier; > > int level; > > int sei; > > + int low_delay_b; > > > > // Derived settings. > > int fixed_qp_idr; > > @@ -894,6 +895,9 @@ static int > vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, > > > > sh->slice_type = hpic->slice_type; > > > > + if (sh->slice_type == HEVC_SLICE_P && priv->low_delay_b) > > + sh->slice_type = HEVC_SLICE_B; > Add a trace or info message in this, it's will help to reduce the > sudden surprises, Yes, considered this and the trace/info message for low_delay_b change was added in vaapi_encode_h265_init() as a AV_LOG_VERBOSE Information, so I didn't prompt a debug information for each slice. > the other question is, if enable the low_delay_b in > ICL- platform, what is the result of this? Possibly leads to encoding failure. And that's why a query support is necessary for this option/feature on ICL-. While replying this mail, I noticed this patch is insufficient. Will update a new one, sorry for the noise. - Linjie
On 13/04/2020 05:32, Linjie Fu wrote: > Low delay B-frame is supported on ICL+ platform. > > For low power encoding, low_delay_b should be enabled by default. > > Low delay B: > <http://what-when-how.com/Tutorial/topic-397pct9eq3/High-Efficiency-Video-Coding-HEVC-288.html> > > There is an on-going work in libva and media-driver to add querys > support for low delay b, would add it once it's ready: > https://github.com/intel/libva/pull/220 > https://github.com/intel/libva/pull/364 > https://github.com/intel/media-driver/issues/721 > > Signed-off-by: Linjie Fu <linjie.fu@intel.com> > --- > doc/encoders.texi | 8 ++++++++ > libavcodec/vaapi_encode_h265.c | 19 +++++++++++++++++-- > 2 files changed, 25 insertions(+), 2 deletions(-) My understanding was that the only reason for this stuff existing was to work around broken hardware which didn't support P frames. Is this no longer true? > > diff --git a/doc/encoders.texi b/doc/encoders.texi > index e23b6b3..b0812be 100644 > --- a/doc/encoders.texi > +++ b/doc/encoders.texi > @@ -3089,6 +3089,14 @@ Some combination of the following values: > Include HDR metadata if the input frames have it > (@emph{mastering_display_colour_volume} and @emph{content_light_level} > messages). > + > +@item low_delay_b > +Use low delay B-frames instead of P frames. Reordering of pictures is > +not allowed. The first picture is encoded as an I picture and subsequent > +pictures are encoded as B pictures. Moreover, since past B pictures are > +used for prediction, a low coding delay but with higher coding efficiency > +(because of bi-prediction) is achieved. This sounds like a marketing blurb. Not for the manual, but can you explain in detail what might actually make this better, with actual numbers if possible? Intuitively the coding efficiency will be worse, because a number of the B-picture syntax elements are just redundant but still have to be coded (picking between two options which are actually identical). The gain of allowing biprediction between two identical pictures doesn't seem like a useful feature. - Mark
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > Mark Thompson > Sent: Monday, April 13, 2020 20:20 > To: ffmpeg-devel@ffmpeg.org > Subject: Re: [FFmpeg-devel] [PATCH] lavc/vaapi_encode_h265: add > low_delay_b option for HEVC > > On 13/04/2020 05:32, Linjie Fu wrote: > > Low delay B-frame is supported on ICL+ platform. > > > > For low power encoding, low_delay_b should be enabled by default. > > > > Low delay B: > > <http://what-when-how.com/Tutorial/topic-397pct9eq3/High-Efficiency- > Video-Coding-HEVC-288.html> > > > > There is an on-going work in libva and media-driver to add querys > > support for low delay b, would add it once it's ready: > > https://github.com/intel/libva/pull/220 > > https://github.com/intel/libva/pull/364 > > https://github.com/intel/media-driver/issues/721 > > > > Signed-off-by: Linjie Fu <linjie.fu@intel.com> > > --- > > doc/encoders.texi | 8 ++++++++ > > libavcodec/vaapi_encode_h265.c | 19 +++++++++++++++++-- > > 2 files changed, 25 insertions(+), 2 deletions(-) > > My understanding was that the only reason for this stuff existing was to work > around broken hardware which didn't support P frames. Is this no longer > true? > > > > > diff --git a/doc/encoders.texi b/doc/encoders.texi > > index e23b6b3..b0812be 100644 > > --- a/doc/encoders.texi > > +++ b/doc/encoders.texi > > @@ -3089,6 +3089,14 @@ Some combination of the following values: > > Include HDR metadata if the input frames have it > > (@emph{mastering_display_colour_volume} and > @emph{content_light_level} > > messages). > > + > > +@item low_delay_b > > +Use low delay B-frames instead of P frames. Reordering of pictures is > > +not allowed. The first picture is encoded as an I picture and subsequent > > +pictures are encoded as B pictures. Moreover, since past B pictures are > > +used for prediction, a low coding delay but with higher coding efficiency > > +(because of bi-prediction) is achieved. > > This sounds like a marketing blurb. > > Not for the manual, but can you explain in detail what might actually make > this better, with actual numbers if possible? Intuitively the coding efficiency > will be worse, because a number of the B-picture syntax elements are just > redundant but still have to be coded (picking between two options which are > actually identical). The gain of allowing biprediction between two identical > pictures doesn't seem like a useful feature. > The story is, this is kind of the hardware limitation for VDENC as we've discussed on IRC. As to the performance/efficiency, I'm curious too and would take some experiments. This patch is kind of insufficient, updated a new version, to distinguish low_delay_b and reference B frames, thx. - Linjie
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Fu, > Linjie > Sent: Tuesday, April 14, 2020 17:15 > To: FFmpeg development discussions and patches <ffmpeg- > devel@ffmpeg.org> > Subject: Re: [FFmpeg-devel] [PATCH] lavc/vaapi_encode_h265: add > low_delay_b option for HEVC > > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > > Mark Thompson > > Sent: Monday, April 13, 2020 20:20 > > To: ffmpeg-devel@ffmpeg.org > > Subject: Re: [FFmpeg-devel] [PATCH] lavc/vaapi_encode_h265: add > > low_delay_b option for HEVC > > > > On 13/04/2020 05:32, Linjie Fu wrote: > > > Low delay B-frame is supported on ICL+ platform. > > > > > > For low power encoding, low_delay_b should be enabled by default. > > > > > > Low delay B: > > > <http://what-when-how.com/Tutorial/topic-397pct9eq3/High-Efficiency- > > Video-Coding-HEVC-288.html> > > > > > > There is an on-going work in libva and media-driver to add querys > > > support for low delay b, would add it once it's ready: > > > https://github.com/intel/libva/pull/220 > > > https://github.com/intel/libva/pull/364 > > > https://github.com/intel/media-driver/issues/721 > > > > > > Signed-off-by: Linjie Fu <linjie.fu@intel.com> > > > --- > > > doc/encoders.texi | 8 ++++++++ > > > libavcodec/vaapi_encode_h265.c | 19 +++++++++++++++++-- > > > 2 files changed, 25 insertions(+), 2 deletions(-) > > > > My understanding was that the only reason for this stuff existing was to > work > > around broken hardware which didn't support P frames. Is this no longer > > true? > > > > > > > > diff --git a/doc/encoders.texi b/doc/encoders.texi > > > index e23b6b3..b0812be 100644 > > > --- a/doc/encoders.texi > > > +++ b/doc/encoders.texi > > > @@ -3089,6 +3089,14 @@ Some combination of the following values: > > > Include HDR metadata if the input frames have it > > > (@emph{mastering_display_colour_volume} and > > @emph{content_light_level} > > > messages). > > > + > > > +@item low_delay_b > > > +Use low delay B-frames instead of P frames. Reordering of pictures is > > > +not allowed. The first picture is encoded as an I picture and subsequent > > > +pictures are encoded as B pictures. Moreover, since past B pictures are > > > +used for prediction, a low coding delay but with higher coding efficiency > > > +(because of bi-prediction) is achieved. > > > > This sounds like a marketing blurb. > > > > Not for the manual, but can you explain in detail what might actually make > > this better, with actual numbers if possible? Intuitively the coding > efficiency > > will be worse, because a number of the B-picture syntax elements are just > > redundant but still have to be coded (picking between two options which > are > > actually identical). The gain of allowing biprediction between two identical > > pictures doesn't seem like a useful feature. > > > The story is, this is kind of the hardware limitation for VDENC as we've > discussed > on IRC. Since the dependency in LIBVA and media-driver is addressed, updated and resent the patch. > As to the performance/efficiency, I'm curious too and would take some > experiments. Did some quick experiments on ICL with IBBPBBP reference structure (not official data, just to show the possible benefits): Bdrate : -1.570% based on normal IPB structure if we convert P frames to B frames. biprediction between two identical pictures seem to benefit the quality slightly. Please help to comment: https://patchwork.ffmpeg.org/project/ffmpeg/patch/1591019384-21910-1-git-send-email-linjie.fu@intel.com/ - Linjie
diff --git a/doc/encoders.texi b/doc/encoders.texi index e23b6b3..b0812be 100644 --- a/doc/encoders.texi +++ b/doc/encoders.texi @@ -3089,6 +3089,14 @@ Some combination of the following values: Include HDR metadata if the input frames have it (@emph{mastering_display_colour_volume} and @emph{content_light_level} messages). + +@item low_delay_b +Use low delay B-frames instead of P frames. Reordering of pictures is +not allowed. The first picture is encoded as an I picture and subsequent +pictures are encoded as B pictures. Moreover, since past B pictures are +used for prediction, a low coding delay but with higher coding efficiency +(because of bi-prediction) is achieved. + @end table @end table diff --git a/libavcodec/vaapi_encode_h265.c b/libavcodec/vaapi_encode_h265.c index 97dc5a7..cd48545 100644 --- a/libavcodec/vaapi_encode_h265.c +++ b/libavcodec/vaapi_encode_h265.c @@ -62,6 +62,7 @@ typedef struct VAAPIEncodeH265Context { int tier; int level; int sei; + int low_delay_b; // Derived settings. int fixed_qp_idr; @@ -894,6 +895,9 @@ static int vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, sh->slice_type = hpic->slice_type; + if (sh->slice_type == HEVC_SLICE_P && priv->low_delay_b) + sh->slice_type = HEVC_SLICE_B; + sh->slice_pic_order_cnt_lsb = hpic->pic_order_cnt & (1 << (sps->log2_max_pic_order_cnt_lsb_minus4 + 4)) - 1; @@ -1054,9 +1058,13 @@ static int vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, vslice->ref_pic_list0[0] = vpic->reference_frames[0]; } if (pic->nb_refs >= 2) { - // Forward reference for B-frame. av_assert0(pic->type == PICTURE_TYPE_B); - vslice->ref_pic_list1[0] = vpic->reference_frames[1]; + if (priv->low_delay_b) + // Reference for low delay B-frame + vslice->ref_pic_list1[0] = vpic->reference_frames[0]; + else + // Forward reference for B-frame. + vslice->ref_pic_list1[0] = vpic->reference_frames[1]; } return 0; @@ -1181,6 +1189,11 @@ static av_cold int vaapi_encode_h265_init(AVCodecContext *avctx) if (priv->qp > 0) ctx->explicit_qp = priv->qp; + /* low_delay_b is required for low power encoding */ + priv->low_delay_b = ctx->low_power ? 1 : priv->low_delay_b; + if (priv->low_delay_b) + av_log(avctx, AV_LOG_VERBOSE, "Low delay B-frame enabled.\n"); + return ff_vaapi_encode_init(avctx); } @@ -1256,6 +1269,8 @@ static const AVOption vaapi_encode_h265_options[] = { 0, AV_OPT_TYPE_CONST, { .i64 = SEI_MASTERING_DISPLAY | SEI_CONTENT_LIGHT_LEVEL }, INT_MIN, INT_MAX, FLAGS, "sei" }, + { "low_delay_b", "Use low delay B frames instead of P frames", + OFFSET(low_delay_b), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, FLAGS }, { NULL }, };
Low delay B-frame is supported on ICL+ platform. For low power encoding, low_delay_b should be enabled by default. Low delay B: <http://what-when-how.com/Tutorial/topic-397pct9eq3/High-Efficiency-Video-Coding-HEVC-288.html> There is an on-going work in libva and media-driver to add querys support for low delay b, would add it once it's ready: https://github.com/intel/libva/pull/220 https://github.com/intel/libva/pull/364 https://github.com/intel/media-driver/issues/721 Signed-off-by: Linjie Fu <linjie.fu@intel.com> --- doc/encoders.texi | 8 ++++++++ libavcodec/vaapi_encode_h265.c | 19 +++++++++++++++++-- 2 files changed, 25 insertions(+), 2 deletions(-)