mbox series

[FFmpeg-devel,v5,00/20] clean-up QSV filters

Message ID 20210805081910.9386-1-haihao.xiang@intel.com
Headers show
Series clean-up QSV filters
Related show

Message

Xiang, Haihao Aug. 5, 2021, 8:18 a.m. UTC
This patchset clean up scale_qsv and deinterlace_qsv filters, and take
the two filters as the special cases of vpp_qsv, so vf_scale_qsv.c and
vf_deinterlace_qsv.c can be deleted from FFmpeg. In addition, a few
small features are added in this patchset.
---
v5:
* Rebased this patchset against the latest master branch and fixed conflicts

Haihao Xiang (20):
  lavfi/qsv: use QSVVPPContext as base context in
    vf_vpp_qsv/vf_overlay_qsv
  lavfi/scale_qsv: simplify scale_qsv filter
  lavfi/scale_qsv: don't need variables for constants in FFmpeg
  lavfi/vpp_qsv: add "a", "dar" and "sar" variables
  lavfi/vpp_qsv: handle NULL pointer when evaluating an expression
  lavfi/vpp_qsv: allow special values for the output dimensions
  lavfi/vpp_qsv: factorize extra MFX configuration
  lavfi/vpp_qsv: allow user to set scale_mode with constant
  lavfi/vpp_qsv: add vpp_preinit callback
  lavfi/scale_qsv: re-use VPPContext for scale_qsv filter
  lavfi/vpp_qsv: factor common QSV filter definition
  lavfi/scale_qsv: add new options for scale_qsv filter
  lavfi/scale_qsv: add more input / output pixel formats
  lavfi/qsvvpp: avoid overriding the returned value
  lavfi/qsvvpp: set PTS for output frame
  lavfi/vpp_qsv: check output format string against NULL pointer
  lavfi/deinterlace_qsv: simplify deinterlace_qsv filter
  lavfi/deinterlace_qsv: re-use VPPContext for deinterlace_qsv filter
  lavfi/deinterlace_qsv: add async_depth option
  lavfi/deinterlace_qsv: add more input / output pixel formats

 libavfilter/Makefile             |   4 +-
 libavfilter/qsvvpp.c             |  57 ++-
 libavfilter/qsvvpp.h             |  11 +-
 libavfilter/vf_deinterlace_qsv.c | 611 ---------------------------
 libavfilter/vf_overlay_qsv.c     |  11 +-
 libavfilter/vf_scale_qsv.c       | 685 -------------------------------
 libavfilter/vf_vpp_qsv.c         | 473 +++++++++++++--------
 7 files changed, 347 insertions(+), 1505 deletions(-)
 delete mode 100644 libavfilter/vf_deinterlace_qsv.c
 delete mode 100644 libavfilter/vf_scale_qsv.c

Comments

Soft Works Aug. 5, 2021, 4:21 p.m. UTC | #1
> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Haihao Xiang
> Sent: Thursday, 5 August 2021 10:19
> To: ffmpeg-devel@ffmpeg.org
> Cc: Haihao Xiang <haihao.xiang@intel.com>
> Subject: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> 
> This patchset clean up scale_qsv and deinterlace_qsv filters, and take the
> two filters as the special cases of vpp_qsv, so vf_scale_qsv.c and
> vf_deinterlace_qsv.c can be deleted from FFmpeg. In addition, a few small
> features are added in this patchset.
> ---
> v5:
> * Rebased this patchset against the latest master branch and fixed conflicts
> 
> Haihao Xiang (20):
>   lavfi/qsv: use QSVVPPContext as base context in
>     vf_vpp_qsv/vf_overlay_qsv
>   lavfi/scale_qsv: simplify scale_qsv filter
>   lavfi/scale_qsv: don't need variables for constants in FFmpeg
>   lavfi/vpp_qsv: add "a", "dar" and "sar" variables
>   lavfi/vpp_qsv: handle NULL pointer when evaluating an expression
>   lavfi/vpp_qsv: allow special values for the output dimensions
>   lavfi/vpp_qsv: factorize extra MFX configuration
>   lavfi/vpp_qsv: allow user to set scale_mode with constant
>   lavfi/vpp_qsv: add vpp_preinit callback
>   lavfi/scale_qsv: re-use VPPContext for scale_qsv filter
>   lavfi/vpp_qsv: factor common QSV filter definition
>   lavfi/scale_qsv: add new options for scale_qsv filter
>   lavfi/scale_qsv: add more input / output pixel formats
>   lavfi/qsvvpp: avoid overriding the returned value
>   lavfi/qsvvpp: set PTS for output frame
>   lavfi/vpp_qsv: check output format string against NULL pointer
>   lavfi/deinterlace_qsv: simplify deinterlace_qsv filter
>   lavfi/deinterlace_qsv: re-use VPPContext for deinterlace_qsv filter
>   lavfi/deinterlace_qsv: add async_depth option
>   lavfi/deinterlace_qsv: add more input / output pixel formats
> 
>  libavfilter/Makefile             |   4 +-
>  libavfilter/qsvvpp.c             |  57 ++-
>  libavfilter/qsvvpp.h             |  11 +-
>  libavfilter/vf_deinterlace_qsv.c | 611 ---------------------------
>  libavfilter/vf_overlay_qsv.c     |  11 +-
>  libavfilter/vf_scale_qsv.c       | 685 -------------------------------
>  libavfilter/vf_vpp_qsv.c         | 473 +++++++++++++--------
>  7 files changed, 347 insertions(+), 1505 deletions(-)  delete mode 100644
> libavfilter/vf_deinterlace_qsv.c  delete mode 100644
> libavfilter/vf_scale_qsv.c
> 
> --
> 2.17.1

Hi Hihao,

The general idea of this patch makes sense to me.

Currently there are implementation differences between these filters, 
and there are cases where vpp_qsv  doesn't work and I need to use 
scale_qsv instead.

I have never analyzed the actual reason, but this should be done 
before replacing scale_qsv with an aliased vpp_qsv.

I'll try to dig out an example..

softworkz
Xiang, Haihao Aug. 30, 2021, 4:20 a.m. UTC | #2
On Thu, 2021-08-05 at 16:21 +0000, Soft Works wrote:
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Haihao Xiang
> > Sent: Thursday, 5 August 2021 10:19
> > To: ffmpeg-devel@ffmpeg.org
> > Cc: Haihao Xiang <haihao.xiang@intel.com>
> > Subject: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> > 
> > This patchset clean up scale_qsv and deinterlace_qsv filters, and take the
> > two filters as the special cases of vpp_qsv, so vf_scale_qsv.c and
> > vf_deinterlace_qsv.c can be deleted from FFmpeg. In addition, a few small
> > features are added in this patchset.
> > ---
> > v5:
> > * Rebased this patchset against the latest master branch and fixed conflicts
> > 
> > Haihao Xiang (20):
> >   lavfi/qsv: use QSVVPPContext as base context in
> >     vf_vpp_qsv/vf_overlay_qsv
> >   lavfi/scale_qsv: simplify scale_qsv filter
> >   lavfi/scale_qsv: don't need variables for constants in FFmpeg
> >   lavfi/vpp_qsv: add "a", "dar" and "sar" variables
> >   lavfi/vpp_qsv: handle NULL pointer when evaluating an expression
> >   lavfi/vpp_qsv: allow special values for the output dimensions
> >   lavfi/vpp_qsv: factorize extra MFX configuration
> >   lavfi/vpp_qsv: allow user to set scale_mode with constant
> >   lavfi/vpp_qsv: add vpp_preinit callback
> >   lavfi/scale_qsv: re-use VPPContext for scale_qsv filter
> >   lavfi/vpp_qsv: factor common QSV filter definition
> >   lavfi/scale_qsv: add new options for scale_qsv filter
> >   lavfi/scale_qsv: add more input / output pixel formats
> >   lavfi/qsvvpp: avoid overriding the returned value
> >   lavfi/qsvvpp: set PTS for output frame
> >   lavfi/vpp_qsv: check output format string against NULL pointer
> >   lavfi/deinterlace_qsv: simplify deinterlace_qsv filter
> >   lavfi/deinterlace_qsv: re-use VPPContext for deinterlace_qsv filter
> >   lavfi/deinterlace_qsv: add async_depth option
> >   lavfi/deinterlace_qsv: add more input / output pixel formats
> > 
> >  libavfilter/Makefile             |   4 +-
> >  libavfilter/qsvvpp.c             |  57 ++-
> >  libavfilter/qsvvpp.h             |  11 +-
> >  libavfilter/vf_deinterlace_qsv.c | 611 ---------------------------
> >  libavfilter/vf_overlay_qsv.c     |  11 +-
> >  libavfilter/vf_scale_qsv.c       | 685 -------------------------------
> >  libavfilter/vf_vpp_qsv.c         | 473 +++++++++++++--------
> >  7 files changed, 347 insertions(+), 1505 deletions(-)  delete mode 100644
> > libavfilter/vf_deinterlace_qsv.c  delete mode 100644
> > libavfilter/vf_scale_qsv.c
> > 
> > --
> > 2.17.1
> 
> Hi Hihao,
> 
> The general idea of this patch makes sense to me.
> 
> Currently there are implementation differences between these filters, 
> and there are cases where vpp_qsv  doesn't work and I need to use 
> scale_qsv instead.
> 
> I have never analyzed the actual reason, but this should be done 
> before replacing scale_qsv with an aliased vpp_qsv.
> 
> I'll try to dig out an example..


Hi Softworkz,

Could you provide the cases when you have time ? I may look into the issues.

Thanks
Haihao
Soft Works Aug. 30, 2021, 5:52 a.m. UTC | #3
> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Xiang, Haihao
> Sent: Monday, 30 August 2021 06:20
> To: ffmpeg-devel@ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> 
> On Thu, 2021-08-05 at 16:21 +0000, Soft Works wrote:
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > Haihao Xiang
> > > Sent: Thursday, 5 August 2021 10:19
> > > To: ffmpeg-devel@ffmpeg.org
> > > Cc: Haihao Xiang <haihao.xiang@intel.com>
> > > Subject: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> > >
> > > This patchset clean up scale_qsv and deinterlace_qsv filters, and
> take the
> > > two filters as the special cases of vpp_qsv, so vf_scale_qsv.c
> and
> > > vf_deinterlace_qsv.c can be deleted from FFmpeg. In addition, a
> few small
> > > features are added in this patchset.
> > > ---
> > > v5:
> > > * Rebased this patchset against the latest master branch and
> fixed conflicts
> > >
> > > Haihao Xiang (20):
> > >   lavfi/qsv: use QSVVPPContext as base context in
> > >     vf_vpp_qsv/vf_overlay_qsv
> > >   lavfi/scale_qsv: simplify scale_qsv filter
> > >   lavfi/scale_qsv: don't need variables for constants in FFmpeg
> > >   lavfi/vpp_qsv: add "a", "dar" and "sar" variables
> > >   lavfi/vpp_qsv: handle NULL pointer when evaluating an
> expression
> > >   lavfi/vpp_qsv: allow special values for the output dimensions
> > >   lavfi/vpp_qsv: factorize extra MFX configuration
> > >   lavfi/vpp_qsv: allow user to set scale_mode with constant
> > >   lavfi/vpp_qsv: add vpp_preinit callback
> > >   lavfi/scale_qsv: re-use VPPContext for scale_qsv filter
> > >   lavfi/vpp_qsv: factor common QSV filter definition
> > >   lavfi/scale_qsv: add new options for scale_qsv filter
> > >   lavfi/scale_qsv: add more input / output pixel formats
> > >   lavfi/qsvvpp: avoid overriding the returned value
> > >   lavfi/qsvvpp: set PTS for output frame
> > >   lavfi/vpp_qsv: check output format string against NULL pointer
> > >   lavfi/deinterlace_qsv: simplify deinterlace_qsv filter
> > >   lavfi/deinterlace_qsv: re-use VPPContext for deinterlace_qsv
> filter
> > >   lavfi/deinterlace_qsv: add async_depth option
> > >   lavfi/deinterlace_qsv: add more input / output pixel formats
> > >
> > >  libavfilter/Makefile             |   4 +-
> > >  libavfilter/qsvvpp.c             |  57 ++-
> > >  libavfilter/qsvvpp.h             |  11 +-
> > >  libavfilter/vf_deinterlace_qsv.c | 611 -------------------------
> --
> > >  libavfilter/vf_overlay_qsv.c     |  11 +-
> > >  libavfilter/vf_scale_qsv.c       | 685 -------------------------
> ------
> > >  libavfilter/vf_vpp_qsv.c         | 473 +++++++++++++--------
> > >  7 files changed, 347 insertions(+), 1505 deletions(-)  delete
> mode 100644
> > > libavfilter/vf_deinterlace_qsv.c  delete mode 100644
> > > libavfilter/vf_scale_qsv.c
> > >
> > > --
> > > 2.17.1
> >
> > Hi Hihao,
> >
> > The general idea of this patch makes sense to me.
> >
> > Currently there are implementation differences between these
> filters,
> > and there are cases where vpp_qsv  doesn't work and I need to use
> > scale_qsv instead.
> >
> > I have never analyzed the actual reason, but this should be done
> > before replacing scale_qsv with an aliased vpp_qsv.
> >
> > I'll try to dig out an example..
> 
> 
> Hi Softworkz,
> 
> Could you provide the cases when you have time ? I may look into the
> issues.
> 
> Thanks
> Haihao


Hi Haihao,

IIRC think it can be reproduced easily by using the filters without parameters,
where scale_qsv works and vpp_qsv doesn't work.

If it doesn't reproduce, try this in combination with hwupload, e.g.

- "hwupload@f1=extra_hw_frames=32,vpp_qsv"
vs.
- "hwupload@f1=extra_hw_frames=32,scale_qsv"


Please let me know whether you can reproduce.

Kind regards,
softworkz
Xiang, Haihao Aug. 30, 2021, 8:04 a.m. UTC | #4
On Mon, 2021-08-30 at 05:52 +0000, Soft Works wrote:
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Xiang, Haihao
> > Sent: Monday, 30 August 2021 06:20
> > To: ffmpeg-devel@ffmpeg.org
> > Subject: Re: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> > 
> > On Thu, 2021-08-05 at 16:21 +0000, Soft Works wrote:
> > > > -----Original Message-----
> > > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > > Haihao Xiang
> > > > Sent: Thursday, 5 August 2021 10:19
> > > > To: ffmpeg-devel@ffmpeg.org
> > > > Cc: Haihao Xiang <haihao.xiang@intel.com>
> > > > Subject: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> > > > 
> > > > This patchset clean up scale_qsv and deinterlace_qsv filters, and
> > 
> > take the
> > > > two filters as the special cases of vpp_qsv, so vf_scale_qsv.c
> > 
> > and
> > > > vf_deinterlace_qsv.c can be deleted from FFmpeg. In addition, a
> > 
> > few small
> > > > features are added in this patchset.
> > > > ---
> > > > v5:
> > > > * Rebased this patchset against the latest master branch and
> > 
> > fixed conflicts
> > > > 
> > > > Haihao Xiang (20):
> > > >   lavfi/qsv: use QSVVPPContext as base context in
> > > >     vf_vpp_qsv/vf_overlay_qsv
> > > >   lavfi/scale_qsv: simplify scale_qsv filter
> > > >   lavfi/scale_qsv: don't need variables for constants in FFmpeg
> > > >   lavfi/vpp_qsv: add "a", "dar" and "sar" variables
> > > >   lavfi/vpp_qsv: handle NULL pointer when evaluating an
> > 
> > expression
> > > >   lavfi/vpp_qsv: allow special values for the output dimensions
> > > >   lavfi/vpp_qsv: factorize extra MFX configuration
> > > >   lavfi/vpp_qsv: allow user to set scale_mode with constant
> > > >   lavfi/vpp_qsv: add vpp_preinit callback
> > > >   lavfi/scale_qsv: re-use VPPContext for scale_qsv filter
> > > >   lavfi/vpp_qsv: factor common QSV filter definition
> > > >   lavfi/scale_qsv: add new options for scale_qsv filter
> > > >   lavfi/scale_qsv: add more input / output pixel formats
> > > >   lavfi/qsvvpp: avoid overriding the returned value
> > > >   lavfi/qsvvpp: set PTS for output frame
> > > >   lavfi/vpp_qsv: check output format string against NULL pointer
> > > >   lavfi/deinterlace_qsv: simplify deinterlace_qsv filter
> > > >   lavfi/deinterlace_qsv: re-use VPPContext for deinterlace_qsv
> > 
> > filter
> > > >   lavfi/deinterlace_qsv: add async_depth option
> > > >   lavfi/deinterlace_qsv: add more input / output pixel formats
> > > > 
> > > >  libavfilter/Makefile             |   4 +-
> > > >  libavfilter/qsvvpp.c             |  57 ++-
> > > >  libavfilter/qsvvpp.h             |  11 +-
> > > >  libavfilter/vf_deinterlace_qsv.c | 611 -------------------------
> > 
> > --
> > > >  libavfilter/vf_overlay_qsv.c     |  11 +-
> > > >  libavfilter/vf_scale_qsv.c       | 685 -------------------------
> > 
> > ------
> > > >  libavfilter/vf_vpp_qsv.c         | 473 +++++++++++++--------
> > > >  7 files changed, 347 insertions(+), 1505 deletions(-)  delete
> > 
> > mode 100644
> > > > libavfilter/vf_deinterlace_qsv.c  delete mode 100644
> > > > libavfilter/vf_scale_qsv.c
> > > > 
> > > > --
> > > > 2.17.1
> > > 
> > > Hi Hihao,
> > > 
> > > The general idea of this patch makes sense to me.
> > > 
> > > Currently there are implementation differences between these
> > 
> > filters,
> > > and there are cases where vpp_qsv  doesn't work and I need to use
> > > scale_qsv instead.
> > > 
> > > I have never analyzed the actual reason, but this should be done
> > > before replacing scale_qsv with an aliased vpp_qsv.
> > > 
> > > I'll try to dig out an example..
> > 
> > 
> > Hi Softworkz,
> > 
> > Could you provide the cases when you have time ? I may look into the
> > issues.
> > 
> > Thanks
> > Haihao
> 
> 
> Hi Haihao,
> 
> IIRC think it can be reproduced easily by using the filters without
> parameters,
> where scale_qsv works and vpp_qsv doesn't work.

scale_qsv only supports video memory input and output however vpp_qsv may
support both video memory and system memory, there is a case where scale_qsv
doesn't work but vpp_qsv may work. 

$> ffmpeg -c:v h264_qsv -i input.mp4 -vf scale_qsv -f null -

Impossible to convert between the formats supported by the filter 'graph 0 input
from stream 0:0' and the filter 'auto_scale_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
Conversion failed!

After applying this patchset, scale_qsv may work with both video memory and
system memory.

In addition, vpp_qsv may support crop however scale_qsv doesn't support crop.
After applying  this patchset, scale_qsv may support crop too. 

> 
> If it doesn't reproduce, try this in combination with hwupload, e.g.
> 
> - "hwupload@f1=extra_hw_frames=32,vpp_qsv"
> vs.
> - "hwupload@f1=extra_hw_frames=32,scale_qsv"
> 

The above two combinations work for me

Thanks
Haihao
Xiang, Haihao Sept. 9, 2021, 2:56 p.m. UTC | #5
On Mon, 2021-08-30 at 08:04 +0000, Xiang, Haihao wrote:
> On Mon, 2021-08-30 at 05:52 +0000, Soft Works wrote:
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > Xiang, Haihao
> > > Sent: Monday, 30 August 2021 06:20
> > > To: ffmpeg-devel@ffmpeg.org
> > > Subject: Re: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> > > 
> > > On Thu, 2021-08-05 at 16:21 +0000, Soft Works wrote:
> > > > > -----Original Message-----
> > > > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > > > Haihao Xiang
> > > > > Sent: Thursday, 5 August 2021 10:19
> > > > > To: ffmpeg-devel@ffmpeg.org
> > > > > Cc: Haihao Xiang <haihao.xiang@intel.com>
> > > > > Subject: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> > > > > 
> > > > > This patchset clean up scale_qsv and deinterlace_qsv filters, and
> > > 
> > > take the
> > > > > two filters as the special cases of vpp_qsv, so vf_scale_qsv.c
> > > 
> > > and
> > > > > vf_deinterlace_qsv.c can be deleted from FFmpeg. In addition, a
> > > 
> > > few small
> > > > > features are added in this patchset.
> > > > > ---
> > > > > v5:
> > > > > * Rebased this patchset against the latest master branch and
> > > 
> > > fixed conflicts
> > > > > 
> > > > > Haihao Xiang (20):
> > > > >   lavfi/qsv: use QSVVPPContext as base context in
> > > > >     vf_vpp_qsv/vf_overlay_qsv
> > > > >   lavfi/scale_qsv: simplify scale_qsv filter
> > > > >   lavfi/scale_qsv: don't need variables for constants in FFmpeg
> > > > >   lavfi/vpp_qsv: add "a", "dar" and "sar" variables
> > > > >   lavfi/vpp_qsv: handle NULL pointer when evaluating an
> > > 
> > > expression
> > > > >   lavfi/vpp_qsv: allow special values for the output dimensions
> > > > >   lavfi/vpp_qsv: factorize extra MFX configuration
> > > > >   lavfi/vpp_qsv: allow user to set scale_mode with constant
> > > > >   lavfi/vpp_qsv: add vpp_preinit callback
> > > > >   lavfi/scale_qsv: re-use VPPContext for scale_qsv filter
> > > > >   lavfi/vpp_qsv: factor common QSV filter definition
> > > > >   lavfi/scale_qsv: add new options for scale_qsv filter
> > > > >   lavfi/scale_qsv: add more input / output pixel formats
> > > > >   lavfi/qsvvpp: avoid overriding the returned value
> > > > >   lavfi/qsvvpp: set PTS for output frame
> > > > >   lavfi/vpp_qsv: check output format string against NULL pointer
> > > > >   lavfi/deinterlace_qsv: simplify deinterlace_qsv filter
> > > > >   lavfi/deinterlace_qsv: re-use VPPContext for deinterlace_qsv
> > > 
> > > filter
> > > > >   lavfi/deinterlace_qsv: add async_depth option
> > > > >   lavfi/deinterlace_qsv: add more input / output pixel formats
> > > > > 
> > > > >  libavfilter/Makefile             |   4 +-
> > > > >  libavfilter/qsvvpp.c             |  57 ++-
> > > > >  libavfilter/qsvvpp.h             |  11 +-
> > > > >  libavfilter/vf_deinterlace_qsv.c | 611 -------------------------
> > > 
> > > --
> > > > >  libavfilter/vf_overlay_qsv.c     |  11 +-
> > > > >  libavfilter/vf_scale_qsv.c       | 685 -------------------------
> > > 
> > > ------
> > > > >  libavfilter/vf_vpp_qsv.c         | 473 +++++++++++++--------
> > > > >  7 files changed, 347 insertions(+), 1505 deletions(-)  delete
> > > 
> > > mode 100644
> > > > > libavfilter/vf_deinterlace_qsv.c  delete mode 100644
> > > > > libavfilter/vf_scale_qsv.c
> > > > > 
> > > > > --
> > > > > 2.17.1
> > > > 
> > > > Hi Hihao,
> > > > 
> > > > The general idea of this patch makes sense to me.
> > > > 
> > > > Currently there are implementation differences between these
> > > 
> > > filters,
> > > > and there are cases where vpp_qsv  doesn't work and I need to use
> > > > scale_qsv instead.
> > > > 
> > > > I have never analyzed the actual reason, but this should be done
> > > > before replacing scale_qsv with an aliased vpp_qsv.
> > > > 
> > > > I'll try to dig out an example..
> > > 
> > > 
> > > Hi Softworkz,
> > > 
> > > Could you provide the cases when you have time ? I may look into the
> > > issues.
> > > 
> > > Thanks
> > > Haihao
> > 
> > 
> > Hi Haihao,
> > 
> > IIRC think it can be reproduced easily by using the filters without
> > parameters,
> > where scale_qsv works and vpp_qsv doesn't work.
> 
> scale_qsv only supports video memory input and output however vpp_qsv may
> support both video memory and system memory, there is a case where scale_qsv
> doesn't work but vpp_qsv may work. 
> 
> $> ffmpeg -c:v h264_qsv -i input.mp4 -vf scale_qsv -f null -
> 
> Impossible to convert between the formats supported by the filter 'graph 0
> input
> from stream 0:0' and the filter 'auto_scale_0'
> Error reinitializing filters!
> Failed to inject frame into filter network: Function not implemented
> Error while processing the decoded data for stream #0:0
> Conversion failed!
> 
> After applying this patchset, scale_qsv may work with both video memory and
> system memory.
> 
> In addition, vpp_qsv may support crop however scale_qsv doesn't support crop.
> After applying  this patchset, scale_qsv may support crop too. 
> 
> > 
> > If it doesn't reproduce, try this in combination with hwupload, e.g.
> > 
> > - "hwupload@f1=extra_hw_frames=32,vpp_qsv"
> > vs.
> > - "hwupload@f1=extra_hw_frames=32,scale_qsv"
> > 
> 
> The above two combinations work for me

Hi Softworkz,

I failed to reproduce your issue, could you provide more info if possible? 

Thanks
Haihao
Xiang, Haihao Sept. 18, 2021, 1:27 a.m. UTC | #6
On Thu, 2021-09-09 at 14:56 +0000, Xiang, Haihao wrote:
> On Mon, 2021-08-30 at 08:04 +0000, Xiang, Haihao wrote:
> > On Mon, 2021-08-30 at 05:52 +0000, Soft Works wrote:
> > > > -----Original Message-----
> > > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > > Xiang, Haihao
> > > > Sent: Monday, 30 August 2021 06:20
> > > > To: ffmpeg-devel@ffmpeg.org
> > > > Subject: Re: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> > > > 
> > > > On Thu, 2021-08-05 at 16:21 +0000, Soft Works wrote:
> > > > > > -----Original Message-----
> > > > > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > > > > Haihao Xiang
> > > > > > Sent: Thursday, 5 August 2021 10:19
> > > > > > To: ffmpeg-devel@ffmpeg.org
> > > > > > Cc: Haihao Xiang <haihao.xiang@intel.com>
> > > > > > Subject: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> > > > > > 
> > > > > > This patchset clean up scale_qsv and deinterlace_qsv filters, and
> > > > 
> > > > take the
> > > > > > two filters as the special cases of vpp_qsv, so vf_scale_qsv.c
> > > > 
> > > > and
> > > > > > vf_deinterlace_qsv.c can be deleted from FFmpeg. In addition, a
> > > > 
> > > > few small
> > > > > > features are added in this patchset.
> > > > > > ---
> > > > > > v5:
> > > > > > * Rebased this patchset against the latest master branch and
> > > > 
> > > > fixed conflicts
> > > > > > 
> > > > > > Haihao Xiang (20):
> > > > > >   lavfi/qsv: use QSVVPPContext as base context in
> > > > > >     vf_vpp_qsv/vf_overlay_qsv
> > > > > >   lavfi/scale_qsv: simplify scale_qsv filter
> > > > > >   lavfi/scale_qsv: don't need variables for constants in FFmpeg
> > > > > >   lavfi/vpp_qsv: add "a", "dar" and "sar" variables
> > > > > >   lavfi/vpp_qsv: handle NULL pointer when evaluating an
> > > > 
> > > > expression
> > > > > >   lavfi/vpp_qsv: allow special values for the output dimensions
> > > > > >   lavfi/vpp_qsv: factorize extra MFX configuration
> > > > > >   lavfi/vpp_qsv: allow user to set scale_mode with constant
> > > > > >   lavfi/vpp_qsv: add vpp_preinit callback
> > > > > >   lavfi/scale_qsv: re-use VPPContext for scale_qsv filter
> > > > > >   lavfi/vpp_qsv: factor common QSV filter definition
> > > > > >   lavfi/scale_qsv: add new options for scale_qsv filter
> > > > > >   lavfi/scale_qsv: add more input / output pixel formats
> > > > > >   lavfi/qsvvpp: avoid overriding the returned value
> > > > > >   lavfi/qsvvpp: set PTS for output frame
> > > > > >   lavfi/vpp_qsv: check output format string against NULL pointer
> > > > > >   lavfi/deinterlace_qsv: simplify deinterlace_qsv filter
> > > > > >   lavfi/deinterlace_qsv: re-use VPPContext for deinterlace_qsv
> > > > 
> > > > filter
> > > > > >   lavfi/deinterlace_qsv: add async_depth option
> > > > > >   lavfi/deinterlace_qsv: add more input / output pixel formats
> > > > > > 
> > > > > >  libavfilter/Makefile             |   4 +-
> > > > > >  libavfilter/qsvvpp.c             |  57 ++-
> > > > > >  libavfilter/qsvvpp.h             |  11 +-
> > > > > >  libavfilter/vf_deinterlace_qsv.c | 611 -------------------------
> > > > 
> > > > --
> > > > > >  libavfilter/vf_overlay_qsv.c     |  11 +-
> > > > > >  libavfilter/vf_scale_qsv.c       | 685 -------------------------
> > > > 
> > > > ------
> > > > > >  libavfilter/vf_vpp_qsv.c         | 473 +++++++++++++--------
> > > > > >  7 files changed, 347 insertions(+), 1505 deletions(-)  delete
> > > > 
> > > > mode 100644
> > > > > > libavfilter/vf_deinterlace_qsv.c  delete mode 100644
> > > > > > libavfilter/vf_scale_qsv.c
> > > > > > 
> > > > > > --
> > > > > > 2.17.1
> > > > > 
> > > > > Hi Hihao,
> > > > > 
> > > > > The general idea of this patch makes sense to me.
> > > > > 
> > > > > Currently there are implementation differences between these
> > > > 
> > > > filters,
> > > > > and there are cases where vpp_qsv  doesn't work and I need to use
> > > > > scale_qsv instead.
> > > > > 
> > > > > I have never analyzed the actual reason, but this should be done
> > > > > before replacing scale_qsv with an aliased vpp_qsv.
> > > > > 
> > > > > I'll try to dig out an example..
> > > > 
> > > > 
> > > > Hi Softworkz,
> > > > 
> > > > Could you provide the cases when you have time ? I may look into the
> > > > issues.
> > > > 
> > > > Thanks
> > > > Haihao
> > > 
> > > 
> > > Hi Haihao,
> > > 
> > > IIRC think it can be reproduced easily by using the filters without
> > > parameters,
> > > where scale_qsv works and vpp_qsv doesn't work.
> > 
> > scale_qsv only supports video memory input and output however vpp_qsv may
> > support both video memory and system memory, there is a case where scale_qsv
> > doesn't work but vpp_qsv may work. 
> > 
> > $> ffmpeg -c:v h264_qsv -i input.mp4 -vf scale_qsv -f null -
> > 
> > Impossible to convert between the formats supported by the filter 'graph 0
> > input
> > from stream 0:0' and the filter 'auto_scale_0'
> > Error reinitializing filters!
> > Failed to inject frame into filter network: Function not implemented
> > Error while processing the decoded data for stream #0:0
> > Conversion failed!
> > 
> > After applying this patchset, scale_qsv may work with both video memory and
> > system memory.
> > 
> > In addition, vpp_qsv may support crop however scale_qsv doesn't support
> > crop.
> > After applying  this patchset, scale_qsv may support crop too. 
> > 
> > > 
> > > If it doesn't reproduce, try this in combination with hwupload, e.g.
> > > 
> > > - "hwupload@f1=extra_hw_frames=32,vpp_qsv"
> > > vs.
> > > - "hwupload@f1=extra_hw_frames=32,scale_qsv"
> > > 
> > 
> > The above two combinations work for me
> 
> Hi Softworkz,
> 
> I failed to reproduce your issue, could you provide more info if possible? 

Hi Softworkz,

I didn't reproduce your issue, maybe the issue has been fixed on master. How
about to merge this patchset first? I will rebase the patchset if no objection. 
 BTW we didn't see regression in our internal testing. 

Thanks
Haihao
Soft Works Oct. 10, 2021, 10:37 p.m. UTC | #7
> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Xiang, Haihao
> Sent: Saturday, September 18, 2021 3:27 AM
> To: ffmpeg-devel@ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> 
> On Thu, 2021-09-09 at 14:56 +0000, Xiang, Haihao wrote:
> > On Mon, 2021-08-30 at 08:04 +0000, Xiang, Haihao wrote:
> > > On Mon, 2021-08-30 at 05:52 +0000, Soft Works wrote:


[..]

> > > > > >
> > > > > > Hi Hihao,
> > > > > >
> > > > > > The general idea of this patch makes sense to me.
> > > > > >
> > > > > > Currently there are implementation differences between
> these
> > > > >
> > > > > filters,
> > > > > > and there are cases where vpp_qsv  doesn't work and I need
> to use
> > > > > > scale_qsv instead.
> > > > > >
> > > > > > I have never analyzed the actual reason, but this should be
> done
> > > > > > before replacing scale_qsv with an aliased vpp_qsv.
> > > > > >
> > > > > > I'll try to dig out an example..
> > > > >
> > > > >
> > > > > Hi Softworkz,
> > > > >
> > > > > Could you provide the cases when you have time ? I may look
> into the
> > > > > issues.
> > > > >
> > > > > Thanks
> > > > > Haihao
> > > >
> > > >
> > > > Hi Haihao,
> > > >
> > > > IIRC think it can be reproduced easily by using the filters
> without
> > > > parameters,
> > > > where scale_qsv works and vpp_qsv doesn't work.
> > >
> > > scale_qsv only supports video memory input and output however
> vpp_qsv may
> > > support both video memory and system memory, there is a case
> where scale_qsv
> > > doesn't work but vpp_qsv may work.
> > >
> > > $> ffmpeg -c:v h264_qsv -i input.mp4 -vf scale_qsv -f null -
> > >
> > > Impossible to convert between the formats supported by the filter
> 'graph 0
> > > input
> > > from stream 0:0' and the filter 'auto_scale_0'
> > > Error reinitializing filters!
> > > Failed to inject frame into filter network: Function not
> implemented
> > > Error while processing the decoded data for stream #0:0
> > > Conversion failed!
> > >
> > > After applying this patchset, scale_qsv may work with both video
> memory and
> > > system memory.
> > >
> > > In addition, vpp_qsv may support crop however scale_qsv doesn't
> support
> > > crop.
> > > After applying  this patchset, scale_qsv may support crop too.
> > >
> > > >
> > > > If it doesn't reproduce, try this in combination with hwupload,
> e.g.
> > > >
> > > > - "hwupload@f1=extra_hw_frames=32,vpp_qsv"
> > > > vs.
> > > > - "hwupload@f1=extra_hw_frames=32,scale_qsv"
> > > >
> > >
> > > The above two combinations work for me
> >
> > Hi Softworkz,
> >
> > I failed to reproduce your issue, could you provide more info if
> possible?
> 
> Hi Softworkz,
> 
> I didn't reproduce your issue, maybe the issue has been fixed on
> master. How
> about to merge this patchset first? I will rebase the patchset if no
> objection.
>  BTW we didn't see regression in our internal testing.


Hi Haihao,
(sending a CC to the ML allowing others to follow)

I found an example for the cases I was referring to. 

An example command line is this:

ffmpeg.exe -y -loglevel verbose -init_hw_device "qsv=qd1:hw_any,child_device=1,child_device_type=d3d11va"  -hwaccel qsv -c:v h264_qsv -canvas_size:s:0 "1920:1080" -i red2.mkv -filter_complex "[0:0]vpp_qsv[vid];[0:3]format=nv12,hwupload@f1=extra_hw_frames=16,vpp_qsv=w=400:h=200[subs];[vid][subs]overlay_qsv[out]" -map [out] -c:v h264_qsv -b:v 5M output.mp4

It contains two uses of vpp_qsv: one in “passthrough mode” and 
the other one with scaling.
Only when replacing BOTH usages of vpp_qsv with scale_qsv, 
the command will work.

(test file on request)

Kind regards,
softworkz
Xiang, Haihao Oct. 11, 2021, 1:45 a.m. UTC | #8
On Sun, 2021-10-10 at 22:37 +0000, Soft Works wrote:
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Xiang, Haihao
> > Sent: Saturday, September 18, 2021 3:27 AM
> > To: ffmpeg-devel@ffmpeg.org
> > Subject: Re: [FFmpeg-devel] [PATCH v5 00/20] clean-up QSV filters
> > 
> > On Thu, 2021-09-09 at 14:56 +0000, Xiang, Haihao wrote:
> > > On Mon, 2021-08-30 at 08:04 +0000, Xiang, Haihao wrote:
> > > > On Mon, 2021-08-30 at 05:52 +0000, Soft Works wrote:
> 
> 
> [..]
> 
> > > > > > > 
> > > > > > > Hi Hihao,
> > > > > > > 
> > > > > > > The general idea of this patch makes sense to me.
> > > > > > > 
> > > > > > > Currently there are implementation differences between
> > 
> > these
> > > > > > 
> > > > > > filters,
> > > > > > > and there are cases where vpp_qsv  doesn't work and I need
> > 
> > to use
> > > > > > > scale_qsv instead.
> > > > > > > 
> > > > > > > I have never analyzed the actual reason, but this should be
> > 
> > done
> > > > > > > before replacing scale_qsv with an aliased vpp_qsv.
> > > > > > > 
> > > > > > > I'll try to dig out an example..
> > > > > > 
> > > > > > 
> > > > > > Hi Softworkz,
> > > > > > 
> > > > > > Could you provide the cases when you have time ? I may look
> > 
> > into the
> > > > > > issues.
> > > > > > 
> > > > > > Thanks
> > > > > > Haihao
> > > > > 
> > > > > 
> > > > > Hi Haihao,
> > > > > 
> > > > > IIRC think it can be reproduced easily by using the filters
> > 
> > without
> > > > > parameters,
> > > > > where scale_qsv works and vpp_qsv doesn't work.
> > > > 
> > > > scale_qsv only supports video memory input and output however
> > 
> > vpp_qsv may
> > > > support both video memory and system memory, there is a case
> > 
> > where scale_qsv
> > > > doesn't work but vpp_qsv may work.
> > > > 
> > > > $> ffmpeg -c:v h264_qsv -i input.mp4 -vf scale_qsv -f null -
> > > > 
> > > > Impossible to convert between the formats supported by the filter
> > 
> > 'graph 0
> > > > input
> > > > from stream 0:0' and the filter 'auto_scale_0'
> > > > Error reinitializing filters!
> > > > Failed to inject frame into filter network: Function not
> > 
> > implemented
> > > > Error while processing the decoded data for stream #0:0
> > > > Conversion failed!
> > > > 
> > > > After applying this patchset, scale_qsv may work with both video
> > 
> > memory and
> > > > system memory.
> > > > 
> > > > In addition, vpp_qsv may support crop however scale_qsv doesn't
> > 
> > support
> > > > crop.
> > > > After applying  this patchset, scale_qsv may support crop too.
> > > > 
> > > > > 
> > > > > If it doesn't reproduce, try this in combination with hwupload,
> > 
> > e.g.
> > > > > 
> > > > > - "hwupload@f1=extra_hw_frames=32,vpp_qsv"
> > > > > vs.
> > > > > - "hwupload@f1=extra_hw_frames=32,scale_qsv"
> > > > > 
> > > > 
> > > > The above two combinations work for me
> > > 
> > > Hi Softworkz,
> > > 
> > > I failed to reproduce your issue, could you provide more info if
> > 
> > possible?
> > 
> > Hi Softworkz,
> > 
> > I didn't reproduce your issue, maybe the issue has been fixed on
> > master. How
> > about to merge this patchset first? I will rebase the patchset if no
> > objection.
> >  BTW we didn't see regression in our internal testing.
> 
> 
> Hi Haihao,
> (sending a CC to the ML allowing others to follow)
> 
> I found an example for the cases I was referring to. 
> 
> An example command line is this:
> 
> ffmpeg.exe -y -loglevel verbose -init_hw_device
> "qsv=qd1:hw_any,child_device=1,child_device_type=d3d11va"  -hwaccel qsv -c:v
> h264_qsv -canvas_size:s:0 "1920:1080" -i red2.mkv -filter_complex
> "[0:0]vpp_qsv[vid];[0:3]format=nv12,hwupload@f1=extra_hw_frames=16,vpp_qsv=w=4
> 00:h=200[subs];[vid][subs]overlay_qsv[out]" -map [out] -c:v h264_qsv -b:v 5M
> output.mp4
> 
> It contains two uses of vpp_qsv: one in “passthrough mode” and 
> the other one with scaling.
> Only when replacing BOTH usages of vpp_qsv with scale_qsv, 
> the command will work.
> 
> (test file on request)

Thanks for the info, I'll look into it.

BRs
Haihao