mbox series

[FFmpeg-devel,v6,00/10] make QSV works with the Intel's oneVPL

Message ID 20211015133915.13624-1-haihao.xiang@intel.com
Headers show
Series make QSV works with the Intel's oneVPL
Related show

Message

Xiang, Haihao Oct. 15, 2021, 1:39 p.m. UTC
The oneAPI Video Processing Library (oneVPL) is a single interface for
encode, decode and video processing[1]. oneVPL is a successor to Intel(R) Media
SDK, but removed obsolete features. Intel(R) Media SDK lifetime comes to an
end now, new features for new Intel Gen platforms will be supported in oneVPL
only[2].

It is recommended to use oneVPL for new work, even for currently available
hardwares[3]. Hence, this patchset added a new option --enable-onevpl to bring
the support for oneVPL in QSV. New features for oneVPL will be implemented in
other patchset. --enble-libmfx option still works with Intel(R) Media SDK.

Note user can't enable onevpl and libmfx together.

oneVPL dispatcher source code:
https://github.com/oneapi-src/oneVPL

oneVPL GPU runtime for new Intel Gen platforms:
https://github.com/oneapi-src/oneVPL-intel-gpu

v6:
  - Use ${libmfx_incdir} in configure
  - Don't define mfx related function in an exported header
  - Rebased this patchset against the latest master and fixed bugs

[1] https://spec.oneapi.io/versions/latest/elements/oneVPL/source/index.html
[2] https://github.com/Intel-Media-SDK/MediaSDK/#media-sdk-support-matrix
[3] https://software.intel.com/content/www/us/en/develop/articles/upgrading-from-msdk-to-onevpl.html

Haihao Xiang (10):
  configure: ensure --enable-libmfx uses libmfx 1.x
  configure: fix the check for MFX_CODEC_VP9
  qsv: remove mfx/ prefix from mfx headers
  qsv: load user plugin for MFX_VERSION < 2.0
  qsv: build audio related code when MFX_VERSION < 2.0
  qsvenc: support multi-frame encode when MFX_VERSION < 2.0
  qsvenc: support MFX_RATECONTROL_LA_EXT when MFX_VERSION < 2.0
  qsv: support OPAQUE memory when MFX_VERSION < 2.0
  qsv: use a new method to create mfx session when using oneVPL
  configure: add --enable-libvpl option

 configure                        |  29 ++-
 libavcodec/qsv.c                 | 220 ++++++++++++++--
 libavcodec/qsv.h                 |   4 +-
 libavcodec/qsv_internal.h        |   6 +-
 libavcodec/qsvdec.c              |  21 +-
 libavcodec/qsvenc.c              |  25 +-
 libavcodec/qsvenc.h              |   9 +-
 libavcodec/qsvenc_h264.c         |   3 +-
 libavcodec/qsvenc_hevc.c         |   3 +-
 libavcodec/qsvenc_jpeg.c         |   3 +-
 libavcodec/qsvenc_mpeg2.c        |   3 +-
 libavcodec/qsvenc_vp9.c          |   3 +-
 libavfilter/qsvvpp.c             | 145 ++++++++++-
 libavfilter/qsvvpp.h             |  12 +-
 libavfilter/vf_deinterlace_qsv.c |  73 +++---
 libavfilter/vf_scale_qsv.c       |  88 ++++---
 libavutil/hwcontext_d3d11va.c    |  13 +
 libavutil/hwcontext_d3d11va.h    |   5 +
 libavutil/hwcontext_dxva2.c      |   8 +
 libavutil/hwcontext_dxva2.h      |   4 +
 libavutil/hwcontext_opencl.c     |   2 +-
 libavutil/hwcontext_qsv.c        | 423 +++++++++++++++++++++++++++----
 libavutil/hwcontext_qsv.h        |   3 +-
 libavutil/hwcontext_vaapi.c      |  13 +
 libavutil/hwcontext_vaapi.h      |   4 +
 25 files changed, 934 insertions(+), 188 deletions(-)

Comments

Soft Works Oct. 15, 2021, 8:23 p.m. UTC | #1
> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Haihao Xiang
> Sent: Friday, October 15, 2021 3:39 PM
> To: ffmpeg-devel@ffmpeg.org
> Cc: Haihao Xiang <haihao.xiang@intel.com>
> Subject: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the
> Intel's oneVPL
> 
> The oneAPI Video Processing Library (oneVPL) is a single interface
> for
> encode, decode and video processing[1]. oneVPL is a successor to
> Intel(R) Media
> SDK, but removed obsolete features. Intel(R) Media SDK lifetime comes
> to an
> end now, new features for new Intel Gen platforms will be supported
> in oneVPL
> only[2].
> 
> It is recommended to use oneVPL for new work, even for currently
> available
> hardwares[3]. Hence, this patchset added a new option --enable-onevpl
> to bring
> the support for oneVPL in QSV. New features for oneVPL will be
> implemented in
> other patchset. --enble-libmfx option still works with Intel(R) Media
> SDK.
> 
> Note user can't enable onevpl and libmfx together.
> 
> oneVPL dispatcher source code:
> https://github.com/oneapi-src/oneVPL
> 
> oneVPL GPU runtime for new Intel Gen platforms:
> https://github.com/oneapi-src/oneVPL-intel-gpu
> 
> v6:
>   - Use ${libmfx_incdir} in configure
>   - Don't define mfx related function in an exported header
>   - Rebased this patchset against the latest master and fixed bugs
> 
> [1]
> https://spec.oneapi.io/versions/latest/elements/oneVPL/source/index.h
> tml
> [2] https://github.com/Intel-Media-SDK/MediaSDK/#media-sdk-support-
> matrix
> [3]
> https://software.intel.com/content/www/us/en/develop/articles/upgradi
> ng-from-msdk-to-onevpl.html
> 

Hi Haihao,

I really appreciate all the effort that Intel is taking to improve 
ffmppeg for QSV hw acceleration, but in this case I'm not convinced
that this should be merged into ffmpeg at this time.

Let's look at the messages from your individual commits:

> User plugin isn't supported for MFX_VERSION >= 2.0[1][2]. This is in
> preparation for oneVPL Support

> OPAQUE memory isn't supported for MFX_VERSION >= 2.0[1][2]. This is
> in preparation for oneVPL support

> Audio isn't supported for MFX_VERSION >= 2.0[1][2]. This is in
> preparation for oneVPL support

> Multi-frame encode isn't supported for MFX_VERSION >= 2.0[1][2]. This
> is in preparation for oneVPL support

> MFX_RATECONTROL_LA_EXT isn't supported for MFX_VERSION >= 2.0[1][2].
> This is in preparation for oneVPL support

In its current state, oneVPL is all about removal of features that
have existed for a long time.

In its current state, it does not provide a single benefit over
libmfx 1.x

New CPU gens are still supported when using the latest libmfx 1.35

All-in-all, there is no single reason for anybody to compile 
ffmpeg with oneVPL instead of libmfx, but several reasons
against doing so.

When I look at your patchset that is in large parts dealing with
the removal of features that are "in preparation for oneVPL", 
I wonder why the ffmpeg code should go through all those 
changes, which are creating a lot of unnecessary noise.

While I don't like the way those recent changes are performed 
by Intel, I'm not opposed to oneVPL in general.
But from my point of view, this is not the right time to deal 
with it at the side of ffmpeg for the reasons above.

My suggestion is to postpone this until:

- the removed features are added to oneVPL
  and
- there exist some actual benefits in using oneVPL over libmfx 1.x

Kind regards,
softworkz
Jean-Baptiste Kempf Oct. 19, 2021, 9:19 p.m. UTC | #2
Hello,

On Fri, 15 Oct 2021, at 22:23, Soft Works wrote:
> I really appreciate all the effort that Intel is taking to improve 
> ffmppeg for QSV hw acceleration, but in this case I'm not convinced
> that this should be merged into ffmpeg at this time.
>
> [...]
> In its current state, oneVPL is all about removal of features that
> have existed for a long time.
>
> In its current state, it does not provide a single benefit over
> libmfx 1.x

Indeed.
But libmfx is not supported anymore by upstream, who's moving to oneVPL, aka mfx 2.0.

You can think this is a bad move from Intel to remove feature, but, de facto, they don't support libmfx anymore, the new features will come only on oneVPL and the new hardware will only be supported in oneVPL.

The patchset allows to support libmfx if people want, and have the old featureset.
But it also allows to support oneVPL (mfx 2.0) if people want.

As long as this is a user choice, I don't see why this shouldn't be merged;
knowing also that upstream tells us that they only support 2.0, with less features.
(This is the issue always with upgrades of external libraries, even worse when they are hardware linked)

jb
Soft Works Oct. 19, 2021, 11 p.m. UTC | #3
> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Jean-Baptiste Kempf
> Sent: Tuesday, October 19, 2021 11:19 PM
> To: ffmpeg-devel@ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the
> Intel's oneVPL
> 
> Hello,
> 
> On Fri, 15 Oct 2021, at 22:23, Soft Works wrote:
> > I really appreciate all the effort that Intel is taking to improve
> > ffmppeg for QSV hw acceleration, but in this case I'm not convinced
> > that this should be merged into ffmpeg at this time.
> >
> > [...]
> > In its current state, oneVPL is all about removal of features that
> > have existed for a long time.
> >
> > In its current state, it does not provide a single benefit over
> > libmfx 1.x
> 
> Indeed.
> But libmfx is not supported anymore by upstream, who's moving to
> oneVPL, aka mfx 2.0.
> 
> You can think this is a bad move from Intel to remove feature, but,
> de facto, they don't support libmfx anymore, the new features will
> come only on oneVPL and the new hardware will only be supported in
> oneVPL.

That's not exactly true. There might no new features be added
but it will still be supported because oneVPL doesn't replace it,
at least not the runtime.

Let's look at it in detail, there are 4 components involved:

1. Dispatchers

The dispatcher is what ffmpeg is interfacing with. It loads the
runtimes (there are GPU and sw runtimes) and forwards calls to 
the runtimes.


1.1. libmfx Dispatcher (Intel Media SDK)

This is the dispatcher that ffmpeg is currently using, latest
version is 1.35:

https://github.com/Intel-Media-SDK/MediaSDK/tree/master/api/mfx_dispatch

Many are also using this variant for ffmpeg compilation:
https://github.com/lu-zero/mfx_dispatch


1.2 oneVPL Dispatcher

This is the new oneVPL dispatcher which has those features 
removed and doesn't include any new ones:

https://github.com/oneapi-src/oneVPL/tree/master/dispatcher



2. Runtimes

The runtimes are containing the actual implementations, which 
are partially CPU gen specific and include a range of hardware
kernels/shaders that provide functionality that VAAPI doesn't
have.

The following table shows which hardware is supported by each
runtime and which runtimes are loaded by the dispatchers:

https://github.com/Intel-Media-SDK/MediaSDK#media-sdk-support-matrix

[I'm leaving out the CPU runtimes]


2.1. Media SDK Runtime (libmfxhw64)

This is what you get from building the MediaSDK repo. It supports 
all Intel hw up to Tiger Lake.

https://github.com/Intel-Media-SDK/MediaSDK#how-to-build


2.2. oneVPL Runtime (libmfx-gen)

The new oneVPL runtime supports only the very latest hardware and
needs to be built  from this repo:

https://github.com/oneapi-src/oneVPL-intel-gpu


THE CONSEQUENCES

If you want to ship the runtimes as part of an application and you
want to support all Intel hardware, you will always need to

    build and ship BOTH runtimes.

The only good thing is that both dispatchers can work with both
runtimes.

And that means in turn that the currently used dispatcher (libmfx
1.35) will be sufficient to work even with the latest hardware.
That means that there's no pressing reason to use the oneVPL 
dispatcher instead.

New features are promised for oneVPL but there don't exist any
yet. Even when those will be added, it will take a number of
years until they'll get really attractive to be used, because
Intel announced that these would only be available with the 
oneVPL runtime.

The only hardware for which the dispatchers are loading the 
oneVPL runtime instead of the MSDK runtime are Alder Lake CPUs.

From my point of view, it's of very low interest to use a
feature - no matter how great it might be - that is limited
to a single CPU gen.

Others might think different, that's why I only said that we 
should at least wait until any of those new features exists.

Factually there doesn't exist a single benefit or reason
to adopt oneVPL at this time.


Kind regards,
softworkz
Jean-Baptiste Kempf Oct. 20, 2021, 6 a.m. UTC | #4
On Wed, 20 Oct 2021, at 01:00, Soft Works wrote:
>> -----Original Message-----
>> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
>> Jean-Baptiste Kempf
>> Sent: Tuesday, October 19, 2021 11:19 PM
>> To: ffmpeg-devel@ffmpeg.org
>> Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the
>> Intel's oneVPL
>> 
>> Hello,
>> 
>> On Fri, 15 Oct 2021, at 22:23, Soft Works wrote:
>> > I really appreciate all the effort that Intel is taking to improve
>> > ffmppeg for QSV hw acceleration, but in this case I'm not convinced
>> > that this should be merged into ffmpeg at this time.
>> >
>> > [...]
>> > In its current state, oneVPL is all about removal of features that
>> > have existed for a long time.
>> >
>> > In its current state, it does not provide a single benefit over
>> > libmfx 1.x
>> 
>> Indeed.
>> But libmfx is not supported anymore by upstream, who's moving to
>> oneVPL, aka mfx 2.0.
>> 
>> You can think this is a bad move from Intel to remove feature, but,
>> de facto, they don't support libmfx anymore, the new features will
>> come only on oneVPL and the new hardware will only be supported in
>> oneVPL.
>
> That's not exactly true. There might no new features be added
> but it will still be supported because oneVPL doesn't replace it,
> at least not the runtime.

When an upstream library deprecates a feature, including when we do it for libav* , downstreams need to adapt. You can say that is bad and complain, but you adapt.

Intel says that oneVPL is mfx 2.0 and won’t support the old versions, so that is a fact.

And this patchset does not remove any feature, since you can still compile with mfx 1.3 and keep all the features. The person compiling makes the decision.

> That means that there's no pressing reason to use the oneVPL 
> dispatcher instead.

Yes, and so? The patchset does not remove any feature since you can still compile with mfx support.

> From my point of view, it's of very low interest to use a
> feature - no matter how great it might be - that is limited
> to a single CPU gen.

Again, then compile with mfx and without the onevpl switch. Everything will be the same.

> Factually there doesn't exist a single benefit or reason
> to adopt oneVPL at this time.

And while this true, this patchset gives the possibility to use oneVPL and does not remove mfx…

You don’t think Intel strategy is a good one?
Fair enough. But that is not what this patchset is about.

jb
Soft Works Oct. 20, 2021, 6:43 a.m. UTC | #5
> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Jean-Baptiste Kempf
> Sent: Wednesday, October 20, 2021 8:01 AM
> To: ffmpeg-devel@ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the
> Intel's oneVPL
> 
> 
> 
> On Wed, 20 Oct 2021, at 01:00, Soft Works wrote:
> >> -----Original Message-----
> >> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> >> Jean-Baptiste Kempf
> >> Sent: Tuesday, October 19, 2021 11:19 PM
> >> To: ffmpeg-devel@ffmpeg.org
> >> Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with
> the
> >> Intel's oneVPL
> >>
> When an upstream library deprecates a feature, including when we do
> it for libav* , downstreams need to adapt. You can say that is bad
> and complain, but you adapt.
> 
> Intel says that oneVPL is mfx 2.0 and won’t support the old versions,
> so that is a fact.

It's a marketing fact, not a technical fact, because not supporting 
the MSDK runtime anymore would mean to stop supporting all of their
CPUs except the recent 2 or 3 generations.

They also said that D3D9/DXVA2 would not be supported anymore which seems 
to have changed when I'm looking at the latest Windows oneVPL runtime dll.

> And this patchset does not remove any feature, since you can still
> compile with mfx 1.3 and keep all the features. 

Yes, but it adds a lot of clutter to the codebase. Based on the patch
comments "..this is in preparation..", these changes would be 
reduced when waiting until these are available.

> And while this true, this patchset gives the possibility to use
> oneVPL and does not remove mfx…

It just switches the dispatcher, but the runtimes being used are 
always the same (both dispatchers use the same two runtimes).

> You don’t think Intel strategy is a good one?
> Fair enough. But that is not what this patchset is about.

While I don't like the strategy, it would be very stupid to oppose 
those changes for that reason. Without doubt, this change will 
need to be done at some point in time.

I'm only saying that it's a bit too early.

Anyway it's not my decision. I tried to clarify the situation for others
to better understand and laid out my view. 

When the majority wants to merge this, I'll be good as well.. :-)

Kind regards,
softworkz
Soft Works Oct. 20, 2021, 6:53 a.m. UTC | #6
> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Jean-Baptiste Kempf
> Sent: Wednesday, October 20, 2021 8:01 AM
> To: ffmpeg-devel@ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the
> Intel's oneVPL
> 
> 
> 
> On Wed, 20 Oct 2021, at 01:00, Soft Works wrote:
> >> -----Original Message-----
> >> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> >> Jean-Baptiste Kempf
> >> Sent: Tuesday, October 19, 2021 11:19 PM
> >> To: ffmpeg-devel@ffmpeg.org
> >> Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with
> the
> >> Intel's oneVPL
> >>
> >> Hello,
> >>
> >> On Fri, 15 Oct 2021, at 22:23, Soft Works wrote:
> >> > I really appreciate all the effort that Intel is taking to
> improve
> >> > ffmppeg for QSV hw acceleration, but in this case I'm not
> convinced
> >> > that this should be merged into ffmpeg at this time.
> >> >
> >> > [...]
> >> > In its current state, oneVPL is all about removal of features
> that
> >> > have existed for a long time.
> >> >
> >> > In its current state, it does not provide a single benefit over
> >> > libmfx 1.x
> >>
> >> Indeed.
> >> But libmfx is not supported anymore by upstream, who's moving to
> >> oneVPL, aka mfx 2.0.

Intel documents say otherwise:

"Intel® Media SDK API is nearing its final update. Implementation updates,
such as security patches and critical bug fixes, will continue as needed."

https://www.intel.com/content/www/us/en/develop/documentation/upgrading-from-msdk-to-onevpl/top/faq/how-long-will-intel-media-sdk-be-available.html

"For currently available hardware, Intel® Media SDK and oneVPL provide the
same access. Existing applications may continue to use Intel® Media SDK."

https://www.intel.com/content/www/us/en/develop/documentation/upgrading-from-msdk-to-onevpl/top.html

Kind regards,
softworkz