| Message ID | 20211015133915.13624-1-haihao.xiang@intel.com |
|---|---|
| Headers | show |
| Series | make QSV works with the Intel's oneVPL | expand |
> -----Original Message----- > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > Haihao Xiang > Sent: Friday, October 15, 2021 3:39 PM > To: ffmpeg-devel@ffmpeg.org > Cc: Haihao Xiang <haihao.xiang@intel.com> > Subject: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the > Intel's oneVPL > > The oneAPI Video Processing Library (oneVPL) is a single interface > for > encode, decode and video processing[1]. oneVPL is a successor to > Intel(R) Media > SDK, but removed obsolete features. Intel(R) Media SDK lifetime comes > to an > end now, new features for new Intel Gen platforms will be supported > in oneVPL > only[2]. > > It is recommended to use oneVPL for new work, even for currently > available > hardwares[3]. Hence, this patchset added a new option --enable-onevpl > to bring > the support for oneVPL in QSV. New features for oneVPL will be > implemented in > other patchset. --enble-libmfx option still works with Intel(R) Media > SDK. > > Note user can't enable onevpl and libmfx together. > > oneVPL dispatcher source code: > https://github.com/oneapi-src/oneVPL > > oneVPL GPU runtime for new Intel Gen platforms: > https://github.com/oneapi-src/oneVPL-intel-gpu > > v6: > - Use ${libmfx_incdir} in configure > - Don't define mfx related function in an exported header > - Rebased this patchset against the latest master and fixed bugs > > [1] > https://spec.oneapi.io/versions/latest/elements/oneVPL/source/index.h > tml > [2] https://github.com/Intel-Media-SDK/MediaSDK/#media-sdk-support- > matrix > [3] > https://software.intel.com/content/www/us/en/develop/articles/upgradi > ng-from-msdk-to-onevpl.html > Hi Haihao, I really appreciate all the effort that Intel is taking to improve ffmppeg for QSV hw acceleration, but in this case I'm not convinced that this should be merged into ffmpeg at this time. Let's look at the messages from your individual commits: > User plugin isn't supported for MFX_VERSION >= 2.0[1][2]. This is in > preparation for oneVPL Support > OPAQUE memory isn't supported for MFX_VERSION >= 2.0[1][2]. This is > in preparation for oneVPL support > Audio isn't supported for MFX_VERSION >= 2.0[1][2]. This is in > preparation for oneVPL support > Multi-frame encode isn't supported for MFX_VERSION >= 2.0[1][2]. This > is in preparation for oneVPL support > MFX_RATECONTROL_LA_EXT isn't supported for MFX_VERSION >= 2.0[1][2]. > This is in preparation for oneVPL support In its current state, oneVPL is all about removal of features that have existed for a long time. In its current state, it does not provide a single benefit over libmfx 1.x New CPU gens are still supported when using the latest libmfx 1.35 All-in-all, there is no single reason for anybody to compile ffmpeg with oneVPL instead of libmfx, but several reasons against doing so. When I look at your patchset that is in large parts dealing with the removal of features that are "in preparation for oneVPL", I wonder why the ffmpeg code should go through all those changes, which are creating a lot of unnecessary noise. While I don't like the way those recent changes are performed by Intel, I'm not opposed to oneVPL in general. But from my point of view, this is not the right time to deal with it at the side of ffmpeg for the reasons above. My suggestion is to postpone this until: - the removed features are added to oneVPL and - there exist some actual benefits in using oneVPL over libmfx 1.x Kind regards, softworkz
Hello, On Fri, 15 Oct 2021, at 22:23, Soft Works wrote: > I really appreciate all the effort that Intel is taking to improve > ffmppeg for QSV hw acceleration, but in this case I'm not convinced > that this should be merged into ffmpeg at this time. > > [...] > In its current state, oneVPL is all about removal of features that > have existed for a long time. > > In its current state, it does not provide a single benefit over > libmfx 1.x Indeed. But libmfx is not supported anymore by upstream, who's moving to oneVPL, aka mfx 2.0. You can think this is a bad move from Intel to remove feature, but, de facto, they don't support libmfx anymore, the new features will come only on oneVPL and the new hardware will only be supported in oneVPL. The patchset allows to support libmfx if people want, and have the old featureset. But it also allows to support oneVPL (mfx 2.0) if people want. As long as this is a user choice, I don't see why this shouldn't be merged; knowing also that upstream tells us that they only support 2.0, with less features. (This is the issue always with upgrades of external libraries, even worse when they are hardware linked) jb
> -----Original Message----- > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > Jean-Baptiste Kempf > Sent: Tuesday, October 19, 2021 11:19 PM > To: ffmpeg-devel@ffmpeg.org > Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the > Intel's oneVPL > > Hello, > > On Fri, 15 Oct 2021, at 22:23, Soft Works wrote: > > I really appreciate all the effort that Intel is taking to improve > > ffmppeg for QSV hw acceleration, but in this case I'm not convinced > > that this should be merged into ffmpeg at this time. > > > > [...] > > In its current state, oneVPL is all about removal of features that > > have existed for a long time. > > > > In its current state, it does not provide a single benefit over > > libmfx 1.x > > Indeed. > But libmfx is not supported anymore by upstream, who's moving to > oneVPL, aka mfx 2.0. > > You can think this is a bad move from Intel to remove feature, but, > de facto, they don't support libmfx anymore, the new features will > come only on oneVPL and the new hardware will only be supported in > oneVPL. That's not exactly true. There might no new features be added but it will still be supported because oneVPL doesn't replace it, at least not the runtime. Let's look at it in detail, there are 4 components involved: 1. Dispatchers The dispatcher is what ffmpeg is interfacing with. It loads the runtimes (there are GPU and sw runtimes) and forwards calls to the runtimes. 1.1. libmfx Dispatcher (Intel Media SDK) This is the dispatcher that ffmpeg is currently using, latest version is 1.35: https://github.com/Intel-Media-SDK/MediaSDK/tree/master/api/mfx_dispatch Many are also using this variant for ffmpeg compilation: https://github.com/lu-zero/mfx_dispatch 1.2 oneVPL Dispatcher This is the new oneVPL dispatcher which has those features removed and doesn't include any new ones: https://github.com/oneapi-src/oneVPL/tree/master/dispatcher 2. Runtimes The runtimes are containing the actual implementations, which are partially CPU gen specific and include a range of hardware kernels/shaders that provide functionality that VAAPI doesn't have. The following table shows which hardware is supported by each runtime and which runtimes are loaded by the dispatchers: https://github.com/Intel-Media-SDK/MediaSDK#media-sdk-support-matrix [I'm leaving out the CPU runtimes] 2.1. Media SDK Runtime (libmfxhw64) This is what you get from building the MediaSDK repo. It supports all Intel hw up to Tiger Lake. https://github.com/Intel-Media-SDK/MediaSDK#how-to-build 2.2. oneVPL Runtime (libmfx-gen) The new oneVPL runtime supports only the very latest hardware and needs to be built from this repo: https://github.com/oneapi-src/oneVPL-intel-gpu THE CONSEQUENCES If you want to ship the runtimes as part of an application and you want to support all Intel hardware, you will always need to build and ship BOTH runtimes. The only good thing is that both dispatchers can work with both runtimes. And that means in turn that the currently used dispatcher (libmfx 1.35) will be sufficient to work even with the latest hardware. That means that there's no pressing reason to use the oneVPL dispatcher instead. New features are promised for oneVPL but there don't exist any yet. Even when those will be added, it will take a number of years until they'll get really attractive to be used, because Intel announced that these would only be available with the oneVPL runtime. The only hardware for which the dispatchers are loading the oneVPL runtime instead of the MSDK runtime are Alder Lake CPUs. From my point of view, it's of very low interest to use a feature - no matter how great it might be - that is limited to a single CPU gen. Others might think different, that's why I only said that we should at least wait until any of those new features exists. Factually there doesn't exist a single benefit or reason to adopt oneVPL at this time. Kind regards, softworkz
On Wed, 20 Oct 2021, at 01:00, Soft Works wrote: >> -----Original Message----- >> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of >> Jean-Baptiste Kempf >> Sent: Tuesday, October 19, 2021 11:19 PM >> To: ffmpeg-devel@ffmpeg.org >> Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the >> Intel's oneVPL >> >> Hello, >> >> On Fri, 15 Oct 2021, at 22:23, Soft Works wrote: >> > I really appreciate all the effort that Intel is taking to improve >> > ffmppeg for QSV hw acceleration, but in this case I'm not convinced >> > that this should be merged into ffmpeg at this time. >> > >> > [...] >> > In its current state, oneVPL is all about removal of features that >> > have existed for a long time. >> > >> > In its current state, it does not provide a single benefit over >> > libmfx 1.x >> >> Indeed. >> But libmfx is not supported anymore by upstream, who's moving to >> oneVPL, aka mfx 2.0. >> >> You can think this is a bad move from Intel to remove feature, but, >> de facto, they don't support libmfx anymore, the new features will >> come only on oneVPL and the new hardware will only be supported in >> oneVPL. > > That's not exactly true. There might no new features be added > but it will still be supported because oneVPL doesn't replace it, > at least not the runtime. When an upstream library deprecates a feature, including when we do it for libav* , downstreams need to adapt. You can say that is bad and complain, but you adapt. Intel says that oneVPL is mfx 2.0 and won’t support the old versions, so that is a fact. And this patchset does not remove any feature, since you can still compile with mfx 1.3 and keep all the features. The person compiling makes the decision. > That means that there's no pressing reason to use the oneVPL > dispatcher instead. Yes, and so? The patchset does not remove any feature since you can still compile with mfx support. > From my point of view, it's of very low interest to use a > feature - no matter how great it might be - that is limited > to a single CPU gen. Again, then compile with mfx and without the onevpl switch. Everything will be the same. > Factually there doesn't exist a single benefit or reason > to adopt oneVPL at this time. And while this true, this patchset gives the possibility to use oneVPL and does not remove mfx… You don’t think Intel strategy is a good one? Fair enough. But that is not what this patchset is about. jb
> -----Original Message----- > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > Jean-Baptiste Kempf > Sent: Wednesday, October 20, 2021 8:01 AM > To: ffmpeg-devel@ffmpeg.org > Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the > Intel's oneVPL > > > > On Wed, 20 Oct 2021, at 01:00, Soft Works wrote: > >> -----Original Message----- > >> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > >> Jean-Baptiste Kempf > >> Sent: Tuesday, October 19, 2021 11:19 PM > >> To: ffmpeg-devel@ffmpeg.org > >> Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with > the > >> Intel's oneVPL > >> > When an upstream library deprecates a feature, including when we do > it for libav* , downstreams need to adapt. You can say that is bad > and complain, but you adapt. > > Intel says that oneVPL is mfx 2.0 and won’t support the old versions, > so that is a fact. It's a marketing fact, not a technical fact, because not supporting the MSDK runtime anymore would mean to stop supporting all of their CPUs except the recent 2 or 3 generations. They also said that D3D9/DXVA2 would not be supported anymore which seems to have changed when I'm looking at the latest Windows oneVPL runtime dll. > And this patchset does not remove any feature, since you can still > compile with mfx 1.3 and keep all the features. Yes, but it adds a lot of clutter to the codebase. Based on the patch comments "..this is in preparation..", these changes would be reduced when waiting until these are available. > And while this true, this patchset gives the possibility to use > oneVPL and does not remove mfx… It just switches the dispatcher, but the runtimes being used are always the same (both dispatchers use the same two runtimes). > You don’t think Intel strategy is a good one? > Fair enough. But that is not what this patchset is about. While I don't like the strategy, it would be very stupid to oppose those changes for that reason. Without doubt, this change will need to be done at some point in time. I'm only saying that it's a bit too early. Anyway it's not my decision. I tried to clarify the situation for others to better understand and laid out my view. When the majority wants to merge this, I'll be good as well.. :-) Kind regards, softworkz
> -----Original Message----- > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > Jean-Baptiste Kempf > Sent: Wednesday, October 20, 2021 8:01 AM > To: ffmpeg-devel@ffmpeg.org > Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the > Intel's oneVPL > > > > On Wed, 20 Oct 2021, at 01:00, Soft Works wrote: > >> -----Original Message----- > >> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > >> Jean-Baptiste Kempf > >> Sent: Tuesday, October 19, 2021 11:19 PM > >> To: ffmpeg-devel@ffmpeg.org > >> Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with > the > >> Intel's oneVPL > >> > >> Hello, > >> > >> On Fri, 15 Oct 2021, at 22:23, Soft Works wrote: > >> > I really appreciate all the effort that Intel is taking to > improve > >> > ffmppeg for QSV hw acceleration, but in this case I'm not > convinced > >> > that this should be merged into ffmpeg at this time. > >> > > >> > [...] > >> > In its current state, oneVPL is all about removal of features > that > >> > have existed for a long time. > >> > > >> > In its current state, it does not provide a single benefit over > >> > libmfx 1.x > >> > >> Indeed. > >> But libmfx is not supported anymore by upstream, who's moving to > >> oneVPL, aka mfx 2.0. Intel documents say otherwise: "Intel® Media SDK API is nearing its final update. Implementation updates, such as security patches and critical bug fixes, will continue as needed." https://www.intel.com/content/www/us/en/develop/documentation/upgrading-from-msdk-to-onevpl/top/faq/how-long-will-intel-media-sdk-be-available.html "For currently available hardware, Intel® Media SDK and oneVPL provide the same access. Existing applications may continue to use Intel® Media SDK." https://www.intel.com/content/www/us/en/develop/documentation/upgrading-from-msdk-to-onevpl/top.html Kind regards, softworkz
On Wed, 2021-10-20 at 06:53 +0000, Soft Works wrote: > > -----Original Message----- > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > > Jean-Baptiste Kempf > > Sent: Wednesday, October 20, 2021 8:01 AM > > To: ffmpeg-devel@ffmpeg.org > > Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with the > > Intel's oneVPL > > > > > > > > On Wed, 20 Oct 2021, at 01:00, Soft Works wrote: > > > > -----Original Message----- > > > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of > > > > Jean-Baptiste Kempf > > > > Sent: Tuesday, October 19, 2021 11:19 PM > > > > To: ffmpeg-devel@ffmpeg.org > > > > Subject: Re: [FFmpeg-devel] [PATCH v6 00/10] make QSV works with > > > > the > > > > Intel's oneVPL > > > > > > > > Hello, > > > > > > > > On Fri, 15 Oct 2021, at 22:23, Soft Works wrote: > > > > > I really appreciate all the effort that Intel is taking to > > > > improve > > > > > ffmppeg for QSV hw acceleration, but in this case I'm not > > > > convinced > > > > > that this should be merged into ffmpeg at this time. > > > > > > > > > > [...] > > > > > In its current state, oneVPL is all about removal of features > > > > that > > > > > have existed for a long time. > > > > > > > > > > In its current state, it does not provide a single benefit over > > > > > libmfx 1.x > > > > > > > > Indeed. > > > > But libmfx is not supported anymore by upstream, who's moving to > > > > oneVPL, aka mfx 2.0. > > Intel documents say otherwise: > > "Intel® Media SDK API is nearing its final update. Implementation updates, > such as security patches and critical bug fixes, will continue as needed." > > https://www.intel.com/content/www/us/en/develop/documentation/upgrading-from-msdk-to-onevpl/top/faq/how-long-will-intel-media-sdk-be-available.html > > "For currently available hardware, Intel® Media SDK and oneVPL provide the > same access. Existing applications may continue to use Intel® Media SDK." > > https://www.intel.com/content/www/us/en/develop/documentation/upgrading-from-msdk-to-onevpl/top.html > According to this doc, new features are supported in oneVPL only. We will provide av1_qsv encoder which is available for --enable-libvpl only, Please see https://github.com/intel-media-ci/ffmpeg/pull/515/commits/f4b500cb977f9506092831fda8f34b207feae618 if you are interested. Thanks Haihao
The oneAPI Video Processing Library (oneVPL) is a single interface for encode, decode and video processing[1]. oneVPL is a successor to Intel(R) Media SDK, but removed obsolete features. Intel(R) Media SDK lifetime comes to an end now, new features for new Intel Gen platforms will be supported in oneVPL only[2]. It is recommended to use oneVPL for new work, even for currently available hardwares[3]. Hence, this patchset added a new option --enable-onevpl to bring the support for oneVPL in QSV. New features for oneVPL will be implemented in other patchset. --enble-libmfx option still works with Intel(R) Media SDK. Note user can't enable onevpl and libmfx together. oneVPL dispatcher source code: https://github.com/oneapi-src/oneVPL oneVPL GPU runtime for new Intel Gen platforms: https://github.com/oneapi-src/oneVPL-intel-gpu v6: - Use ${libmfx_incdir} in configure - Don't define mfx related function in an exported header - Rebased this patchset against the latest master and fixed bugs [1] https://spec.oneapi.io/versions/latest/elements/oneVPL/source/index.html [2] https://github.com/Intel-Media-SDK/MediaSDK/#media-sdk-support-matrix [3] https://software.intel.com/content/www/us/en/develop/articles/upgrading-from-msdk-to-onevpl.html Haihao Xiang (10): configure: ensure --enable-libmfx uses libmfx 1.x configure: fix the check for MFX_CODEC_VP9 qsv: remove mfx/ prefix from mfx headers qsv: load user plugin for MFX_VERSION < 2.0 qsv: build audio related code when MFX_VERSION < 2.0 qsvenc: support multi-frame encode when MFX_VERSION < 2.0 qsvenc: support MFX_RATECONTROL_LA_EXT when MFX_VERSION < 2.0 qsv: support OPAQUE memory when MFX_VERSION < 2.0 qsv: use a new method to create mfx session when using oneVPL configure: add --enable-libvpl option configure | 29 ++- libavcodec/qsv.c | 220 ++++++++++++++-- libavcodec/qsv.h | 4 +- libavcodec/qsv_internal.h | 6 +- libavcodec/qsvdec.c | 21 +- libavcodec/qsvenc.c | 25 +- libavcodec/qsvenc.h | 9 +- libavcodec/qsvenc_h264.c | 3 +- libavcodec/qsvenc_hevc.c | 3 +- libavcodec/qsvenc_jpeg.c | 3 +- libavcodec/qsvenc_mpeg2.c | 3 +- libavcodec/qsvenc_vp9.c | 3 +- libavfilter/qsvvpp.c | 145 ++++++++++- libavfilter/qsvvpp.h | 12 +- libavfilter/vf_deinterlace_qsv.c | 73 +++--- libavfilter/vf_scale_qsv.c | 88 ++++--- libavutil/hwcontext_d3d11va.c | 13 + libavutil/hwcontext_d3d11va.h | 5 + libavutil/hwcontext_dxva2.c | 8 + libavutil/hwcontext_dxva2.h | 4 + libavutil/hwcontext_opencl.c | 2 +- libavutil/hwcontext_qsv.c | 423 +++++++++++++++++++++++++++---- libavutil/hwcontext_qsv.h | 3 +- libavutil/hwcontext_vaapi.c | 13 + libavutil/hwcontext_vaapi.h | 4 + 25 files changed, 934 insertions(+), 188 deletions(-)