diff mbox series

[FFmpeg-devel] hwcontext_videotoolbox: add vt_device_derive

Message ID 20240222184603.13074-1-gnattuoc@me.com
State New
Headers show
Series [FFmpeg-devel] hwcontext_videotoolbox: add vt_device_derive | expand

Checks

Context Check Description
yinshiyou/make_loongarch64 success Make finished
yinshiyou/make_fate_loongarch64 success Make fate finished
andriy/make_x86 success Make finished
andriy/make_fate_x86 success Make fate finished

Commit Message

gnattu Feb. 22, 2024, 6:46 p.m. UTC
There is no device context to be setup, nor devices to be
selected with VideoToolbox. Just a simple return would allow
us to use derived device in filters like
`hwupload=derive_device=videotoolbox`

Signed-off-by: Gnattu OC <gnattuoc@me.com>
---
 libavutil/hwcontext_videotoolbox.c | 9 +++++++++
 1 file changed, 9 insertions(+)

Comments

Mark Thompson Feb. 22, 2024, 7:28 p.m. UTC | #1
On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
> There is no device context to be setup, nor devices to be
> selected with VideoToolbox. Just a simple return would allow
> us to use derived device in filters like
> `hwupload=derive_device=videotoolbox`
> 
> Signed-off-by: Gnattu OC <gnattuoc@me.com>
> ---
>   libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 
> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
> index fe469dc161..d13199eca7 100644
> --- a/libavutil/hwcontext_videotoolbox.c
> +++ b/libavutil/hwcontext_videotoolbox.c
> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>       return 0;
>   }
>   
> +static int vt_device_derive(AVHWDeviceContext *device_ctx,
> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
> +                            int flags)
> +{
> +    // There is no context to be setup with VT, just return.
> +    return 0;
> +}
> +
>   const HWContextType ff_hwcontext_type_videotoolbox = {
>       .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>       .name                 = "videotoolbox",
> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>       .frames_priv_size     = sizeof(VTFramesContext),
>   
>       .device_create        = vt_device_create,
> +    .device_derive        = vt_device_derive,
>       .frames_hwctx_size    = sizeof(AVVTFramesContext),
>       .frames_init          = vt_frames_init,
>       .frames_get_buffer    = vt_get_buffer,

This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.

(Also, try -init_hw_device.)

Thanks,

- Mark
gnattu Feb. 22, 2024, 7:39 p.m. UTC | #2
Well this does make sense. 

If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.

> On Feb 23, 2024, at 03:28, Mark Thompson <sw@jkqxz.net> wrote:
> 
> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>> There is no device context to be setup, nor devices to be
>> selected with VideoToolbox. Just a simple return would allow
>> us to use derived device in filters like
>> `hwupload=derive_device=videotoolbox`
>> Signed-off-by: Gnattu OC <gnattuoc@me.com>
>> ---
>>  libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>  1 file changed, 9 insertions(+)
>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>> index fe469dc161..d13199eca7 100644
>> --- a/libavutil/hwcontext_videotoolbox.c
>> +++ b/libavutil/hwcontext_videotoolbox.c
>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>      return 0;
>>  }
>>  +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>> +                            int flags)
>> +{
>> +    // There is no context to be setup with VT, just return.
>> +    return 0;
>> +}
>> +
>>  const HWContextType ff_hwcontext_type_videotoolbox = {
>>      .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>      .name                 = "videotoolbox",
>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>      .frames_priv_size     = sizeof(VTFramesContext),
>>        .device_create        = vt_device_create,
>> +    .device_derive        = vt_device_derive,
>>      .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>      .frames_init          = vt_frames_init,
>>      .frames_get_buffer    = vt_get_buffer,
> 
> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
> 
> (Also, try -init_hw_device.)
> 
> Thanks,
> 
> - Mark
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org <mailto:ffmpeg-devel@ffmpeg.org>
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org <mailto:ffmpeg-devel-request@ffmpeg.org> with subject "unsubscribe".
Mark Thompson Feb. 22, 2024, 7:50 p.m. UTC | #3
On 22/02/2024 19:39, ChenLiucheng via ffmpeg-devel wrote:
>> On Feb 23, 2024, at 03:28, Mark Thompson <sw@jkqxz.net> wrote:
>>
>> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>>> There is no device context to be setup, nor devices to be
>>> selected with VideoToolbox. Just a simple return would allow
>>> us to use derived device in filters like
>>> `hwupload=derive_device=videotoolbox`
>>> Signed-off-by: Gnattu OC <gnattuoc@me.com>
>>> ---
>>>   libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>>   1 file changed, 9 insertions(+)
>>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>>> index fe469dc161..d13199eca7 100644
>>> --- a/libavutil/hwcontext_videotoolbox.c
>>> +++ b/libavutil/hwcontext_videotoolbox.c
>>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>>       return 0;
>>>   }
>>>   +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>>> +                            int flags)
>>> +{
>>> +    // There is no context to be setup with VT, just return.
>>> +    return 0;
>>> +}
>>> +
>>>   const HWContextType ff_hwcontext_type_videotoolbox = {
>>>       .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>>       .name                 = "videotoolbox",
>>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>>       .frames_priv_size     = sizeof(VTFramesContext),
>>>         .device_create        = vt_device_create,
>>> +    .device_derive        = vt_device_derive,
>>>       .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>>       .frames_init          = vt_frames_init,
>>>       .frames_get_buffer    = vt_get_buffer,
>>
>> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
>>
>> (Also, try -init_hw_device.)
>>
 > Well this does make sense.
 >
 > If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.
 >

Yes, it is unfortunate that the ffmpeg utility is missing this feature.

A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly, but the ffmpeg utility does not fully expose the functionality of libavfilter to be able to do this.

This is mostly for syntax reasons; implementation would be straightforward.  If you can think of a good way to represent this in options to the ffmpeg utility (which doesn't become too horrible with escaping) then I would very much welcome suggestions.

Thanks,

- Mark
gnattu Feb. 22, 2024, 8:19 p.m. UTC | #4
Actually, if you examine the `hwcontext_videotoolbox.c` file, you will find that the current function for `vt_device_create` is essentially a no-op: it only returns an error if a user attempts to select a device, otherwise it just returns. This is generally how VideoToolbox works. If the `create_device` function is already a no-op, then the derive operation will certainly also be a no-op. All other hardware filters can utilize the derive API, and there are use cases that depend on this API. Thus, it seems inappropriate to me that VideoToolbox users cannot use the derive API just because VideoToolbox does not require any device creation.

> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly

Can't this just be the modification of the `hwupload` filter? Instead of derive, the user just set the which initialized hardware to use, as `-init_hw_device` can already be called multiple times and creating alias for each hardware.

> On Feb 23, 2024, at 03:50, Mark Thompson <sw@jkqxz.net> wrote:
> 
> On 22/02/2024 19:39, ChenLiucheng via ffmpeg-devel wrote:
>>> On Feb 23, 2024, at 03:28, Mark Thompson <sw@jkqxz.net> wrote:
>>> 
>>> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>>>> There is no device context to be setup, nor devices to be
>>>> selected with VideoToolbox. Just a simple return would allow
>>>> us to use derived device in filters like
>>>> `hwupload=derive_device=videotoolbox`
>>>> Signed-off-by: Gnattu OC <gnattuoc@me.com>
>>>> ---
>>>>  libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>>>  1 file changed, 9 insertions(+)
>>>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>>>> index fe469dc161..d13199eca7 100644
>>>> --- a/libavutil/hwcontext_videotoolbox.c
>>>> +++ b/libavutil/hwcontext_videotoolbox.c
>>>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>>>      return 0;
>>>>  }
>>>>  +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>>>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>>>> +                            int flags)
>>>> +{
>>>> +    // There is no context to be setup with VT, just return.
>>>> +    return 0;
>>>> +}
>>>> +
>>>>  const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>      .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>>>      .name                 = "videotoolbox",
>>>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>      .frames_priv_size     = sizeof(VTFramesContext),
>>>>        .device_create        = vt_device_create,
>>>> +    .device_derive        = vt_device_derive,
>>>>      .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>>>      .frames_init          = vt_frames_init,
>>>>      .frames_get_buffer    = vt_get_buffer,
>>> 
>>> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
>>> 
>>> (Also, try -init_hw_device.)
>>> 
> > Well this does make sense.
> >
> > If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.
> >
> 
> Yes, it is unfortunate that the ffmpeg utility is missing this feature.
> 
> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly, but the ffmpeg utility does not fully expose the functionality of libavfilter to be able to do this.
> 
> This is mostly for syntax reasons; implementation would be straightforward.  If you can think of a good way to represent this in options to the ffmpeg utility (which doesn't become too horrible with escaping) then I would very much welcome suggestions.
> 
> Thanks,
> 
> - Mark
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Zhao Zhili Feb. 23, 2024, 1:20 a.m. UTC | #5
> 在 2024年2月23日,上午4:19,Gnattu OC via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> 写道:
> 
> Actually, if you examine the `hwcontext_videotoolbox.c` file, you will find that the current function for `vt_device_create` is essentially a no-op: it only returns an error if a user attempts to select a device, otherwise it just returns. This is generally how VideoToolbox works. If the `create_device` function is already a no-op, then the derive operation will certainly also be a no-op.

Device derive means inter operation between different hwcontext, which doesn’t supported by videotoolbox yet, so a no-op implementation for now is misleading.

I’m trying to implement opencl and videotoolbox inter operation, the no-op implementation can make sense afterwards.

> All other hardware filters can utilize the derive API, and there are use cases that depend on this API. Thus, it seems inappropriate to me that VideoToolbox users cannot use the derive API just because VideoToolbox does not require any device creation.
> 
>> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly
> 
> Can't this just be the modification of the `hwupload` filter? Instead of derive, the user just set the which initialized hardware to use, as `-init_hw_device` can already be called multiple times and creating alias for each hardware.
> 
>> On Feb 23, 2024, at 03:50, Mark Thompson <sw@jkqxz.net> wrote:
>> 
>> On 22/02/2024 19:39, ChenLiucheng via ffmpeg-devel wrote:
>>>> On Feb 23, 2024, at 03:28, Mark Thompson <sw@jkqxz.net> wrote:
>>>> 
>>>>> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>>>>>> There is no device context to be setup, nor devices to be
>>>>>> selected with VideoToolbox. Just a simple return would allow
>>>>>> us to use derived device in filters like
>>>>>> `hwupload=derive_device=videotoolbox`
>>>>>> Signed-off-by: Gnattu OC <gnattuoc@me.com>
>>>>>> ---
>>>>>> libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>>>>> 1 file changed, 9 insertions(+)
>>>>>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>>>>>> index fe469dc161..d13199eca7 100644
>>>>>> --- a/libavutil/hwcontext_videotoolbox.c
>>>>>> +++ b/libavutil/hwcontext_videotoolbox.c
>>>>>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>>>>>     return 0;
>>>>>> }
>>>>>> +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>>>>>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>>>>>> +                            int flags)
>>>>>> +{
>>>>>> +    // There is no context to be setup with VT, just return.
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>>>     .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>>>>>     .name                 = "videotoolbox",
>>>>>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>>>     .frames_priv_size     = sizeof(VTFramesContext),
>>>>>>       .device_create        = vt_device_create,
>>>>>> +    .device_derive        = vt_device_derive,
>>>>>>     .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>>>>>     .frames_init          = vt_frames_init,
>>>>>>     .frames_get_buffer    = vt_get_buffer,
>>>>> 
>>>>> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
>>>>> 
>>>>> (Also, try -init_hw_device.)
>>>>> 
>>> Well this does make sense.
>>> 
>>> If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.
>>> 
>> 
>> Yes, it is unfortunate that the ffmpeg utility is missing this feature.
>> 
>> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly, but the ffmpeg utility does not fully expose the functionality of libavfilter to be able to do this.
>> 
>> This is mostly for syntax reasons; implementation would be straightforward.  If you can think of a good way to represent this in options to the ffmpeg utility (which doesn't become too horrible with escaping) then I would very much welcome suggestions.
>> 
>> Thanks,
>> 
>> - Mark
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> 
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
Zhao Zhili Feb. 23, 2024, 1:31 a.m. UTC | #6
> 在 2024年2月23日,上午9:20,Zhao Zhili <quinkblack@foxmail.com> 写道:
> 
> 
>> 在 2024年2月23日,上午4:19,Gnattu OC via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> 写道:
>> 
>> Actually, if you examine the `hwcontext_videotoolbox.c` file, you will find that the current function for `vt_device_create` is essentially a no-op: it only returns an error if a user attempts to select a device, otherwise it just returns. This is generally how VideoToolbox works. If the `create_device` function is already a no-op, then the derive operation will certainly also be a no-op.
> 
> Device derive means inter operation between different hwcontext, which doesn’t supported by videotoolbox yet, so a no-op implementation for now is misleading.
> 
> I’m trying to implement opencl and videotoolbox inter operation, the no-op implementation can make sense afterwards.

I mean a real implementation of device derive.

> 
>> All other hardware filters can utilize the derive API, and there are use cases that depend on this API. Thus, it seems inappropriate to me that VideoToolbox users cannot use the derive API just because VideoToolbox does not require any device creation.
>> 
>>> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly
>> 
>> Can't this just be the modification of the `hwupload` filter? Instead of derive, the user just set the which initialized hardware to use, as `-init_hw_device` can already be called multiple times and creating alias for each hardware.
>> 
>>>> On Feb 23, 2024, at 03:50, Mark Thompson <sw@jkqxz.net> wrote:
>>> 
>>> On 22/02/2024 19:39, ChenLiucheng via ffmpeg-devel wrote:
>>>>> On Feb 23, 2024, at 03:28, Mark Thompson <sw@jkqxz.net> wrote:
>>>>> 
>>>>>> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>>>>>>> There is no device context to be setup, nor devices to be
>>>>>>> selected with VideoToolbox. Just a simple return would allow
>>>>>>> us to use derived device in filters like
>>>>>>> `hwupload=derive_device=videotoolbox`
>>>>>>> Signed-off-by: Gnattu OC <gnattuoc@me.com>
>>>>>>> ---
>>>>>>> libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>>>>>> 1 file changed, 9 insertions(+)
>>>>>>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>>>>>>> index fe469dc161..d13199eca7 100644
>>>>>>> --- a/libavutil/hwcontext_videotoolbox.c
>>>>>>> +++ b/libavutil/hwcontext_videotoolbox.c
>>>>>>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>>>>>>    return 0;
>>>>>>> }
>>>>>>> +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>>>>>>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>>>>>>> +                            int flags)
>>>>>>> +{
>>>>>>> +    // There is no context to be setup with VT, just return.
>>>>>>> +    return 0;
>>>>>>> +}
>>>>>>> +
>>>>>>> const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>>>>    .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>>>>>>    .name                 = "videotoolbox",
>>>>>>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>>>>    .frames_priv_size     = sizeof(VTFramesContext),
>>>>>>>      .device_create        = vt_device_create,
>>>>>>> +    .device_derive        = vt_device_derive,
>>>>>>>    .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>>>>>>    .frames_init          = vt_frames_init,
>>>>>>>    .frames_get_buffer    = vt_get_buffer,
>>>>>> 
>>>>>> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
>>>>>> 
>>>>>> (Also, try -init_hw_device.)
>>>>>> 
>>>> Well this does make sense.
>>>> 
>>>> If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.
>>>> 
>>> 
>>> Yes, it is unfortunate that the ffmpeg utility is missing this feature.
>>> 
>>> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly, but the ffmpeg utility does not fully expose the functionality of libavfilter to be able to do this.
>>> 
>>> This is mostly for syntax reasons; implementation would be straightforward.  If you can think of a good way to represent this in options to the ffmpeg utility (which doesn't become too horrible with escaping) then I would very much welcome suggestions.
>>> 
>>> Thanks,
>>> 
>>> - Mark
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel@ffmpeg.org
>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>> 
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>> 
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> 
>> To unsubscribe, visit link above, or email
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.o
gnattu Feb. 23, 2024, 5:20 p.m. UTC | #7
I’ve tested and confirmed to work, and `reverse=1` also works for chaining to other videotoolbox filters. 

> On Feb 23, 2024, at 22:25, Zhao Zhili <quinkblack@foxmail.com> wrote:
> 
> <favicon.ico>_______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
diff mbox series

Patch

diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
index fe469dc161..d13199eca7 100644
--- a/libavutil/hwcontext_videotoolbox.c
+++ b/libavutil/hwcontext_videotoolbox.c
@@ -759,6 +759,14 @@  static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
     return 0;
 }
 
+static int vt_device_derive(AVHWDeviceContext *device_ctx,
+                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
+                            int flags)
+{
+    // There is no context to be setup with VT, just return.
+    return 0;
+}
+
 const HWContextType ff_hwcontext_type_videotoolbox = {
     .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
     .name                 = "videotoolbox",
@@ -766,6 +774,7 @@  const HWContextType ff_hwcontext_type_videotoolbox = {
     .frames_priv_size     = sizeof(VTFramesContext),
 
     .device_create        = vt_device_create,
+    .device_derive        = vt_device_derive,
     .frames_hwctx_size    = sizeof(AVVTFramesContext),
     .frames_init          = vt_frames_init,
     .frames_get_buffer    = vt_get_buffer,