[FFmpeg-devel] avdevice/avdevice: Deprecate AVDevice Capabilities API

It has been added in 6db42a2b6b22e6f1928fafcf3faa67ed78201004,
yet since then none of the necessary create/free_device_capabilities
functions has been implemented, making this API completely useless.

Because of this one can already simplify
avdevice_capabilities_free/create and can already remove the function
pointers at the next major bump.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
---
 doc/APIchanges         |  4 ++++
 libavdevice/avdevice.c | 41 ++++++-----------------------------------
 libavdevice/avdevice.h |  5 +++++
 libavdevice/version.h  |  7 +++++--
 libavformat/avformat.h |  4 ++++
 5 files changed, 24 insertions(+), 37 deletions(-)

On 1/24/2021 5:16 PM, Andreas Rheinhardt wrote:
> It has been added in 6db42a2b6b22e6f1928fafcf3faa67ed78201004,
> yet since then none of the necessary create/free_device_capabilities
> functions has been implemented, making this API completely useless.
> 
> Because of this one can already simplify
> avdevice_capabilities_free/create and can already remove the function
> pointers at the next major bump.

Not against it if nobody will bother to implement this on existing 
devices, but since the long term idea is to merge lavd into lavf, I 
assume the entirety of the library's API will be eventually deprecated 
and replaced.
Worth if anything to remove those AVOutputFormat/AVInputFormat function 
pointers, though.

> 
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
> ---
>   doc/APIchanges         |  4 ++++
>   libavdevice/avdevice.c | 41 ++++++-----------------------------------
>   libavdevice/avdevice.h |  5 +++++
>   libavdevice/version.h  |  7 +++++--
>   libavformat/avformat.h |  4 ++++
>   5 files changed, 24 insertions(+), 37 deletions(-)
> 
> diff --git a/doc/APIchanges b/doc/APIchanges
> index bbf56a5385..8eeb6ba70f 100644
> --- a/doc/APIchanges
> +++ b/doc/APIchanges
> @@ -15,6 +15,10 @@ libavutil:     2017-10-21
>   
>   API changes, most recent first:
>   
> +2021-01-24 - xxxxxxxxxx - lavd 58.12.100 - avdevice.h
> +  Deprecated avdevice_capabilities_create() and
> +  Add avdevice_capabilities_free(). They

They?

James Almer:
> On 1/24/2021 5:16 PM, Andreas Rheinhardt wrote:
>> It has been added in 6db42a2b6b22e6f1928fafcf3faa67ed78201004,
>> yet since then none of the necessary create/free_device_capabilities
>> functions has been implemented, making this API completely useless.
>>
>> Because of this one can already simplify
>> avdevice_capabilities_free/create and can already remove the function
>> pointers at the next major bump.
> 
> Not against it if nobody will bother to implement this on existing
> devices, but since the long term idea is to merge lavd into lavf, I
> assume the entirety of the library's API will be eventually deprecated
> and replaced.

I know that Anton is currently working on merging them, but I thought
that this would only involve compiling lavd into libavformat.so; not
that the API would be changed or the libavdevice/ source folder merged
into libavformat.

> Worth if anything to remove those AVOutputFormat/AVInputFormat function
> pointers, though.
> 
>>
>> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
>> ---
>>   doc/APIchanges         |  4 ++++
>>   libavdevice/avdevice.c | 41 ++++++-----------------------------------
>>   libavdevice/avdevice.h |  5 +++++
>>   libavdevice/version.h  |  7 +++++--
>>   libavformat/avformat.h |  4 ++++
>>   5 files changed, 24 insertions(+), 37 deletions(-)
>>
>> diff --git a/doc/APIchanges b/doc/APIchanges
>> index bbf56a5385..8eeb6ba70f 100644
>> --- a/doc/APIchanges
>> +++ b/doc/APIchanges
>> @@ -15,6 +15,10 @@ libavutil:     2017-10-21
>>     API changes, most recent first:
>>   +2021-01-24 - xxxxxxxxxx - lavd 58.12.100 - avdevice.h
>> +  Deprecated avdevice_capabilities_create() and
>> +  Add avdevice_capabilities_free(). They
> 
> They?

"Add" is even bigger bullshit. Sorry.
(I swear I read this more than a dozen times.)

- Andreas

Andreas Rheinhardt (12021-01-24):
> I know that Anton is currently working on merging them, but I thought
> that this would only involve compiling lavd into libavformat.so; not
> that the API would be changed or the libavdevice/ source folder merged
> into libavformat.

Considering Anton's open dislike of libavdevice, I find that news quite
worrisome. I hope he will consult with the people who care about it and
have ideas for its evolution before investing time in this work;
otherwise it may mean a lot of wasted effort, and I must say it would
also be quite disrespectful.

Regards,

Quoting Andreas Rheinhardt (2021-01-24 21:42:10)
> James Almer:
> > On 1/24/2021 5:16 PM, Andreas Rheinhardt wrote:
> >> It has been added in 6db42a2b6b22e6f1928fafcf3faa67ed78201004,
> >> yet since then none of the necessary create/free_device_capabilities
> >> functions has been implemented, making this API completely useless.
> >>
> >> Because of this one can already simplify
> >> avdevice_capabilities_free/create and can already remove the function
> >> pointers at the next major bump.
> > 
> > Not against it if nobody will bother to implement this on existing
> > devices, but since the long term idea is to merge lavd into lavf, I
> > assume the entirety of the library's API will be eventually deprecated
> > and replaced.
> 
> I know that Anton is currently working on merging them, but I thought
> that this would only involve compiling lavd into libavformat.so; not
> that the API would be changed or the libavdevice/ source folder merged
> into libavformat.

My plan is currently to move the sources into a subdirectory of
libavformat/. The question of the API is for the most part separte.

I like this patch BTW.

Anton Khirnov (12021-01-25):
> My plan is currently to move the sources into a subdirectory of
> libavformat/.

If that is all, then no. Please spend your time in more useful manners.
As somebody who maintains parts of libavdevice, I do not want tracking
history made more difficult and brittle, I do not to have to type an
extra path component, and I see no benefit.

In fact, since you dislike libavdevice so much, I suggest you add
--disable-avdevice to your configure line, and let the people who do not
hate it work on it in peace.

Thanks in advance.

On 1/25/2021 1:18 PM, Nicolas George wrote:
> Anton Khirnov (12021-01-25):
>> My plan is currently to move the sources into a subdirectory of
>> libavformat/.
> 
> If that is all, then no. Please spend your time in more useful manners.
> As somebody who maintains parts of libavdevice, I do not want tracking
> history made more difficult and brittle, I do not to have to type an
> extra path component, and I see no benefit.
> 
> In fact, since you dislike libavdevice so much, I suggest you add
> --disable-avdevice to your configure line, and let the people who do not
> hate it work on it in peace.
> 
> Thanks in advance.

It being within a subfolder or directly in libavformat's root is 
secondary, really.
Suggestions for alternatives are obviously welcome, if you dislike 
Anton's current approach (admittedly, it reminds me a bit of libmpcodecs 
within libavfilter).

James Almer (12021-01-25):
> Suggestions for alternatives are obviously welcome, if you dislike Anton's
> current approach

Simple: if you do not have plans to make it better, leave it alone and
work on something useful.

Regards,

On 1/25/2021 1:38 PM, Nicolas George wrote:
> James Almer (12021-01-25):
>> Suggestions for alternatives are obviously welcome, if you dislike Anton's
>> current approach
> 
> Simple: if you do not have plans to make it better, leave it alone and
> work on something useful.

Solving the hacky state of lavd <-> lavf *is* useful. And it would help 
the project a lot if you could be a tiny bit less aggressive and 
dismissive of other people's personal opinions.

James Almer (12021-01-25):
> Solving the hacky state of lavd <-> lavf *is* useful.

The only "hacky state" comes from the useless split of the libraries.

>							And it would help the
> project a lot if you could be a tiny bit less aggressive and dismissive of
> other people's personal opinions.

Anton is dismissive of other people's efforts — not just opinions — yet
you said him nothing. Double standards.

Regards,

On 1/25/2021 1:46 PM, Nicolas George wrote:
> James Almer (12021-01-25):
>> Solving the hacky state of lavd <-> lavf *is* useful.
> 
> The only "hacky state" comes from the useless split of the libraries.

And that's why we're trying to merge them. So again, a subdirectory is 
not obligatory for this purpose, and alternative suggestions are welcome.
Even just saying to keep them in the root folder would be more useful 
than saying "Work on something else".

> 
>> 							And it would help the
>> project a lot if you could be a tiny bit less aggressive and dismissive of
>> other people's personal opinions.
> 
> Anton is dismissive of other people's efforts — not just opinions — yet
> you said him nothing. Double standards.

Saying and thinking that lavd devices with their current capabilities 
and API are not very useful is not being "dismissive of other people's 
efforts", it's giving his opinion. He did it in fact as a reply to 
Mark's comment that the KMS/DRM output device patch didn't seem like a 
good idea.
Mark then suggested to directly replace/extend the API with one that's 
more useful in the long run, instead of removing them as Anton suggested 
since you and some other devs stated they in fact do have users, and you 
agreed to that. So how about we try to move in that direction? The first 
step is merging the libraries, and not trying to stop any relevant effort.

And for that matter, unlike you, Anton did not resort to ad hominem 
attacks to back his argument.

> 
> Regards,
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>

James Almer (12021-01-25):
> And that's why we're trying to merge them. So again, a subdirectory is not
> obligatory for this purpose, and alternative suggestions are welcome.
> Even just saying to keep them in the root folder would be more useful than

That is what leaving it alone implies.

> saying "Work on something else".

As a general principle, I think no member of this project should touch
an area of the code they hate: hate is not compatible with good work. Do
you not agree?

> Saying and thinking that lavd devices with their current capabilities and
> API are not very useful is not being "dismissive of other people's efforts",
> it's giving his opinion.

But saying it is "worse than useless" is dismissive. Double standards.

> Mark then suggested to directly replace/extend the API with one that's more
> useful in the long run, instead of removing them as Anton suggested since
> you and some other devs stated they in fact do have users, and you agreed to
> that. So how about we try to move in that direction? The first step is
> merging the libraries, and not trying to stop any relevant effort.

No. The first step is to acknowledge that libavdevice is useful, already
as it is. I am still waiting for that.

The second step would be to think about how it can be made even more
useful. For that, we need people who understand its usefulness. People
who do not should stay out of it, or at least take second role.

I have detailed thoughts on making libavdevice much more useful. And
they do not involve merging with libavformat; and they require a few
infrastructure steps before they can happen. But I will not discuss them
in this hostile environment, as a defense. If somebody is interested and
sympathetic, I will gladly explain them.

> And for that matter, unlike you, Anton did not resort to ad hominem attacks
> to back his argument.

Unfortunately, this point is significant of a wider attitude towards
features versus purity. It needs to be addressed or the project will
wither.

Regards,

On 25/01/2021 17:11, James Almer wrote:
> Mark then suggested to directly replace/extend the API with one that's more useful in the long run, instead of removing them as Anton suggested since you and some other devs stated they in fact do have users, and you agreed to that. So how about we try to move in that direction?

Yes.

> The first step is merging the libraries, ...

?  I don't see how this follows at all.

Merging the libraries (in source form, orthogonal to merging the binaries) only makes sense if we are going to continue using the libavformat internals, and that is exactly the thing we are trying to get rid of.

If we replace the libavdevice API with a proper one oriented towards devices rather than being hacked on to the libavformat API then we actively don't want that merger.

So, what do people actually want here?  The situation as I see it is (please correct me on anything you feel is in error here):

* The features which libavdevice offers are useful to a lot of people.
* Many of those are using it via the ffmpeg utility, but not all.
* The libavdevice API is the libavformat API because it was originally split out from libavformat, and it has the nice property that devices and files end up being interchangable in some contexts.
* The libavdevice API, being the libavformat API for files, is not particularly well-suited in other contexts, because devices may not have the same properties as files.
* Some odd things like the completely-unused capabilities API and the almost-never-used message API are hacked on top of that to try to avoid some libavformat issues, but are not actually useful to anyone (hence the lack of use).
* To implement devices as AVInputFormat/AVOutputFormat instances, libavdevice currently needs access to the internals of libavformat.
* Many developers want to get rid of that dependency on libavformat internals, because it creates a corresponding ugliness on the libavformat side which has to leave those parts exposed in an ABI-constrained way.

What might we do about this:

* Delete libavdevice, with a statement that its capabilities are outside the scope of ffmpeg.  Obviously this isn't sensible, the ffmpeg utility would then need to depend on some external library to offer the same features it does now.
* Merge libavdevice into the ffmpeg utility.  Since it is the primary user this would neatly sidestep the previous objection, but now external users are in the same boat so again unacceptable.
* Merge libavdevice into libavformat.  This works to eliminate the dependency problem, but it only really makes sense if we accept that libavdevice is finished and will never want new API which isn't repurposed libavformat.
* Make a new libavdevice-specific API and switch to that.  The eliminates the dependency problem, but there is a nasty transition period and it would break the interchangability of the two libraries.  It also would be the most work, but leave libavdevice in the best state for doing more things with in future.
* Nothing.  People will continue to be annoyed about the problems above.

Opinions?

Personally, I think the right option here is to make a new libavdevice-specific API and cut the connection to libavformat entirely, and I would be prepared to do some work towards that goal.  Exactly what form it would take is unclear, but I would be happy to talk about that further if people agree that it is the right approach.

Thanks,

- Mark

Mark Thompson (12021-01-25):
> Merging the libraries (in source form, orthogonal to merging the
> binaries) only makes sense if we are going to continue using the
> libavformat internals, and that is exactly the thing we are trying to
> get rid of.

"We"?

It seems to me that many people here have an agenda about libavdevice
while they have no reason to.

> If we replace the libavdevice API with a proper one oriented towards
> devices rather than being hacked on to the libavformat API then we
> actively don't want that merger.
> 
> So, what do people actually want here?  The situation as I see it is
> (please correct me on anything you feel is in error here):
> 
> * The features which libavdevice offers are useful to a lot of people.

Thank you for acknowledging it.

> * Many of those are using it via the ffmpeg utility, but not all.
> * The libavdevice API is the libavformat API because it was originally
> split out from libavformat, and it has the nice property that devices
> and files end up being interchangable in some contexts.
> * The libavdevice API, being the libavformat API for files, is not
> particularly well-suited in other contexts, because devices may not
> have the same properties as files.
> * Some odd things like the completely-unused capabilities API and the
> almost-never-used message API are hacked on top of that to try to
> avoid some libavformat issues, but are not actually useful to anyone
> (hence the lack of use).
> * To implement devices as AVInputFormat/AVOutputFormat instances,
> libavdevice currently needs access to the internals of libavformat.
> * Many developers want to get rid of that dependency on libavformat
> internals, because it creates a corresponding ugliness on the
> libavformat side which has to leave those parts exposed in an
> ABI-constrained way.
> 
> What might we do about this:
> 
> * Delete libavdevice, with a statement that its capabilities are
> outside the scope of ffmpeg.  Obviously this isn't sensible, the
> ffmpeg utility would then need to depend on some external library to
> offer the same features it does now.
> * Merge libavdevice into the ffmpeg utility.  Since it is the primary
> user this would neatly sidestep the previous objection, but now
> external users are in the same boat so again unacceptable.

One of the main reasons the fork failed while original ffmpeg kept going
was that while ffmpeg has a policy of being useful, the fork had an
unspoken rule about code purity that lead to removing many useful
features. IIRC, Anton was one of the most vocal proponent of this
harmful unspoken rule.

I find worrisome to see the same attitude start to appear here. We need
to be very careful.

Also, if the project actually starts removing features for purity, like
some people discussed here, I'll stop bothering about this dying
project.

> * Merge libavdevice into libavformat.  This works to eliminate the
> dependency problem, but it only really makes sense if we accept that
> libavdevice is finished and will never want new API which isn't
> repurposed libavformat.
> * Make a new libavdevice-specific API and switch to that.  The
> eliminates the dependency problem, but there is a nasty transition
> period and it would break the interchangability of the two libraries.
> It also would be the most work, but leave libavdevice in the best
> state for doing more things with in future.

One of the important features of libavdevice, as you mentioned earlier,
is that devices can be a drop-in replacement for demuxers and muxers. An
application that is not designed for devices can be made to use them,
and that is immensely convenient. It does not work always, but it work
often enough to be, I repeat, immensely useful.

If a solution does not have this feature, we can just skip it.

> * Nothing.  People will continue to be annoyed about the problems above.

Duh. People often are annoyed at the wrong things, let them be annoyed.
I have had proof in the past that some of the annoyance on this subject
was actually a deep misunderstanding of how linking works.

There is another option:

* Stop imposing ourselves constraints that bring absolutely no benefit.
Having internals is useful for a library because we can change them
without bothering about compatibility: if we change the internals, we
can change all places that use them at the same time, and everything
keeps working.

It is a good hygiene that the fftools are written to use no internals,
because that proves the public API is complete enough to build powerful
tools.

But for the libraries, it makes no sense to have this constraint. What
benefit is there in it? Nobody has been able to answer this question,
because there are none. Libraries should be able to use each other
internals (typical example: sharing header decoding and format-specific
utility functions between lavc and lavf).

> Opinions?
> 
> Personally, I think the right option here is to make a new
> libavdevice-specific API and cut the connection to libavformat
> entirely, and I would be prepared to do some work towards that goal.
> Exactly what form it would take is unclear, but I would be happy to
> talk about that further if people agree that it is the right approach.

One of the very important things you missed:

A lot of devices would also be very useful as libavfilter sources and
sinks. It allows automatic format negotiation, direct rendering and many
other benefits.

But it cannot work for now, because the API to run a filter graph only
works with buffer sinks, no other kinds of sinks.

Therefore, before considering enhancing the API of libavdevice, we need
a proper API tu run a filter graph.

And since we want inter-filters parallelism, a feature that has been
wanted for a long time, for that we need a proper event loop and
parallel task scheduler. Which brings us back to this mail, on which I
intend to follow-up soon:
https://ffmpeg.org/pipermail/ffmpeg-devel/2020-December/274172.html

Yes, libavdevice's API is far from perfect, but it is part of a much
wider problem: everything in FFmpeg that is not directly codecs and
formats, and secondarily filters, has been let to slowly rot, because
"we don't need this", "it's not our core mission", "it's unusual",
whatever.

Continue like that, and FFmpeg will become irrelevant. Or change
completely, embrace originality, create the necessary infrastructure,
and FFmpeg can be again a leader in efficient audio-video programming.

Regards,

On 1/25/2021 8:20 PM, Nicolas George wrote:
> Mark Thompson (12021-01-25):
>> Merging the libraries (in source form, orthogonal to merging the
>> binaries) only makes sense if we are going to continue using the
>> libavformat internals, and that is exactly the thing we are trying to
>> get rid of.
> "We"?
> 
> It seems to me that many people here have an agenda about libavdevice
> while they have no reason to.

If by agenda you mean trying to solve the hacky state of the library, 
then sure. Otherwise, please get rid of any potential slanderous 
thoughts you may be have about it, because it sounds like you may be 
having trouble distinguishing expressed personal opinions with actual 
unrelated coding efforts.

So again, focus on working with people to find a solution and a future 
for the library, in whatever form we ultimately decide is best, instead 
of attacking them for not sharing your opinions. So far, I've seen 
plenty of the latter and nothing of the former.

James Almer (12021-01-25):
> If by agenda you mean trying to solve the hacky state of the library, then
> sure. Otherwise, please get rid of any potential slanderous thoughts you may
> be have about it, because it sounds like you may be having trouble
> distinguishing expressed personal opinions with actual unrelated coding
> efforts.
> 
> So again, focus on working with people to find a solution and a future for
> the library, in whatever form we ultimately decide is best, instead of
> attacking them for not sharing your opinions. So far, I've seen plenty of
> the latter and nothing of the former.

I will accept other people's opinion when it is founded on knowledge.
Voicing an opinion when knowing almost nothing on the topic is
disrespectful in the first place, and it is even more disrespectful when
the opinion itself is disrespectful, as was Anton's "worse than useless".

Regards,

On Tue, 26 Jan 2021, at 11:28, Nicolas George wrote:
> I will accept other people's opinion when it is founded on knowledge.

Unfortunately, that's not how it works.
We need to accept the will of majority of developers, even if you (or someone else) is not convinced.


--
Jean-Baptiste Kempf -  President
+33 672 704 734

On 25/01/2021 23:20, Nicolas George wrote:
> Mark Thompson (12021-01-25):
>> Merging the libraries (in source form, orthogonal to merging the
>> binaries) only makes sense if we are going to continue using the
>> libavformat internals, and that is exactly the thing we are trying to
>> get rid of.
> 
> "We"?
> 
> It seems to me that many people here have an agenda about libavdevice
> while they have no reason to.

A number of the people in the project have expressed this opinion, hence I believe it is something which "we" want to do.  ("we" may also not want to do it; "we" do not need to be consistent.)

>> If we replace the libavdevice API with a proper one oriented towards
>> devices rather than being hacked on to the libavformat API then we
>> actively don't want that merger.
>>
>> So, what do people actually want here?  The situation as I see it is
>> (please correct me on anything you feel is in error here):
>>
>> * The features which libavdevice offers are useful to a lot of people.
> 
> Thank you for acknowledging it.
> 
>> * Many of those are using it via the ffmpeg utility, but not all.
>> * The libavdevice API is the libavformat API because it was originally
>> split out from libavformat, and it has the nice property that devices
>> and files end up being interchangable in some contexts.
>> * The libavdevice API, being the libavformat API for files, is not
>> particularly well-suited in other contexts, because devices may not
>> have the same properties as files.
>> * Some odd things like the completely-unused capabilities API and the
>> almost-never-used message API are hacked on top of that to try to
>> avoid some libavformat issues, but are not actually useful to anyone
>> (hence the lack of use).
>> * To implement devices as AVInputFormat/AVOutputFormat instances,
>> libavdevice currently needs access to the internals of libavformat.
>> * Many developers want to get rid of that dependency on libavformat
>> internals, because it creates a corresponding ugliness on the
>> libavformat side which has to leave those parts exposed in an
>> ABI-constrained way.
>>
>> What might we do about this:
>>
>> * Delete libavdevice, with a statement that its capabilities are
>> outside the scope of ffmpeg.  Obviously this isn't sensible, the
>> ffmpeg utility would then need to depend on some external library to
>> offer the same features it does now.
>> * Merge libavdevice into the ffmpeg utility.  Since it is the primary
>> user this would neatly sidestep the previous objection, but now
>> external users are in the same boat so again unacceptable.
> 
> One of the main reasons the fork failed while original ffmpeg kept going
> was that while ffmpeg has a policy of being useful, the fork had an
> unspoken rule about code purity that lead to removing many useful
> features. IIRC, Anton was one of the most vocal proponent of this
> harmful unspoken rule.
> 
> I find worrisome to see the same attitude start to appear here. We need
> to be very careful.
> 
> Also, if the project actually starts removing features for purity, like
> some people discussed here, I'll stop bothering about this dying
> project.
> 
>> * Merge libavdevice into libavformat.  This works to eliminate the
>> dependency problem, but it only really makes sense if we accept that
>> libavdevice is finished and will never want new API which isn't
>> repurposed libavformat.
>> * Make a new libavdevice-specific API and switch to that.  The
>> eliminates the dependency problem, but there is a nasty transition
>> period and it would break the interchangability of the two libraries.
>> It also would be the most work, but leave libavdevice in the best
>> state for doing more things with in future.
> 
> One of the important features of libavdevice, as you mentioned earlier,
> is that devices can be a drop-in replacement for demuxers and muxers. An
> application that is not designed for devices can be made to use them,
> and that is immensely convenient. It does not work always, but it work
> often enough to be, I repeat, immensely useful.

I'm afraid I disagree with this sentiment.  Each library should be designed around an API which makes sense for the abstraction it is trying to create.  Forcing one library into the ill-suited API of another to work makes it less useful, not more.  (OpenMAX is a less useful API than libavcodec/libavfilter, despite covering more cases.)

> If a solution does not have this feature, we can just skip it.

Nothing here excludes the possibility of making a libavdevice psuedo-format which covers the device use-cases possible within the libavformat API (compare the lavfi pseudo-demuxer in the ffmpeg utility).

>> * Nothing.  People will continue to be annoyed about the problems above.
> 
> Duh. People often are annoyed at the wrong things, let them be annoyed.
> I have had proof in the past that some of the annoyance on this subject
> was actually a deep misunderstanding of how linking works.
> 
> There is another option:
> 
> * Stop imposing ourselves constraints that bring absolutely no benefit.
> Having internals is useful for a library because we can change them
> without bothering about compatibility: if we change the internals, we
> can change all places that use them at the same time, and everything
> keeps working.
> 
> It is a good hygiene that the fftools are written to use no internals,
> because that proves the public API is complete enough to build powerful
> tools.
> 
> But for the libraries, it makes no sense to have this constraint. What
> benefit is there in it? Nobody has been able to answer this question,
> because there are none. Libraries should be able to use each other
> internals (typical example: sharing header decoding and format-specific
> utility functions between lavc and lavf).

Even after such a merge, the libavdevice API is still a problem.

>> Opinions?
>>
>> Personally, I think the right option here is to make a new
>> libavdevice-specific API and cut the connection to libavformat
>> entirely, and I would be prepared to do some work towards that goal.
>> Exactly what form it would take is unclear, but I would be happy to
>> talk about that further if people agree that it is the right approach.
> 
> One of the very important things you missed:
> 
> A lot of devices would also be very useful as libavfilter sources and
> sinks. It allows automatic format negotiation, direct rendering and many
> other benefits.

Direct rendering with devices is not possible without a proper libavdevice API.  Making an internal special case allowing it in libavfilter only does not seem like the most useful step to take in this direction to me.

> But it cannot work for now, because the API to run a filter graph only
> works with buffer sinks, no other kinds of sinks.

I agree that this change makes sense for the use-cases you describe, but I'm unsure why it has to be related (though I suspect I may be missing some subtlety).

I'm also unclear why it would need to mess with libavdevice internals - a pseudo-filter which acts as a source is already possible if you want to write one.

> Therefore, before considering enhancing the API of libavdevice, we need
> a proper API tu run a filter graph.

Why is this a dependency?  libavdevice is useful in a context with no libavfilter at all.

> And since we want inter-filters parallelism, a feature that has been
> wanted for a long time, for that we need a proper event loop and
> parallel task scheduler. Which brings us back to this mail, on which I
> intend to follow-up soon:
> https://ffmpeg.org/pipermail/ffmpeg-devel/2020-December/274172.html

I agree that careful movements towards an event loop are a good idea, but again not sure why it has to be related.

> Yes, libavdevice's API is far from perfect, but it is part of a much
> wider problem: everything in FFmpeg that is not directly codecs and
> formats, and secondarily filters, has been let to slowly rot, because
> "we don't need this", "it's not our core mission", "it's unusual",
> whatever.

Or lack of developer interest, which in the playback case which prompted this discussion originally is I think driven by "other projects do it better, so why should we include it here".

> Continue like that, and FFmpeg will become irrelevant. Or change
> completely, embrace originality, create the necessary infrastructure,
> and FFmpeg can be again a leader in efficient audio-video programming.

- Mark

Mark Thompson (12021-01-26):
> Even after such a merge, the libavdevice API is still a problem.

I will re-center the discussion on this alone.

And I ask, simply: what problem exactly?

You, who AFAIK, do not maintain anything in libavdevice and have never
used it in a project, say there is a problem with its API. I, who
maintain parts of libavdevice and have been using it in projects for
years, say there is no problem, there are ways to enhance it, but on the
whole it works well enough.

Who might be right? Maybe the person with the most experience, don't you
think?

Therefore, I will cut short this discussion:

If you want to know what my ideas are to enhance the API of libavdevice,
just ask, and if you'll read the answer I'll be happy to explain at
lengths.

If you have a diagnostic of an actual problem, please share it.

If you have a detailed plan to make libavdevice's API significantly
better, then please explain it, I am keenly interested.

Another point: remember that in this project, we have a policy of no new
API without user: we do not add a new function or API just for the fun
of it; we add a new function or API when and if it allows to add a
feature, to simplify existing code, etc.

So, if you invent a new design for devices, and you implement an useful
new feature thanks to this new design, or even better, several useful
new features, then excellent.

But if you propose to just add a grand new design for devices, and add a
wrapper so that existing devices can be used unchanged, and another
wrapper so that existing applications can still use the old API, and
that's it, then you are making it worse.

So, since AFAIK, you do not have plans to add new useful features to
devices, or to libavfilter as a whole, my advice is: for now, leave it
alone, until you have given it much more thought.

(In return, I will continue to refrain from criticizing the design of
the VAAPI stuff.)

Regards,

Jean-Baptiste Kempf (12021-01-26):
> Unfortunately, that's not how it works.
> We need to accept the will of majority of developers, even if you (or
> someone else) is not convinced.

We will accept the will of the majority of developers when the time for
making a decision comes if no consensus has been reached.

But before this point, every single developer in the project should
listen to the advice of whoever has the most experience and knowledge
about a particular topic.

This is an appeal to authority, but the good kind, not "listen to me
because I'm famous" but "listen to me because I know more about this
particular issue".

Regards,

On 27/01/2021 14:36, Nicolas George wrote:
> Mark Thompson (12021-01-26):
>> Even after such a merge, the libavdevice API is still a problem.
> 
> I will re-center the discussion on this alone.
> 
> And I ask, simply: what problem exactly?

See for example the list of suggestions I made for improving the API a few days ago.

On 20/01/2021 12:41, Mark Thompson wrote:
 > * Handle frames as well as packets.
 >    * Including hardware frames - DRM objects from KMS/V4L2, D3D surfaces from Windows desktop duplication (which doesn't currently exist but should).
 > * Clear core option set - currently almost everything is set by inconsistent private options; things like pixel/sample format, frame/sample rate, geometry and hardware device should be common options to all.
 > * Asynchronicity - a big annoyance in current recording scenarios with the ffmpeg utility is that both audio and video capture block, and do so on the same thread which results in skipped frames.
 > * Capability probing - the existing method of options which log the capabilities are not very useful for API users.

> You, who AFAIK, do not maintain anything in libavdevice and have never
> used it in a project, say there is a problem with its API. I, who
> maintain parts of libavdevice and have been using it in projects for
> years, say there is no problem, there are ways to enhance it, but on the
> whole it works well enough.
> 
> Who might be right? Maybe the person with the most experience, don't you
> think?

How are we to determine who has the most relevant experience, so that we avoid appealing to a false authority?

For example, we could ask who has made the most changes to the library in the last few years:

$ git log --since 2015 libavdevice/ | grep "^Author:" | grep 'Nicolas\|Mark' | sort | uniq -c | sort -n
       9 Author: Mark Thompson <sw@jkqxz.net>
       7 Author: Nicolas George <george@nsup.org>

Or maybe we could go by lines of code?

$ git log --since 2015 --author='sw@jkqxz.net' --patch libavdevice/ | diffstat -s
  9 files changed, 827 insertions(+), 121 deletions(-)
$ git log --since 2015 --author='george@nsup.org' --patch libavdevice/ | diffstat -s
  6 files changed, 73 insertions(+), 61 deletions(-)

Or maybe these are completely meaningless measures which can be cherry-picked to show whatever answer we want and we should instead consider the merits of the proposals involved rather than trying to dismiss the person making them?

> Therefore, I will cut short this discussion:
> 
> If you want to know what my ideas are to enhance the API of libavdevice,
> just ask, and if you'll read the answer I'll be happy to explain at
> lengths.

Please will you explain what your ideas are about how to enhance the API of libavdevice.

Even if we disagree about exactly where such changes should be implemented, I would very much welcome hearing about the improvements you would like to make underneath.

> If you have a diagnostic of an actual problem, please share it.
> 
> If you have a detailed plan to make libavdevice's API significantly
> better, then please explain it, I am keenly interested.
> 
> Another point: remember that in this project, we have a policy of no new
> API without user: we do not add a new function or API just for the fun
> of it; we add a new function or API when and if it allows to add a
> feature, to simplify existing code, etc.
> 
> So, if you invent a new design for devices, and you implement an useful
> new feature thanks to this new design, or even better, several useful
> new features, then excellent.

Indeed.  The most obvious use-case for much of what I have said is of course the ffmpeg utility itself - the ability to avoid pointless copies when working with devices and to be able to record video and audio at the same time without weird interactions would both be useful features.

I should note that my original intent in engaging with this discussion was to gather thoughts from other members of the project wrt this sort of improvement before doing significant work on it, to avoid proceeding down a path which wouldn't go anywhere useful.

> But if you propose to just add a grand new design for devices, and add a
> wrapper so that existing devices can be used unchanged, and another
> wrapper so that existing applications can still use the old API, and
> that's it, then you are making it worse.
> 
> So, since AFAIK, you do not have plans to add new useful features to
> devices, or to libavfilter as a whole, my advice is: for now, leave it
> alone, until you have given it much more thought.
> 
> (In return, I will continue to refrain from criticizing the design of
> the VAAPI stuff.)

I would prefer that you do not refrain from offering constructive criticism of "the VAAPI stuff", or anything else that I work on, should you have any.

Thanks,

- Mark

Mark Thompson (12021-01-27):
> See for example the list of suggestions I made for improving the API a few days ago.

I intended to reply to them, but the conversation never came back to it
before now.

> * Handle frames as well as packets.

Already possible.

>   * Including hardware frames - DRM objects from KMS/V4L2, D3D
>   surfaces from Windows desktop duplication (which doesn't currentl y
>   exist but should).

I do not know if it is possible, you know this stuff better than me.

> * Clear core option set - currently almost everything is set by
> inconsistent private options; things like pixel/sample format,
> frame/sample rate, geometry and hardware device should be common
> options to all.

That would be good, but can be done with any API.

> * Asynchronicity - a big annoyance in current recording scenarios with
> the ffmpeg utility is that both audio and video capture block, and do
> so on the same thread which results in skipped frames.

This is a problem for devices, protocols and demuxers, and even filters,
and is therefore not an argument for a separate device API.

Please know that I am working on fixing this issue. This is the event
loop stuff. But it is a complex task.

> * Capability probing - the existing method of options which log the
> capabilities are not very useful for API users.

This is true, but it is not a problem with the API, just a missing
feature. See below for details about what I mean.

> How are we to determine who has the most relevant experience, so that
> we avoid appealing to a false authority?

Please accept my apology. I mistakenly assumed that you were speaking
from the same of lack of experience as the other people who want to
destroy libavdevice.

Did you use libavdevice in projects, not through ffmpeg.c, as well?

> Please will you explain what your ideas are about how to enhance the
> API of libavdevice.
> 
> Even if we disagree about exactly where such changes should be
> implemented, I would very much welcome hearing about the improvements
> you would like to make underneath.

First, I need to say that I agree with you that intuitively libavdevice
needs a dedicated API: devices are not the same thing as muxers and
demuxers, they have special need.

BUT...

But it means yet another f...ing API, and that is not good either. It
requires learning another API, special code, etc. Or it will be used
only through wrappers, and as such useless in itself.

Devices have special needs, but at the core, they are components that
produce or consume frames.

We already have APIs for components that produce or consume frames. We
already have several of them in fact.

So, my thoughts on devices is:

Let us choose the most nimble and extensible such API, and extend it as
needed.

This is, in essence, object-oriented design: instead of inventing a
completely new interface for devices, let the interface for devices
inherit from an existing interface, choosing the best one.

How does it sound for now?

> Indeed.  The most obvious use-case for much of what I have said is of
> course the ffmpeg utility itself - the ability to avoid pointless
> copies when working with devices and to be able to record video and

That would be nice, but I wonder how urgent it is.

> audio at the same time without weird interactions would both be useful
> features.

This would be very useful, but it is not specific to devices. We need to
be able to capture from the sound device and from a networked webcam
too.

If you try to fix this with a device API, you will be missing the bigger
picture.

> I should note that my original intent in engaging with this discussion
> was to gather thoughts from other members of the project wrt this sort
> of improvement before doing significant work on it, to avoid
> proceeding down a path which wouldn't go anywhere useful.

My advice is: before starting to work on code, discuss concrete details
of your project.

> I would prefer that you do not refrain from offering constructive
> criticism of "the VAAPI stuff", or anything else that I work on,
> should you have any.

I will refrain still, because I do not know anything about them.

Regards,

Andreas Rheinhardt:
> It has been added in 6db42a2b6b22e6f1928fafcf3faa67ed78201004,
> yet since then none of the necessary create/free_device_capabilities
> functions has been implemented, making this API completely useless.
> 
> Because of this one can already simplify
> avdevice_capabilities_free/create and can already remove the function
> pointers at the next major bump.
> 
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
> ---
>  doc/APIchanges         |  4 ++++
>  libavdevice/avdevice.c | 41 ++++++-----------------------------------
>  libavdevice/avdevice.h |  5 +++++
>  libavdevice/version.h  |  7 +++++--
>  libavformat/avformat.h |  4 ++++
>  5 files changed, 24 insertions(+), 37 deletions(-)
> 
> diff --git a/doc/APIchanges b/doc/APIchanges
> index bbf56a5385..8eeb6ba70f 100644
> --- a/doc/APIchanges
> +++ b/doc/APIchanges
> @@ -15,6 +15,10 @@ libavutil:     2017-10-21
>  
>  API changes, most recent first:
>  
> +2021-01-24 - xxxxxxxxxx - lavd 58.12.100 - avdevice.h
> +  Deprecated avdevice_capabilities_create() and
> +  Add avdevice_capabilities_free(). They
> +
>  2021-01-11 - xxxxxxxxxx - lavc 58.116.100 - avcodec.h
>    Add FF_PROFILE_VVC_MAIN_10 and FF_PROFILE_VVC_MAIN_10_444.
>  
> diff --git a/libavdevice/avdevice.c b/libavdevice/avdevice.c
> index ec84d3b990..fbcf4a4ab2 100644
> --- a/libavdevice/avdevice.c
> +++ b/libavdevice/avdevice.c
> @@ -27,6 +27,7 @@
>  #include "libavutil/ffversion.h"
>  const char av_device_ffversion[] = "FFmpeg version " FFMPEG_VERSION;
>  
> +#if FF_API_DEVICE_CAPABILITIES
>  #define E AV_OPT_FLAG_ENCODING_PARAM
>  #define D AV_OPT_FLAG_DECODING_PARAM
>  #define A AV_OPT_FLAG_AUDIO_PARAM
> @@ -60,6 +61,7 @@ const AVOption av_device_capabilities[] = {
>  #undef A
>  #undef V
>  #undef OFFSET
> +#endif
>  
>  unsigned avdevice_version(void)
>  {
> @@ -94,49 +96,18 @@ int avdevice_dev_to_app_control_message(struct AVFormatContext *s, enum AVDevToA
>      return s->control_message_cb(s, type, data, data_size);
>  }
>  
> +#if FF_API_DEVICE_CAPABILITIES
>  int avdevice_capabilities_create(AVDeviceCapabilitiesQuery **caps, AVFormatContext *s,
>                                   AVDictionary **device_options)
>  {
> -    int ret;
> -    av_assert0(s && caps);
> -    av_assert0(s->iformat || s->oformat);
> -    if ((s->oformat && !s->oformat->create_device_capabilities) ||
> -        (s->iformat && !s->iformat->create_device_capabilities))
> -        return AVERROR(ENOSYS);
> -    *caps = av_mallocz(sizeof(**caps));
> -    if (!(*caps))
> -        return AVERROR(ENOMEM);
> -    (*caps)->device_context = s;
> -    if (((ret = av_opt_set_dict(s->priv_data, device_options)) < 0))
> -        goto fail;
> -    if (s->iformat) {
> -        if ((ret = s->iformat->create_device_capabilities(s, *caps)) < 0)
> -            goto fail;
> -    } else {
> -        if ((ret = s->oformat->create_device_capabilities(s, *caps)) < 0)
> -            goto fail;
> -    }
> -    av_opt_set_defaults(*caps);
> -    return 0;
> -  fail:
> -    av_freep(caps);
> -    return ret;
> +    return AVERROR(ENOSYS);
>  }
>  
>  void avdevice_capabilities_free(AVDeviceCapabilitiesQuery **caps, AVFormatContext *s)
>  {
> -    if (!s || !caps || !(*caps))
> -        return;
> -    av_assert0(s->iformat || s->oformat);
> -    if (s->iformat) {
> -        if (s->iformat->free_device_capabilities)
> -            s->iformat->free_device_capabilities(s, *caps);
> -    } else {
> -        if (s->oformat->free_device_capabilities)
> -            s->oformat->free_device_capabilities(s, *caps);
> -    }
> -    av_freep(caps);
> +    return;
>  }
> +#endif
>  
>  int avdevice_list_devices(AVFormatContext *s, AVDeviceInfoList **device_list)
>  {
> diff --git a/libavdevice/avdevice.h b/libavdevice/avdevice.h
> index ee9462480e..85a4dcc6df 100644
> --- a/libavdevice/avdevice.h
> +++ b/libavdevice/avdevice.h
> @@ -321,6 +321,7 @@ int avdevice_dev_to_app_control_message(struct AVFormatContext *s,
>                                          enum AVDevToAppMessageType type,
>                                          void *data, size_t data_size);
>  
> +#if FF_API_DEVICE_CAPABILITIES
>  /**
>   * Following API allows user to probe device capabilities (supported codecs,
>   * pixel formats, sample formats, resolutions, channel counts, etc).
> @@ -416,6 +417,7 @@ typedef struct AVDeviceCapabilitiesQuery {
>  /**
>   * AVOption table used by devices to implement device capabilities API. Should not be used by a user.
>   */
> +attribute_deprecated
>  extern const AVOption av_device_capabilities[];
>  
>  /**
> @@ -435,6 +437,7 @@ extern const AVOption av_device_capabilities[];
>   *
>   * @return >= 0 on success, negative otherwise.
>   */
> +attribute_deprecated
>  int avdevice_capabilities_create(AVDeviceCapabilitiesQuery **caps, AVFormatContext *s,
>                                   AVDictionary **device_options);
>  
> @@ -444,7 +447,9 @@ int avdevice_capabilities_create(AVDeviceCapabilitiesQuery **caps, AVFormatConte
>   * @param caps Device capabilities data to be freed.
>   * @param s    Context of the device.
>   */
> +attribute_deprecated
>  void avdevice_capabilities_free(AVDeviceCapabilitiesQuery **caps, AVFormatContext *s);
> +#endif
>  
>  /**
>   * Structure describes basic parameters of the device.
> diff --git a/libavdevice/version.h b/libavdevice/version.h
> index 7022fdbf2a..f5aaa168af 100644
> --- a/libavdevice/version.h
> +++ b/libavdevice/version.h
> @@ -28,8 +28,8 @@
>  #include "libavutil/version.h"
>  
>  #define LIBAVDEVICE_VERSION_MAJOR  58
> -#define LIBAVDEVICE_VERSION_MINOR  11
> -#define LIBAVDEVICE_VERSION_MICRO 103
> +#define LIBAVDEVICE_VERSION_MINOR  12
> +#define LIBAVDEVICE_VERSION_MICRO 100
>  
>  #define LIBAVDEVICE_VERSION_INT AV_VERSION_INT(LIBAVDEVICE_VERSION_MAJOR, \
>                                                 LIBAVDEVICE_VERSION_MINOR, \
> @@ -46,5 +46,8 @@
>   * dropped at a future version bump. The defines themselves are not part of
>   * the public API and may change, break or disappear at any time.
>   */
> +#ifndef FF_API_DEVICE_CAPABILITIES
> +#define FF_API_DEVICE_CAPABILITIES (LIBAVDEVICE_VERSION_MAJOR < 60)
> +#endif
>  
>  #endif /* AVDEVICE_VERSION_H */
> diff --git a/libavformat/avformat.h b/libavformat/avformat.h
> index 523cf34d55..41482328f6 100644
> --- a/libavformat/avformat.h
> +++ b/libavformat/avformat.h
> @@ -590,6 +590,7 @@ typedef struct AVOutputFormat {
>       * @see avdevice_list_devices() for more details.
>       */
>      int (*get_device_list)(struct AVFormatContext *s, struct AVDeviceInfoList *device_list);
> +#if LIBAVFORMAT_VERSION_MAJOR < 59
>      /**
>       * Initialize device capabilities submodule.
>       * @see avdevice_capabilities_create() for more details.
> @@ -600,6 +601,7 @@ typedef struct AVOutputFormat {
>       * @see avdevice_capabilities_free() for more details.
>       */
>      int (*free_device_capabilities)(struct AVFormatContext *s, struct AVDeviceCapabilitiesQuery *caps);
> +#endif
>      enum AVCodecID data_codec; /**< default data codec */
>      /**
>       * Initialize format. May allocate data here, and set any AVFormatContext or
> @@ -769,6 +771,7 @@ typedef struct AVInputFormat {
>       */
>      int (*get_device_list)(struct AVFormatContext *s, struct AVDeviceInfoList *device_list);
>  
> +#if LIBAVFORMAT_VERSION_MAJOR < 59
>      /**
>       * Initialize device capabilities submodule.
>       * @see avdevice_capabilities_create() for more details.
> @@ -780,6 +783,7 @@ typedef struct AVInputFormat {
>       * @see avdevice_capabilities_free() for more details.
>       */
>      int (*free_device_capabilities)(struct AVFormatContext *s, struct AVDeviceCapabilitiesQuery *caps);
> +#endif
>  } AVInputFormat;
>  /**
>   * @}
> 
Can I get an update on how to proceed with this patch?

- Andreas

PS: I could already remove all the av_device_capabilities options
(except the sentinel) at the next major bump, couldn't I?

Quoting Andreas Rheinhardt (2021-02-09 09:04:23)
> Can I get an update on how to proceed with this patch?

It doesn't seem that anyone actually objected to this patch, so go ahead
and push IMO.

> 
> - Andreas
> 
> PS: I could already remove all the av_device_capabilities options
> (except the sentinel) at the next major bump, couldn't I?

Doxy says should not be used by a user, so I guess yes.

Anton Khirnov:
> Quoting Andreas Rheinhardt (2021-02-09 09:04:23)
>> Can I get an update on how to proceed with this patch?
> 
> It doesn't seem that anyone actually objected to this patch, so go ahead
> and push IMO.
> 
>>
>> - Andreas
>>
>> PS: I could already remove all the av_device_capabilities options
>> (except the sentinel) at the next major bump, couldn't I?
> 
> Doxy says should not be used by a user, so I guess yes.
> 
Then I could actually remove these options now.

- Andreas

Andreas Rheinhardt:
> Anton Khirnov:
>> Quoting Andreas Rheinhardt (2021-02-09 09:04:23)
>>> Can I get an update on how to proceed with this patch?
>>
>> It doesn't seem that anyone actually objected to this patch, so go ahead
>> and push IMO.
>>
>>>
>>> - Andreas
>>>
>>> PS: I could already remove all the av_device_capabilities options
>>> (except the sentinel) at the next major bump, couldn't I?
>>
>> Doxy says should not be used by a user, so I guess yes.
>>
> Then I could actually remove these options now.
> 
> - Andreas
> 
Will apply with that change the day after tomorrow unless there are
objections.

- Andreas

I (new contributor) have been making a push to get avdevice/dshow to where
it is great to use programmatically (see recent patches, more coming). One
thing on my list is to be able to through the API query the capabilities of
my device, as right now I'd have to ask users to run a command like ffmpeg
-list_options true -f dshow -i video="Camera" to get that info and manually
enter it into my program, let alone that I am unable to make a nicer GUI. I
see that this capability has just been deprecated. Instead of waiting for a
new avdevice interface that may or may not come (I personally like that they
are just formats like anything else, super easy to use), would it be
possible to undeprecate when I provide an implementation of
create_device_capabilities for the dshow avdevice?

Thanks!

-----Original Message-----
From: andreas.rheinhardt at gmail.com (Andreas Rheinhardt)
<andreas.rheinhardt at gmail.com (Andreas Rheinhardt)> 
Sent: 24 January 2021 21:16
Subject: [FFmpeg-devel] [PATCH] avdevice/avdevice: Deprecate AVDevice
Capabilities API

It has been added in 6db42a2b6b22e6f1928fafcf3faa67ed78201004,
yet since then none of the necessary create/free_device_capabilities
functions has been implemented, making this API completely useless.

Because of this one can already simplify avdevice_capabilities_free/create
and can already remove the function pointers at the next major bump.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at gmail.com>
---
 doc/APIchanges         |  4 ++++
 libavdevice/avdevice.c | 41 ++++++-----------------------------------
 libavdevice/avdevice.h |  5 +++++
 libavdevice/version.h  |  7 +++++--
 libavformat/avformat.h |  4 ++++
 5 files changed, 24 insertions(+), 37 deletions(-)

diff --git a/doc/APIchanges b/doc/APIchanges index bbf56a5385..8eeb6ba70f
100644
--- a/doc/APIchanges
+++ b/doc/APIchanges
@@ -15,6 +15,10 @@ libavutil:     2017-10-21
 
 API changes, most recent first:
 
+2021-01-24 - xxxxxxxxxx - lavd 58.12.100 - avdevice.h
+  Deprecated avdevice_capabilities_create() and
+  Add avdevice_capabilities_free(). They
+
 2021-01-11 - xxxxxxxxxx - lavc 58.116.100 - avcodec.h
   Add FF_PROFILE_VVC_MAIN_10 and FF_PROFILE_VVC_MAIN_10_444.
 
diff --git a/libavdevice/avdevice.c b/libavdevice/avdevice.c index
ec84d3b990..fbcf4a4ab2 100644
--- a/libavdevice/avdevice.c
+++ b/libavdevice/avdevice.c
@@ -27,6 +27,7 @@
 #include "libavutil/ffversion.h"
 const char av_device_ffversion[] = "FFmpeg version " FFMPEG_VERSION;
 
+#if FF_API_DEVICE_CAPABILITIES
 #define E AV_OPT_FLAG_ENCODING_PARAM
 #define D AV_OPT_FLAG_DECODING_PARAM
 #define A AV_OPT_FLAG_AUDIO_PARAM
@@ -60,6 +61,7 @@ const AVOption av_device_capabilities[] = {  #undef A
#undef V  #undef OFFSET
+#endif
 
 unsigned avdevice_version(void)
 {
@@ -94,49 +96,18 @@ int avdevice_dev_to_app_control_message(struct
AVFormatContext *s, enum AVDevToA
     return s->control_message_cb(s, type, data, data_size);  }
 
+#if FF_API_DEVICE_CAPABILITIES
 int avdevice_capabilities_create(AVDeviceCapabilitiesQuery **caps,
AVFormatContext *s,
                                  AVDictionary **device_options)  {
-    int ret;
-    av_assert0(s && caps);
-    av_assert0(s->iformat || s->oformat);
-    if ((s->oformat && !s->oformat->create_device_capabilities) ||
-        (s->iformat && !s->iformat->create_device_capabilities))
-        return AVERROR(ENOSYS);
-    *caps = av_mallocz(sizeof(**caps));
-    if (!(*caps))
-        return AVERROR(ENOMEM);
-    (*caps)->device_context = s;
-    if (((ret = av_opt_set_dict(s->priv_data, device_options)) < 0))
-        goto fail;
-    if (s->iformat) {
-        if ((ret = s->iformat->create_device_capabilities(s, *caps)) < 0)
-            goto fail;
-    } else {
-        if ((ret = s->oformat->create_device_capabilities(s, *caps)) < 0)
-            goto fail;
-    }
-    av_opt_set_defaults(*caps);
-    return 0;
-  fail:
-    av_freep(caps);
-    return ret;
+    return AVERROR(ENOSYS);
 }
 
 void avdevice_capabilities_free(AVDeviceCapabilitiesQuery **caps,
AVFormatContext *s)  {
-    if (!s || !caps || !(*caps))
-        return;
-    av_assert0(s->iformat || s->oformat);
-    if (s->iformat) {
-        if (s->iformat->free_device_capabilities)
-            s->iformat->free_device_capabilities(s, *caps);
-    } else {
-        if (s->oformat->free_device_capabilities)
-            s->oformat->free_device_capabilities(s, *caps);
-    }
-    av_freep(caps);
+    return;
 }
+#endif
 
 int avdevice_list_devices(AVFormatContext *s, AVDeviceInfoList
**device_list)  { diff --git a/libavdevice/avdevice.h
b/libavdevice/avdevice.h index ee9462480e..85a4dcc6df 100644
--- a/libavdevice/avdevice.h
+++ b/libavdevice/avdevice.h
@@ -321,6 +321,7 @@ int avdevice_dev_to_app_control_message(struct
AVFormatContext *s,
                                         enum AVDevToAppMessageType type,
                                         void *data, size_t data_size);
 
+#if FF_API_DEVICE_CAPABILITIES
 /**
  * Following API allows user to probe device capabilities (supported
codecs,
  * pixel formats, sample formats, resolutions, channel counts, etc).
@@ -416,6 +417,7 @@ typedef struct AVDeviceCapabilitiesQuery {
 /**
  * AVOption table used by devices to implement device capabilities API.
Should not be used by a user.
  */
+attribute_deprecated
 extern const AVOption av_device_capabilities[];
 
 /**
@@ -435,6 +437,7 @@ extern const AVOption av_device_capabilities[];
  *
  * @return >= 0 on success, negative otherwise.
  */
+attribute_deprecated
 int avdevice_capabilities_create(AVDeviceCapabilitiesQuery **caps,
AVFormatContext *s,
                                  AVDictionary **device_options);
 
@@ -444,7 +447,9 @@ int
avdevice_capabilities_create(AVDeviceCapabilitiesQuery **caps, AVFormatConte
  * @param caps Device capabilities data to be freed.
  * @param s    Context of the device.
  */
+attribute_deprecated
 void avdevice_capabilities_free(AVDeviceCapabilitiesQuery **caps,
AVFormatContext *s);
+#endif
 
 /**
  * Structure describes basic parameters of the device.
diff --git a/libavdevice/version.h b/libavdevice/version.h index
7022fdbf2a..f5aaa168af 100644
--- a/libavdevice/version.h
+++ b/libavdevice/version.h
@@ -28,8 +28,8 @@
 #include "libavutil/version.h"
 
 #define LIBAVDEVICE_VERSION_MAJOR  58
-#define LIBAVDEVICE_VERSION_MINOR  11
-#define LIBAVDEVICE_VERSION_MICRO 103
+#define LIBAVDEVICE_VERSION_MINOR  12
+#define LIBAVDEVICE_VERSION_MICRO 100
 
 #define LIBAVDEVICE_VERSION_INT AV_VERSION_INT(LIBAVDEVICE_VERSION_MAJOR, \
                                                LIBAVDEVICE_VERSION_MINOR, \
@@ -46,5 +46,8 @@
  * dropped at a future version bump. The defines themselves are not part of
  * the public API and may change, break or disappear at any time.
  */
+#ifndef FF_API_DEVICE_CAPABILITIES
+#define FF_API_DEVICE_CAPABILITIES (LIBAVDEVICE_VERSION_MAJOR < 60) 
+#endif
 
 #endif /* AVDEVICE_VERSION_H */
diff --git a/libavformat/avformat.h b/libavformat/avformat.h index
523cf34d55..41482328f6 100644
--- a/libavformat/avformat.h
+++ b/libavformat/avformat.h
@@ -590,6 +590,7 @@ typedef struct AVOutputFormat {
      * @see avdevice_list_devices() for more details.
      */
     int (*get_device_list)(struct AVFormatContext *s, struct
AVDeviceInfoList *device_list);
+#if LIBAVFORMAT_VERSION_MAJOR < 59
     /**
      * Initialize device capabilities submodule.
      * @see avdevice_capabilities_create() for more details.
@@ -600,6 +601,7 @@ typedef struct AVOutputFormat {
      * @see avdevice_capabilities_free() for more details.
      */
     int (*free_device_capabilities)(struct AVFormatContext *s, struct
AVDeviceCapabilitiesQuery *caps);
+#endif
     enum AVCodecID data_codec; /**< default data codec */
     /**
      * Initialize format. May allocate data here, and set any
AVFormatContext or @@ -769,6 +771,7 @@ typedef struct AVInputFormat {
      */
     int (*get_device_list)(struct AVFormatContext *s, struct
AVDeviceInfoList *device_list);
 
+#if LIBAVFORMAT_VERSION_MAJOR < 59
     /**
      * Initialize device capabilities submodule.
      * @see avdevice_capabilities_create() for more details.
@@ -780,6 +783,7 @@ typedef struct AVInputFormat {
      * @see avdevice_capabilities_free() for more details.
      */
     int (*free_device_capabilities)(struct AVFormatContext *s, struct
AVDeviceCapabilitiesQuery *caps);
+#endif
 } AVInputFormat;
 /**
  * @}
--
2.25.1

dcnieho@gmail.com (12021-06-02):
> I (new contributor) have been making a push to get avdevice/dshow to where
> it is great to use programmatically (see recent patches, more coming). One
> thing on my list is to be able to through the API query the capabilities of
> my device, as right now I'd have to ask users to run a command like ffmpeg
> -list_options true -f dshow -i video="Camera" to get that info and manually
> enter it into my program, let alone that I am unable to make a nicer GUI. I
> see that this capability has just been deprecated. Instead of waiting for a
> new avdevice interface that may or may not come (I personally like that they
> are just formats like anything else, super easy to use), would it be
> possible to undeprecate when I provide an implementation of
> create_device_capabilities for the dshow avdevice?

Excellent. Applications that use the advanced features of libavdevice
and serve as test beds for these features are sorely needed.

The project has no real system to make engagements about something like
this, but I can say that if you propose a patch series that de-deprecate
the API and implements it in dshow, I would personally support it.

Regards,

On Wed, Jun 2, 2021 at 2:37 PM Nicolas George <george@nsup.org> wrote:
>
> dcnieho@gmail.com (12021-06-02):
> > I (new contributor) have been making a push to get avdevice/dshow to where
> > it is great to use programmatically (see recent patches, more coming). One
> > thing on my list is to be able to through the API query the capabilities of
> > my device, as right now I'd have to ask users to run a command like ffmpeg
> > -list_options true -f dshow -i video="Camera" to get that info and manually
> > enter it into my program, let alone that I am unable to make a nicer GUI. I
> > see that this capability has just been deprecated. Instead of waiting for a
> > new avdevice interface that may or may not come (I personally like that they
> > are just formats like anything else, super easy to use), would it be
> > possible to undeprecate when I provide an implementation of
> > create_device_capabilities for the dshow avdevice?
>
> Excellent. Applications that use the advanced features of libavdevice
> and serve as test beds for these features are sorely needed.
>
> The project has no real system to make engagements about something like
> this, but I can say that if you propose a patch series that de-deprecate
> the API and implements it in dshow, I would personally support it.

Great to hear! What should i do to de-deprecate?
1. bump LIBAVDEVICE_VERSION_MINOR
2. remove FF_API_DEVICE_CAPABILITIES define
3. remove the #if LIBAVFORMAT_VERSION_MAJOR < 59 in avformat.h
around
int (*create_device_capabilities)(struct AVFormatContext *s, struct
AVDeviceCapabilitiesQuery *caps);
and
int (*free_device_capabilities)(struct AVFormatContext *s, struct
AVDeviceCapabilitiesQuery *caps);
?

All the best,
Dee

Hi Nicolas,

On Wed, Jun 2, 2021 at 2:37 PM Nicolas George <george@nsup.org> wrote:
> Excellent. Applications that use the advanced features of libavdevice
> and serve as test beds for these features are sorely needed.
>
> The project has no real system to make engagements about something like
> this, but I can say that if you propose a patch series that de-deprecate
> the API and implements it in dshow, I would personally support it.

Just sent the patch, it completes my push together with my other
patches of the last few days to make the dshow device fully
programmatically controllable and discoverable.

By the way, each of these patch series applies to master, but not on
top of each other (there'd be massive conflicts). What is custom?
Should i instead send them all as one large patch series?

Thanks and all the best,
Dee

Diederick C. Niehorster (12021-06-04):
> Just sent the patch, it completes my push together with my other
> patches of the last few days to make the dshow device fully
> programmatically controllable and discoverable.
> 
> By the way, each of these patch series applies to master, but not on
> top of each other (there'd be massive conflicts). What is custom?
> Should i instead send them all as one large patch series?

I do not understand: you did send them as a large patch series. Twice,
by the way, which is confusing.

It is the way to do it: make clean commits in your Git tree, using
rebase if necessary to make them really clean. Then let git format-patch
or send-email make a patch series, with each patch stacking on top of
the other.

Regards,

Hi Nicolas,

On Fri, Jun 4, 2021 at 11:06 AM Nicolas George <george@nsup.org> wrote:
> I do not understand: you did send them as a large patch series. Twice,
> by the way, which is confusing.

Yes, the first series got messed up, send it a second time correctly.
I've cleaned up patchwork, it only shows the right ones.

I have sent multiple patch series, all of which apply cleanly to
master, but some of which heavily conflict with each other. E.g.
https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=4090,
https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=4088 and
https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=4100
implement different features, but each heavily edit overlapping code.
Once one is merged, the others will conflict. Should i put these all
in one large series implementing multiple features?

> It is the way to do it: make clean commits in your Git tree, using
> rebase if necessary to make them really clean. Then let git format-patch
> or send-email make a patch series, with each patch stacking on top of
> the other.

Thanks! send-email I have made some mistakes with my first times here,
sorry again for the double patches.

Cheers,
Dee

Hi All,

If instead of the various separate patch series i have sent the last
few days, you would like to see one integrated series where all are
applied on top of each other and conflicts resolves, please see all
the commits ahead of master (currently 22) on the develop branch here:
https://github.com/dcnieho/ffmpeg/tree/develop

In making this branch, i have also done some small refactoring, and
fixed some issues i came across in my implementation.

As said before, I hope one of you can advise me on how to submit all
this (one big series so it all applies cleanly?). The brunt is letting
dshow provide more information about what video data  it provides, and
especially making it way more controllable programmatically/through
the API. It is now at a point where i can use ffmpeg/dshow as a
backend to flexible, reconfigurable use of e.g. a webcam. Exciting to
me:)

All the best,
Dee

On Fri, Jun 4, 2021 at 2:22 PM Diederick C. Niehorster
<dcnieho@gmail.com> wrote:
>
> Hi Nicolas,
>
> On Fri, Jun 4, 2021 at 11:06 AM Nicolas George <george@nsup.org> wrote:
> > I do not understand: you did send them as a large patch series. Twice,
> > by the way, which is confusing.
>
> Yes, the first series got messed up, send it a second time correctly.
> I've cleaned up patchwork, it only shows the right ones.
>
> I have sent multiple patch series, all of which apply cleanly to
> master, but some of which heavily conflict with each other. E.g.
> https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=4090,
> https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=4088 and
> https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=4100
> implement different features, but each heavily edit overlapping code.
> Once one is merged, the others will conflict. Should i put these all
> in one large series implementing multiple features?
>
> > It is the way to do it: make clean commits in your Git tree, using
> > rebase if necessary to make them really clean. Then let git format-patch
> > or send-email make a patch series, with each patch stacking on top of
> > the other.
>
> Thanks! send-email I have made some mistakes with my first times here,
> sorry again for the double patches.
>
> Cheers,
> Dee

You may want to wait for at least some review and then of course, Andreas,
for example, sometimes sends 200 patch series.
https://patchwork.ffmpeg.org/project/ffmpeg/patch/20201203003628.778278-6-andreas.rheinhardt@gmail.com/