[FFmpeg-devel] avfilter: add audio surround upmixer

Submitted by Paul B Mahol on May 25, 2017, 2:45 p.m.

Details

Message ID 20170525144546.15459-1-onemda@gmail.com
State New
Headers show

Commit Message

Paul B Mahol May 25, 2017, 2:45 p.m.
Signed-off-by: Paul B Mahol <onemda@gmail.com>
---
 doc/filters.texi          |  27 ++
 libavfilter/Makefile      |   1 +
 libavfilter/af_surround.c | 853 ++++++++++++++++++++++++++++++++++++++++++++++
 libavfilter/allfilters.c  |   1 +
 4 files changed, 882 insertions(+)
 create mode 100644 libavfilter/af_surround.c

Comments

Carl Eugen Hoyos May 25, 2017, 6:34 p.m.
2017-05-25 16:45 GMT+02:00 Paul B Mahol <onemda@gmail.com>:

> +@section surround
> +Apply audio surround upmix filter.
> +
> +This filter allows to produce multichannel output from
> stereo audio stream.

Does this work on Pro Logic-encoded files (ticket #4085)
or does this filter use a different algorithm?
Does a filter have advantages over an implementation in
libswresample?

Iirc, our Pro Logic downmixer has an issue with "phase shift"
(ticket 3455 / 4175): Does this filter give expected output
for FFmpeg-encoded files or original Pro Logic files (or is
there no difference on upmixing)?

Thank you, I believe this is a very important feature,
Carl Eugen
Paul B Mahol May 25, 2017, 8:39 p.m.
On 5/25/17, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
> 2017-05-25 16:45 GMT+02:00 Paul B Mahol <onemda@gmail.com>:
>
>> +@section surround
>> +Apply audio surround upmix filter.
>> +
>> +This filter allows to produce multichannel output from
>> stereo audio stream.
>
> Does this work on Pro Logic-encoded files (ticket #4085)
> or does this filter use a different algorithm?

It works with sample from that ticket.
It uses own algorithm.

> Does a filter have advantages over an implementation in
> libswresample?

libswresample doesnt do this.

>
> Iirc, our Pro Logic downmixer has an issue with "phase shift"
> (ticket 3455 / 4175): Does this filter give expected output
> for FFmpeg-encoded files or original Pro Logic files (or is
> there no difference on upmixing)?

FFmpeg downmixing, while certainly not correct, is giving reasonable
output when used with this filter.
Moritz Barsnick May 25, 2017, 8:53 p.m.
The same questions as Carl Eugen posted came to mind, furthermore:

On Thu, May 25, 2017 at 16:45:46 +0200, Paul B Mahol wrote:

> +Set output channel layout. By default is @var{5.1}.

The grammar of the second sentence here (and all the following
descriptions) should either by:
"By default, this is @var{}."
or
"Default is @var{}."

Regarding the channel layout: Could you please link to the section
describing the supported layouts? As such:

"See @ref{channel layout syntax,,the Channel Layout section in the ffmpeg-utils(1) manual,ffmpeg-utils}
for the required syntax."
 
> +@item lfe
> +Enable LFE channel output if output channel layout have it. By default is enabled.

"has it"

Moritz
Paul B Mahol May 25, 2017, 9:32 p.m.
On 5/25/17, Moritz Barsnick <barsnick@gmx.net> wrote:
> The same questions as Carl Eugen posted came to mind, furthermore:
>
> On Thu, May 25, 2017 at 16:45:46 +0200, Paul B Mahol wrote:
>
>> +Set output channel layout. By default is @var{5.1}.
>
> The grammar of the second sentence here (and all the following
> descriptions) should either by:
> "By default, this is @var{}."
> or
> "Default is @var{}."
>
> Regarding the channel layout: Could you please link to the section
> describing the supported layouts? As such:
>
> "See @ref{channel layout syntax,,the Channel Layout section in the
> ffmpeg-utils(1) manual,ffmpeg-utils}
> for the required syntax."
>
>> +@item lfe
>> +Enable LFE channel output if output channel layout have it. By default is
>> enabled.
>
> "has it"
>
> Moritz
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>

Done.
Carl Eugen Hoyos May 25, 2017, 10:59 p.m.
2017-05-25 22:39 GMT+02:00 Paul B Mahol <onemda@gmail.com>:
> On 5/25/17, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>> 2017-05-25 16:45 GMT+02:00 Paul B Mahol <onemda@gmail.com>:
>>
>>> +@section surround
>>> +Apply audio surround upmix filter.
>>> +
>>> +This filter allows to produce multichannel output from
>>> stereo audio stream.
>>
>> Does this work on Pro Logic-encoded files (ticket #4085)
>> or does this filter use a different algorithm?
>
> It works with sample from that ticket.

> It uses own algorithm.

Is it related to one of the algorithms listed here?
https://en.wikipedia.org/wiki/Matrix_decoder
It may make sense to mention the used algorithm in the
documentation (or the source code).

>> Does a filter have advantages over an implementation in
>> libswresample?
>
> libswresample doesnt do this.

Yes.
My question was if your implementation shouldn't be
part of libswresample, or rather what would be the
disadvantage.

>> Iirc, our Pro Logic downmixer has an issue with "phase shift"
>> (ticket 3455 / 4175): Does this filter give expected output
>> for FFmpeg-encoded files or original Pro Logic files (or is
>> there no difference on upmixing)?
>
> FFmpeg downmixing, while certainly not correct, is giving
> reasonable output when used with this filter.

But it works better for Pro Logic files?

Thanks, Carl Eugen
Paul B Mahol May 26, 2017, 9:16 a.m.
On 5/26/17, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
> 2017-05-25 22:39 GMT+02:00 Paul B Mahol <onemda@gmail.com>:
>> On 5/25/17, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>>> 2017-05-25 16:45 GMT+02:00 Paul B Mahol <onemda@gmail.com>:
>>>
>>>> +@section surround
>>>> +Apply audio surround upmix filter.
>>>> +
>>>> +This filter allows to produce multichannel output from
>>>> stereo audio stream.
>>>
>>> Does this work on Pro Logic-encoded files (ticket #4085)
>>> or does this filter use a different algorithm?
>>
>> It works with sample from that ticket.
>
>> It uses own algorithm.
>
> Is it related to one of the algorithms listed here?
> https://en.wikipedia.org/wiki/Matrix_decoder
> It may make sense to mention the used algorithm in the
> documentation (or the source code).

All those are passive algorithms, while this one is active.

>
>>> Does a filter have advantages over an implementation in
>>> libswresample?
>>
>> libswresample doesnt do this.
>
> Yes.
> My question was if your implementation shouldn't be
> part of libswresample, or rather what would be the
> disadvantage.

This filter relies on FFT, and it should probably stay in libavfilter library.

>
>>> Iirc, our Pro Logic downmixer has an issue with "phase shift"
>>> (ticket 3455 / 4175): Does this filter give expected output
>>> for FFmpeg-encoded files or original Pro Logic files (or is
>>> there no difference on upmixing)?
>>
>> FFmpeg downmixing, while certainly not correct, is giving
>> reasonable output when used with this filter.
>
> But it works better for Pro Logic files?

I only have one such file, and its not that useful.

FFmpeg have only downmixing from 5.1 layout to stereo.

I also plan to add 5.1 to 7.1 upmixing.
Michael Niedermayer May 26, 2017, 11 a.m.
On Thu, May 25, 2017 at 04:45:46PM +0200, Paul B Mahol wrote:
> Signed-off-by: Paul B Mahol <onemda@gmail.com>
> ---
>  doc/filters.texi          |  27 ++
>  libavfilter/Makefile      |   1 +
>  libavfilter/af_surround.c | 853 ++++++++++++++++++++++++++++++++++++++++++++++
>  libavfilter/allfilters.c  |   1 +
>  4 files changed, 882 insertions(+)
>  create mode 100644 libavfilter/af_surround.c

This belongs in libswresample

[...]
Paul B Mahol May 26, 2017, 11:02 a.m.
On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> On Thu, May 25, 2017 at 04:45:46PM +0200, Paul B Mahol wrote:
>> Signed-off-by: Paul B Mahol <onemda@gmail.com>
>> ---
>>  doc/filters.texi          |  27 ++
>>  libavfilter/Makefile      |   1 +
>>  libavfilter/af_surround.c | 853
>> ++++++++++++++++++++++++++++++++++++++++++++++
>>  libavfilter/allfilters.c  |   1 +
>>  4 files changed, 882 insertions(+)
>>  create mode 100644 libavfilter/af_surround.c
>
> This belongs in libswresample

No it does not.
Nicolas George May 26, 2017, 11:08 a.m.
Le septidi 7 prairial, an CCXXV, Paul B Mahol a écrit :
> > This belongs in libswresample
> No it does not.

I think it does too.

Regards,
Paul B Mahol May 26, 2017, 11:11 a.m.
On 5/26/17, Nicolas George <george@nsup.org> wrote:
> Le septidi 7 prairial, an CCXXV, Paul B Mahol a ecrit :
>> > This belongs in libswresample
>> No it does not.
>
> I think it does too.

You want to link libswresample with libavcodec?

Besides libswresample is quite simple.
And point of this filter it to not be simple
at all or even to be some sort of standard.
wm4 May 26, 2017, 11:11 a.m.
On Fri, 26 May 2017 13:08:10 +0200
Nicolas George <george@nsup.org> wrote:

> Le septidi 7 prairial, an CCXXV, Paul B Mahol a écrit :
> > > This belongs in libswresample  
> > No it does not.  
> 
> I think it does too.

There's plenty of precedent of not moving filters into libswresample or
libswscale, even if in theory it could have been done.
Nicolas George May 26, 2017, 11:13 a.m.
Le septidi 7 prairial, an CCXXV, Paul B Mahol a écrit :
> You want to link libswresample with libavcodec?

I want to merge all libraries, I do not make a secret of it. The
internal separation is nothing but trouble. But API-wise, this feature
belongs in resampling.

Regards,
Hendrik Leppkes May 26, 2017, 11:36 a.m.
On Fri, May 26, 2017 at 1:13 PM, Nicolas George <george@nsup.org> wrote:
> Le septidi 7 prairial, an CCXXV, Paul B Mahol a écrit :
>> You want to link libswresample with libavcodec?
>
> I want to merge all libraries, I do not make a secret of it. The
> internal separation is nothing but trouble. But API-wise, this feature
> belongs in resampling.
>

Implementation-wise, swresample has nothing of the required primitives
already, and as Paul pointed out it relies on the FFT from avcodec,
linking swresample to avcodec is a big no-no (a breaking change, and a
circular dependency since avcodec links to swresample already), while
avfilter does already link to avcodec for a variety of things.

Until such a day when everything becomes merged, if it ever happens,
there is several technical aspects that make it very impractical to
put this into swresample.

- Hendrik
Michael Niedermayer May 26, 2017, 11:43 a.m.
On Fri, May 26, 2017 at 01:11:38PM +0200, Paul B Mahol wrote:
> On 5/26/17, Nicolas George <george@nsup.org> wrote:
> > Le septidi 7 prairial, an CCXXV, Paul B Mahol a ecrit :
> >> > This belongs in libswresample
> >> No it does not.
> >
> > I think it does too.
> 
> You want to link libswresample with libavcodec?

While this question was directed at nicolas ...

I dont think audio upmix code should depend on a lib of encoders and
decoders (libavcodec)
No matter if the upmix code would be in libavfilter or libswresample

I belive a temporary dependancy would be ok, if there is intend to
factor/move things to remove the dependancy in the future.

But IMO libavcodec is the wrong place to export generic FFT
functionality.
We need FFTs in codecs, we need them in filters, we need them in
swresample (the soxr resampler we support also uses a FFT internally)

Also moving FFT to a different lib should be quite easy compared to
other ugly dependancies we have (as in snow motion estimation, which
is not as easy to move. none the less none of these ugly dependancies
should be there except temporary)

[...]
Paul B Mahol May 26, 2017, 4:54 p.m.
On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> On Fri, May 26, 2017 at 01:11:38PM +0200, Paul B Mahol wrote:
>> On 5/26/17, Nicolas George <george@nsup.org> wrote:
>> > Le septidi 7 prairial, an CCXXV, Paul B Mahol a ecrit :
>> >> > This belongs in libswresample
>> >> No it does not.
>> >
>> > I think it does too.
>>
>> You want to link libswresample with libavcodec?
>
> While this question was directed at nicolas ...
>
> I dont think audio upmix code should depend on a lib of encoders and
> decoders (libavcodec)
> No matter if the upmix code would be in libavfilter or libswresample
>
> I belive a temporary dependancy would be ok, if there is intend to
> factor/move things to remove the dependancy in the future.
>
> But IMO libavcodec is the wrong place to export generic FFT
> functionality.
> We need FFTs in codecs, we need them in filters, we need them in
> swresample (the soxr resampler we support also uses a FFT internally)
>
> Also moving FFT to a different lib should be quite easy compared to
> other ugly dependancies we have (as in snow motion estimation, which
> is not as easy to move. none the less none of these ugly dependancies
> should be there except temporary)

This code does upmixing, and there could by myriad variants of upmixing.

Having it in libswresample is flawed design.

So I will not do the transitions.

If you still object to leave it as it is, in lavfi. You will need to take
care of the necessary changes by yourself.
Michael Niedermayer May 26, 2017, 7:55 p.m.
On Fri, May 26, 2017 at 06:54:33PM +0200, Paul B Mahol wrote:
> On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> > On Fri, May 26, 2017 at 01:11:38PM +0200, Paul B Mahol wrote:
> >> On 5/26/17, Nicolas George <george@nsup.org> wrote:
> >> > Le septidi 7 prairial, an CCXXV, Paul B Mahol a ecrit :
> >> >> > This belongs in libswresample
> >> >> No it does not.
> >> >
> >> > I think it does too.
> >>
> >> You want to link libswresample with libavcodec?
> >
> > While this question was directed at nicolas ...
> >
> > I dont think audio upmix code should depend on a lib of encoders and
> > decoders (libavcodec)
> > No matter if the upmix code would be in libavfilter or libswresample
> >
> > I belive a temporary dependancy would be ok, if there is intend to
> > factor/move things to remove the dependancy in the future.
> >
> > But IMO libavcodec is the wrong place to export generic FFT
> > functionality.
> > We need FFTs in codecs, we need them in filters, we need them in
> > swresample (the soxr resampler we support also uses a FFT internally)
> >
> > Also moving FFT to a different lib should be quite easy compared to
> > other ugly dependancies we have (as in snow motion estimation, which
> > is not as easy to move. none the less none of these ugly dependancies
> > should be there except temporary)
> 

> This code does upmixing, and there could by myriad variants of upmixing.

This is true for any format convertion or more genericaly for anything.
there are always many ways to do something.

the way FFmpeg is modularized is that we have a lib for audio format
convertion, resampling, rematrixing including upmixing and
downmixing, ...


> 
> Having it in libswresample is flawed design.
> 
> So I will not do the transitions.
> 
> If you still object to leave it as it is, in lavfi. You will need to take
> care of the necessary changes by yourself.

Do you agree that we need a part of code and API that does
audio format convertion, amongth it upmixing and downmixing ?
Something thats used by default

if you agree on the need of such code, why would it be flawed design
to add a improved upmixing implementation in there so it gets used ?
(be that by default or users choice of what to use by default or
 by specific choice for a specific instance)

I want the best and most correct code to be used.
I dont want to object or demand anything. I belive though that putting
upmixing code in 2 different places and 2 different libs will give us
headaches in the future

[...]
Paul B Mahol May 26, 2017, 8:04 p.m.
On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> On Fri, May 26, 2017 at 06:54:33PM +0200, Paul B Mahol wrote:
>> On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
>> > On Fri, May 26, 2017 at 01:11:38PM +0200, Paul B Mahol wrote:
>> >> On 5/26/17, Nicolas George <george@nsup.org> wrote:
>> >> > Le septidi 7 prairial, an CCXXV, Paul B Mahol a ecrit :
>> >> >> > This belongs in libswresample
>> >> >> No it does not.
>> >> >
>> >> > I think it does too.
>> >>
>> >> You want to link libswresample with libavcodec?
>> >
>> > While this question was directed at nicolas ...
>> >
>> > I dont think audio upmix code should depend on a lib of encoders and
>> > decoders (libavcodec)
>> > No matter if the upmix code would be in libavfilter or libswresample
>> >
>> > I belive a temporary dependancy would be ok, if there is intend to
>> > factor/move things to remove the dependancy in the future.
>> >
>> > But IMO libavcodec is the wrong place to export generic FFT
>> > functionality.
>> > We need FFTs in codecs, we need them in filters, we need them in
>> > swresample (the soxr resampler we support also uses a FFT internally)
>> >
>> > Also moving FFT to a different lib should be quite easy compared to
>> > other ugly dependancies we have (as in snow motion estimation, which
>> > is not as easy to move. none the less none of these ugly dependancies
>> > should be there except temporary)
>>
>
>> This code does upmixing, and there could by myriad variants of upmixing.
>
> This is true for any format convertion or more genericaly for anything.
> there are always many ways to do something.
>
> the way FFmpeg is modularized is that we have a lib for audio format
> convertion, resampling, rematrixing including upmixing and
> downmixing, ...
>
>
>>
>> Having it in libswresample is flawed design.
>>
>> So I will not do the transitions.
>>
>> If you still object to leave it as it is, in lavfi. You will need to take
>> care of the necessary changes by yourself.
>
> Do you agree that we need a part of code and API that does
> audio format convertion, amongth it upmixing and downmixing ?
> Something thats used by default
>
> if you agree on the need of such code, why would it be flawed design
> to add a improved upmixing implementation in there so it gets used ?
> (be that by default or users choice of what to use by default or
>  by specific choice for a specific instance)
>
> I want the best and most correct code to be used.
> I dont want to object or demand anything. I belive though that putting
> upmixing code in 2 different places and 2 different libs will give us
> headaches in the future

Why is pan filter not part of libswresample? It does upmixing and downmixing.
Therefore it should be in libswresample by your logic.

Why is stereotools not part of libswresample?

Why is volume not part of libswresample?

Hey why not put there ambisonics decoder too?

Or equalizer?

Why is eq filter not part of libswscale?

Or any damn A->A or V->V filter?

Besides the surround filter is not "correct". There is no "correct"
way to do upmixing.

Consider mono upmixing, which is just spreading different bands to
different channels,
do you want that too in libswresample?

What about synthesizers (we have anoisesrc) but I plan to add
wavetable, do you want that
in libswresample too?
Michael Niedermayer May 26, 2017, 8:39 p.m.
On Fri, May 26, 2017 at 10:04:40PM +0200, Paul B Mahol wrote:
> On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> > On Fri, May 26, 2017 at 06:54:33PM +0200, Paul B Mahol wrote:
> >> On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> >> > On Fri, May 26, 2017 at 01:11:38PM +0200, Paul B Mahol wrote:
> >> >> On 5/26/17, Nicolas George <george@nsup.org> wrote:
> >> >> > Le septidi 7 prairial, an CCXXV, Paul B Mahol a ecrit :
> >> >> >> > This belongs in libswresample
> >> >> >> No it does not.
> >> >> >
> >> >> > I think it does too.
> >> >>
> >> >> You want to link libswresample with libavcodec?
> >> >
> >> > While this question was directed at nicolas ...
> >> >
> >> > I dont think audio upmix code should depend on a lib of encoders and
> >> > decoders (libavcodec)
> >> > No matter if the upmix code would be in libavfilter or libswresample
> >> >
> >> > I belive a temporary dependancy would be ok, if there is intend to
> >> > factor/move things to remove the dependancy in the future.
> >> >
> >> > But IMO libavcodec is the wrong place to export generic FFT
> >> > functionality.
> >> > We need FFTs in codecs, we need them in filters, we need them in
> >> > swresample (the soxr resampler we support also uses a FFT internally)
> >> >
> >> > Also moving FFT to a different lib should be quite easy compared to
> >> > other ugly dependancies we have (as in snow motion estimation, which
> >> > is not as easy to move. none the less none of these ugly dependancies
> >> > should be there except temporary)
> >>
> >
> >> This code does upmixing, and there could by myriad variants of upmixing.
> >
> > This is true for any format convertion or more genericaly for anything.
> > there are always many ways to do something.
> >
> > the way FFmpeg is modularized is that we have a lib for audio format
> > convertion, resampling, rematrixing including upmixing and
> > downmixing, ...
> >
> >
> >>
> >> Having it in libswresample is flawed design.
> >>
> >> So I will not do the transitions.
> >>
> >> If you still object to leave it as it is, in lavfi. You will need to take
> >> care of the necessary changes by yourself.
> >
> > Do you agree that we need a part of code and API that does
> > audio format convertion, amongth it upmixing and downmixing ?
> > Something thats used by default
> >
> > if you agree on the need of such code, why would it be flawed design
> > to add a improved upmixing implementation in there so it gets used ?
> > (be that by default or users choice of what to use by default or
> >  by specific choice for a specific instance)
> >
> > I want the best and most correct code to be used.
> > I dont want to object or demand anything. I belive though that putting
> > upmixing code in 2 different places and 2 different libs will give us
> > headaches in the future
> 
> Why is pan filter not part of libswresample? It does upmixing and downmixing.
> Therefore it should be in libswresample by your logic.

libswresample supports what pan does through the C API
pan in libavfilter provides a convenient way to access the same
functionality through a filtergraph description string

> 
> Why is stereotools not part of libswresample?

> 
> Why is volume not part of libswresample?

same as with pan


> 
> Hey why not put there ambisonics decoder too?
> 
> Or equalizer?
> 
> Why is eq filter not part of libswscale?
> 
> Or any damn A->A or V->V filter?

This is becoming increasingly specific and outside the scope these
libs.


> 
> Besides the surround filter is not "correct". There is no "correct"
> way to do upmixing.
> 
> Consider mono upmixing, which is just spreading different bands to
> different channels,
> do you want that too in libswresample?
> 
> What about synthesizers (we have anoisesrc) but I plan to add
> wavetable, do you want that
> in libswresample too?

You misunderstand me i think.

The filter patch you posted is descibed by you in the patch as:
    +@section surround
    +Apply audio surround upmix filter.
    +
    +This filter allows to produce multichannel output from stereo audio stream.
    +
    +The filter accepts the following options:

That is not a special case or something that falls outside the scope.
(which is what you argue about above with listing other filters)
this is a description of a generic stereo upmix.
In fact this very description could be used for the stereo upmix code
in swresample

[...]
Paul B Mahol May 27, 2017, 6:54 a.m.
On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> On Fri, May 26, 2017 at 10:04:40PM +0200, Paul B Mahol wrote:
>> On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
>> > On Fri, May 26, 2017 at 06:54:33PM +0200, Paul B Mahol wrote:
>> >> On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
>> >> > On Fri, May 26, 2017 at 01:11:38PM +0200, Paul B Mahol wrote:
>> >> >> On 5/26/17, Nicolas George <george@nsup.org> wrote:
>> >> >> > Le septidi 7 prairial, an CCXXV, Paul B Mahol a ecrit :
>> >> >> >> > This belongs in libswresample
>> >> >> >> No it does not.
>> >> >> >
>> >> >> > I think it does too.
>> >> >>
>> >> >> You want to link libswresample with libavcodec?
>> >> >
>> >> > While this question was directed at nicolas ...
>> >> >
>> >> > I dont think audio upmix code should depend on a lib of encoders and
>> >> > decoders (libavcodec)
>> >> > No matter if the upmix code would be in libavfilter or libswresample
>> >> >
>> >> > I belive a temporary dependancy would be ok, if there is intend to
>> >> > factor/move things to remove the dependancy in the future.
>> >> >
>> >> > But IMO libavcodec is the wrong place to export generic FFT
>> >> > functionality.
>> >> > We need FFTs in codecs, we need them in filters, we need them in
>> >> > swresample (the soxr resampler we support also uses a FFT
>> >> > internally)
>> >> >
>> >> > Also moving FFT to a different lib should be quite easy compared to
>> >> > other ugly dependancies we have (as in snow motion estimation, which
>> >> > is not as easy to move. none the less none of these ugly
>> >> > dependancies
>> >> > should be there except temporary)
>> >>
>> >
>> >> This code does upmixing, and there could by myriad variants of
>> >> upmixing.
>> >
>> > This is true for any format convertion or more genericaly for anything.
>> > there are always many ways to do something.
>> >
>> > the way FFmpeg is modularized is that we have a lib for audio format
>> > convertion, resampling, rematrixing including upmixing and
>> > downmixing, ...
>> >
>> >
>> >>
>> >> Having it in libswresample is flawed design.
>> >>
>> >> So I will not do the transitions.
>> >>
>> >> If you still object to leave it as it is, in lavfi. You will need to
>> >> take
>> >> care of the necessary changes by yourself.
>> >
>> > Do you agree that we need a part of code and API that does
>> > audio format convertion, amongth it upmixing and downmixing ?
>> > Something thats used by default
>> >
>> > if you agree on the need of such code, why would it be flawed design
>> > to add a improved upmixing implementation in there so it gets used ?
>> > (be that by default or users choice of what to use by default or
>> >  by specific choice for a specific instance)
>> >
>> > I want the best and most correct code to be used.
>> > I dont want to object or demand anything. I belive though that putting
>> > upmixing code in 2 different places and 2 different libs will give us
>> > headaches in the future
>>
>> Why is pan filter not part of libswresample? It does upmixing and
>> downmixing.
>> Therefore it should be in libswresample by your logic.
>
> libswresample supports what pan does through the C API
> pan in libavfilter provides a convenient way to access the same
> functionality through a filtergraph description string
>
>>
>> Why is stereotools not part of libswresample?
>
>>
>> Why is volume not part of libswresample?
>
> same as with pan
>
>
>>
>> Hey why not put there ambisonics decoder too?
>>
>> Or equalizer?
>>
>> Why is eq filter not part of libswscale?
>>
>> Or any damn A->A or V->V filter?
>
> This is becoming increasingly specific and outside the scope these
> libs.
>
>
>>
>> Besides the surround filter is not "correct". There is no "correct"
>> way to do upmixing.
>>
>> Consider mono upmixing, which is just spreading different bands to
>> different channels,
>> do you want that too in libswresample?
>>
>> What about synthesizers (we have anoisesrc) but I plan to add
>> wavetable, do you want that
>> in libswresample too?
>
> You misunderstand me i think.
>
> The filter patch you posted is descibed by you in the patch as:
>     +@section surround
>     +Apply audio surround upmix filter.
>     +
>     +This filter allows to produce multichannel output from stereo audio
> stream.
>     +
>     +The filter accepts the following options:
>
> That is not a special case or something that falls outside the scope.
> (which is what you argue about above with listing other filters)
> this is a description of a generic stereo upmix.
> In fact this very description could be used for the stereo upmix code
> in swresample

Look, i will not waste my time on this nonsense, including SUINT,

I'm going to fork FFmpeg, everyone is welcome to join.
Michael Niedermayer May 27, 2017, 9:29 a.m.
On Sat, May 27, 2017 at 08:54:31AM +0200, Paul B Mahol wrote:
> On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> > On Fri, May 26, 2017 at 10:04:40PM +0200, Paul B Mahol wrote:
> >> On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> >> > On Fri, May 26, 2017 at 06:54:33PM +0200, Paul B Mahol wrote:
> >> >> On 5/26/17, Michael Niedermayer <michael@niedermayer.cc> wrote:
> >> >> > On Fri, May 26, 2017 at 01:11:38PM +0200, Paul B Mahol wrote:
> >> >> >> On 5/26/17, Nicolas George <george@nsup.org> wrote:
> >> >> >> > Le septidi 7 prairial, an CCXXV, Paul B Mahol a ecrit :
> >> >> >> >> > This belongs in libswresample
> >> >> >> >> No it does not.
> >> >> >> >
> >> >> >> > I think it does too.
> >> >> >>
> >> >> >> You want to link libswresample with libavcodec?
> >> >> >
> >> >> > While this question was directed at nicolas ...
> >> >> >
> >> >> > I dont think audio upmix code should depend on a lib of encoders and
> >> >> > decoders (libavcodec)
> >> >> > No matter if the upmix code would be in libavfilter or libswresample
> >> >> >
> >> >> > I belive a temporary dependancy would be ok, if there is intend to
> >> >> > factor/move things to remove the dependancy in the future.
> >> >> >
> >> >> > But IMO libavcodec is the wrong place to export generic FFT
> >> >> > functionality.
> >> >> > We need FFTs in codecs, we need them in filters, we need them in
> >> >> > swresample (the soxr resampler we support also uses a FFT
> >> >> > internally)
> >> >> >
> >> >> > Also moving FFT to a different lib should be quite easy compared to
> >> >> > other ugly dependancies we have (as in snow motion estimation, which
> >> >> > is not as easy to move. none the less none of these ugly
> >> >> > dependancies
> >> >> > should be there except temporary)
> >> >>
> >> >
> >> >> This code does upmixing, and there could by myriad variants of
> >> >> upmixing.
> >> >
> >> > This is true for any format convertion or more genericaly for anything.
> >> > there are always many ways to do something.
> >> >
> >> > the way FFmpeg is modularized is that we have a lib for audio format
> >> > convertion, resampling, rematrixing including upmixing and
> >> > downmixing, ...
> >> >
> >> >
> >> >>
> >> >> Having it in libswresample is flawed design.
> >> >>
> >> >> So I will not do the transitions.
> >> >>
> >> >> If you still object to leave it as it is, in lavfi. You will need to
> >> >> take
> >> >> care of the necessary changes by yourself.
> >> >
> >> > Do you agree that we need a part of code and API that does
> >> > audio format convertion, amongth it upmixing and downmixing ?
> >> > Something thats used by default
> >> >
> >> > if you agree on the need of such code, why would it be flawed design
> >> > to add a improved upmixing implementation in there so it gets used ?
> >> > (be that by default or users choice of what to use by default or
> >> >  by specific choice for a specific instance)
> >> >
> >> > I want the best and most correct code to be used.
> >> > I dont want to object or demand anything. I belive though that putting
> >> > upmixing code in 2 different places and 2 different libs will give us
> >> > headaches in the future
> >>
> >> Why is pan filter not part of libswresample? It does upmixing and
> >> downmixing.
> >> Therefore it should be in libswresample by your logic.
> >
> > libswresample supports what pan does through the C API
> > pan in libavfilter provides a convenient way to access the same
> > functionality through a filtergraph description string
> >
> >>
> >> Why is stereotools not part of libswresample?
> >
> >>
> >> Why is volume not part of libswresample?
> >
> > same as with pan
> >
> >
> >>
> >> Hey why not put there ambisonics decoder too?
> >>
> >> Or equalizer?
> >>
> >> Why is eq filter not part of libswscale?
> >>
> >> Or any damn A->A or V->V filter?
> >
> > This is becoming increasingly specific and outside the scope these
> > libs.
> >
> >
> >>
> >> Besides the surround filter is not "correct". There is no "correct"
> >> way to do upmixing.
> >>
> >> Consider mono upmixing, which is just spreading different bands to
> >> different channels,
> >> do you want that too in libswresample?
> >>
> >> What about synthesizers (we have anoisesrc) but I plan to add
> >> wavetable, do you want that
> >> in libswresample too?
> >
> > You misunderstand me i think.
> >
> > The filter patch you posted is descibed by you in the patch as:
> >     +@section surround
> >     +Apply audio surround upmix filter.
> >     +
> >     +This filter allows to produce multichannel output from stereo audio
> > stream.
> >     +
> >     +The filter accepts the following options:
> >
> > That is not a special case or something that falls outside the scope.
> > (which is what you argue about above with listing other filters)
> > this is a description of a generic stereo upmix.
> > In fact this very description could be used for the stereo upmix code
> > in swresample
> 
> Look, i will not waste my time on this nonsense, including SUINT,
> 
> I'm going to fork FFmpeg, everyone is welcome to join.

If you feel so strongly about it, then please add the filter to
libavfilter.
Its my oppinion it would be better to keep all audio up/down mix
code in one place. This is only my oppinion, i do not object to
people doing something else. Of course i cannot speak for anyone
else in this thread.

[...]

Patch hide | download patch | download mbox

diff --git a/doc/filters.texi b/doc/filters.texi
index a0ab2fb..19e4bfe 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -3792,6 +3792,33 @@  channels. Default is 0.3.
 Set level of input signal of original channel. Default is 0.8.
 @end table
 
+@section surround
+Apply audio surround upmix filter.
+
+This filter allows to produce multichannel output from stereo audio stream.
+
+The filter accepts the following options:
+
+@table @option
+@item chl_out
+Set output channel layout. By default is @var{5.1}.
+
+@item level_in
+Set input volume level. By default is @var{1}.
+
+@item level_out
+Set output volume level. By default is @var{1}.
+
+@item lfe
+Enable LFE channel output if output channel layout have it. By default is enabled.
+
+@item lfe_low
+Set LFE low cut off frequency. By default is @var{128} Hz.
+
+@item lfe_high
+Set LFE high cut off frequency. By default is @var{256} Hz.
+@end table
+
 @section treble
 
 Boost or cut treble (upper) frequencies of the audio using a two-pole
diff --git a/libavfilter/Makefile b/libavfilter/Makefile
index 434a989..c88dfb3 100644
--- a/libavfilter/Makefile
+++ b/libavfilter/Makefile
@@ -108,6 +108,7 @@  OBJS-$(CONFIG_SILENCEREMOVE_FILTER)          += af_silenceremove.o
 OBJS-$(CONFIG_SOFALIZER_FILTER)              += af_sofalizer.o
 OBJS-$(CONFIG_STEREOTOOLS_FILTER)            += af_stereotools.o
 OBJS-$(CONFIG_STEREOWIDEN_FILTER)            += af_stereowiden.o
+OBJS-$(CONFIG_SURROUND_FILTER)               += af_surround.o
 OBJS-$(CONFIG_TREBLE_FILTER)                 += af_biquads.o
 OBJS-$(CONFIG_TREMOLO_FILTER)                += af_tremolo.o
 OBJS-$(CONFIG_VIBRATO_FILTER)                += af_vibrato.o generate_wave_table.o
diff --git a/libavfilter/af_surround.c b/libavfilter/af_surround.c
new file mode 100644
index 0000000..310b8d7
--- /dev/null
+++ b/libavfilter/af_surround.c
@@ -0,0 +1,853 @@ 
+/*
+ * Copyright (c) 2017 Paul B Mahol
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include "libavutil/audio_fifo.h"
+#include "libavutil/channel_layout.h"
+#include "libavutil/opt.h"
+#include "libavcodec/avfft.h"
+#include "avfilter.h"
+#include "audio.h"
+#include "formats.h"
+
+typedef struct AudioSurroundContext {
+    const AVClass *class;
+
+    char *out_channel_layout_str;
+    float level_in;
+    float level_out;
+    int output_lfe;
+    int lowcutf;
+    int highcutf;
+
+    float lowcut;
+    float highcut;
+
+    uint64_t out_channel_layout;
+    int nb_in_channels;
+    int nb_out_channels;
+
+    AVFrame *input;
+    AVFrame *output;
+    AVFrame *overlap_buffer;
+
+    int buf_size;
+    int hop_size;
+    AVAudioFifo *fifo;
+    RDFTContext **rdft, **irdft;
+    float *window_func_lut;
+
+    int64_t pts;
+
+    void (*upmix)(AVFilterContext *ctx,
+                  float l_phase,
+                  float r_phase,
+                  float c_phase,
+                  float mag_total,
+                  float x, float y,
+                  int n);
+} AudioSurroundContext;
+
+static int query_formats(AVFilterContext *ctx)
+{
+    AudioSurroundContext *s = ctx->priv;
+    AVFilterFormats *formats = NULL;
+    AVFilterChannelLayouts *layouts = NULL;
+    int ret;
+
+    ret = ff_add_format(&formats, AV_SAMPLE_FMT_FLTP);
+    if (ret)
+        return ret;
+    ret = ff_set_common_formats(ctx, formats);
+    if (ret)
+        return ret;
+
+    layouts = NULL;
+    ret = ff_add_channel_layout(&layouts, s->out_channel_layout);
+    if (ret)
+        return ret;
+
+    ret = ff_channel_layouts_ref(layouts, &ctx->outputs[0]->in_channel_layouts);
+    if (ret)
+        return ret;
+
+    layouts = NULL;
+    ret = ff_add_channel_layout(&layouts, AV_CH_LAYOUT_STEREO);
+    if (ret)
+        return ret;
+
+    ret = ff_channel_layouts_ref(layouts, &ctx->inputs[0]->out_channel_layouts);
+    if (ret)
+        return ret;
+
+    formats = ff_all_samplerates();
+    if (!formats)
+        return AVERROR(ENOMEM);
+    return ff_set_common_samplerates(ctx, formats);
+}
+
+static int config_input(AVFilterLink *inlink)
+{
+    AVFilterContext *ctx = inlink->dst;
+    AudioSurroundContext *s = ctx->priv;
+    int ch;
+
+    s->rdft = av_calloc(inlink->channels, sizeof(*s->rdft));
+    if (!s->rdft)
+        return AVERROR(ENOMEM);
+
+    for (ch = 0; ch < inlink->channels; ch++) {
+        s->rdft[ch]  = av_rdft_init(ff_log2(s->buf_size), DFT_R2C);
+        if (!s->rdft[ch])
+            return AVERROR(ENOMEM);
+    }
+    s->nb_in_channels = inlink->channels;
+
+    s->input = ff_get_audio_buffer(inlink, s->buf_size * 2);
+    if (!s->input)
+        return AVERROR(ENOMEM);
+
+    s->fifo = av_audio_fifo_alloc(inlink->format, inlink->channels, s->buf_size);
+    if (!s->fifo)
+        return AVERROR(ENOMEM);
+
+    s->lowcut = 1.f * s->lowcutf / (inlink->sample_rate * 0.5) * (s->buf_size / 2);
+    s->highcut = 1.f * s->highcutf / (inlink->sample_rate * 0.5) * (s->buf_size / 2);
+
+    return 0;
+}
+
+static int config_output(AVFilterLink *outlink)
+{
+    AVFilterContext *ctx = outlink->src;
+    AudioSurroundContext *s = ctx->priv;
+    int ch;
+
+    s->irdft = av_calloc(outlink->channels, sizeof(*s->irdft));
+    if (!s->irdft)
+        return AVERROR(ENOMEM);
+
+    for (ch = 0; ch < outlink->channels; ch++) {
+        s->irdft[ch] = av_rdft_init(ff_log2(s->buf_size), IDFT_C2R);
+        if (!s->irdft[ch])
+            return AVERROR(ENOMEM);
+    }
+    s->nb_out_channels = outlink->channels;
+
+    s->output = ff_get_audio_buffer(outlink, s->buf_size * 2);
+    s->overlap_buffer = ff_get_audio_buffer(outlink, s->buf_size * 2);
+    if (!s->overlap_buffer || !s->output)
+        return AVERROR(ENOMEM);
+
+    return 0;
+}
+
+static void stereo_position(float a, float p, float *x, float *y)
+{
+      *x = av_clipf(a+FFMAX(0, sinf(p-M_PI_2))*FFDIFFSIGN(a,0), -1, 1);
+      *y = av_clipf(cosf(a*M_PI_2+M_PI)*cosf(M_PI_2-p/M_PI)*M_LN10+1, -1, 1);
+}
+
+static void upmix_1_0(AVFilterContext *ctx,
+                      float l_phase,
+                      float r_phase,
+                      float c_phase,
+                      float mag_total,
+                      float x, float y,
+                      int n)
+{
+    float mag, *dst;
+    AudioSurroundContext *s = ctx->priv;
+
+    dst = (float *)s->output->extended_data[0];
+
+    mag = sqrtf(1 - fabsf(x)) * ((y + 1) * .5) * mag_total;
+
+    dst[2 * n    ] = mag * cosf(c_phase);
+    dst[2 * n + 1] = mag * sinf(c_phase);
+}
+
+static void upmix_stereo(AVFilterContext *ctx,
+                         float l_phase,
+                         float r_phase,
+                         float c_phase,
+                         float mag_total,
+                         float x, float y,
+                         int n)
+{
+    float l_mag, r_mag, *dstl, *dstr;
+    AudioSurroundContext *s = ctx->priv;
+
+    dstl = (float *)s->output->extended_data[0];
+    dstr = (float *)s->output->extended_data[1];
+
+    l_mag = sqrtf(0.5 * ( x + 1)) * ((y + 1) * .5) * mag_total;
+    r_mag = sqrtf(0.5 * (-x + 1)) * ((y + 1) * .5) * mag_total;
+
+    dstl[2 * n    ] = l_mag * cosf(l_phase);
+    dstl[2 * n + 1] = l_mag * sinf(l_phase);
+
+    dstr[2 * n    ] = r_mag * cosf(r_phase);
+    dstr[2 * n + 1] = r_mag * sinf(r_phase);
+}
+
+static void upmix_2_1(AVFilterContext *ctx,
+                      float l_phase,
+                      float r_phase,
+                      float c_phase,
+                      float mag_total,
+                      float x, float y,
+                      int n)
+{
+    AudioSurroundContext *s = ctx->priv;
+    float lfe_mag, l_mag, r_mag, *dstl, *dstr, *dstlfe;
+
+    dstl = (float *)s->output->extended_data[0];
+    dstr = (float *)s->output->extended_data[1];
+    dstlfe = (float *)s->output->extended_data[2];
+
+    if (s->output_lfe && n < s->highcut) {
+        lfe_mag = n < s->lowcut ? 1.f : .5f*(1.f+cosf(M_PI*(s->lowcut-n)/(s->lowcut-s->highcut)));
+        lfe_mag *= mag_total;
+        mag_total -= lfe_mag;
+    } else {
+        lfe_mag = 0.f;
+    }
+
+    l_mag = sqrtf(0.5 * ( x + 1)) * ((y + 1) * .5) * mag_total;
+    r_mag = sqrtf(0.5 * (-x + 1)) * ((y + 1) * .5) * mag_total;
+
+    dstl[2 * n    ] = l_mag * cosf(l_phase);
+    dstl[2 * n + 1] = l_mag * sinf(l_phase);
+
+    dstr[2 * n    ] = r_mag * cosf(r_phase);
+    dstr[2 * n + 1] = r_mag * sinf(r_phase);
+
+    dstlfe[2 * n    ] = lfe_mag * cosf(c_phase);
+    dstlfe[2 * n + 1] = lfe_mag * sinf(c_phase);
+}
+
+static void upmix_3_0(AVFilterContext *ctx,
+                      float l_phase,
+                      float r_phase,
+                      float c_phase,
+                      float mag_total,
+                      float x, float y,
+                      int n)
+{
+    AudioSurroundContext *s = ctx->priv;
+    float l_mag, r_mag, c_mag, *dstc, *dstl, *dstr;
+
+    dstl = (float *)s->output->extended_data[0];
+    dstr = (float *)s->output->extended_data[1];
+    dstc = (float *)s->output->extended_data[2];
+
+    c_mag = sqrtf(1 - fabsf(x)) * ((y + 1) * .5) * mag_total;
+    l_mag = sqrtf(0.5 * ( x + 1)) * ((y + 1) * .5) * mag_total;
+    r_mag = sqrtf(0.5 * (-x + 1)) * ((y + 1) * .5) * mag_total;
+
+    dstl[2 * n    ] = l_mag * cosf(l_phase);
+    dstl[2 * n + 1] = l_mag * sinf(l_phase);
+
+    dstr[2 * n    ] = r_mag * cosf(r_phase);
+    dstr[2 * n + 1] = r_mag * sinf(r_phase);
+
+    dstc[2 * n    ] = c_mag * cosf(c_phase);
+    dstc[2 * n + 1] = c_mag * sinf(c_phase);
+}
+
+static void upmix_3_1(AVFilterContext *ctx,
+                      float l_phase,
+                      float r_phase,
+                      float c_phase,
+                      float mag_total,
+                      float x, float y,
+                      int n)
+{
+    AudioSurroundContext *s = ctx->priv;
+    float lfe_mag, l_mag, r_mag, c_mag, *dstc, *dstl, *dstr, *dstlfe;
+
+    dstl = (float *)s->output->extended_data[0];
+    dstr = (float *)s->output->extended_data[1];
+    dstc = (float *)s->output->extended_data[2];
+    dstlfe = (float *)s->output->extended_data[3];
+
+    if (s->output_lfe && n < s->highcut) {
+        lfe_mag = n < s->lowcut ? 1.f : .5f*(1.f+cosf(M_PI*(s->lowcut-n)/(s->lowcut-s->highcut)));
+        lfe_mag *= mag_total;
+        mag_total -= lfe_mag;
+    } else {
+        lfe_mag = 0.f;
+    }
+
+    c_mag = sqrtf(1 - fabsf(x))   * ((y + 1) * .5) * mag_total;
+    l_mag = sqrtf(0.5 * ( x + 1)) * ((y + 1) * .5) * mag_total;
+    r_mag = sqrtf(0.5 * (-x + 1)) * ((y + 1) * .5) * mag_total;
+
+    dstl[2 * n    ] = l_mag * cosf(l_phase);
+    dstl[2 * n + 1] = l_mag * sinf(l_phase);
+
+    dstr[2 * n    ] = r_mag * cosf(r_phase);
+    dstr[2 * n + 1] = r_mag * sinf(r_phase);
+
+    dstc[2 * n    ] = c_mag * cosf(c_phase);
+    dstc[2 * n + 1] = c_mag * sinf(c_phase);
+
+    dstlfe[2 * n    ] = lfe_mag * cosf(c_phase);
+    dstlfe[2 * n + 1] = lfe_mag * sinf(c_phase);
+}
+
+static void upmix_4_0(AVFilterContext *ctx,
+                      float l_phase,
+                      float r_phase,
+                      float c_phase,
+                      float mag_total,
+                      float x, float y,
+                      int n)
+{
+    float b_mag, l_mag, r_mag, c_mag, *dstc, *dstl, *dstr, *dstb;
+    AudioSurroundContext *s = ctx->priv;
+
+    dstl = (float *)s->output->extended_data[0];
+    dstr = (float *)s->output->extended_data[1];
+    dstc = (float *)s->output->extended_data[2];
+    dstb = (float *)s->output->extended_data[3];
+
+    c_mag = sqrtf(1 - fabsf(x))   * ((y + 1) * .5) * mag_total;
+    b_mag = sqrtf(1 - fabsf(x))   * ((1 - y) * .5) * mag_total;
+    l_mag = sqrtf(0.5 * ( x + 1)) * ((y + 1) * .5) * mag_total;
+    r_mag = sqrtf(0.5 * (-x + 1)) * ((y + 1) * .5) * mag_total;
+
+    dstl[2 * n    ] = l_mag * cosf(l_phase);
+    dstl[2 * n + 1] = l_mag * sinf(l_phase);
+
+    dstr[2 * n    ] = r_mag * cosf(r_phase);
+    dstr[2 * n + 1] = r_mag * sinf(r_phase);
+
+    dstc[2 * n    ] = c_mag * cosf(c_phase);
+    dstc[2 * n + 1] = c_mag * sinf(c_phase);
+
+    dstb[2 * n    ] = b_mag * cosf(c_phase);
+    dstb[2 * n + 1] = b_mag * sinf(c_phase);
+}
+
+static void upmix_4_1(AVFilterContext *ctx,
+                      float l_phase,
+                      float r_phase,
+                      float c_phase,
+                      float mag_total,
+                      float x, float y,
+                      int n)
+{
+    float lfe_mag, b_mag, l_mag, r_mag, c_mag, *dstc, *dstl, *dstr, *dstb, *dstlfe;
+    AudioSurroundContext *s = ctx->priv;
+
+    dstl = (float *)s->output->extended_data[0];
+    dstr = (float *)s->output->extended_data[1];
+    dstc = (float *)s->output->extended_data[2];
+    dstlfe = (float *)s->output->extended_data[3];
+    dstb = (float *)s->output->extended_data[4];
+
+    if (s->output_lfe && n < s->highcut) {
+        lfe_mag = n < s->lowcut ? 1.f : .5f*(1.f+cosf(M_PI*(s->lowcut-n)/(s->lowcut-s->highcut)));
+        lfe_mag *= mag_total;
+        mag_total -= lfe_mag;
+    } else {
+        lfe_mag = 0.f;
+    }
+
+    dstlfe[2 * n    ] = lfe_mag * cosf(c_phase);
+    dstlfe[2 * n + 1] = lfe_mag * sinf(c_phase);
+
+    c_mag = sqrtf(1 - fabsf(x))   * ((y + 1) * .5) * mag_total;
+    b_mag = sqrtf(1 - fabsf(x))   * ((1 - y) * .5) * mag_total;
+    l_mag = sqrtf(0.5 * ( x + 1)) * ((y + 1) * .5) * mag_total;
+    r_mag = sqrtf(0.5 * (-x + 1)) * ((y + 1) * .5) * mag_total;
+
+    dstl[2 * n    ] = l_mag * cosf(l_phase);
+    dstl[2 * n + 1] = l_mag * sinf(l_phase);
+
+    dstr[2 * n    ] = r_mag * cosf(r_phase);
+    dstr[2 * n + 1] = r_mag * sinf(r_phase);
+
+    dstc[2 * n    ] = c_mag * cosf(c_phase);
+    dstc[2 * n + 1] = c_mag * sinf(c_phase);
+
+    dstb[2 * n    ] = b_mag * cosf(c_phase);
+    dstb[2 * n + 1] = b_mag * sinf(c_phase);
+}
+
+static void upmix_5_0_back(AVFilterContext *ctx,
+                           float l_phase,
+                           float r_phase,
+                           float c_phase,
+                           float mag_total,
+                           float x, float y,
+                           int n)
+{
+    float l_mag, r_mag, ls_mag, rs_mag, c_mag, *dstc, *dstl, *dstr, *dstls, *dstrs;
+    AudioSurroundContext *s = ctx->priv;
+
+    dstl  = (float *)s->output->extended_data[0];
+    dstr  = (float *)s->output->extended_data[1];
+    dstc  = (float *)s->output->extended_data[2];
+    dstls = (float *)s->output->extended_data[3];
+    dstrs = (float *)s->output->extended_data[4];
+
+    c_mag  = sqrtf(1 - fabsf(x))   * ((y + 1) * .5) * mag_total;
+    l_mag  = sqrtf(0.5 * ( x + 1)) * ((y + 1) * .5) * mag_total;
+    r_mag  = sqrtf(0.5 * (-x + 1)) * ((y + 1) * .5) * mag_total;
+    ls_mag = sqrtf(0.5 * ( x + 1)) * (1 - ((y + 1) * .5)) * mag_total;
+    rs_mag = sqrtf(0.5 * (-x + 1)) * (1 - ((y + 1) * .5)) * mag_total;
+
+    dstl[2 * n    ] = l_mag * cosf(l_phase);
+    dstl[2 * n + 1] = l_mag * sinf(l_phase);
+
+    dstr[2 * n    ] = r_mag * cosf(r_phase);
+    dstr[2 * n + 1] = r_mag * sinf(r_phase);
+
+    dstc[2 * n    ] = c_mag * cosf(c_phase);
+    dstc[2 * n + 1] = c_mag * sinf(c_phase);
+
+    dstls[2 * n    ] = ls_mag * cosf(l_phase);
+    dstls[2 * n + 1] = ls_mag * sinf(l_phase);
+
+    dstrs[2 * n    ] = rs_mag * cosf(r_phase);
+    dstrs[2 * n + 1] = rs_mag * sinf(r_phase);
+}
+
+static void upmix_5_1_back(AVFilterContext *ctx,
+                           float l_phase,
+                           float r_phase,
+                           float c_phase,
+                           float mag_total,
+                           float x, float y,
+                           int n)
+{
+    float lfe_mag, l_mag, r_mag, ls_mag, rs_mag, c_mag, *dstc, *dstl, *dstr, *dstls, *dstrs, *dstlfe;
+    AudioSurroundContext *s = ctx->priv;
+
+    dstl  = (float *)s->output->extended_data[0];
+    dstr  = (float *)s->output->extended_data[1];
+    dstc  = (float *)s->output->extended_data[2];
+    dstlfe = (float *)s->output->extended_data[3];
+    dstls = (float *)s->output->extended_data[4];
+    dstrs = (float *)s->output->extended_data[5];
+
+    if (s->output_lfe && n < s->highcut) {
+        lfe_mag = n < s->lowcut ? 1.f : .5f*(1.f+cosf(M_PI*(s->lowcut-n)/(s->lowcut-s->highcut)));
+        lfe_mag *= mag_total;
+        mag_total -= lfe_mag;
+    } else {
+        lfe_mag = 0.f;
+    }
+
+    c_mag  = sqrtf(1 - fabsf(x))   * ((y + 1) * .5) * mag_total;
+    l_mag  = sqrtf(0.5 * ( x + 1)) * ((y + 1) * .5) * mag_total;
+    r_mag  = sqrtf(0.5 * (-x + 1)) * ((y + 1) * .5) * mag_total;
+    ls_mag = sqrtf(0.5 * ( x + 1)) * (1 - ((y + 1) * .5)) * mag_total;
+    rs_mag = sqrtf(0.5 * (-x + 1)) * (1 - ((y + 1) * .5)) * mag_total;
+
+    dstl[2 * n    ] = l_mag * cosf(l_phase);
+    dstl[2 * n + 1] = l_mag * sinf(l_phase);
+
+    dstr[2 * n    ] = r_mag * cosf(r_phase);
+    dstr[2 * n + 1] = r_mag * sinf(r_phase);
+
+    dstc[2 * n    ] = c_mag * cosf(c_phase);
+    dstc[2 * n + 1] = c_mag * sinf(c_phase);
+
+    dstlfe[2 * n    ] = lfe_mag * cosf(c_phase);
+    dstlfe[2 * n + 1] = lfe_mag * sinf(c_phase);
+
+    dstls[2 * n    ] = ls_mag * cosf(l_phase);
+    dstls[2 * n + 1] = ls_mag * sinf(l_phase);
+
+    dstrs[2 * n    ] = rs_mag * cosf(r_phase);
+    dstrs[2 * n + 1] = rs_mag * sinf(r_phase);
+}
+
+static void upmix_7_0(AVFilterContext *ctx,
+                      float l_phase,
+                      float r_phase,
+                      float c_phase,
+                      float mag_total,
+                      float x, float y,
+                      int n)
+{
+    float l_mag, r_mag, ls_mag, rs_mag, c_mag, lb_mag, rb_mag;
+    float *dstc, *dstl, *dstr, *dstls, *dstrs, *dstlb, *dstrb;
+    AudioSurroundContext *s = ctx->priv;
+
+    dstl  = (float *)s->output->extended_data[0];
+    dstr  = (float *)s->output->extended_data[1];
+    dstc  = (float *)s->output->extended_data[2];
+    dstlb = (float *)s->output->extended_data[3];
+    dstrb = (float *)s->output->extended_data[4];
+    dstls = (float *)s->output->extended_data[5];
+    dstrs = (float *)s->output->extended_data[6];
+
+    c_mag  = sqrtf(1 - fabsf(x))   * ((y + 1) * .5) * mag_total;
+    l_mag  = sqrtf(0.5 * ( x + 1)) * ((y + 1) * .5) * mag_total;
+    r_mag  = sqrtf(0.5 * (-x + 1)) * ((y + 1) * .5) * mag_total;
+    lb_mag = sqrtf(0.5 * ( x + 1)) * (1 - ((y + 1) * .5)) * mag_total;
+    rb_mag = sqrtf(0.5 * (-x + 1)) * (1 - ((y + 1) * .5)) * mag_total;
+    ls_mag = sqrtf(0.5 * ( x + 1)) * (1 - fabsf(y)) * mag_total;
+    rs_mag = sqrtf(0.5 * (-x + 1)) * (1 - fabsf(y)) * mag_total;
+
+    dstl[2 * n    ] = l_mag * cosf(l_phase);
+    dstl[2 * n + 1] = l_mag * sinf(l_phase);
+
+    dstr[2 * n    ] = r_mag * cosf(r_phase);
+    dstr[2 * n + 1] = r_mag * sinf(r_phase);
+
+    dstc[2 * n    ] = c_mag * cosf(c_phase);
+    dstc[2 * n + 1] = c_mag * sinf(c_phase);
+
+    dstlb[2 * n    ] = lb_mag * cosf(l_phase);
+    dstlb[2 * n + 1] = lb_mag * sinf(l_phase);
+
+    dstrb[2 * n    ] = rb_mag * cosf(r_phase);
+    dstrb[2 * n + 1] = rb_mag * sinf(r_phase);
+
+    dstls[2 * n    ] = ls_mag * cosf(l_phase);
+    dstls[2 * n + 1] = ls_mag * sinf(l_phase);
+
+    dstrs[2 * n    ] = rs_mag * cosf(r_phase);
+    dstrs[2 * n + 1] = rs_mag * sinf(r_phase);
+}
+
+static void upmix_7_1(AVFilterContext *ctx,
+                      float l_phase,
+                      float r_phase,
+                      float c_phase,
+                      float mag_total,
+                      float x, float y,
+                      int n)
+{
+    float lfe_mag, l_mag, r_mag, ls_mag, rs_mag, c_mag, lb_mag, rb_mag;
+    float *dstc, *dstl, *dstr, *dstls, *dstrs, *dstlb, *dstrb, *dstlfe;
+    AudioSurroundContext *s = ctx->priv;
+
+    dstl  = (float *)s->output->extended_data[0];
+    dstr  = (float *)s->output->extended_data[1];
+    dstc  = (float *)s->output->extended_data[2];
+    dstlfe = (float *)s->output->extended_data[3];
+    dstlb = (float *)s->output->extended_data[4];
+    dstrb = (float *)s->output->extended_data[5];
+    dstls = (float *)s->output->extended_data[6];
+    dstrs = (float *)s->output->extended_data[7];
+
+    if (s->output_lfe && n < s->highcut) {
+        lfe_mag = n < s->lowcut ? 1.f : .5f*(1.f+cosf(M_PI*(s->lowcut-n)/(s->lowcut-s->highcut)));
+        lfe_mag *= mag_total;
+        mag_total -= lfe_mag;
+    } else {
+        lfe_mag = 0.f;
+    }
+
+    c_mag  = sqrtf(1 - fabsf(x))   * ((y + 1) * .5) * mag_total;
+    l_mag  = sqrtf(0.5 * ( x + 1)) * ((y + 1) * .5) * mag_total;
+    r_mag  = sqrtf(0.5 * (-x + 1)) * ((y + 1) * .5) * mag_total;
+    lb_mag = sqrtf(0.5 * ( x + 1)) * (1 - ((y + 1) * .5)) * mag_total;
+    rb_mag = sqrtf(0.5 * (-x + 1)) * (1 - ((y + 1) * .5)) * mag_total;
+    ls_mag = sqrtf(0.5 * ( x + 1)) * (1 - fabsf(y)) * mag_total;
+    rs_mag = sqrtf(0.5 * (-x + 1)) * (1 - fabsf(y)) * mag_total;
+
+    dstl[2 * n    ] = l_mag * cosf(l_phase);
+    dstl[2 * n + 1] = l_mag * sinf(l_phase);
+
+    dstr[2 * n    ] = r_mag * cosf(r_phase);
+    dstr[2 * n + 1] = r_mag * sinf(r_phase);
+
+    dstc[2 * n    ] = c_mag * cosf(c_phase);
+    dstc[2 * n + 1] = c_mag * sinf(c_phase);
+
+    dstlfe[2 * n    ] = lfe_mag * cosf(c_phase);
+    dstlfe[2 * n + 1] = lfe_mag * sinf(c_phase);
+
+    dstlb[2 * n    ] = lb_mag * cosf(l_phase);
+    dstlb[2 * n + 1] = lb_mag * sinf(l_phase);
+
+    dstrb[2 * n    ] = rb_mag * cosf(r_phase);
+    dstrb[2 * n + 1] = rb_mag * sinf(r_phase);
+
+    dstls[2 * n    ] = ls_mag * cosf(l_phase);
+    dstls[2 * n + 1] = ls_mag * sinf(l_phase);
+
+    dstrs[2 * n    ] = rs_mag * cosf(r_phase);
+    dstrs[2 * n + 1] = rs_mag * sinf(r_phase);
+}
+
+static int init(AVFilterContext *ctx)
+{
+    AudioSurroundContext *s = ctx->priv;
+    float overlap;
+    int i;
+
+    if (!(s->out_channel_layout = av_get_channel_layout(s->out_channel_layout_str))) {
+        av_log(ctx, AV_LOG_ERROR, "Error parsing channel layout '%s'.\n",
+               s->out_channel_layout_str);
+        return AVERROR(EINVAL);
+    }
+
+    if (s->lowcutf >= s->highcutf) {
+        av_log(ctx, AV_LOG_ERROR, "Low cut-off '%d' should be less than high cut-off '%d'.\n",
+               s->lowcutf, s->highcutf);
+        return AVERROR(EINVAL);
+    }
+
+    switch (s->out_channel_layout) {
+    case AV_CH_LAYOUT_MONO:
+        s->upmix = upmix_1_0;
+        break;
+    case AV_CH_LAYOUT_STEREO:
+        s->upmix = upmix_stereo;
+        break;
+    case AV_CH_LAYOUT_2POINT1:
+        s->upmix = upmix_2_1;
+        break;
+    case AV_CH_LAYOUT_SURROUND:
+        s->upmix = upmix_3_0;
+        break;
+    case AV_CH_LAYOUT_3POINT1:
+        s->upmix = upmix_3_1;
+        break;
+    case AV_CH_LAYOUT_4POINT0:
+        s->upmix = upmix_4_0;
+        break;
+    case AV_CH_LAYOUT_4POINT1:
+        s->upmix = upmix_4_1;
+        break;
+    case AV_CH_LAYOUT_5POINT0_BACK:
+        s->upmix = upmix_5_0_back;
+        break;
+    case AV_CH_LAYOUT_5POINT1_BACK:
+        s->upmix = upmix_5_1_back;
+        break;
+    case AV_CH_LAYOUT_7POINT0:
+        s->upmix = upmix_7_0;
+        break;
+    case AV_CH_LAYOUT_7POINT1:
+        s->upmix = upmix_7_1;
+        break;
+    default:
+        av_log(ctx, AV_LOG_ERROR, "Unsupported output channel layout '%s'.\n",
+               s->out_channel_layout_str);
+        return AVERROR(EINVAL);
+    }
+
+    s->buf_size = 4096;
+    s->pts = AV_NOPTS_VALUE;
+
+    s->window_func_lut = av_calloc(s->buf_size, sizeof(*s->window_func_lut));
+    if (!s->window_func_lut)
+        return AVERROR(ENOMEM);
+
+    for (i = 0; i < s->buf_size; i++)
+        s->window_func_lut[i] = sqrtf(0.5 * (1 - cosf(2 * M_PI * i / s->buf_size)) / s->buf_size);
+    overlap = .5;
+    s->hop_size = s->buf_size * (1. - overlap);
+
+    return 0;
+}
+
+static int fft_channel(AVFilterContext *ctx, void *arg, int ch, int nb_jobs)
+{
+    AudioSurroundContext *s = ctx->priv;
+    const float level_in = s->level_in;
+    float *dst;
+    int n;
+
+    memset(s->input->extended_data[ch] + s->buf_size * sizeof(float), 0, s->buf_size * sizeof(float));
+
+    dst = (float *)s->input->extended_data[ch];
+    for (n = 0; n < s->buf_size; n++) {
+        dst[n] *= s->window_func_lut[n] * level_in;
+    }
+
+    av_rdft_calc(s->rdft[ch], (float *)s->input->extended_data[ch]);
+
+    return 0;
+}
+
+static int ifft_channel(AVFilterContext *ctx, void *arg, int ch, int nb_jobs)
+{
+    AudioSurroundContext *s = ctx->priv;
+    const float level_out = s->level_out;
+    AVFrame *out = arg;
+    float *dst, *ptr;
+    int n;
+
+    av_rdft_calc(s->irdft[ch], (float *)s->output->extended_data[ch]);
+
+    dst = (float *)s->output->extended_data[ch];
+    ptr = (float *)s->overlap_buffer->extended_data[ch];
+
+    memmove(s->overlap_buffer->extended_data[ch],
+            s->overlap_buffer->extended_data[ch] + s->hop_size * sizeof(float),
+            s->buf_size * sizeof(float));
+    memset(s->overlap_buffer->extended_data[ch] + s->buf_size * sizeof(float),
+           0, s->hop_size * sizeof(float));
+
+    for (n = 0; n < s->buf_size; n++) {
+        ptr[n] += dst[n] * s->window_func_lut[n] * level_out;
+    }
+
+    ptr = (float *)s->overlap_buffer->extended_data[ch];
+    dst = (float *)out->extended_data[ch];
+    memcpy(dst, ptr, s->hop_size * sizeof(float));
+
+    return 0;
+}
+
+static int filter_frame(AVFilterLink *inlink, AVFrame *in)
+{
+    AVFilterContext *ctx = inlink->dst;
+    AVFilterLink *outlink = ctx->outputs[0];
+    AudioSurroundContext *s = ctx->priv;
+
+    av_audio_fifo_write(s->fifo, (void **)in->extended_data,
+                        in->nb_samples);
+
+    if (s->pts == AV_NOPTS_VALUE)
+        s->pts = in->pts;
+
+    av_frame_free(&in);
+
+    while (av_audio_fifo_size(s->fifo) >= s->buf_size) {
+        float *srcl, *srcr;
+        AVFrame *out;
+        int n, ret;
+
+        ret = av_audio_fifo_peek(s->fifo, (void **)s->input->extended_data, s->buf_size);
+        if (ret < 0)
+            return ret;
+
+        ctx->internal->execute(ctx, fft_channel, NULL, NULL, inlink->channels);
+
+        srcl = (float *)s->input->extended_data[0];
+        srcr = (float *)s->input->extended_data[1];
+
+        for (n = 0; n < s->buf_size; n++) {
+            float l_re = srcl[2 * n], r_re = srcr[2 * n];
+            float l_im = srcl[2 * n + 1], r_im = srcr[2 * n + 1];
+            float c_phase = atan2f(l_im + r_im, l_re + r_re);
+            float l_mag = hypotf(l_re, l_im);
+            float r_mag = hypotf(r_re, r_im);
+            float l_phase = atan2f(l_im, l_re);
+            float r_phase = atan2f(r_im, r_re);
+            float phase_dif = fabsf(l_phase - r_phase);
+            float mag_dif = (l_mag - r_mag) / (l_mag + r_mag);
+            float mag_total = hypotf(l_mag, r_mag);
+            float x, y;
+
+            if (phase_dif > M_PI)
+                phase_dif = 2 * M_PI - phase_dif;
+
+            stereo_position(mag_dif, phase_dif, &x, &y);
+
+            s->upmix(ctx, l_phase, r_phase, c_phase, mag_total, x, y, n);
+        }
+
+        out = ff_get_audio_buffer(outlink, s->hop_size);
+        if (!out)
+            return AVERROR(ENOMEM);
+
+        ctx->internal->execute(ctx, ifft_channel, out, NULL, outlink->channels);
+
+        out->pts = s->pts;
+        if (s->pts != AV_NOPTS_VALUE)
+            s->pts += av_rescale_q(out->nb_samples, (AVRational){1, outlink->sample_rate}, outlink->time_base);
+        av_audio_fifo_drain(s->fifo, s->hop_size);
+        ret = ff_filter_frame(outlink, out);
+        if (ret < 0)
+            return ret;
+    }
+
+    return 0;
+}
+
+static av_cold void uninit(AVFilterContext *ctx)
+{
+    AudioSurroundContext *s = ctx->priv;
+    int ch;
+
+    av_frame_free(&s->input);
+    av_frame_free(&s->output);
+    av_frame_free(&s->overlap_buffer);
+
+    for (ch = 0; ch < s->nb_in_channels; ch++) {
+        av_rdft_end(s->rdft[ch]);
+    }
+    for (ch = 0; ch < s->nb_out_channels; ch++) {
+        av_rdft_end(s->irdft[ch]);
+    }
+    av_freep(&s->rdft);
+    av_freep(&s->irdft);
+    av_audio_fifo_free(s->fifo);
+    av_freep(&s->window_func_lut);
+}
+
+#define OFFSET(x) offsetof(AudioSurroundContext, x)
+#define FLAGS AV_OPT_FLAG_AUDIO_PARAM|AV_OPT_FLAG_FILTERING_PARAM
+
+static const AVOption surround_options[] = {
+    { "chl_out",   "set output channel layout", OFFSET(out_channel_layout_str), AV_OPT_TYPE_STRING, {.str="5.1"}, 0,   0, FLAGS },
+    { "level_in",  "set input level",           OFFSET(level_in),               AV_OPT_TYPE_FLOAT,  {.dbl=1},     0,  10, FLAGS },
+    { "level_out", "set output level",          OFFSET(level_out),              AV_OPT_TYPE_FLOAT,  {.dbl=1},     0,  10, FLAGS },
+    { "lfe",       "output LFE",                OFFSET(output_lfe),             AV_OPT_TYPE_BOOL,   {.i64=1},     0,   1, FLAGS },
+    { "lfe_low",   "LFE low cut off",           OFFSET(lowcutf),                AV_OPT_TYPE_INT,    {.i64=128},   0, 256, FLAGS },
+    { "lfe_high",  "LFE high cut off",          OFFSET(highcutf),               AV_OPT_TYPE_INT,    {.i64=256},   0, 512, FLAGS },
+    { NULL }
+};
+
+AVFILTER_DEFINE_CLASS(surround);
+
+static const AVFilterPad inputs[] = {
+    {
+        .name         = "default",
+        .type         = AVMEDIA_TYPE_AUDIO,
+        .filter_frame = filter_frame,
+        .config_props = config_input,
+    },
+    { NULL }
+};
+
+static const AVFilterPad outputs[] = {
+    {
+        .name         = "default",
+        .type         = AVMEDIA_TYPE_AUDIO,
+        .config_props = config_output,
+    },
+    { NULL }
+};
+
+AVFilter ff_af_surround = {
+    .name           = "surround",
+    .description    = NULL_IF_CONFIG_SMALL("Apply audio surround upmix filter."),
+    .query_formats  = query_formats,
+    .priv_size      = sizeof(AudioSurroundContext),
+    .priv_class     = &surround_class,
+    .init           = init,
+    .uninit         = uninit,
+    .inputs         = inputs,
+    .outputs        = outputs,
+    .flags          = AVFILTER_FLAG_SLICE_THREADS,
+};
diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
index f8cd193..534c340 100644
--- a/libavfilter/allfilters.c
+++ b/libavfilter/allfilters.c
@@ -121,6 +121,7 @@  static void register_all(void)
     REGISTER_FILTER(SOFALIZER,      sofalizer,      af);
     REGISTER_FILTER(STEREOTOOLS,    stereotools,    af);
     REGISTER_FILTER(STEREOWIDEN,    stereowiden,    af);
+    REGISTER_FILTER(SURROUND,       surround,       af);
     REGISTER_FILTER(TREBLE,         treble,         af);
     REGISTER_FILTER(TREMOLO,        tremolo,        af);
     REGISTER_FILTER(VIBRATO,        vibrato,        af);