diff mbox

[FFmpeg-devel] Support for Ambisonics and OpusProjection* API.

Message ID CABQ9DctqO5rqYgcT63+Xi8Azz6MH4BjZWfVKs1XcNZz7wLdwxA@mail.gmail.com
State New
Headers show

Commit Message

Drew Allen April 20, 2018, 5:23 p.m. UTC
Hello all,

Attached is a revised patch based on your suggestions.

Rostislav Pehlivanov - I'm not sure how to address your concerns. My patch
provides support for the ambisonics features already approved and present
in Opus 1.3. If there are ways I can change my patch, please let me know.

Cheers,
Drew

On Mon, Apr 9, 2018 at 10:57 AM Drew Allen <bitllama@google.com> wrote:

> Friendly weekly ping about this patch.
>
> Can someone please let me know if I need to do anything more to submit
> this patch? Thanks!
>
> On Mon, Apr 2, 2018 at 9:13 AM Drew Allen <bitllama@google.com> wrote:
>
>> Friendly ping to allow this to push to the member list, please.
>>
>> On Wed, Mar 28, 2018 at 2:58 PM Drew Allen <bitllama@google.com> wrote:
>>
>>> Hello all,
>>>
>>> My name is Andrew Allen and I'm a contributor to Opus, supporting new
>>> channel mappings 2 and 3 for ambisonics compression. I've attached a patch
>>> to support the new OpusProjectionEncoder and OpusProjectionDecoder APIs for
>>> handling the new channel mapping 3 in OPUS.
>>>
>>> Please let me know of any changes I should make or if there are any
>>> questions I can help answer.
>>>
>>> Cheers,
>>> Drew
>>>
>>

Comments

Rostislav Pehlivanov April 20, 2018, 6:41 p.m. UTC | #1
On 20 April 2018 at 18:23, Drew Allen <bitllama-at-google.com@ffmpeg.org>
wrote:

> Hello all,
>
> Attached is a revised patch based on your suggestions.
>
> Rostislav Pehlivanov - I'm not sure how to address your concerns. My patch
> provides support for the ambisonics features already approved and present
> in Opus 1.3. If there are ways I can change my patch, please let me know.
>
> Cheers,
> Drew
>
> On Mon, Apr 9, 2018 at 10:57 AM Drew Allen <bitllama@google.com> wrote:
>
> > Friendly weekly ping about this patch.
> >
> > Can someone please let me know if I need to do anything more to submit
> > this patch? Thanks!
> >
> > On Mon, Apr 2, 2018 at 9:13 AM Drew Allen <bitllama@google.com> wrote:
> >
> >> Friendly ping to allow this to push to the member list, please.
> >>
> >> On Wed, Mar 28, 2018 at 2:58 PM Drew Allen <bitllama@google.com> wrote:
> >>
> >>> Hello all,
> >>>
> >>> My name is Andrew Allen and I'm a contributor to Opus, supporting new
> >>> channel mappings 2 and 3 for ambisonics compression. I've attached a
> patch
> >>> to support the new OpusProjectionEncoder and OpusProjectionDecoder
> APIs for
> >>> handling the new channel mapping 3 in OPUS.
> >>>
> >>> Please let me know of any changes I should make or if there are any
> >>> questions I can help answer.
> >>>
> >>> Cheers,
> >>> Drew
> >>>
> >>
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
>
I am sorry, I don't agree that this should be merged quite yet.

First of all, the draft upon which how the channel family is interpreted
(draft-ietf-codec-ambisonics-04) isn't finalized yet. What if it changes
after we make a release? We can't really change it. This affects both
encoding and decoding.

Second, the API isn't in any libopus release yet. What if we make a
release, the draft changes, so the API in libopus needs to be changed. Or
the API in the current git master of libopus changes? We can't rely on an
unstable API in a library.

Third, this patch makes the decoder always do demixing with the data in
extradata. What if someone wants to just decode and do their own demixing
with positional information? We'd break decoding for anyone who's depending
on the behaviour if we were to change this so the decoder outputs raw
ambisonics. We never do any mixing or conversions in decoders. Hence
ambisonics data needs to be exposed directly, and anyone wanting to demix
it should use a filter (there's an unmerged filter to do that) or do it
themselves.
What we need to do to properly handle ambisonics is:
1. Be able to describe ambisonics. This needs a new API.
2. Be able to expose the demixing matrix via frame side data.
3. Have a filter which can use both to provide a demixed output, better yet
with positional information.

I think the draft should become an RFC first. That gives us enough time to
work on 1.), which should take the longest time to do and agree on. 2.) is
trivial and 3.) is from what I know mostly done.
Drew Allen April 23, 2018, 4:02 p.m. UTC | #2
Hi Rostislav,

Here is my feedback:

I am sorry, I don't agree that this should be merged quite yet.

First of all, the draft upon which how the channel family is interpreted
(draft-ietf-codec-ambisonics-04) isn't finalized yet. What if it changes
after we make a release? We can't really change it. This affects both
encoding and decoding.

We have spent the past 2 years with the draft relatively unchanged aside
from minor edits on the draft. It is headed to a working group for
finalization very soon and no one has yet raised a single issue regarding
any proposed changes that this patch introduces. I wrote the
OpusProjection* API and it has been adopted in all Opus-related xiph master
branches.

Second, the API isn't in any libopus release yet. What if we make a
release, the draft changes, so the API in libopus needs to be changed. Or
the API in the current git master of libopus changes? We can't rely on an
unstable API in a library.

I worked closely with Jean-Marc Valin to design the API in Opus 1.3 to his
specification. Opus 1.3 beta already contains this new API and upon
release, I have 100% assurance from Jean-Marc that the OpusProjection* API
will be supported in 1.3 RC.

Third, this patch makes the decoder always do demixing with the data in
extradata. What if someone wants to just decode and do their own demixing
with positional information? We'd break decoding for anyone who's depending
on the behaviour if we were to change this so the decoder outputs raw
ambisonics. We never do any mixing or conversions in decoders. Hence
ambisonics data needs to be exposed directly, and anyone wanting to demix
it should use a filter (there's an unmerged filter to do that) or do it
themselves.
What we need to do to properly handle ambisonics is:
1. Be able to describe ambisonics. This needs a new API.
2. Be able to expose the demixing matrix via frame side data.
3. Have a filter which can use both to provide a demixed output, better yet
with positional information.

I disagree that a filter or some other layer of abstraction is necessary
here. OpusProjection* does not code the ambisonic channels directly.
Instead, they are mixed using a mixing matrix that minimizes coding
artifacts over the sphere. The demixing matrix on the decoder is vital in
order to get back the original ambisonic channels and OpusProjectionDecoder
handles this automatically.

I think the draft should become an RFC first. That gives us enough time to
work on 1.), which should take the longest time to do and agree on. 2.) is
trivial and 3.) is from what I know mostly done.

I completely disagree. The IETF draft has been stable for over a year and
these same changes to support the new API are already present in Opus,
libopusenc, opusfile and opus-tools.
Rostislav Pehlivanov April 23, 2018, 7:28 p.m. UTC | #3
On 23 April 2018 at 17:02, Drew Allen <bitllama-at-google.com@ffmpeg.org>
wrote:

> We have spent the past 2 years with the draft relatively unchanged aside
> from minor edits on the draft. It is headed to a working group for
> finalization very soon and no one has yet raised a single issue regarding
> any proposed changes that this patch introduces. I wrote the
> OpusProjection* API and it has been adopted in all Opus-related xiph master
> branches.
>

Good to hear. I know the IETF can take an extraordinarily long amount of
time to publish a draft and make it an RFC and wish you all the luck on
getting it completed quickly.


I worked closely with Jean-Marc Valin to design the API in Opus 1.3 to his
> specification. Opus 1.3 beta already contains this new API and upon
> release, I have 100% assurance from Jean-Marc that the OpusProjection* API
> will be supported in 1.3 RC.
>

...hidden behind a configure flag and not enabled by default. No one will
enable it until its ready and becomes accepted, which is why it isn't
enabled by default.



> I disagree that a filter or some other layer of abstraction is necessary
> here. OpusProjection* does not code the ambisonic channels directly.
> Instead, they are mixed using a mixing matrix that minimizes coding
> artifacts over the sphere. The demixing matrix on the decoder is vital in
> order to get back the original ambisonic channels and OpusProjectionDecoder
> handles this automatically.
>

Ah, okay. In which case what we need still is an API to signal the
ambisonics order and any other metadata needed. We can't just say "here's a
frame with a bunch of channels, based on their numer you could probably
figure its some ambisonics" and clients shouldn't use heuristics like "this
frame has 16 channels, its probably ambisonics". This information needs to
be indicated properly.


I completely disagree. The IETF draft has been stable for over a year and
> these same changes to support the new API are already present in Opus,
> libopusenc, opusfile and opus-tools.


Stable != accepted. The option to enable the API is still hidden behind a
configure flag for libopus. And it will remain like this until it becomes
an RFC, just like the previous RFC which butc... updated the codec with
some fixes.

If libopus had the new API enabled by default I wouldn't mind this patch at
all and would apply it, since I'd be sure it wouldn't change. But as it
stands, both it and the signalling changes in the RFC can change at any
time despite being stable by yourself.

Also your mail client stripped the quote marks and made things confusing.
Drew Allen May 11, 2018, 2:55 p.m. UTC | #4
Hi Rostislav et all,

The IETF document has just been moved to a working group last call.

Do you think it would it be possible to land this patch under an
experimental flag?

Cheers,
Drew

On Mon, Apr 23, 2018 at 12:28 PM Rostislav Pehlivanov <atomnuker@gmail.com>
wrote:

> On 23 April 2018 at 17:02, Drew Allen <bitllama-at-google.com@ffmpeg.org>
> wrote:
>
> > We have spent the past 2 years with the draft relatively unchanged aside
> > from minor edits on the draft. It is headed to a working group for
> > finalization very soon and no one has yet raised a single issue regarding
> > any proposed changes that this patch introduces. I wrote the
> > OpusProjection* API and it has been adopted in all Opus-related xiph
> master
> > branches.
> >
>
> Good to hear. I know the IETF can take an extraordinarily long amount of
> time to publish a draft and make it an RFC and wish you all the luck on
> getting it completed quickly.
>
>
> I worked closely with Jean-Marc Valin to design the API in Opus 1.3 to his
> > specification. Opus 1.3 beta already contains this new API and upon
> > release, I have 100% assurance from Jean-Marc that the OpusProjection*
> API
> > will be supported in 1.3 RC.
> >
>
> ...hidden behind a configure flag and not enabled by default. No one will
> enable it until its ready and becomes accepted, which is why it isn't
> enabled by default.
>
>
>
> > I disagree that a filter or some other layer of abstraction is necessary
> > here. OpusProjection* does not code the ambisonic channels directly.
> > Instead, they are mixed using a mixing matrix that minimizes coding
> > artifacts over the sphere. The demixing matrix on the decoder is vital in
> > order to get back the original ambisonic channels and
> OpusProjectionDecoder
> > handles this automatically.
> >
>
> Ah, okay. In which case what we need still is an API to signal the
> ambisonics order and any other metadata needed. We can't just say "here's a
> frame with a bunch of channels, based on their numer you could probably
> figure its some ambisonics" and clients shouldn't use heuristics like "this
> frame has 16 channels, its probably ambisonics". This information needs to
> be indicated properly.
>
>
> I completely disagree. The IETF draft has been stable for over a year and
> > these same changes to support the new API are already present in Opus,
> > libopusenc, opusfile and opus-tools.
>
>
> Stable != accepted. The option to enable the API is still hidden behind a
> configure flag for libopus. And it will remain like this until it becomes
> an RFC, just like the previous RFC which butc... updated the codec with
> some fixes.
>
> If libopus had the new API enabled by default I wouldn't mind this patch at
> all and would apply it, since I'd be sure it wouldn't change. But as it
> stands, both it and the signalling changes in the RFC can change at any
> time despite being stable by yourself.
>
> Also your mail client stripped the quote marks and made things confusing.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
Rostislav Pehlivanov May 11, 2018, 4:39 p.m. UTC | #5
On 11 May 2018 at 15:55, Drew Allen <bitllama-at-google.com@ffmpeg.org>
wrote:

> Hi Rostislav et all,
>
> The IETF document has just been moved to a working group last call.
>
> Do you think it would it be possible to land this patch under an
> experimental flag?


No, libopus itself still hides the new API under an experimental flag. We
also don't do experimental flags for decoders and there's no API yet to
signal a channel layout is ambisonics.
Rostislav Pehlivanov July 26, 2018, 2:15 p.m. UTC | #6
Hey,

As of now, the ambisonics API is enabled by default in libopus. We still
don't have a way to signal ambisonics yet.
We still have plenty of bits left in libavutil/channel_layout.h to signal
many orders of ambisonics but some people have had opinions against
extending that API. We could instead extend AVMatrixEncoding but I don't
think that's entirely appropriate.
What opinions do people have on this?
Vittorio Giovara July 26, 2018, 2:57 p.m. UTC | #7
On Thu, Jul 26, 2018 at 4:15 PM, Rostislav Pehlivanov <atomnuker@gmail.com>
wrote:

> Hey,
>
> As of now, the ambisonics API is enabled by default in libopus. We still
> don't have a way to signal ambisonics yet.
> We still have plenty of bits left in libavutil/channel_layout.h to signal
> many orders of ambisonics but some people have had opinions against
> extending that API. We could instead extend AVMatrixEncoding but I don't
> think that's entirely appropriate.
> What opinions do people have on this?
>

I had been working on a new API that would encompass ambisonic ordering
(see
https://github.com/kodabb/libav/commit/98d9b0a7b28525b29e40ae4c564e51e7c94449eb).
The downside is that it requires updating the whole channel layout API (see
https://github.com/kodabb/libav/commit/c023b553e6ad6da5af6d0d4ff067ff844b2fcfac
)
I got it mostly working but ran into issues during backward compatibility,
and didn't have time to debug and fix it.

If anyone wants to finish the set, backport it, and add the missing lswr
part it would be easy work. I'm available to help in the process just to
get this completed.
The full branch is available at https://github.com/kodabb/libav/commits/chl
(I hope this will be a mature discussion even though the patches belong to
another tree).
diff mbox

Patch

From 35f9b25fbe5dd086b6fab0276d00491b4783cd64 Mon Sep 17 00:00:00 2001
From: Andrew Allen <bitllama@google.com>
Date: Wed, 28 Mar 2018 14:48:46 -0700
Subject: [PATCH] Support for Ambisonics and OpusProjection* API.

---
 libavcodec/libopusdec.c | 159 ++++++++++++++++++++-----
 libavcodec/libopusenc.c | 253 ++++++++++++++++++++++++++++++++++------
 libavcodec/opus.c       |  18 ++-
 3 files changed, 350 insertions(+), 80 deletions(-)

diff --git a/libavcodec/libopusdec.c b/libavcodec/libopusdec.c
index 2a97811d18..5caf8f3f53 100644
--- a/libavcodec/libopusdec.c
+++ b/libavcodec/libopusdec.c
@@ -21,6 +21,9 @@ 
 
 #include <opus.h>
 #include <opus_multistream.h>
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+#include <opus_projection.h>
+#endif
 
 #include "libavutil/internal.h"
 #include "libavutil/intreadwrite.h"
@@ -33,9 +36,89 @@ 
 #include "mathops.h"
 #include "libopus.h"
 
+typedef struct OpusGenericDecoder {
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+    OpusProjectionDecoder *pr;
+#endif
+    OpusMSDecoder *ms;
+} OpusGenericDecoder;
+
+static int libopus_generic_decoder_init(OpusGenericDecoder *enc, int Fs,
+                                        int channels, int nb_streams,
+                                        int nb_coupled, uint8_t *mapping,
+                                        uint8_t *dmatrix) {
+    int err;
+    if (dmatrix != NULL) {
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+        opus_int32 size;
+        size = 2 * channels * (nb_streams + nb_coupled);
+        enc->pr = opus_projection_decoder_create(Fs, channels, nb_streams,
+            nb_coupled, dmatrix, size, &err);
+#else
+        err = OPUS_UNIMPLEMENTED;
+#endif
+        return err;
+    }
+    enc->ms = opus_multistream_decoder_create(Fs, channels, nb_streams,
+        nb_coupled, mapping, &err);
+    return err;
+}
+
+static void libopus_generic_decoder_cleanup(OpusGenericDecoder *enc)
+{
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+    if (enc->pr) opus_projection_decoder_destroy(enc->pr);
+#endif
+    if (enc->ms) opus_multistream_decoder_destroy(enc->ms);
+}
+
+static int libopus_generic_decode(OpusGenericDecoder *enc,
+        const unsigned char *data, opus_int32 len, opus_int16 *pcm,
+        int frame_size, int decode_fec) {
+    int ret;
+
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+    if (enc->pr){
+        ret=opus_projection_decode(enc->pr, data, len, pcm, frame_size,
+            decode_fec);
+        return ret;
+    }
+#endif
+    ret=opus_multistream_decode(enc->ms, data, len, pcm, frame_size,
+        decode_fec);
+    return ret;
+}
+
+static int libopus_generic_decode_float(OpusGenericDecoder *enc,
+        const unsigned char *data, opus_int32 len, float *pcm, int frame_size,
+        int decode_fec) {
+    int ret;
+
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+    if (enc->pr){
+        ret=opus_projection_decode_float(enc->pr, data, len, pcm, frame_size,
+            decode_fec);
+        return ret;
+    }
+#endif
+    ret=opus_multistream_decode_float(enc->ms, data, len, pcm, frame_size,
+        decode_fec);
+    return ret;
+}
+
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+# define libopus_generic_decoder_ctl(enc, request) \
+    ((enc)->pr != NULL ? \
+    opus_projection_decoder_ctl((enc)->pr, request) : \
+    opus_multistream_decoder_ctl((enc)->ms, request))
+#else
+# define libopus_generic_decoder_ctl(enc, request) \
+    opus_multistream_decoder_ctl((enc)->ms, request)
+#endif
+
 struct libopus_context {
     AVClass *class;
-    OpusMSDecoder *dec;
+    OpusGenericDecoder dec;
     int pre_skip;
 #ifndef OPUS_SET_GAIN
     union { int i; double d; } gain;
@@ -46,12 +129,17 @@  struct libopus_context {
 };
 
 #define OPUS_HEAD_SIZE 19
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+# define OPUS_MAX_CHANNELS 18
+#else
+# define OPUS_MAX_CHANNELS 8
+#endif
 
 static av_cold int libopus_decode_init(AVCodecContext *avc)
 {
     struct libopus_context *opus = avc->priv_data;
     int ret, channel_map = 0, gain_db = 0, nb_streams, nb_coupled;
-    uint8_t mapping_arr[8] = { 0, 1 }, *mapping;
+    uint8_t mapping_arr[OPUS_MAX_CHANNELS] = { 0, 1 }, *mapping, *dmatrix = NULL;
 
     avc->channels = avc->extradata_size >= 10 ? avc->extradata[9] : (avc->channels == 1) ? 1 : 2;
     if (avc->channels <= 0) {
@@ -74,7 +162,21 @@  static av_cold int libopus_decode_init(AVCodecContext *avc)
         nb_coupled = avc->extradata[OPUS_HEAD_SIZE + 1];
         if (nb_streams + nb_coupled != avc->channels)
             av_log(avc, AV_LOG_WARNING, "Inconsistent channel mapping.\n");
-        mapping = avc->extradata + OPUS_HEAD_SIZE + 2;
+        if (channel_map == 3) {
+            int ch;
+            if (avc->extradata_size >= OPUS_HEAD_SIZE + 2 + 2 * avc->channels * (nb_streams + nb_coupled)) {
+                dmatrix =avc->extradata + OPUS_HEAD_SIZE + 2;
+            } else {
+                av_log(avc, AV_LOG_ERROR,
+                    "Demixing matrix not present.\n");
+                return AVERROR_INVALIDDATA;
+            }
+            for (ch = 0; ch < avc->channels; ch++)
+                mapping_arr[ch] = ch;
+            mapping = mapping_arr;
+        } else {
+            mapping = avc->extradata + OPUS_HEAD_SIZE + 2;
+        }
     } else {
         if (avc->channels > 2 || channel_map) {
             av_log(avc, AV_LOG_ERROR,
@@ -98,18 +200,15 @@  static av_cold int libopus_decode_init(AVCodecContext *avc)
                 mapping_arr[ch] = mapping[vorbis_offset[ch]];
             mapping = mapping_arr;
         }
-    } else if (channel_map == 2) {
-        int ambisonic_order = ff_sqrt(avc->channels) - 1;
-        if (avc->channels != (ambisonic_order + 1) * (ambisonic_order + 1) &&
-            avc->channels != (ambisonic_order + 1) * (ambisonic_order + 1) + 2) {
+    } else if (channel_map == 2 || channel_map == 3) {
+        int order_plus_one = ff_sqrt(avc->channels);
+        int nondiegetic_channels = avc->channels - order_plus_one * order_plus_one;
+        if (order_plus_one < 1 || order_plus_one > 15 ||
+           (nondiegetic_channels != 0 && nondiegetic_channels != 2)) {
             av_log(avc, AV_LOG_ERROR,
-                   "Channel mapping 2 is only specified for channel counts"
-                   " which can be written as (n + 1)^2 or (n + 2)^2 + 2"
-                   " for nonnegative integer n\n");
-            return AVERROR_INVALIDDATA;
-        }
-        if (avc->channels > 227) {
-            av_log(avc, AV_LOG_ERROR, "Too many channels\n");
+                "This channel mapping is only specified for channel counts"
+                " which can be written as (n + 1)^2 + 2j, where n is a"
+                " non-negative integar from 0 to 14 and j is either 0 or 1.\n");
             return AVERROR_INVALIDDATA;
         }
         avc->channel_layout = 0;
@@ -117,17 +216,16 @@  static av_cold int libopus_decode_init(AVCodecContext *avc)
         avc->channel_layout = 0;
     }
 
-    opus->dec = opus_multistream_decoder_create(avc->sample_rate, avc->channels,
-                                                nb_streams, nb_coupled,
-                                                mapping, &ret);
-    if (!opus->dec) {
+    ret = libopus_generic_decoder_init(&opus->dec, avc->sample_rate, avc->channels,
+                                       nb_streams, nb_coupled, mapping, dmatrix);
+    if (ret != OPUS_OK) {
         av_log(avc, AV_LOG_ERROR, "Unable to create decoder: %s\n",
                opus_strerror(ret));
         return ff_opus_error_to_averror(ret);
     }
 
 #ifdef OPUS_SET_GAIN
-    ret = opus_multistream_decoder_ctl(opus->dec, OPUS_SET_GAIN(gain_db));
+    ret = libopus_generic_decoder_ctl(&opus->dec, OPUS_SET_GAIN(gain_db));
     if (ret != OPUS_OK)
         av_log(avc, AV_LOG_WARNING, "Failed to set gain: %s\n",
                opus_strerror(ret));
@@ -142,8 +240,8 @@  static av_cold int libopus_decode_init(AVCodecContext *avc)
 #endif
 
 #ifdef OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST
-    ret = opus_multistream_decoder_ctl(opus->dec,
-                                       OPUS_SET_PHASE_INVERSION_DISABLED(!opus->apply_phase_inv));
+    ret = libopus_generic_decoder_ctl(&opus->dec,
+                                      OPUS_SET_PHASE_INVERSION_DISABLED(!opus->apply_phase_inv));
     if (ret != OPUS_OK)
         av_log(avc, AV_LOG_WARNING,
                "Unable to set phase inversion: %s\n",
@@ -160,10 +258,7 @@  static av_cold int libopus_decode_close(AVCodecContext *avc)
 {
     struct libopus_context *opus = avc->priv_data;
 
-    if (opus->dec) {
-        opus_multistream_decoder_destroy(opus->dec);
-        opus->dec = NULL;
-    }
+    libopus_generic_decoder_cleanup(&opus->dec);
     return 0;
 }
 
@@ -181,13 +276,13 @@  static int libopus_decode(AVCodecContext *avc, void *data,
         return ret;
 
     if (avc->sample_fmt == AV_SAMPLE_FMT_S16)
-        nb_samples = opus_multistream_decode(opus->dec, pkt->data, pkt->size,
-                                             (opus_int16 *)frame->data[0],
-                                             frame->nb_samples, 0);
+        nb_samples = libopus_generic_decode(&opus->dec, pkt->data, pkt->size,
+                                            (opus_int16 *)frame->data[0],
+                                            frame->nb_samples, 0);
     else
-        nb_samples = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size,
-                                                   (float *)frame->data[0],
-                                                   frame->nb_samples, 0);
+        nb_samples = libopus_generic_decode_float(&opus->dec, pkt->data, pkt->size,
+                                                  (float *)frame->data[0],
+                                                  frame->nb_samples, 0);
 
     if (nb_samples < 0) {
         av_log(avc, AV_LOG_ERROR, "Decoding error: %s\n",
@@ -220,7 +315,7 @@  static void libopus_flush(AVCodecContext *avc)
 {
     struct libopus_context *opus = avc->priv_data;
 
-    opus_multistream_decoder_ctl(opus->dec, OPUS_RESET_STATE);
+    libopus_generic_decoder_ctl(&opus->dec, OPUS_RESET_STATE);
     /* The stream can have been extracted by a tool that is not Opus-aware.
        Therefore, any packet can become the first of the stream. */
     avc->internal->skip_samples = opus->pre_skip;
diff --git a/libavcodec/libopusenc.c b/libavcodec/libopusenc.c
index 4ae81b0bb2..d3305a1459 100644
--- a/libavcodec/libopusenc.c
+++ b/libavcodec/libopusenc.c
@@ -21,15 +21,135 @@ 
 
 #include <opus.h>
 #include <opus_multistream.h>
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+#include <opus_projection.h>
+#endif
 
 #include "libavutil/opt.h"
 #include "avcodec.h"
 #include "bytestream.h"
 #include "internal.h"
 #include "libopus.h"
+#include "mathops.h"
 #include "vorbis.h"
 #include "audio_frame_queue.h"
 
+typedef struct OpusGenericEncoder {
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+    OpusProjectionEncoder *pr;
+#endif
+    OpusMSEncoder *ms;
+} OpusGenericEncoder;
+
+static int libopus_generic_encoder_surround_init(OpusGenericEncoder *enc,
+                                                 int Fs,
+                                                 int channels,
+                                                 int mapping_family,
+                                                 int *nb_streams,
+                                                 int *nb_coupled,
+                                                 unsigned char *stream_map,
+                                                 int application)
+{
+    int ret;
+    if (mapping_family == 3) {
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+        int ci;
+        enc->pr = opus_projection_ambisonics_encoder_create(
+            Fs, channels, mapping_family, nb_streams, nb_coupled,
+            application, &ret);
+        for (ci = 0; ci < channels; ci++)
+            stream_map[ci] = ci;
+#else
+        ret = OPUS_UNIMPLEMENTED;
+#endif
+        return ret;
+    }
+    enc->ms = opus_multistream_surround_encoder_create(
+        Fs, channels, mapping_family, nb_streams, nb_coupled, stream_map,
+        application, &ret);
+    return ret;
+}
+
+static int libopus_generic_encoder_init(OpusGenericEncoder *enc, int Fs,
+                                        int channels, int streams,
+                                        int coupled_streams,
+                                        const unsigned char *mapping,
+                                        int application)
+{
+    int ret;
+    enc->ms = opus_multistream_encoder_create(Fs, channels, streams,
+        coupled_streams, mapping, application, &ret);
+    return ret;
+}
+
+static int libopus_generic_encode(OpusGenericEncoder *enc,
+                                  const opus_int16 *pcm,
+                                  int frame_size, unsigned char *data,
+                                  opus_int32 max_data_bytes)
+{
+    int ret;
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+    if (enc->pr) {
+        ret = opus_projection_encode(enc->pr, pcm, frame_size, data,
+            max_data_bytes);
+        return ret;
+    }
+#endif
+    ret = opus_multistream_encode(enc->ms, pcm, frame_size, data,
+        max_data_bytes);
+    return ret;
+}
+
+static int libopus_generic_encode_float(OpusGenericEncoder *enc,
+                                        const float *pcm,
+                                        int frame_size, unsigned char *data,
+                                        opus_int32 max_data_bytes)
+{
+    int ret;
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+    if (enc->pr) {
+        ret = opus_projection_encode_float(enc->pr, pcm, frame_size, data,
+                                           max_data_bytes);
+        return ret;
+    }
+#endif
+    ret = opus_multistream_encode_float(enc->ms, pcm, frame_size, data,
+                                        max_data_bytes);
+    return ret;
+}
+
+static void libous_generic_encoder_cleanup(OpusGenericEncoder *enc)
+{
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+    if (enc->pr) opus_projection_encoder_destroy(enc->pr);
+#endif
+    if (enc->ms) opus_multistream_encoder_destroy(enc->ms);
+}
+
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+# define libopus_generic_encoder_ctl(enc, request) \
+    ((enc)->pr != NULL ? \
+    opus_projection_encoder_ctl((enc)->pr, request) : \
+    opus_multistream_encoder_ctl((enc)->ms, request))
+#else
+# define libopus_generic_encoder_ctl(enc, request) \
+    opus_multistream_encoder_ctl((enc)->ms, request)
+#endif
+
+static int libopus_generic_encoder_get_header_size(int mapping_family,
+                                                   int channels, int streams,
+                                                   int coupled_streams)
+{
+    int size = 19;
+    if (mapping_family == 1 || mapping_family == 2 || mapping_family == 255) {
+        return size + 2 + channels;
+    }
+    else if (mapping_family == 3) {
+        return size + 2 + 2 * channels * (streams + coupled_streams);
+    }
+    return size;
+}
+
 typedef struct LibopusEncOpts {
     int vbr;
     int application;
@@ -46,7 +166,7 @@  typedef struct LibopusEncOpts {
 
 typedef struct LibopusEncContext {
     AVClass *class;
-    OpusMSEncoder *enc;
+    OpusGenericEncoder enc;
     int stream_count;
     uint8_t *samples;
     LibopusEncOpts opts;
@@ -85,28 +205,68 @@  static const uint8_t libavcodec_libopus_channel_map[8][8] = {
 static void libopus_write_header(AVCodecContext *avctx, int stream_count,
                                  int coupled_stream_count,
                                  int mapping_family,
-                                 const uint8_t *channel_mapping)
+                                 const uint8_t *channel_mapping,
+                                 OpusGenericEncoder *enc)
 {
     uint8_t *p   = avctx->extradata;
     int channels = avctx->channels;
+    int gain = 0;
 
     bytestream_put_buffer(&p, "OpusHead", 8);
     bytestream_put_byte(&p, 1); /* Version */
     bytestream_put_byte(&p, channels);
     bytestream_put_le16(&p, avctx->initial_padding); /* Lookahead samples at 48kHz */
     bytestream_put_le32(&p, avctx->sample_rate); /* Original sample rate */
-    bytestream_put_le16(&p, 0); /* Gain of 0dB is recommended. */
+    if (mapping_family == 3) {
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+        int ret;
+        ret = libopus_generic_encoder_ctl(enc,
+                                          OPUS_PROJECTION_GET_DEMIXING_MATRIX_GAIN(&gain));
+        if (ret != OPUS_OK) {
+            av_log(avctx, AV_LOG_ERROR,
+                   "Unable to write header, demixing matrix gain not found.\n");
+            return;
+        }
+#endif
+    }
+    bytestream_put_le16(&p, gain); /* Gain of 0dB is recommended. */
 
     /* Channel mapping */
     bytestream_put_byte(&p, mapping_family);
-    if (mapping_family != 0) {
+    if (mapping_family == 3) {
+        int ret;
+        int32_t size;
+        size = 2 * channels * (stream_count + coupled_stream_count);
+        bytestream_put_byte(&p, stream_count);
+        bytestream_put_byte(&p, coupled_stream_count);
+        bytestream_put_byte(&p, stream_count);
+#ifdef OPUS_HAVE_OPUS_PROJECTION_H
+        ret = libopus_generic_encoder_ctl(enc,
+                                          OPUS_PROJECTION_GET_DEMIXING_MATRIX_SIZE(&size));
+        if (ret != OPUS_OK) {
+            av_log(avctx, AV_LOG_ERROR,
+                   "Unable to write header, demixing matrix size not found.\n");
+            return;
+        }
+        ret = libopus_generic_encoder_ctl(enc,
+                                          OPUS_PROJECTION_GET_DEMIXING_MATRIX(p, size));
+        if (ret != OPUS_OK) {
+            av_log(avctx, AV_LOG_ERROR,
+                   "Unable to write header, demixing matrix not found.\n");
+            return;
+        }
+        (*(&p)) += size;
+#endif
+    }
+    else if (mapping_family != 0) {
         bytestream_put_byte(&p, stream_count);
         bytestream_put_byte(&p, coupled_stream_count);
         bytestream_put_buffer(&p, channel_mapping, channels);
     }
 }
 
-static int libopus_configure_encoder(AVCodecContext *avctx, OpusMSEncoder *enc,
+static int libopus_configure_encoder(AVCodecContext *avctx,
+                                     OpusGenericEncoder *enc,
                                      LibopusEncOpts *opts)
 {
     int ret;
@@ -118,48 +278,48 @@  static int libopus_configure_encoder(AVCodecContext *avctx, OpusMSEncoder *enc,
         return AVERROR(EINVAL);
     }
 
-    ret = opus_multistream_encoder_ctl(enc, OPUS_SET_BITRATE(avctx->bit_rate));
+    ret = libopus_generic_encoder_ctl(enc, OPUS_SET_BITRATE(avctx->bit_rate));
     if (ret != OPUS_OK) {
         av_log(avctx, AV_LOG_ERROR,
                "Failed to set bitrate: %s\n", opus_strerror(ret));
         return ret;
     }
 
-    ret = opus_multistream_encoder_ctl(enc,
-                                       OPUS_SET_COMPLEXITY(opts->complexity));
+    ret = libopus_generic_encoder_ctl(enc,
+                                      OPUS_SET_COMPLEXITY(opts->complexity));
     if (ret != OPUS_OK)
         av_log(avctx, AV_LOG_WARNING,
                "Unable to set complexity: %s\n", opus_strerror(ret));
 
-    ret = opus_multistream_encoder_ctl(enc, OPUS_SET_VBR(!!opts->vbr));
+    ret = libopus_generic_encoder_ctl(enc, OPUS_SET_VBR(!!opts->vbr));
     if (ret != OPUS_OK)
         av_log(avctx, AV_LOG_WARNING,
                "Unable to set VBR: %s\n", opus_strerror(ret));
 
-    ret = opus_multistream_encoder_ctl(enc,
-                                       OPUS_SET_VBR_CONSTRAINT(opts->vbr == 2));
+    ret = libopus_generic_encoder_ctl(enc,
+                                      OPUS_SET_VBR_CONSTRAINT(opts->vbr == 2));
     if (ret != OPUS_OK)
         av_log(avctx, AV_LOG_WARNING,
                "Unable to set constrained VBR: %s\n", opus_strerror(ret));
 
-    ret = opus_multistream_encoder_ctl(enc,
-                                       OPUS_SET_PACKET_LOSS_PERC(opts->packet_loss));
+    ret = libopus_generic_encoder_ctl(enc,
+                                      OPUS_SET_PACKET_LOSS_PERC(opts->packet_loss));
     if (ret != OPUS_OK)
         av_log(avctx, AV_LOG_WARNING,
                "Unable to set expected packet loss percentage: %s\n",
                opus_strerror(ret));
 
     if (avctx->cutoff) {
-        ret = opus_multistream_encoder_ctl(enc,
-                                           OPUS_SET_MAX_BANDWIDTH(opts->max_bandwidth));
+        ret = libopus_generic_encoder_ctl(enc,
+                                          OPUS_SET_MAX_BANDWIDTH(opts->max_bandwidth));
         if (ret != OPUS_OK)
             av_log(avctx, AV_LOG_WARNING,
                    "Unable to set maximum bandwidth: %s\n", opus_strerror(ret));
     }
 
 #ifdef OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST
-    ret = opus_multistream_encoder_ctl(enc,
-                                       OPUS_SET_PHASE_INVERSION_DISABLED(!opts->apply_phase_inv));
+    ret = libopus_generic_encoder_ctl(enc,
+                                      OPUS_SET_PHASE_INVERSION_DISABLED(!opts->apply_phase_inv));
     if (ret != OPUS_OK)
         av_log(avctx, AV_LOG_WARNING,
                "Unable to set phase inversion: %s\n",
@@ -207,6 +367,8 @@  static int libopus_validate_layout_and_get_channel_map(
 {
     const uint8_t * channel_map = NULL;
     int ret;
+    int order_plus_one;
+    int nondiegetic_channels;
 
     switch (mapping_family) {
     case -1:
@@ -231,6 +393,23 @@  static int libopus_validate_layout_and_get_channel_map(
             channel_map = ff_vorbis_channel_layout_offsets[avctx->channels - 1];
         }
         break;
+    case 2:
+    case 3:
+        order_plus_one = ff_sqrt(avctx->channels);
+        nondiegetic_channels = avctx->channels - order_plus_one * order_plus_one;
+        if (order_plus_one < 1 || order_plus_one > 15 ||
+           (nondiegetic_channels != 0 && nondiegetic_channels != 2)) {
+            av_log(avctx, AV_LOG_ERROR,
+                   "This channel mapping is only specified for channel counts"
+                   " which can be written as (n + 1)^2 + 2j, where n is a"
+                   " non-negative integar from 0 to 14 and j is either 0 or 1.\n");
+            ret = AVERROR_INVALIDDATA;
+        } else {
+            ret = 0;
+        }
+
+        /* Channels do not need to be reordered. */
+        break;
     case 255:
         ret = libopus_check_max_channels(avctx, 254);
         break;
@@ -248,7 +427,7 @@  static int libopus_validate_layout_and_get_channel_map(
 static av_cold int libopus_encode_init(AVCodecContext *avctx)
 {
     LibopusEncContext *opus = avctx->priv_data;
-    OpusMSEncoder *enc;
+    OpusGenericEncoder *enc = &opus->enc;
     uint8_t libopus_channel_mapping[255];
     int ret = OPUS_OK;
     int av_ret;
@@ -335,20 +514,20 @@  static av_cold int libopus_encode_init(AVCodecContext *avctx)
                opus_vorbis_channel_map[avctx->channels - 1],
                avctx->channels * sizeof(*libopus_channel_mapping));
 
-        enc = opus_multistream_encoder_create(
-            avctx->sample_rate, avctx->channels, opus->stream_count,
+        ret = libopus_generic_encoder_init(
+            enc, avctx->sample_rate, avctx->channels, opus->stream_count,
             coupled_stream_count,
             libavcodec_libopus_channel_map[avctx->channels - 1],
-            opus->opts.application, &ret);
+            opus->opts.application);
     } else {
         /* Use the newer multistream API. The encoder will set the channel
          * mapping and coupled stream counts to its internal defaults and will
          * use surround masking analysis to save bits. */
         mapping_family = opus->opts.mapping_family;
-        enc = opus_multistream_surround_encoder_create(
-            avctx->sample_rate, avctx->channels, mapping_family,
+        ret = libopus_generic_encoder_surround_init(
+            enc, avctx->sample_rate, avctx->channels, mapping_family,
             &opus->stream_count, &coupled_stream_count, libopus_channel_mapping,
-            opus->opts.application, &ret);
+            opus->opts.application);
     }
 
     if (ret != OPUS_OK) {
@@ -380,7 +559,9 @@  static av_cold int libopus_encode_init(AVCodecContext *avctx)
     }
 
     /* Header includes channel mapping table if and only if mapping family is NOT 0 */
-    header_size = 19 + (mapping_family == 0 ? 0 : 2 + avctx->channels);
+    header_size = libopus_generic_encoder_get_header_size(
+        mapping_family, avctx->channels, opus->stream_count,
+        coupled_stream_count);
     avctx->extradata = av_malloc(header_size + AV_INPUT_BUFFER_PADDING_SIZE);
     if (!avctx->extradata) {
         av_log(avctx, AV_LOG_ERROR, "Failed to allocate extradata.\n");
@@ -397,23 +578,21 @@  static av_cold int libopus_encode_init(AVCodecContext *avctx)
         goto fail;
     }
 
-    ret = opus_multistream_encoder_ctl(enc, OPUS_GET_LOOKAHEAD(&avctx->initial_padding));
+    ret = libopus_generic_encoder_ctl(enc, OPUS_GET_LOOKAHEAD(&avctx->initial_padding));
     if (ret != OPUS_OK)
         av_log(avctx, AV_LOG_WARNING,
                "Unable to get number of lookahead samples: %s\n",
                opus_strerror(ret));
 
     libopus_write_header(avctx, opus->stream_count, coupled_stream_count,
-                         mapping_family, libopus_channel_mapping);
+                         mapping_family, libopus_channel_mapping, enc);
 
     ff_af_queue_init(avctx, &opus->afq);
 
-    opus->enc = enc;
-
     return 0;
 
 fail:
-    opus_multistream_encoder_destroy(enc);
+    libous_generic_encoder_cleanup(enc);
     av_freep(&avctx->extradata);
     return ret;
 }
@@ -470,13 +649,13 @@  static int libopus_encode(AVCodecContext *avctx, AVPacket *avpkt,
         return ret;
 
     if (avctx->sample_fmt == AV_SAMPLE_FMT_FLT)
-        ret = opus_multistream_encode_float(opus->enc, (float *)audio,
-                                            opus->opts.packet_size,
-                                            avpkt->data, avpkt->size);
+        ret = libopus_generic_encode_float(&opus->enc, (float *)audio,
+                                           opus->opts.packet_size,
+                                           avpkt->data, avpkt->size);
     else
-        ret = opus_multistream_encode(opus->enc, (opus_int16 *)audio,
-                                      opus->opts.packet_size,
-                                      avpkt->data, avpkt->size);
+        ret = libopus_generic_encode(&opus->enc, (opus_int16 *)audio,
+                                     opus->opts.packet_size,
+                                     avpkt->data, avpkt->size);
 
     if (ret < 0) {
         av_log(avctx, AV_LOG_ERROR,
@@ -517,7 +696,7 @@  static av_cold int libopus_encode_close(AVCodecContext *avctx)
 {
     LibopusEncContext *opus = avctx->priv_data;
 
-    opus_multistream_encoder_destroy(opus->enc);
+    libous_generic_encoder_cleanup(&opus->enc);
 
     ff_af_queue_close(&opus->afq);
 
diff --git a/libavcodec/opus.c b/libavcodec/opus.c
index aa827b604c..f8877233af 100644
--- a/libavcodec/opus.c
+++ b/libavcodec/opus.c
@@ -373,18 +373,14 @@  av_cold int ff_opus_parse_extradata(AVCodecContext *avctx,
             layout = ff_vorbis_channel_layouts[channels - 1];
             channel_reorder = channel_reorder_vorbis;
         } else if (map_type == 2) {
-            int ambisonic_order = ff_sqrt(channels) - 1;
-            if (channels != ((ambisonic_order + 1) * (ambisonic_order + 1)) &&
-                channels != ((ambisonic_order + 1) * (ambisonic_order + 1) + 2)) {
+            int order_plus_one = ff_sqrt(channels);
+            int nondiegetic_channels = channels - order_plus_one * order_plus_one;
+            if (order_plus_one < 1 || order_plus_one > 15 ||
+               (nondiegetic_channels != 0 && nondiegetic_channels != 2)) {
                 av_log(avctx, AV_LOG_ERROR,
-                       "Channel mapping 2 is only specified for channel counts"
-                       " which can be written as (n + 1)^2 or (n + 1)^2 + 2"
-                       " for nonnegative integer n\n");
-                return AVERROR_INVALIDDATA;
-            }
-            if (channels > 227) {
-                av_log(avctx, AV_LOG_ERROR, "Too many channels\n");
-                return AVERROR_INVALIDDATA;
+                    "This channel mapping is only specified for channel counts"
+                    " which can be written as (n + 1)^2 + 2j, where n is a"
+                    " non-negative integar from 0 to 14 and j is either 0 or 1.\n");
             }
             layout = 0;
         } else
-- 
2.17.0.484.g0c8726318c-goog