From patchwork Wed Dec 9 17:20:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul B Mahol X-Patchwork-Id: 24455 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 3F20344B722 for ; Wed, 9 Dec 2020 19:46:10 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1714268A0D0; Wed, 9 Dec 2020 19:46:10 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f181.google.com (mail-lj1-f181.google.com [209.85.208.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3B338689972 for ; Wed, 9 Dec 2020 19:46:04 +0200 (EET) Received: by mail-lj1-f181.google.com with SMTP id q8so3363966ljc.12 for ; Wed, 09 Dec 2020 09:46:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=fHdksO62ng6Kbiqv7gsNJ5/6zjjwxFuXDUGVf8E4Sx4=; b=b9mgnegH/ZKb8WZkOk9GXLLu8v8i+SwEqMmAbwqvg1IopdmY6SDaEO7DtmivT6WExE egWgbTUTlzM4d0yOy1zkWBCPvaS01J2ItNBuDURZxo/WiivXcWILoOjwUb/0/SRQnsAG zZ5PpoEzZGItlweXFf2sK+We8Ei1b4i4itGjOa0xhVjzlSVZn0V1TgvUjcG5LIxYDDWa YqtOnu0OzTSD+gocyBvCNUVhsXUB6Z+waFhbE9A7bdo1EKKJcnp5Vpnxg6cNmR68lv3A +SgFpNCoFRHhXK3Y1RWMFJywl7ktoqaeK9QM3ERpehRqi02282mqD9kPQhxtfq9aU1Ax eoOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=fHdksO62ng6Kbiqv7gsNJ5/6zjjwxFuXDUGVf8E4Sx4=; b=YnVn2mnkcNXxVuhd98cjwug97e1lsgkNAq+4im9x/D5i3imVdl5MBucZvBlBwe9OGH oESUox8Pkx9q9tWkXsRVCcd7J7Ou4S8Gd8aX3twZGhY++bCzST4XmcOTFtVCKvQgizGV ZsB7UovcgW3G9aD1nz86AQtZ//Ll/pyF/ZjeGJdf2rGOgjI/8rNFR5t6+wcYVxODMwZJ VHC8h5ZvylhxkMR8nSNxI47R0bptOaHS4mNXh1gXqgbdOfJI0OGJd6jYu9bhT1UVaAu6 whwetdV7s6Q5t8gJBrumjNgVQrdAe8Zv1i4vthdg81KgLrg8RVc8REWngjJ7m4bgkdm5 q9Kg== X-Gm-Message-State: AOAM531eMBNk+EW5HJXL9c3b8Qb/1H9GD7KN+Qp4YtMreIKAtA3zTvQR bfeN7TSwPrlAQSdrq9sA+F5gqfIaGCo= X-Google-Smtp-Source: ABdhPJwD7EhqLnWCxqVY0J96bqI95J9BFGM4vfROakoPGZST8Nk3DBBTs+pUzCuQkkRbCmEidTPZfg== X-Received: by 2002:a17:906:4881:: with SMTP id v1mr2801497ejq.465.1607534418324; Wed, 09 Dec 2020 09:20:18 -0800 (PST) Received: from localhost.localdomain ([77.237.101.115]) by smtp.gmail.com with ESMTPSA id x9sm2057806ejd.99.2020.12.09.09.20.17 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 09:20:17 -0800 (PST) From: Paul B Mahol To: ffmpeg-devel@ffmpeg.org Date: Wed, 9 Dec 2020 18:20:07 +0100 Message-Id: <20201209172007.14867-1-onemda@gmail.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH] avfilter: add stereoupmix X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Paul B Mahol --- doc/filters.texi | 34 ++++ libavfilter/Makefile | 1 + libavfilter/af_stereoupmix.c | 352 +++++++++++++++++++++++++++++++++++ libavfilter/allfilters.c | 1 + 4 files changed, 388 insertions(+) create mode 100644 libavfilter/af_stereoupmix.c diff --git a/doc/filters.texi b/doc/filters.texi index 9dfe95f40d..325753c8f4 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -5817,6 +5817,40 @@ Convert M/S signal to L/R: @end example @end itemize +@section stereoupmix +Upmix stereo audio. + +This filter upmixes stereo audio using adaptive panning method. + +The filter accepts the following options: + +@table @option +@item upmix +Set the upmix mode. Can be one of the following: +@table @samp +@item 2.1 +@item 3.0 +@item 3.1 +@item 4.0 +@item 4.1 +@item 5.0 +@item 5.1 +@end table +Default value is @var{5.1}. + +@item center +Set the center audio strength. Allowed range is from 0.0 to 1.0. +Default value is 0.5. + +@item ambience +Set the ambience audio strength. Allowed range is from 0.0 to 1.0. +Default value is 0.5. +@end table + +@subsection Commands + +This filter supports the all above options except @code{upmix} as @ref{commands}. + @section stereowiden This filter enhance the stereo effect by suppressing signal common to both diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 1af85a71a0..a9d76a1eaf 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -145,6 +145,7 @@ OBJS-$(CONFIG_SILENCEREMOVE_FILTER) += af_silenceremove.o OBJS-$(CONFIG_SOFALIZER_FILTER) += af_sofalizer.o OBJS-$(CONFIG_SPEECHNORM_FILTER) += af_speechnorm.o OBJS-$(CONFIG_STEREOTOOLS_FILTER) += af_stereotools.o +OBJS-$(CONFIG_STEREOUPMIX_FILTER) += af_stereoupmix.o OBJS-$(CONFIG_STEREOWIDEN_FILTER) += af_stereowiden.o OBJS-$(CONFIG_SUPEREQUALIZER_FILTER) += af_superequalizer.o OBJS-$(CONFIG_SURROUND_FILTER) += af_surround.o diff --git a/libavfilter/af_stereoupmix.c b/libavfilter/af_stereoupmix.c new file mode 100644 index 0000000000..813f21b088 --- /dev/null +++ b/libavfilter/af_stereoupmix.c @@ -0,0 +1,352 @@ +/* + * Copyright (c) 2020 Paul B Mahol + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/avassert.h" +#include "libavutil/channel_layout.h" +#include "libavutil/opt.h" +#include "avfilter.h" +#include "audio.h" +#include "formats.h" + +enum UpmixMode { + UPMIX_2_1, + UPMIX_3_0, + UPMIX_3_1, + UPMIX_4_0, + UPMIX_4_1, + UPMIX_5_0, + UPMIX_5_1, + NB_UPMIX +}; + +typedef struct StereoUpmixContext { + const AVClass *class; + + int upmix; + float center; + float ambience; + + uint64_t out_layout; + + float fl, fr; + float y; + float pk; + float wl, wr; + + float a[2]; + float b[3]; + float z[2]; +} StereoUpmixContext; + +#define OFFSET(x) offsetof(StereoUpmixContext, x) +#define FLAGS AV_OPT_FLAG_AUDIO_PARAM|AV_OPT_FLAG_FILTERING_PARAM +#define TFLAGS AV_OPT_FLAG_AUDIO_PARAM|AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_RUNTIME_PARAM + +static const AVOption stereoupmix_options[] = { + { "upmix", "set upmix mode", OFFSET(upmix), AV_OPT_TYPE_INT, {.i64=UPMIX_5_1}, 0, NB_UPMIX-1, FLAGS, "upmix" }, + { "2.1", NULL, 0, AV_OPT_TYPE_CONST, {.i64=UPMIX_2_1}, 0, 0, FLAGS, "upmix" }, + { "3.0", NULL, 0, AV_OPT_TYPE_CONST, {.i64=UPMIX_3_0}, 0, 0, FLAGS, "upmix" }, + { "3.1", NULL, 0, AV_OPT_TYPE_CONST, {.i64=UPMIX_3_1}, 0, 0, FLAGS, "upmix" }, + { "4.0", NULL, 0, AV_OPT_TYPE_CONST, {.i64=UPMIX_4_0}, 0, 0, FLAGS, "upmix" }, + { "4.1", NULL, 0, AV_OPT_TYPE_CONST, {.i64=UPMIX_4_1}, 0, 0, FLAGS, "upmix" }, + { "5.0", NULL, 0, AV_OPT_TYPE_CONST, {.i64=UPMIX_5_0}, 0, 0, FLAGS, "upmix" }, + { "5.1", NULL, 0, AV_OPT_TYPE_CONST, {.i64=UPMIX_5_1}, 0, 0, FLAGS, "upmix" }, + { "center", "set center strength", OFFSET(center), AV_OPT_TYPE_FLOAT, {.dbl=0.5}, 0, 1, TFLAGS }, + { "ambience", "set ambience strength", OFFSET(ambience), AV_OPT_TYPE_FLOAT, {.dbl=0.5}, 0, 1, TFLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(stereoupmix); + +static av_cold int init(AVFilterContext *ctx) +{ + StereoUpmixContext *s = ctx->priv; + + s->wl = s->wr = M_SQRT1_2; + + switch (s->upmix) { + case UPMIX_2_1: + s->out_layout = AV_CH_LAYOUT_2POINT1; + break; + case UPMIX_3_0: + s->out_layout = AV_CH_LAYOUT_SURROUND; + break; + case UPMIX_3_1: + s->out_layout = AV_CH_LAYOUT_3POINT1; + break; + case UPMIX_4_0: + s->out_layout = AV_CH_LAYOUT_4POINT0; + break; + case UPMIX_4_1: + s->out_layout = AV_CH_LAYOUT_4POINT1; + break; + case UPMIX_5_0: + s->out_layout = AV_CH_LAYOUT_5POINT0_BACK; + break; + case UPMIX_5_1: + s->out_layout = AV_CH_LAYOUT_5POINT1_BACK; + break; + default: + av_assert0(0); + } + + return 0; +} + +static int query_formats(AVFilterContext *ctx) +{ + StereoUpmixContext *s = ctx->priv; + AVFilterFormats *formats = NULL; + AVFilterChannelLayouts *layouts = NULL; + int ret; + + ret = ff_add_format(&formats, AV_SAMPLE_FMT_FLT); + if (ret) + return ret; + ret = ff_set_common_formats(ctx, formats); + if (ret) + return ret; + + layouts = NULL; + ret = ff_add_channel_layout(&layouts, s->out_layout); + if (ret) + return ret; + + ret = ff_channel_layouts_ref(layouts, &ctx->outputs[0]->incfg.channel_layouts); + if (ret) + return ret; + + layouts = NULL; + ret = ff_add_channel_layout(&layouts, AV_CH_LAYOUT_STEREO); + if (ret) + return ret; + + ret = ff_channel_layouts_ref(layouts, &ctx->inputs[0]->outcfg.channel_layouts); + if (ret) + return ret; + + formats = ff_all_samplerates(); + if (!formats) + return AVERROR(ENOMEM); + return ff_set_common_samplerates(ctx, formats); +} + +static void upmix(AVFilterContext *ctx, AVFrame *out, AVFrame *in) +{ + StereoUpmixContext *s = ctx->priv; + const float center = s->center; + const float ambience = s->ambience; + const float *src = (const float *)in->data[0]; + const float b0 = s->b[0]; + const float b1 = s->b[1]; + const float b2 = s->b[2]; + const float a1 = s->a[0]; + const float a2 = s->a[1]; + float *dst = (float *)out->data[0]; + float wl = s->wl, wr = s->wr; + float fl = s->fl, fr = s->fr; + float pk = s->pk; + float y = s->y; + float z0 = s->z[0]; + float z1 = s->z[1]; + + for (int n = 0; n < in->nb_samples; n++) { + float nwl, nwr; + float clr, cl, cr; + float slr, sl, sr; + float beta, gamma, q; + float ro = 0.001f; + float sing, cosb, sinb; + float FL, FR, FC, LFE, SL, SR, SC; + float mid, side; + float g, ig2, gg; + + nwl = wl + ro * y * (fl - wl * y); + nwr = wr + ro * y * (fr - wr * y); + fl = src[n * 2 ]; + fr = src[n * 2 + 1]; + y = nwl * fl + nwr * fr; + q = nwr * fl - nwl * fr; + pk = pk + ro * (2.f * fl * fr - (fl * fl + fr * fr) * pk); + av_assert0(pk >= -1.f && pk <= 1.f); + beta = asinf(1.f - fmaxf(pk, 0.f)); + gamma = acosf(1.f + fminf(pk, 0.f)); + cosb = cosf(beta); + sinb = sinf(beta); + sing = sinf(gamma); + g = cosb * cosb; + gg = sinb * sinb; + ig2 = (1.f - g) * (1.f - g); + + clr = (nwr * nwr - nwl * nwl) * cosb; + slr = (nwr * nwr - nwl * nwl) * sing; + cl = clr < 0 ? -clr : 0.f; + cr = clr >= 0 ? clr : 0.f; + sl = slr < 0 ? -slr : 0.f; + sr = slr >= 0 ? slr : 0.f; + mid = 0.5f * (fl + fr); + side = 0.5f * (fr - fl); + FC = cosb * nwl * nwr * y + gg * mid * center; + FL = cl * y + g * nwl * q + ig2 * fl; + FR = cr * y + g * nwr * q + ig2 * fr; + SC = sinb * q; + SL = SC + sl * side * ambience; + SR = SC + sr * side * ambience; + LFE = FC * b0 + z0; + z0 = b1 * FC + z1 + a1 * LFE; + z1 = b2 * FC + a2 * LFE; + + switch (s->upmix) { + case UPMIX_2_1: + dst[n * 3 + 0] = fl; + dst[n * 3 + 1] = fr; + dst[n * 3 + 2] = LFE; + break; + case UPMIX_3_0: + dst[n * 3 + 0] = fl; + dst[n * 3 + 1] = fr; + dst[n * 3 + 2] = FC; + break; + case UPMIX_3_1: + dst[n * 4 + 0] = fl; + dst[n * 4 + 1] = fr; + dst[n * 4 + 2] = FC; + dst[n * 4 + 3] = LFE; + break; + case UPMIX_4_0: + dst[n * 4 + 0] = FL; + dst[n * 4 + 1] = FR; + dst[n * 4 + 2] = FC; + dst[n * 4 + 3] = SC; + break; + case UPMIX_4_1: + dst[n * 5 + 0] = FL; + dst[n * 5 + 1] = FR; + dst[n * 5 + 2] = FC; + dst[n * 5 + 3] = LFE; + dst[n * 5 + 4] = SC; + break; + case UPMIX_5_0: + dst[n * 5 + 0] = FL; + dst[n * 5 + 1] = FR; + dst[n * 5 + 2] = FC; + dst[n * 5 + 3] = SL; + dst[n * 5 + 4] = SR; + break; + case UPMIX_5_1: + dst[n * 6 + 0] = FL; + dst[n * 6 + 1] = FR; + dst[n * 6 + 2] = FC; + dst[n * 6 + 3] = LFE; + dst[n * 6 + 4] = SL; + dst[n * 6 + 5] = SR; + break; + default: + av_assert0(0); + } + + wl = nwl; + wr = nwr; + } + + s->pk = pk; + s->y = y; + + s->fl = fl; + s->fr = fr; + + s->wl = wl; + s->wr = wr; + + s->z[0] = z0; + s->z[1] = z1; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx = inlink->dst; + AVFilterLink *outlink = ctx->outputs[0]; + AVFrame *out; + + out = ff_get_audio_buffer(outlink, in->nb_samples); + if (!out) { + av_frame_free(&in); + return AVERROR(ENOMEM); + } + av_frame_copy_props(out, in); + + upmix(ctx, out, in); + + av_frame_free(&in); + return ff_filter_frame(outlink, out); +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + StereoUpmixContext *s = ctx->priv; + double w0 = 2. * M_PI * 120. / outlink->sample_rate; + double alpha = sin(w0) / 2. * sqrt(2.); + double a0 = 1. + alpha; + + s->a[0] = 2. * cos(w0); + s->a[1] = alpha - 1.; + s->b[0] = (1. - cos(w0)) / 2.; + s->b[1] = 1. - cos(w0); + s->b[2] = (1. - cos(w0)) / 2.; + + s->a[0] /= a0; + s->a[1] /= a0; + s->b[0] /= a0; + s->b[1] /= a0; + s->b[2] /= a0; + + return 0; +} + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_AUDIO, + .filter_frame = filter_frame, + }, + { NULL } +}; + +static const AVFilterPad outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_AUDIO, + .config_props = config_output, + }, + { NULL } +}; + +AVFilter ff_af_stereoupmix = { + .name = "stereoupmix", + .description = NULL_IF_CONFIG_SMALL("Upmix stereo audio."), + .query_formats = query_formats, + .priv_size = sizeof(StereoUpmixContext), + .priv_class = &stereoupmix_class, + .init = init, + .inputs = inputs, + .outputs = outputs, + .process_command = ff_filter_process_command, +}; diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 16e1f08a29..bb96ca88ab 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -139,6 +139,7 @@ extern AVFilter ff_af_silenceremove; extern AVFilter ff_af_sofalizer; extern AVFilter ff_af_speechnorm; extern AVFilter ff_af_stereotools; +extern AVFilter ff_af_stereoupmix; extern AVFilter ff_af_stereowiden; extern AVFilter ff_af_superequalizer; extern AVFilter ff_af_surround;