From patchwork Sat May 25 11:55:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul B Mahol X-Patchwork-Id: 13290 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id D4E52447638 for ; Sat, 25 May 2019 14:55:18 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B57B96809AC; Sat, 25 May 2019 14:55:18 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B0890680545 for ; Sat, 25 May 2019 14:55:11 +0300 (EEST) Received: by mail-wr1-f66.google.com with SMTP id r7so12384515wrr.13 for ; Sat, 25 May 2019 04:55:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=McNibavsEl8Kwv59RZwg+y8F4iuNIQAQEzyDTE8qlXU=; b=FBjwV+S/McDCui3WWkd63Qr3trnSH2pV4APERlr0aTefQaanWnUuoEfNlzA1pH94YM 0QCycIbuaC13Bo14DE6eix3rwP4CfmVoDUXIoh9PH5y7Cpj8WZRoDQN0h/vazzKkKFLe 8FRM5B3U2DLIcHxHENz/+dLkaxoFKnitEF0CZXBDdzfKs7PyLJqSkn+vzy2nR1190zMF TOgDqT08ol39qM2KpZeYKSBXnKLe604HIYByDTMpNO9aF+s8VARG6AiC2SnKjOKcNpia ZJfQ/uYyPCV6sUyHhwsY5cGLaqR4ANqSrgX0FmiA4OhN7H0sP2a5S4pLjF69MtnvA3K9 kzIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=McNibavsEl8Kwv59RZwg+y8F4iuNIQAQEzyDTE8qlXU=; b=BnMMN1PwhJzh+ej/WxkoIR+/wfZRey3ZBoGYDZw2v71v+AouGKIyq7g+tN94+MyuYm 8BVDxG3U2MA86pLKYlMgw4S++3nwR2IL89jZQ4rT2Z4AoA9n9S4pqOzMb0l0/egJAiCf KvzQySLRPQ+cLw11GPTPUTHbF6G8rwUoKCeciCHaiMIu1PsXuW+4u4hn1i1ac0Gku2Qm XDtNw81BcYAKSa079c6oCeqH3IBAJ7hNrBz6CTf3L21/bL+DDAoqMDdGLOMkJ/6bQcYd 5E50aS0OCVsfJBUKNNU2DES5BdMNtDJ2hSZ4/wsbQ/YbmFUckjiL0/xGz4ISYM8YuQrn OuqQ== X-Gm-Message-State: APjAAAU8TqAOUsXG1BkT0cKNf8ztOlXyCCh+PB3xLtepN1mNduiLGBOc Kf1JH96o2guppfCf0uOzLw9ZwEWU X-Google-Smtp-Source: APXvYqzCAesMfMZMJcVYNm/z+E5m7UCWOR3KIoEtMhsNqlQmeQaym4C8W9+pOrD03z7+DfZLZXU89Q== X-Received: by 2002:a5d:4604:: with SMTP id t4mr65819576wrq.93.1558785310620; Sat, 25 May 2019 04:55:10 -0700 (PDT) Received: from localhost.localdomain ([37.244.236.102]) by smtp.gmail.com with ESMTPSA id n5sm7660330wrj.27.2019.05.25.04.55.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 25 May 2019 04:55:09 -0700 (PDT) From: Paul B Mahol To: ffmpeg-devel@ffmpeg.org Date: Sat, 25 May 2019 13:55:00 +0200 Message-Id: <20190525115500.5830-1-onemda@gmail.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH] avfilter: add showspatial multimedia filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Paul B Mahol --- doc/filters.texi | 52 +++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/avf_showspatial.c | 384 ++++++++++++++++++++++++++++++++++ 4 files changed, 438 insertions(+) create mode 100644 libavfilter/avf_showspatial.c diff --git a/doc/filters.texi b/doc/filters.texi index 4fdcfe919e..60df5bf35b 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -22248,6 +22248,58 @@ Set minimum amplitude used in @code{log} amplitude scaler. @end table +@section showspatial + +Convert stereo input audio to a video output, representing the spatial relationship +between two channels. + +The filter accepts the following options: + +@table @option +@item size, s +Specify the video size for the output. For the syntax of this option, check the +@ref{video size syntax,,"Video size" section in the ffmpeg-utils manual,ffmpeg-utils}. +Default value is @code{512x512}. + +@item win_size +Set window size. Allowed range is from @var{1024} to @var{65536}. Default size is @var{4096}. + +@item win_func +Set window function. + +It accepts the following values: +@table @samp +@item rect +@item bartlett +@item hann +@item hanning +@item hamming +@item blackman +@item welch +@item flattop +@item bharris +@item bnuttall +@item bhann +@item sine +@item nuttall +@item lanczos +@item gauss +@item tukey +@item dolph +@item cauchy +@item parzen +@item poisson +@item bohman +@end table + +Default value is @code{hann}. + +@item overlap +Set ratio of overlap window. Default value is @code{0.5}. +When value is @code{1} overlap is set to recommended size for specific +window function currently used. +@end table + @anchor{showspectrum} @section showspectrum diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 9a61c25b05..a99362b3ee 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -465,6 +465,7 @@ OBJS-$(CONFIG_AVECTORSCOPE_FILTER) += avf_avectorscope.o OBJS-$(CONFIG_CONCAT_FILTER) += avf_concat.o OBJS-$(CONFIG_SHOWCQT_FILTER) += avf_showcqt.o lswsutils.o lavfutils.o OBJS-$(CONFIG_SHOWFREQS_FILTER) += avf_showfreqs.o +OBJS-$(CONFIG_SHOWSPATIAL_FILTER) += avf_showspatial.o OBJS-$(CONFIG_SHOWSPECTRUM_FILTER) += avf_showspectrum.o OBJS-$(CONFIG_SHOWSPECTRUMPIC_FILTER) += avf_showspectrum.o OBJS-$(CONFIG_SHOWVOLUME_FILTER) += avf_showvolume.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 40534738ee..858ed1cf78 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -443,6 +443,7 @@ extern AVFilter ff_avf_avectorscope; extern AVFilter ff_avf_concat; extern AVFilter ff_avf_showcqt; extern AVFilter ff_avf_showfreqs; +extern AVFilter ff_avf_showspatial; extern AVFilter ff_avf_showspectrum; extern AVFilter ff_avf_showspectrumpic; extern AVFilter ff_avf_showvolume; diff --git a/libavfilter/avf_showspatial.c b/libavfilter/avf_showspatial.c new file mode 100644 index 0000000000..974c9154b3 --- /dev/null +++ b/libavfilter/avf_showspatial.c @@ -0,0 +1,384 @@ +/* + * Copyright (c) 2019 Paul B Mahol + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include + +#include "libavcodec/avfft.h" +#include "libavutil/audio_fifo.h" +#include "libavutil/avassert.h" +#include "libavutil/channel_layout.h" +#include "libavutil/opt.h" +#include "libavutil/parseutils.h" +#include "audio.h" +#include "video.h" +#include "avfilter.h" +#include "filters.h" +#include "internal.h" +#include "window_func.h" + +typedef struct ShowSpatialContext { + const AVClass *class; + int w, h; + AVRational frame_rate; + FFTContext *fft[2]; ///< Fast Fourier Transform context + FFTContext *ifft[2]; ///< Inverse Fast Fourier Transform context + int fft_bits; ///< number of bits (FFT window size = 1<priv; + int i; + + for (i = 0; i < 2; i++) + av_fft_end(s->fft[i]); + for (i = 0; i < 2; i++) + av_fft_end(s->ifft[i]); + for (i = 0; i < 2; i++) + av_freep(&s->fft_data[i]); + av_freep(&s->window_func_lut); + av_audio_fifo_free(s->fifo); +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats = NULL; + AVFilterChannelLayouts *layouts = NULL; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSampleFormat sample_fmts[] = { AV_SAMPLE_FMT_FLTP, AV_SAMPLE_FMT_NONE }; + static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_GBRP, AV_PIX_FMT_NONE }; + int ret; + + /* set input audio formats */ + formats = ff_make_format_list(sample_fmts); + if ((ret = ff_formats_ref(formats, &inlink->out_formats)) < 0) + return ret; + + layouts = ff_all_channel_layouts(); + if ((ret = ff_channel_layouts_ref(layouts, &inlink->out_channel_layouts)) < 0) + return ret; + + formats = ff_all_samplerates(); + if ((ret = ff_formats_ref(formats, &inlink->out_samplerates)) < 0) + return ret; + + /* set output video format */ + formats = ff_make_format_list(pix_fmts); + if ((ret = ff_formats_ref(formats, &outlink->in_formats)) < 0) + return ret; + + return 0; +} + +static int run_channel_fft(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) +{ + ShowSpatialContext *s = ctx->priv; + const float *window_func_lut = s->window_func_lut; + AVFrame *fin = arg; + const int ch = jobnr; + int n; + + /* fill FFT input with the number of samples available */ + const float *p = (float *)fin->extended_data[ch]; + + for (n = 0; n < s->win_size; n++) { + s->fft_data[ch][n].re = p[n] * window_func_lut[n]; + s->fft_data[ch][n].im = 0; + } + + av_fft_permute(s->fft[ch], s->fft_data[ch]); + av_fft_calc(s->fft[ch], s->fft_data[ch]); + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + AVFilterLink *inlink = ctx->inputs[0]; + ShowSpatialContext *s = ctx->priv; + int i, fft_bits, h, w; + float overlap; + + outlink->w = s->w; + outlink->h = s->h; + outlink->sample_aspect_ratio = (AVRational){1,1}; + + h = s->h; + w = s->w; + + s->buf_size = 1 << av_log2(s->win_size); + s->win_size = s->buf_size; + fft_bits = av_log2(s->win_size); + + /* (re-)configuration if the video output changed (or first init) */ + if (fft_bits != s->fft_bits) { + s->fft_bits = fft_bits; + + /* FFT buffers: x2 for each channel buffer. + * Note: we use free and malloc instead of a realloc-like function to + * make sure the buffer is aligned in memory for the FFT functions. */ + for (i = 0; i < 2; i++) { + av_fft_end(s->fft[i]); + av_freep(&s->fft_data[i]); + } + for (i = 0; i < 2; i++) { + s->fft[i] = av_fft_init(fft_bits, 0); + if (!s->fft[i]) { + av_log(ctx, AV_LOG_ERROR, "Unable to create FFT context. " + "The window size might be too high.\n"); + return AVERROR(EINVAL); + } + } + + for (i = 0; i < 2; i++) { + s->fft_data[i] = av_calloc(s->buf_size, sizeof(**s->fft_data)); + if (!s->fft_data[i]) + return AVERROR(ENOMEM); + } + + /* pre-calc windowing function */ + s->window_func_lut = + av_realloc_f(s->window_func_lut, s->win_size, + sizeof(*s->window_func_lut)); + if (!s->window_func_lut) + return AVERROR(ENOMEM); + generate_window_func(s->window_func_lut, s->win_size, s->win_func, &overlap); + if (s->overlap == 1) + s->overlap = overlap; + + s->hop_size = (1.f - s->overlap) * s->win_size; + if (s->hop_size < 1) { + av_log(ctx, AV_LOG_ERROR, "overlap %f too big\n", s->overlap); + return AVERROR(EINVAL); + } + } + + outlink->time_base = av_inv_q(outlink->frame_rate); + + av_audio_fifo_free(s->fifo); + s->fifo = av_audio_fifo_alloc(inlink->format, inlink->channels, s->win_size); + if (!s->fifo) + return AVERROR(ENOMEM); + return 0; +} + +#define RE(y, ch) s->fft_data[ch][y].re +#define IM(y, ch) s->fft_data[ch][y].im + +static void draw_dot(uint8_t *dst, int linesize, int value) +{ + dst[0] = value; + dst[1] = value; + dst[-1] = value; + dst[linesize] = value; + dst[-linesize] = value; +} + +static int draw_spatial(AVFilterLink *inlink, AVFrame *insamples) +{ + AVFilterContext *ctx = inlink->dst; + AVFilterLink *outlink = ctx->outputs[0]; + ShowSpatialContext *s = ctx->priv; + AVFrame *outpicref; + int h = s->h - 2; + int w = s->w - 2; + int z = s->win_size / 2; + + outpicref = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!outpicref) + return AVERROR(ENOMEM); + + outpicref->sample_aspect_ratio = (AVRational){1,1}; + for (int i = 0; i < outlink->h; i++) { + memset(outpicref->data[0] + i * outpicref->linesize[0], 0, outlink->w); + memset(outpicref->data[1] + i * outpicref->linesize[1], 0, outlink->w); + memset(outpicref->data[2] + i * outpicref->linesize[2], 0, outlink->w); + } + + for (int j = 0; j < z; j++) { + const int idx = z - 1 - j; + float l = hypotf(RE(idx, 0), IM(idx, 0)); + float r = hypotf(RE(idx, 1), IM(idx, 1)); + float sum = l + r; + float lp = atan2f(IM(idx, 0), RE(idx, 0)); + float rp = atan2f(IM(idx, 1), RE(idx, 1)); + float diffp = fabsf(lp - rp) / M_PI; + float diff = (sum < 0.000001f ? FFDIFFSIGN(r, l) : (r - l) / sum) * 0.5 + 0.5f; + float cr = av_clipf(cbrtf(l / sum), 0, 1) * 255.f; + float cb = av_clipf(cbrtf(r / sum), 0, 1) * 255.f; + float cg; + int x, y; + + if (diffp > 1.f) + diffp = 2.f - diffp; + + cg = diffp * 255.f; + x = av_clip(w * diff, 0, w - 1) + 1; + y = av_clip(h * diffp, 0, h - 1) + 1; + + draw_dot(outpicref->data[0] + outpicref->linesize[0] * y + x, outpicref->linesize[0], cg); + draw_dot(outpicref->data[1] + outpicref->linesize[1] * y + x, outpicref->linesize[1], cb); + draw_dot(outpicref->data[2] + outpicref->linesize[2] * y + x, outpicref->linesize[2], cr); + } + + outpicref->pts = av_rescale_q(insamples->pts, inlink->time_base, outlink->time_base); + + return ff_filter_frame(outlink, outpicref); +} + +static int spatial_activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + ShowSpatialContext *s = ctx->priv; + int ret; + + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + if (av_audio_fifo_size(s->fifo) < s->win_size) { + AVFrame *frame = NULL; + + ret = ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + if (ret > 0) { + s->pts = frame->pts; + s->consumed = 0; + + av_audio_fifo_write(s->fifo, (void **)frame->extended_data, frame->nb_samples); + av_frame_free(&frame); + } + } + + if (av_audio_fifo_size(s->fifo) >= s->win_size) { + AVFrame *fin = ff_get_audio_buffer(inlink, s->win_size); + if (!fin) + return AVERROR(ENOMEM); + + fin->pts = s->pts + s->consumed; + s->consumed += s->hop_size; + ret = av_audio_fifo_peek(s->fifo, (void **)fin->extended_data, + FFMIN(s->win_size, av_audio_fifo_size(s->fifo))); + if (ret < 0) { + av_frame_free(&fin); + return ret; + } + + av_assert0(fin->nb_samples == s->win_size); + + ctx->internal->execute(ctx, run_channel_fft, fin, NULL, 2); + + ret = draw_spatial(inlink, fin); + + av_frame_free(&fin); + av_audio_fifo_drain(s->fifo, s->hop_size); + if (ret <= 0) + return ret; + } + + FF_FILTER_FORWARD_STATUS(inlink, outlink); + if (ff_outlink_frame_wanted(outlink) && av_audio_fifo_size(s->fifo) < s->win_size) { + ff_inlink_request_frame(inlink); + return 0; + } + + if (av_audio_fifo_size(s->fifo) >= s->win_size) { + ff_filter_set_ready(ctx, 10); + return 0; + } + return FFERROR_NOT_READY; +} + +static const AVFilterPad showspatial_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_AUDIO, + }, + { NULL } +}; + +static const AVFilterPad showspatial_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, + { NULL } +}; + +AVFilter ff_avf_showspatial = { + .name = "showspatial", + .description = NULL_IF_CONFIG_SMALL("Convert input audio to a spatial video output."), + .uninit = uninit, + .query_formats = query_formats, + .priv_size = sizeof(ShowSpatialContext), + .inputs = showspatial_inputs, + .outputs = showspatial_outputs, + .activate = spatial_activate, + .priv_class = &showspatial_class, + .flags = AVFILTER_FLAG_SLICE_THREADS, +};