From patchwork Sat Nov 10 17:10:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul B Mahol X-Patchwork-Id: 10972 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id F070844C8FB for ; Sat, 10 Nov 2018 19:15:44 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5F3B0689BB8; Sat, 10 Nov 2018 19:15:16 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7E003688263 for ; Sat, 10 Nov 2018 19:15:09 +0200 (EET) Received: by mail-wr1-f66.google.com with SMTP id j26-v6so5028899wre.1 for ; Sat, 10 Nov 2018 09:15:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=+e/eGon39iekflVqHN3Xxuc2HFQUyUes8mllhsiohFc=; b=ts1jNOzEoZaaP3gMsQjRA9coTOm4GKJ4LgGc+AQSkftLvU2/d7gVyowLcYLkPisulv 9WB+/L+jYdxhi8WzkubCHaSfV66wcnkQl3i0DpxE0AvgojwFm8vLnm3OLkloX2e0F2k5 oj4vtaD6fcgZQy1BWu9PnlCu6kpgGh8iKlWOOw3msfQwYP+pYIdhXoKmV6D/Cf4PMUYP 7vEMbXWFTzGm0ZLClkH7JlObElxCDlb3qvCcb/d3X4PMl8dVS5pGg+tWtiGqLxWPngEr gbn5kAdBbW9IRu9TEaAy6XUDcnqNXtBH8Ed0O+A7C96/HUSrXw3T83UE0cODsob0v5iA +ebQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=+e/eGon39iekflVqHN3Xxuc2HFQUyUes8mllhsiohFc=; b=BSUgDyiguqN9bUqNN9rXH9Qi1A4wNE73dB3IJOE3b7jeZKnOzNxUL9ri2k7fR+CkHx NUudHyYJoDj0KEr5qxaX6SdW4N3sk7Fi1vyvV4icBFNJWRJhy5OwluZnsjw5ceI6GuAV KeENSU9M3ZUEP7WoQGR7oVOUr+9zYHXGcgVeYsYlDQMmDsT6C9iiMoxZwK+WiTY7eTVB t7BXYKMF226c78udrYlAqA/bTpnbIaDGFe9+umZQF3vSDowAyNLHuxXlwnphmuplnMq0 Yp+k17PpiXGv1kSkT5FOkn08mB6Kl6QOlWX2x3oJQMlzFw3edDNo+173jveItOD5yg26 wnug== X-Gm-Message-State: AGRZ1gJyZWxLRR3Ok4fFE4cLgrOZoAWL4knzRxnKIxvFA+eXvta4KDqQ iqzt8tAuYMYNZV/d7WHhfhvDLtsRB94= X-Google-Smtp-Source: AJdET5fO8w0nvy3aGQEKRZFBZuwIrKGXkW+qfGlssc/IKjhaFaty0IU0vO+nLdHC1BYDGCs8xVA5ZQ== X-Received: by 2002:adf:f589:: with SMTP id f9-v6mr12317350wro.281.1541869830930; Sat, 10 Nov 2018 09:10:30 -0800 (PST) Received: from localhost.localdomain ([94.250.174.60]) by smtp.gmail.com with ESMTPSA id i7-v6sm8755784wrs.55.2018.11.10.09.10.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Nov 2018 09:10:30 -0800 (PST) From: Paul B Mahol To: ffmpeg-devel@ffmpeg.org Date: Sat, 10 Nov 2018 18:10:12 +0100 Message-Id: <20181110171012.15007-1-onemda@gmail.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH] avfilter: add astretch filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Paul B Mahol --- libavfilter/Makefile | 1 + libavfilter/af_astretch.c | 330 ++++++++++++++++++++++++++++++++++++++ libavfilter/allfilters.c | 1 + 3 files changed, 332 insertions(+) create mode 100644 libavfilter/af_astretch.c diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 79a89a1ab1..4a715915fc 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -81,6 +81,7 @@ OBJS-$(CONFIG_ASIDEDATA_FILTER) += f_sidedata.o OBJS-$(CONFIG_ASPLIT_FILTER) += split.o OBJS-$(CONFIG_ASTATS_FILTER) += af_astats.o OBJS-$(CONFIG_ASTREAMSELECT_FILTER) += f_streamselect.o framesync.o +OBJS-$(CONFIG_ASTRETCH_FILTER) += af_astretch.o OBJS-$(CONFIG_ATEMPO_FILTER) += af_atempo.o OBJS-$(CONFIG_ATRIM_FILTER) += trim.o OBJS-$(CONFIG_AZMQ_FILTER) += f_zmq.o diff --git a/libavfilter/af_astretch.c b/libavfilter/af_astretch.c new file mode 100644 index 0000000000..1e39ac3163 --- /dev/null +++ b/libavfilter/af_astretch.c @@ -0,0 +1,330 @@ +/* + * Copyright (c) 2018 Paul B Mahol + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or modify it + * under the terms of the GNU Lesser General Public License as published + * by the Free Software Foundation; either version 2.1 of the License, + * or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/audio_fifo.h" +#include "libavutil/avstring.h" +#include "libavutil/lfg.h" +#include "libavutil/random_seed.h" +#include "libavfilter/internal.h" +#include "libavutil/common.h" +#include "libavutil/opt.h" +#include "libavcodec/avfft.h" +#include "audio.h" +#include "filters.h" +#include "window_func.h" + +typedef struct AStretchContext { + const AVClass *class; + double stretch; + double overlap; + int window_size; + int phase; + int win_func; + int64_t seed; + + int fft_bits; + AVAudioFifo *fifo; + double start_pos; + double displace_pos; + int64_t pts; + AVFrame *buffer; + int start, end; + float win_scale; + float *window_func_lut; + + AVLFG c; + FFTContext *fft; + FFTContext *ifft; + FFTComplex *fft_data; +} AStretchContext; + +#define OFFSET(x) offsetof(AStretchContext, x) +#define A AV_OPT_FLAG_AUDIO_PARAM|AV_OPT_FLAG_FILTERING_PARAM + +static const AVOption astretch_options[] = { + { "stretch", "set stretch factor", OFFSET(stretch), AV_OPT_TYPE_DOUBLE, {.dbl=1}, 0.001, 1000, A }, + { "overlap", "set overlap factor", OFFSET(overlap), AV_OPT_TYPE_DOUBLE, {.dbl=1}, 0, 1, A }, + { "winsize", "set window size", OFFSET(window_size), AV_OPT_TYPE_INT, {.i64=8192}, 16, 65536, A }, + { "phase", "change phase", OFFSET(phase), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, A }, + { "winfunc", "set window function", OFFSET(win_func), AV_OPT_TYPE_INT, {.i64=WFUNC_SINE}, 0, NB_WFUNC-1, A, "winfunc" }, + { "rect", "Rectangular", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_RECT}, 0, 0, A, "winfunc" }, + { "bartlett", "Bartlett", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_BARTLETT}, 0, 0, A, "winfunc" }, + { "hann", "Hann", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_HANNING}, 0, 0, A, "winfunc" }, + { "hanning", "Hanning", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_HANNING}, 0, 0, A, "winfunc" }, + { "hamming", "Hamming", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_HAMMING}, 0, 0, A, "winfunc" }, + { "blackman", "Blackman", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_BLACKMAN}, 0, 0, A, "winfunc" }, + { "welch", "Welch", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_WELCH}, 0, 0, A, "winfunc" }, + { "flattop", "Flat-top", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_FLATTOP}, 0, 0, A, "winfunc" }, + { "bharris", "Blackman-Harris", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_BHARRIS}, 0, 0, A, "winfunc" }, + { "bnuttall", "Blackman-Nuttall", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_BNUTTALL}, 0, 0, A, "winfunc" }, + { "bhann", "Bartlett-Hann", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_BHANN}, 0, 0, A, "winfunc" }, + { "sine", "Sine", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_SINE}, 0, 0, A, "winfunc" }, + { "nuttall", "Nuttall", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_NUTTALL}, 0, 0, A, "winfunc" }, + { "lanczos", "Lanczos", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_LANCZOS}, 0, 0, A, "winfunc" }, + { "gauss", "Gauss", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_GAUSS}, 0, 0, A, "winfunc" }, + { "tukey", "Tukey", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_TUKEY}, 0, 0, A, "winfunc" }, + { "dolph", "Dolph-Chebyshev", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_DOLPH}, 0, 0, A, "winfunc" }, + { "cauchy", "Cauchy", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_CAUCHY}, 0, 0, A, "winfunc" }, + { "parzen", "Parzen", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_PARZEN}, 0, 0, A, "winfunc" }, + { "poisson", "Poisson", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_POISSON}, 0, 0, A, "winfunc" }, + { "bohman", "Bohman", 0, AV_OPT_TYPE_CONST, {.i64=WFUNC_BOHMAN}, 0, 0, A, "winfunc" }, + { "seed", "set random seed", OFFSET(seed), AV_OPT_TYPE_INT64, {.i64 = -1}, -1, UINT_MAX, A }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(astretch); + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + AStretchContext *s = ctx->priv; + float overlap; + + av_lfg_init(&s->c, s->seed); + + s->fft_bits = av_log2(s->window_size); + if (s->phase) + s->window_size = 1 << s->fft_bits; + + s->fft = av_fft_init(s->fft_bits, 0); + s->ifft = av_fft_init(s->fft_bits, 1); + if (!s->ifft || !s->fft) + return AVERROR(ENOMEM); + + s->fft_data = av_calloc(s->window_size, sizeof(*s->fft_data)); + if (!s->fft_data) + return AVERROR(ENOMEM); + + s->fifo = av_audio_fifo_alloc(inlink->format, inlink->channels, s->window_size); + if (!s->fifo) + return AVERROR(ENOMEM); + + s->window_func_lut = av_realloc_f(s->window_func_lut, s->window_size, + sizeof(*s->window_func_lut)); + if (!s->window_func_lut) + return AVERROR(ENOMEM); + generate_window_func(s->window_func_lut, s->window_size, s->win_func, &overlap); + if (s->overlap == 1) + s->overlap = overlap; + + s->buffer = ff_get_audio_buffer(inlink, 2 * s->window_size); + if (!s->buffer) + return AVERROR(ENOMEM); + + s->start_pos = 0.; + s->displace_pos = (s->window_size * (1. - s->overlap)) / s->stretch; + s->pts = AV_NOPTS_VALUE; + + return 0; +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + AStretchContext *s = ctx->priv; + AVFrame *frame = NULL; + int ret = 0; + + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + if (av_audio_fifo_size(s->fifo) < s->window_size) { + ret = ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + } + + if (ret > 0) { + if (s->pts == AV_NOPTS_VALUE) + s->pts = frame->pts; + + ret = av_audio_fifo_write(s->fifo, (void **)frame->extended_data, frame->nb_samples); + av_frame_free(&frame); + if (ret < 0) + return ret; + } + + if (av_audio_fifo_size(s->fifo) >= s->window_size) { + int istart_pos, hwin = s->window_size * (1. - s->overlap); + AVFrame *in = ff_get_audio_buffer(outlink, s->window_size); + AVFrame *out = ff_get_audio_buffer(outlink, hwin); + + if (!out || !in) { + av_frame_free(&in); + av_frame_free(&out); + return AVERROR(ENOMEM); + } + + ret = av_audio_fifo_peek(s->fifo, (void **)in->extended_data, s->window_size); + if (ret < 0) { + av_frame_free(&in); + av_frame_free(&out); + return ret; + } + + if (s->phase) { + for (int ch = 0; ch < outlink->channels; ch++) { + const float *src = (float *)in->extended_data[ch]; + const float *w = s->window_func_lut; + float *ptr = (float *)s->buffer->extended_data[ch]; + FFTComplex *dst = s->fft_data; + float magnitude, phase; + + for (int n = 0; n < s->window_size; n++) { + dst[n].re = src[n] * w[n]; + dst[n].im = 0; + } + + av_fft_permute(s->fft, s->fft_data); + av_fft_calc(s->fft, s->fft_data); + + for (int n = 0; n < s->window_size; n++) { + magnitude = hypotf(dst[n].re, dst[n].im); + phase = M_PI * (((2. * av_lfg_get(&s->c)) / (UINT_MAX + 1.0)) - 1.); + dst[n].re = magnitude * cos(phase); + dst[n].im = magnitude * sin(phase); + } + + av_fft_permute(s->ifft, s->fft_data); + av_fft_calc(s->ifft, s->fft_data); + + for (int n = 0; n < s->window_size; n++) { + const float *w = s->window_func_lut; + + ptr[n] += (dst[n].re / s->window_size) * w[n]; + } + } + } else { + for (int ch = 0; ch < outlink->channels; ch++) { + const float *src = (float *)in->extended_data[ch]; + float *dst = (float *)s->buffer->extended_data[ch]; + + for (int n = 0; n < s->window_size; n++) { + dst[n] += src[n]; + } + } + } + + + s->start_pos += s->displace_pos; + istart_pos = lrint(floor(s->start_pos)); + if (istart_pos > 0) { + av_audio_fifo_drain(s->fifo, istart_pos); + s->start_pos -= istart_pos; + } + out->pts = s->pts; + s->pts += out->nb_samples; + + av_samples_copy(out->extended_data, s->buffer->extended_data, + 0, 0, hwin, s->buffer->channels, s->buffer->format); + + for (int ch = 0; ch < outlink->channels; ch++) { + float *buf = (float *)s->buffer->extended_data[ch]; + memmove(buf, buf + hwin, (s->window_size * 2 - hwin) * sizeof(*buf)); + } + + for (int ch = 0; ch < outlink->channels; ch++) { + float *buf = (float *)s->buffer->extended_data[ch]; + memset(buf + s->window_size * 2 - hwin, 0, hwin * sizeof(*buf)); + } + + av_frame_free(&in); + return ff_filter_frame(outlink, out); + } + + FF_FILTER_FORWARD_STATUS(inlink, outlink); + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterChannelLayouts *layouts; + static const enum AVSampleFormat sample_fmts[] = { + AV_SAMPLE_FMT_FLTP, + AV_SAMPLE_FMT_NONE + }; + int ret; + + layouts = ff_all_channel_counts(); + if (!layouts) + return AVERROR(ENOMEM); + ret = ff_set_common_channel_layouts(ctx, layouts); + if (ret < 0) + return ret; + + formats = ff_make_format_list(sample_fmts); + if (!formats) + return AVERROR(ENOMEM); + ret = ff_set_common_formats(ctx, formats); + if (ret < 0) + return ret; + + formats = ff_all_samplerates(); + if (!formats) + return AVERROR(ENOMEM); + return ff_set_common_samplerates(ctx, formats); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + AStretchContext *s = ctx->priv; + + av_frame_free(&s->buffer); + av_freep(&s->window_func_lut); + av_freep(&s->fft_data); + av_fft_end(s->fft); + s->fft = NULL; + av_fft_end(s->ifft); + s->ifft = NULL; + + av_audio_fifo_free(s->fifo); +} + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_AUDIO, + .config_props = config_input, + }, + { NULL } +}; + +static const AVFilterPad outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_AUDIO, + }, + { NULL } +}; + +AVFilter ff_af_astretch = { + .name = "astretch", + .description = NULL_IF_CONFIG_SMALL("Apply stretch."), + .priv_size = sizeof(AStretchContext), + .priv_class = &astretch_class, + .inputs = inputs, + .outputs = outputs, + .activate = activate, + .query_formats = query_formats, + .uninit = uninit, +}; diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 484b080dea..a43376abc7 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -74,6 +74,7 @@ extern AVFilter ff_af_asidedata; extern AVFilter ff_af_asplit; extern AVFilter ff_af_astats; extern AVFilter ff_af_astreamselect; +extern AVFilter ff_af_astretch; extern AVFilter ff_af_atempo; extern AVFilter ff_af_atrim; extern AVFilter ff_af_azmq;