From patchwork Thu May 5 21:14:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Cao X-Patchwork-Id: 35607 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:a885:b0:7f:4be2:bd17 with SMTP id ca5csp1312787pzb; Thu, 5 May 2022 14:14:38 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyrPaRLpfeNh5SCGnUnKuJnoK2CeUZ0FUYTXtAKLTBlEuU61i8Qa61Vr40nUSTYM0NqPRy/ X-Received: by 2002:a05:6402:5202:b0:427:f1a2:305c with SMTP id s2-20020a056402520200b00427f1a2305cmr144555edd.39.1651785277838; Thu, 05 May 2022 14:14:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1651785277; cv=none; d=google.com; s=arc-20160816; b=03y5gVkPx2N4FFUwhQ9X1t2OsM32sUNPMjzimIDYvOELgGVGzKpJBE5CdYaiOQS9dE XkR0Sa1H9veXtPupKdXBqSq+Bku4WJxSGMCEgYzdlJNg+hUi5Sm7mnePhh9qMAj3yY45 pHZu/MNzeRDExfH1Exeu7foYq2BSPi8KRZ0qDPcSK6FNrjG3eyi+gkn/ThRBaj6ucIBG ewMLcjyH8IB+OHxSV83GTuCkY7e05uep2ImdKVGcc+AUFVI3pGqLeWjshfzTb1swl/D8 eznB2Mk2DsLZEoW23h9p6H+MQb51vJr8d4mb37LxQcLKKswvFUNIv9nV0SqzXCy1bLog AdQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:from:references:mime-version :message-id:in-reply-to:date:dkim-signature:delivered-to; bh=F/lYeSXi5Rd8K5FSdQ3EHcp652mt7h4gP1ZPJjz0gXM=; b=yOjI99vnWf4cevuiO6J+7KE7oKkyYoJ5iN65lmoPeftLEW0gJQlbwEOglZnKyTQpDY GLwLhUJEYYl6BKf1xXGLBqfbc4uXRYkrRUx6ZazGcX9e6QRj0L889VisLkX42TyuYk80 1RqpWE0f7Iz2JeqOGoWW/q+Uxgdk3zIdAYEkjUlacWjp51txbCX5CoAJfHAifyxsK60w a0cx4jqq1gj/z77efAB6yZeDexTf6rFEpZf4qMGoRhEx1ZP5Mnx3C78HviAF4EvYFqS3 QeTDp3bFAEc24KJsgkjNrESNbw1cA5RieD6ni+LAk7MpgFQqzSkvDQfmqdyjZF9E9z9j PFVw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20210112 header.b=hh9DoOu9; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bs12-20020a170906d1cc00b006e88cc5c374si3241834ejb.179.2022.05.05.14.14.36; Thu, 05 May 2022 14:14:37 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20210112 header.b=hh9DoOu9; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 352A768B2D5; Fri, 6 May 2022 00:14:33 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5413D6800AA for ; Fri, 6 May 2022 00:14:26 +0300 (EEST) Received: by mail-yb1-f201.google.com with SMTP id b6-20020a253406000000b006484c081280so4595753yba.5 for ; Thu, 05 May 2022 14:14:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=wJ2BRFH1/Tl7EH5puuId3UVhxK+bIpfLqMwx3TBN4y4=; b=hh9DoOu91+xlBB+/p57359ehC1mS8k9hnozGYIZMMMCtvMcloPuC5zQMhsJN+gdnZs iOr8h7Tiwi5D4xFkMDBwEutWxFcIapzUJuPfVCRAN3s5Xp3LiOl+CXukDk1KJtoUHx1w a2y4THLl9OiKc38OtsbC3aQ6yyWBut9TvFknqu7jkn95ujGAkOWhwMr7dCP4DrIfs9Av C/NGyxyzk6XQ/yxSZtHeurKDyTByu8L8lR1sUXmzbxi6MvDj5RxoTFP2B/LfgM+9DQLf nPN1BN2GhoaASp0BMn+k6LOUy/RWL2Dz9VuJ1NResy8J3ekLSyHrAKYlOqU/BfAb0qZT fTyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=wJ2BRFH1/Tl7EH5puuId3UVhxK+bIpfLqMwx3TBN4y4=; b=uyFLX8TlsQTfbQUXNpvrchJ67oLvRZVKflaspmUl/UlUI9hjozm6XKt+FKZw/t/fUh a/YR+xZg5qZdLyEqd+xaJsahUo+USVZFa2ETm5nEDaIwiJ/iVSodamdYlnlcWiGVsamF KvAwqN7t3FSwkLGfd0xs7vk2Z9GIFwmqdkylzIQA0gixEX2LK2XQDjHarStBz++hF4aR LQ0Fl52jEx+WP4lZhBzPs1BtqrrdxzFT7t0RITjcU1H2fBroh4VTtixp2VzrmZ3JHPHD DSPCFSADr/iWlibs1w3/ejU58Y6T2KJthHY+K99eu1e5SjFMteTIJtfvJ1ra8KWC8uhI WqGw== X-Gm-Message-State: AOAM532MLNRMLICCQzt7pIk6FaRc4K7zo/tPVNe+KWYZiT593py8MxEJ VYat2LuZYG94vgCgyhxfrDWftxQC28J+mJAIdMZ7RV5Pw7oDS1hWjHaFbjcSNHcZFD5z7pZ4UKv S1vZ9sEeHLl0AqpNRW25UTtXXOUOXSl65sHbhUN/b7tjMoca854pIijS8wmRGdPqLdg== X-Received: from wangcao.mtv.corp.google.com ([2620:0:1000:4011:809f:157b:f896:c3a2]) (user=wangcao job=sendgmr) by 2002:a05:6902:1107:b0:649:2758:280d with SMTP id o7-20020a056902110700b006492758280dmr86979ybu.367.1651785264215; Thu, 05 May 2022 14:14:24 -0700 (PDT) Date: Thu, 5 May 2022 14:14:16 -0700 In-Reply-To: Message-Id: <20220505211416.671275-1-wangcao@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.36.0.512.ge40c2bad7a-goog From: Wang Cao To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH] avfilter/alimiter:add latency compensation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Wang Cao Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: CfpOSzzvXJ/R Also added 2 FATE tests to verify delay is compenated correctly Signed-off-by: Wang Cao --- doc/filters.texi | 5 +++ libavfilter/af_alimiter.c | 90 +++++++++++++++++++++++++++++++++++++ tests/fate/filter-audio.mak | 24 ++++++++-- 3 files changed, 116 insertions(+), 3 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index a161754233..75a43edd88 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -1978,6 +1978,11 @@ in release time while 1 produces higher release times. @item level Auto level output signal. Default is enabled. This normalizes audio back to 0dB if enabled. + +@item latency +Compensate the delay introduced by using the lookahead buffer set with attack +parameter. Also flush the valid audio data in the lookahead buffer when the +stream hits EOF @end table Depending on picked setting it is recommended to upsample input 2x or 4x times diff --git a/libavfilter/af_alimiter.c b/libavfilter/af_alimiter.c index 133f98f165..01265758d7 100644 --- a/libavfilter/af_alimiter.c +++ b/libavfilter/af_alimiter.c @@ -26,6 +26,7 @@ #include "libavutil/channel_layout.h" #include "libavutil/common.h" +#include "libavutil/fifo.h" #include "libavutil/opt.h" #include "audio.h" @@ -33,6 +34,11 @@ #include "formats.h" #include "internal.h" +typedef struct MetaItem { + int64_t pts; + int nb_samples; +} MetaItem; + typedef struct AudioLimiterContext { const AVClass *class; @@ -55,6 +61,14 @@ typedef struct AudioLimiterContext { int *nextpos; double *nextdelta; + int in_trim; + int out_pad; + int64_t next_in_pts; + int64_t next_out_pts; + int latency; + + AVFifo *fifo; + double delta; int nextiter; int nextlen; @@ -73,6 +87,7 @@ static const AVOption alimiter_options[] = { { "asc", "enable asc", OFFSET(auto_release), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AF }, { "asc_level", "set asc level", OFFSET(asc_coeff), AV_OPT_TYPE_DOUBLE, {.dbl=0.5}, 0, 1, AF }, { "level", "auto level", OFFSET(auto_level), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, AF }, + { "latency", "compensate delay", OFFSET(latency), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AF }, { NULL } }; @@ -129,6 +144,11 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) AVFrame *out; double *buf; int n, c, i; + int new_out_samples; + int64_t out_duration; + int64_t in_duration; + int64_t in_pts; + MetaItem meta; if (av_frame_is_writable(in)) { out = in; @@ -269,12 +289,69 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) dst += channels; } + in_duration = av_rescale_q(in->nb_samples, inlink->time_base, av_make_q(1, in->sample_rate)); + in_pts = in->pts; + meta = (MetaItem){ in->pts, in->nb_samples }; + av_fifo_write(s->fifo, &meta, 1); if (in != out) av_frame_free(&in); + new_out_samples = out->nb_samples; + if (s->in_trim > 0) { + int trim = FFMIN(new_out_samples, s->in_trim); + new_out_samples -= trim; + s->in_trim -= trim; + } + + if (new_out_samples <= 0) { + av_frame_free(&out); + return 0; + } else if (new_out_samples < out->nb_samples) { + int offset = out->nb_samples - new_out_samples; + memmove(out->extended_data[0], out->extended_data[0] + sizeof(double) * offset * out->ch_layout.nb_channels, + sizeof(double) * new_out_samples * out->ch_layout.nb_channels); + out->nb_samples = new_out_samples; + s->in_trim = 0; + } + + av_fifo_read(s->fifo, &meta, 1); + + out_duration = av_rescale_q(out->nb_samples, inlink->time_base, av_make_q(1, out->sample_rate)); + in_duration = av_rescale_q(meta.nb_samples, inlink->time_base, av_make_q(1, out->sample_rate)); + in_pts = meta.pts; + + if (s->next_out_pts != AV_NOPTS_VALUE && out->pts != s->next_out_pts && + s->next_in_pts != AV_NOPTS_VALUE && in_pts == s->next_in_pts) { + out->pts = s->next_out_pts; + } else { + out->pts = in_pts; + } + s->next_in_pts = in_pts + in_duration; + s->next_out_pts = out->pts + out_duration; + return ff_filter_frame(outlink, out); } +static int request_frame(AVFilterLink* outlink) +{ + AVFilterContext *ctx = outlink->src; + AudioLimiterContext *s = (AudioLimiterContext*)ctx->priv; + int ret; + + ret = ff_request_frame(ctx->inputs[0]); + + if (ret == AVERROR_EOF && s->out_pad > 0) { + AVFrame *frame = ff_get_audio_buffer(outlink, FFMIN(1024, s->out_pad)); + if (!frame) + return AVERROR(ENOMEM); + + s->out_pad -= frame->nb_samples; + frame->pts = s->next_in_pts; + return filter_frame(ctx->inputs[0], frame); + } + return ret; +} + static int config_input(AVFilterLink *inlink) { AVFilterContext *ctx = inlink->dst; @@ -294,6 +371,16 @@ static int config_input(AVFilterLink *inlink) memset(s->nextpos, -1, obuffer_size * sizeof(*s->nextpos)); s->buffer_size = inlink->sample_rate * s->attack * inlink->ch_layout.nb_channels; s->buffer_size -= s->buffer_size % inlink->ch_layout.nb_channels; + if (s->latency) { + s->in_trim = s->out_pad = s->buffer_size / inlink->ch_layout.nb_channels - 1; + } + s->next_out_pts = AV_NOPTS_VALUE; + s->next_in_pts = AV_NOPTS_VALUE; + + s->fifo = av_fifo_alloc2(8, sizeof(MetaItem), AV_FIFO_FLAG_AUTO_GROW); + if (!s->fifo) { + return AVERROR(ENOMEM); + } if (s->buffer_size <= 0) { av_log(ctx, AV_LOG_ERROR, "Attack is too small.\n"); @@ -310,6 +397,8 @@ static av_cold void uninit(AVFilterContext *ctx) av_freep(&s->buffer); av_freep(&s->nextdelta); av_freep(&s->nextpos); + + av_fifo_freep2(&s->fifo); } static const AVFilterPad alimiter_inputs[] = { @@ -325,6 +414,7 @@ static const AVFilterPad alimiter_outputs[] = { { .name = "default", .type = AVMEDIA_TYPE_AUDIO, + .request_frame = request_frame, }, }; diff --git a/tests/fate/filter-audio.mak b/tests/fate/filter-audio.mak index eff32b9f81..e33ffdf37f 100644 --- a/tests/fate/filter-audio.mak +++ b/tests/fate/filter-audio.mak @@ -63,11 +63,29 @@ fate-filter-agate: tests/data/asynth-44100-2.wav fate-filter-agate: SRC = $(TARGET_PATH)/tests/data/asynth-44100-2.wav fate-filter-agate: CMD = framecrc -i $(SRC) -af aresample,agate=level_in=10:range=0:threshold=1:ratio=1:attack=1:knee=1:makeup=4,aresample -FATE_AFILTER-$(call FILTERDEMDECENCMUX, AFADE, WAV, PCM_S16LE, PCM_S16LE, WAV) += fate-filter-alimiter -fate-filter-alimiter: tests/data/asynth-44100-2.wav -fate-filter-alimiter: SRC = $(TARGET_PATH)/tests/data/asynth-44100-2.wav +tests/data/filter-alimiter-passthrough: TAG = GEN +tests/data/filter-alimiter-passthrough: ffmpeg$(PROGSSUF)$(EXESUF) | tests/data + $(M)$(TARGET_EXEC) $(TARGET_PATH)/$< -nostdin \ + -i $(TARGET_PATH)/tests/data/asynth-44100-2.wav -af aresample -f crc $(TARGET_PATH)/$@ -y 2>/dev/null + +FATE_ALIMITER += fate-filter-alimiter-passthrough-default-attack +fate-filter-alimiter-passthrough-default-attack: tests/data/filter-alimiter-passthrough +fate-filter-alimiter-passthrough-default-attack: REF = $(TARGET_PATH)/tests/data/filter-alimiter-passthrough +fate-filter-alimiter-passthrough-default-attack: CMD = crc -i $(SRC) -af aresample,alimiter=level_in=1:level_out=1:limit=1:level=0:latency=1,aresample + +FATE_ALIMITER += fate-filter-alimiter-passthrough-large-attack +fate-filter-alimiter-passthrough-large-attack: tests/data/filter-alimiter-passthrough +fate-filter-alimiter-passthrough-large-attack: REF = $(TARGET_PATH)/tests/data/filter-alimiter-passthrough +fate-filter-alimiter-passthrough-large-attack: CMD = crc -i $(SRC) -af aresample,alimiter=level_in=1:level_out=1:limit=1:level=0:latency=1:attack=80,aresample + +FATE_ALIMITER += fate-filter-alimiter fate-filter-alimiter: CMD = framecrc -i $(SRC) -af aresample,alimiter=level_in=1:level_out=2:limit=0.2,aresample +$(FATE_ALIMITER): tests/data/asynth-44100-2.wav +$(FATE_ALIMITER): SRC = $(TARGET_PATH)/tests/data/asynth-44100-2.wav + +FATE_AFILTER-$(call FILTERDEMDECENCMUX, ATRIM, WAV, PCM_S16LE, PCM_S16LE, WAV) += $(FATE_ALIMITER) + FATE_AFILTER-$(call FILTERDEMDECENCMUX, AMERGE, WAV, PCM_S16LE, PCM_S16LE, WAV) += fate-filter-amerge fate-filter-amerge: tests/data/asynth-44100-1.wav fate-filter-amerge: SRC = $(TARGET_PATH)/tests/data/asynth-44100-1.wav