From patchwork Fri Apr 15 18:49:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Cao X-Patchwork-Id: 35330 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:671c:b0:7c:62c8:b2d1 with SMTP id q28csp414077pzh; Fri, 15 Apr 2022 11:50:53 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw4bPklchrtI73Yn+heZUM7wCgVTQz8rbkkbm9DRO+D007zzjkIg5biqz16h5eM8NWYlJr+ X-Received: by 2002:a05:6402:2065:b0:407:eb07:740 with SMTP id bd5-20020a056402206500b00407eb070740mr563225edb.406.1650048652993; Fri, 15 Apr 2022 11:50:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1650048652; cv=none; d=google.com; s=arc-20160816; b=QOy1AQ80bemyqLww+HaBzh3qBWhyiamQL3tW9EQLzj0UBp1ImAPXs8ar665m8L3dSx JTViQA3KV7zZgtRMO4k1JFu4eVxxWvqpPkKw2KbeBQZFeZP9RiOAV4MzKynB90QwAfCS HCVOvFG0oZHsMQIYWxdC5Ozz3M5mzWimayCcskfpeFC1jsog5PW9HKkRoOapvBJc2BQr yL5C3xu3OvzNHyoF5ZSBgUPkhl6iozzbdpO8aBb43X9pm9ofTgNhrbTIYicTKxtyqdMv wDG++ePAMLCxer0cXe6M7jpyTjU+ERfMmbIf+/apV6Wd9PntCniU4mgGZZCirmtOGTRN /v5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:from:references:mime-version :message-id:in-reply-to:date:dkim-signature:delivered-to; bh=yiKCfeS3x2V8ofZmSPyuzjK3LXkwkEULycx0ewNw3Co=; b=J/N8WT8WzPgrYaGc0u2mjv7D+rHy/WlQ6KvJZ/ZhNtAGJrub++pC+EJwCjsqtiLV2q Qz0JHA7o//rNn6tt1qUKBM1K8iv0pgV36ZQc9DOENQg0DNZ5GmgnyRWtYu+99ycDNGGt FuyqaHXtbsPRvP9pzx9FTXlEzhAzy40s3QiQUT/gyXqriA4MwqNnv5Cnrfv+TpLHnMsQ Ja8475xMXhlHf+gMd1GUX2FU4v+IHd8I0kQJb2IUe6qKS0KLpXwUAs50Cojs6d/WvQlN ZX3bxU4a3BaJLrI+SzyV8b5jBGbNaNFYcWjsHyX3MNkbUMhwg+nUNacWHlzJayZ+o4GF 4XOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20210112 header.b=mR0diJI6; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id g12-20020a056402090c00b0041cdc07ff86si1734106edz.584.2022.04.15.11.50.52; Fri, 15 Apr 2022 11:50:52 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20210112 header.b=mR0diJI6; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CF5D368B4DB; Fri, 15 Apr 2022 21:50:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3657768B45D for ; Fri, 15 Apr 2022 21:50:42 +0300 (EEST) Received: by mail-yb1-f201.google.com with SMTP id q142-20020a25d994000000b00641d23dbeb9so7034306ybg.9 for ; Fri, 15 Apr 2022 11:50:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=wgTJ9HaZdr1a/u3IxQmUww1A4c9hoHbah9pE7e2Z4+0=; b=mR0diJI6w+ANg6rsGvIX3wY7sd6o/1j5Xx90vDY9WPhsKWzHJzry3864yWimJBvZ7x 57X5+ubTh5i+e77MksEO2DFx21c7vtW8GddCB7mHwLHU6FR7ZfUqHualTvO55Bm52/F7 r6jsn8jL/B3uxdu5Pw/qKbs62DH1lJeYXnsz0UWY79EcGtQgX/2QHf+mYtR6RK+ZVd7n NSg10f4/yEBh54M8sj+4lmIU4BUAG/rz9lNRRMUfRlPOSUFJKtqDAHJg/BA4DOKwVlbg nF8Yl5dW1l71xAIsdpUSdNIbN/Il97fxmXZun1XM6MZELFY1qLF92m/+w9THeOpXbjXv cL7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=wgTJ9HaZdr1a/u3IxQmUww1A4c9hoHbah9pE7e2Z4+0=; b=FRMf0TwxfYfu+JzpLhCfZnlV9HdPI9wbPgDpanjcnK1ns21DHNROV2vdULxVnR7B3O nCEzgQfxaNaTmErFfo1qS9qY7c+7fNaCjHojBhS7V9nOfqU5zY2bj7JpTgexDxsC7H0N f/NZ/PFtlgMkC7mb//oQFJxrTvkcNlOsEy3nZACVN5D1E79SopTtlYyPaNcDNGKZLiLN rV4CV/3w92LH0KUJX84qNJfeSU9s7akJ8KWvpS2xk2FgXBDthDp0AhrWQOBh53CZGS8O dkP1qcAbbbKRqMi540djXK8YQg6mjN+ltEZ3uDBbLO7PSAFTMAw+ffeSFE8GEoTib0zZ Gg+Q== X-Gm-Message-State: AOAM530KBPbdde9Vh0++vzMDVHwT9YiGTG3JLnlNzb4Cb/QhJKCb4+Np Fq1FSUPqkVclLLStoRhxehgdaccFQ3Pvu2wBr2Ngte9Oz9qXG112dsg+Rudy50f5139040wKqP6 63ycLxnSHu90iyHGV9XKk84IDML/bXfi1uSuBRMtVsT0nj295wSOCBW6kVmKO0BKtXQ== X-Received: from wangcao.mtv.corp.google.com ([2620:0:1000:4011:9c0:7189:8016:d159]) (user=wangcao job=sendgmr) by 2002:a25:312:0:b0:641:3cfa:372d with SMTP id 18-20020a250312000000b006413cfa372dmr524618ybd.580.1650048640560; Fri, 15 Apr 2022 11:50:40 -0700 (PDT) Date: Fri, 15 Apr 2022 11:49:35 -0700 In-Reply-To: Message-Id: <20220415184934.1891758-1-wangcao@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog From: Wang Cao To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH] avfilter/alimiter: Add an option "comp_delay" that removes the delay introduced by lookahead buffer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Wang Cao Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: zMZYWvJprPQc 1. The option also flushes all the valid audio samples in the lookahead buffer so the audio integrity is preserved. Previously the the output audio will lose the amount of audio samples equal to the size of lookahead buffer 2. Add a FATE test to verify that when the filter is working as passthrough filter, all audio samples are properly handled from the input to the output. Signed-off-by: Wang Cao --- doc/filters.texi | 5 +++ libavfilter/af_alimiter.c | 74 +++++++++++++++++++++++++++++++++++++ tests/fate/filter-audio.mak | 12 ++++++ 3 files changed, 91 insertions(+) -- 2.36.0.rc0.470.gd361397f0d-goog diff --git a/doc/filters.texi b/doc/filters.texi index a161754233..2af0953c89 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -1978,6 +1978,11 @@ in release time while 1 produces higher release times. @item level Auto level output signal. Default is enabled. This normalizes audio back to 0dB if enabled. + +@item comp_delay +Compensate the delay introduced by using the lookahead buffer set with attack +parameter. Also flush the valid audio data in the lookahead buffer when the +stream hits EOF. @end table Depending on picked setting it is recommended to upsample input 2x or 4x times diff --git a/libavfilter/af_alimiter.c b/libavfilter/af_alimiter.c index 133f98f165..d10a90859b 100644 --- a/libavfilter/af_alimiter.c +++ b/libavfilter/af_alimiter.c @@ -55,6 +55,12 @@ typedef struct AudioLimiterContext { int *nextpos; double *nextdelta; + int lookahead_delay_samples; + int lookahead_flush_samples; + int64_t output_pts; + int64_t next_output_pts; + int comp_delay; + double delta; int nextiter; int nextlen; @@ -73,6 +79,7 @@ static const AVOption alimiter_options[] = { { "asc", "enable asc", OFFSET(auto_release), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AF }, { "asc_level", "set asc level", OFFSET(asc_coeff), AV_OPT_TYPE_DOUBLE, {.dbl=0.5}, 0, 1, AF }, { "level", "auto level", OFFSET(auto_level), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, AF }, + { "comp_delay","compensate delay", OFFSET(comp_delay), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AF }, { NULL } }; @@ -129,6 +136,8 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) AVFrame *out; double *buf; int n, c, i; + int num_output_samples = in->nb_samples; + int trim_offset; if (av_frame_is_writable(in)) { out = in; @@ -271,10 +280,71 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) if (in != out) av_frame_free(&in); + + if (!s->comp_delay) { + return ff_filter_frame(outlink, out); + } + + if (s->output_pts == AV_NOPTS_VALUE) { + s->output_pts = in->pts; + } + + if (s->lookahead_delay_samples > 0) { + // The current output frame is completely silence + if (s->lookahead_delay_samples >= in->nb_samples) { + s->lookahead_delay_samples -= in->nb_samples; + return 0; + } + + // Trim the silence part + trim_offset = av_samples_get_buffer_size( + NULL, inlink->ch_layout.nb_channels, s->lookahead_delay_samples, + (enum AVSampleFormat)out->format, 1); + out->data[0] += trim_offset; + out->nb_samples = in->nb_samples - s->lookahead_delay_samples; + s->lookahead_delay_samples = 0; + num_output_samples = out->nb_samples; + } + + if (s->lookahead_delay_samples < 0) { + return AVERROR_BUG; + } + + out->pts = s->output_pts; + s->next_output_pts = s->output_pts + num_output_samples; + s->output_pts = s->next_output_pts; return ff_filter_frame(outlink, out); } +static int request_frame(AVFilterLink* outlink) +{ + AVFilterContext *ctx = outlink->src; + AudioLimiterContext *s = (AudioLimiterContext*)ctx->priv; + int ret; + AVFilterLink *inlink; + AVFrame *silence_frame; + + ret = ff_request_frame(ctx->inputs[0]); + + if (ret != AVERROR_EOF || s->lookahead_flush_samples == 0 || !s->comp_delay) { + // Not necessarily an error, just not EOF. + return ret; + } + + // We reach here when input filters have finished producing data (i.e. EOF), + // but because of the attack param, s->buffer still has meaningful + // audio content that needs flushing. + inlink = ctx->inputs[0]; + // Pushes silence frame to flush valid audio in the s->buffer + silence_frame = ff_get_audio_buffer(inlink, s->lookahead_flush_samples); + ret = filter_frame(inlink, silence_frame); + if (ret < 0) { + return ret; + } + return AVERROR_EOF; +} + static int config_input(AVFilterLink *inlink) { AVFilterContext *ctx = inlink->dst; @@ -294,6 +364,9 @@ static int config_input(AVFilterLink *inlink) memset(s->nextpos, -1, obuffer_size * sizeof(*s->nextpos)); s->buffer_size = inlink->sample_rate * s->attack * inlink->ch_layout.nb_channels; s->buffer_size -= s->buffer_size % inlink->ch_layout.nb_channels; + // the current logic outputs the next sample from the lookahead buffer from the beginning so the amount of delay + // compensation is less than the lookahead buffer size by 1 . + s->lookahead_delay_samples = s->lookahead_flush_samples = s->buffer_size / inlink->ch_layout.nb_channels - 1; if (s->buffer_size <= 0) { av_log(ctx, AV_LOG_ERROR, "Attack is too small.\n"); @@ -325,6 +398,7 @@ static const AVFilterPad alimiter_outputs[] = { { .name = "default", .type = AVMEDIA_TYPE_AUDIO, + .request_frame = request_frame, }, }; diff --git a/tests/fate/filter-audio.mak b/tests/fate/filter-audio.mak index eff32b9f81..3a51ca18a6 100644 --- a/tests/fate/filter-audio.mak +++ b/tests/fate/filter-audio.mak @@ -63,6 +63,18 @@ fate-filter-agate: tests/data/asynth-44100-2.wav fate-filter-agate: SRC = $(TARGET_PATH)/tests/data/asynth-44100-2.wav fate-filter-agate: CMD = framecrc -i $(SRC) -af aresample,agate=level_in=10:range=0:threshold=1:ratio=1:attack=1:knee=1:makeup=4,aresample +tests/data/filter-alimiter-passthrough: TAG = GEN +tests/data/filter-alimiter-passthrough: ffmpeg$(PROGSSUF)$(EXESUF) | tests/data + $(M)$(TARGET_EXEC) $(TARGET_PATH)/$< -nostdin \ + -i $(TARGET_PATH)/tests/data/asynth-44100-2.wav -af aresample -f crc $(TARGET_PATH)/$@ -y 2>/dev/null + +FATE_AFILTER-$(call FILTERDEMDECENCMUX, AFADE, WAV, PCM_S16LE, PCM_S16LE, WAV) += fate-filter-alimiter-passthrough +fate-filter-alimiter-passthrough: tests/data/asynth-44100-2.wav +fate-filter-alimiter-passthrough: tests/data/filter-alimiter-passthrough +fate-filter-alimiter-passthrough: SRC = $(TARGET_PATH)/tests/data/asynth-44100-2.wav +fate-filter-alimiter-passthrough: REF = $(TARGET_PATH)/tests/data/filter-alimiter-passthrough +fate-filter-alimiter-passthrough: CMD = crc -i $(SRC) -af aresample,alimiter=level_in=1:level_out=1:limit=1:level=0:comp_delay=1,aresample + FATE_AFILTER-$(call FILTERDEMDECENCMUX, AFADE, WAV, PCM_S16LE, PCM_S16LE, WAV) += fate-filter-alimiter fate-filter-alimiter: tests/data/asynth-44100-2.wav fate-filter-alimiter: SRC = $(TARGET_PATH)/tests/data/asynth-44100-2.wav