From patchwork Sat Mar 25 19:15:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 40826 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:7a30:b0:df:834d:2c1a with SMTP id t48csp225797pzh; Sat, 25 Mar 2023 12:17:17 -0700 (PDT) X-Google-Smtp-Source: AKy350ZyaLtOZpCBg/kKXv5o2BDHwF3Tx+4+stkoeBPBOfBdaUN/v5sgA0jYHweRvLmV2U9YUZ7B X-Received: by 2002:a17:906:148d:b0:8b1:7e23:5041 with SMTP id x13-20020a170906148d00b008b17e235041mr6564581ejc.39.1679771836842; Sat, 25 Mar 2023 12:17:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679771836; cv=none; d=google.com; s=arc-20160816; b=hpu2/mQGw6ZK3XdgqwZwjIfnH74d2ZGTKJXaILMYWgIJK8HgN8DAvLGyMiEmwWbFQ8 HgOJK3hB1CIirMXD9zdx2tnRLKrhMxvyX9lDjEgK8x+06Xbp2g1ODAs8o0cxcWoMdGpI NU1zB7GG/ClOvKobYrmBVJZ/HpJXtqvtxFbmBXMZDh5qyibP44KI4vGQpCbPH33Fugan qy1Ne0HIoiuF9FSnEEqX+EZ+gsc59Ptt9JjArW7VLtLOE9vhU42yLVJvqZxeGafzGyyS 9XXrPiBQXGUxZv2I29YNzwQPxbFifidvZaMeg60oMStXgwrsbzDvs0G/Znt0+PscGqce Odzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=ZJQ8uz0NEw6vFGbv+zRgksHZTomhS1oBMrnhvf2Zp2Q=; b=VZoq9CVod3VH2L3OqCoDxnu0O2/IqbT/VtSyaAL/Tq4ypWIjAyMnaB1YIBbMVY5BLy qcocVGG5eCbItZSRihhsCHap7rwnih3kbKN1TkbqdrmIowJUb0u9N2WmD1U7qGFaTJfy R56+eefQF3AcaZYvbYtY0harBKbN0dF83QbJ2RiifMvqQiN2f5s4twyuKVBCsFyULPyy U1jj9KA3X5bWyhPEUiIozUCYK8K379SB/TEANYGVZLCOxfvF7sI8v0Q8R+fi9qBSdGNT iLK+2KZyBX+lfiofW6gstU02xTqvL7wy23QlH7OmEjKXGoOaoswfdWxueogvJ0t7E8od PIgw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id la21-20020a170906ad9500b009309c7f5650si2949614ejb.23.2023.03.25.12.17.16; Sat, 25 Mar 2023 12:17:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 91BFF68C9BB; Sat, 25 Mar 2023 21:16:16 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail0.khirnov.net (red.khirnov.net [176.97.15.12]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1191068BEC9 for ; Sat, 25 Mar 2023 21:16:08 +0200 (EET) Received: from localhost (localhost [IPv6:::1]) by mail0.khirnov.net (Postfix) with ESMTP id B5ABC2404F5 for ; Sat, 25 Mar 2023 20:16:08 +0100 (CET) Received: from mail0.khirnov.net ([IPv6:::1]) by localhost (mail0.khirnov.net [IPv6:::1]) (amavisd-new, port 10024) with ESMTP id NTbrRfUVQTIB for ; Sat, 25 Mar 2023 20:16:07 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail0.khirnov.net (Postfix) with ESMTPS id 34F742406D0 for ; Sat, 25 Mar 2023 20:16:01 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:::1]) by libav.khirnov.net (Postfix) with ESMTP id 9DA373A0586 for ; Sat, 25 Mar 2023 20:15:54 +0100 (CET) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Sat, 25 Mar 2023 20:15:13 +0100 Message-Id: <20230325191529.10578-7-anton@khirnov.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230325191529.10578-1-anton@khirnov.net> References: <20230325191529.10578-1-anton@khirnov.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 07/23] fftools/sync_queue: allow requesting a specific number of audio samples X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: dHSZSQfKb9oT This will be made useful in following commits. --- fftools/sync_queue.c | 164 ++++++++++++++++++++++++++++++++++++++++--- fftools/sync_queue.h | 10 +++ 2 files changed, 165 insertions(+), 9 deletions(-) diff --git a/fftools/sync_queue.c b/fftools/sync_queue.c index 5b98253a4a..758357940f 100644 --- a/fftools/sync_queue.c +++ b/fftools/sync_queue.c @@ -20,10 +20,13 @@ #include #include "libavutil/avassert.h" +#include "libavutil/channel_layout.h" +#include "libavutil/cpu.h" #include "libavutil/error.h" #include "libavutil/fifo.h" #include "libavutil/mathematics.h" #include "libavutil/mem.h" +#include "libavutil/samplefmt.h" #include "objpool.h" #include "sync_queue.h" @@ -67,6 +70,8 @@ typedef struct SyncQueueStream { AVFifo *fifo; AVRational tb; + /* number of audio samples in fifo */ + uint64_t samples_queued; /* stream head: largest timestamp seen */ int64_t head_ts; int limiting; @@ -74,7 +79,9 @@ typedef struct SyncQueueStream { int finished; uint64_t frames_sent; + uint64_t samples_sent; uint64_t frames_max; + int frame_samples; } SyncQueueStream; struct SyncQueue { @@ -109,8 +116,18 @@ static void frame_move(const SyncQueue *sq, SyncQueueFrame dst, av_frame_move_ref(dst.f, src.f); } -static int64_t frame_ts(const SyncQueue *sq, SyncQueueFrame frame) +/** + * Compute the end timestamp of a frame. If nb_samples is provided, consider + * the frame to have this number of audio samples, otherwise use frame duration. + */ +static int64_t frame_end(const SyncQueue *sq, SyncQueueFrame frame, int nb_samples) { + if (nb_samples) { + int64_t d = av_rescale_q(nb_samples, (AVRational){ 1, frame.f->sample_rate}, + frame.f->time_base); + return frame.f->pts + d; + } + return (sq->type == SYNC_QUEUE_PACKETS) ? frame.p->pts + frame.p->duration : frame.f->pts + frame.f->duration; @@ -265,7 +282,7 @@ static int overflow_heartbeat(SyncQueue *sq, int stream_idx) /* get the chosen stream's tail timestamp */ for (size_t i = 0; tail_ts == AV_NOPTS_VALUE && av_fifo_peek(st->fifo, &frame, 1, i) >= 0; i++) - tail_ts = frame_ts(sq, frame); + tail_ts = frame_end(sq, frame, 0); /* overflow triggers when the tail is over specified duration behind the head */ if (tail_ts == AV_NOPTS_VALUE || tail_ts >= st->head_ts || @@ -326,7 +343,7 @@ int sq_send(SyncQueue *sq, unsigned int stream_idx, SyncQueueFrame frame) dst.f->time_base); } - ts = frame_ts(sq, dst); + ts = frame_end(sq, dst, 0); ret = av_fifo_write(st->fifo, &dst, 1); if (ret < 0) { @@ -337,13 +354,116 @@ int sq_send(SyncQueue *sq, unsigned int stream_idx, SyncQueueFrame frame) stream_update_ts(sq, stream_idx, ts); - st->frames_sent++; + st->samples_queued += nb_samples; + st->samples_sent += nb_samples; + + if (st->frame_samples) + st->frames_sent = st->samples_sent / st->frame_samples; + else + st->frames_sent++; + if (st->frames_sent >= st->frames_max) finish_stream(sq, stream_idx); return 0; } +static void offset_audio(AVFrame *f, int nb_samples) +{ + const int planar = av_sample_fmt_is_planar(f->format); + const int planes = planar ? f->ch_layout.nb_channels : 1; + const int bps = av_get_bytes_per_sample(f->format); + const int offset = nb_samples * bps * (planar ? 1 : f->ch_layout.nb_channels); + + av_assert0(bps > 0); + av_assert0(nb_samples < f->nb_samples); + + for (int i = 0; i < planes; i++) { + f->extended_data[i] += offset; + if (i < FF_ARRAY_ELEMS(f->data)) + f->data[i] = f->extended_data[i]; + } + f->linesize[0] -= offset; + f->nb_samples -= nb_samples; + f->duration = av_rescale_q(f->nb_samples, (AVRational){ 1, f->sample_rate }, + f->time_base); + f->pts += av_rescale_q(nb_samples, (AVRational){ 1, f->sample_rate }, + f->time_base); +} + +static int receive_samples(SyncQueue *sq, SyncQueueStream *st, + AVFrame *dst, int nb_samples) +{ + SyncQueueFrame src; + int ret; + + av_assert0(st->samples_queued >= nb_samples); + + ret = av_fifo_peek(st->fifo, &src, 1, 0); + av_assert0(ret >= 0); + + // peeked frame has enough samples and its data is aligned + // -> we can just make a reference and limit its sample count + if (src.f->nb_samples > nb_samples && + !((uintptr_t)src.f->data[0] & (av_cpu_max_align() - 1))) { + ret = av_frame_ref(dst, src.f); + if (ret < 0) + return ret; + + dst->nb_samples = nb_samples; + offset_audio(src.f, nb_samples); + st->samples_queued -= nb_samples; + + return 0; + } + + // otherwise allocate a new frame and copy the data + ret = av_channel_layout_copy(&dst->ch_layout, &src.f->ch_layout); + if (ret < 0) + return ret; + + dst->format = src.f->format; + dst->nb_samples = nb_samples; + + ret = av_frame_get_buffer(dst, 0); + if (ret < 0) + goto fail; + + ret = av_frame_copy_props(dst, src.f); + if (ret < 0) + goto fail; + + dst->nb_samples = 0; + while (dst->nb_samples < nb_samples) { + int to_copy; + + ret = av_fifo_peek(st->fifo, &src, 1, 0); + av_assert0(ret >= 0); + + to_copy = FFMIN(nb_samples - dst->nb_samples, src.f->nb_samples); + + av_samples_copy(dst->extended_data, src.f->extended_data, dst->nb_samples, + 0, to_copy, dst->ch_layout.nb_channels, dst->format); + + if (to_copy < src.f->nb_samples) + offset_audio(src.f, to_copy); + else { + av_frame_unref(src.f); + objpool_release(sq->pool, (void**)&src); + av_fifo_drain2(st->fifo, 1); + } + st->samples_queued -= to_copy; + + dst->nb_samples += to_copy; + } + + return 0; + +fail: + av_frame_unref(dst); + return ret; +} + static int receive_for_stream(SyncQueue *sq, unsigned int stream_idx, SyncQueueFrame frame) { @@ -354,13 +474,18 @@ static int receive_for_stream(SyncQueue *sq, unsigned int stream_idx, av_assert0(stream_idx < sq->nb_streams); st = &sq->streams[stream_idx]; - if (av_fifo_can_read(st->fifo)) { + if (av_fifo_can_read(st->fifo) && + (st->frame_samples <= st->samples_queued || st->finished)) { + int nb_samples = st->frame_samples; SyncQueueFrame peek; int64_t ts; int cmp = 1; + if (st->finished) + nb_samples = FFMIN(nb_samples, st->samples_queued); + av_fifo_peek(st->fifo, &peek, 1, 0); - ts = frame_ts(sq, peek); + ts = frame_end(sq, peek, nb_samples); /* check if this stream's tail timestamp does not overtake * the overall queue head */ @@ -372,9 +497,18 @@ static int receive_for_stream(SyncQueue *sq, unsigned int stream_idx, * Frames are also passed through when there are no limiting streams. */ if (cmp <= 0 || ts == AV_NOPTS_VALUE || !sq->have_limiting) { - frame_move(sq, frame, peek); - objpool_release(sq->pool, (void**)&peek); - av_fifo_drain2(st->fifo, 1); + if (nb_samples && nb_samples != peek.f->nb_samples) { + int ret = receive_samples(sq, st, frame.f, nb_samples); + if (ret < 0) + return ret; + } else { + frame_move(sq, frame, peek); + objpool_release(sq->pool, (void**)&peek); + av_fifo_drain2(st->fifo, 1); + av_assert0(st->samples_queued >= frame_samples(sq, frame)); + st->samples_queued -= frame_samples(sq, frame); + } + return 0; } } @@ -460,6 +594,18 @@ void sq_limit_frames(SyncQueue *sq, unsigned int stream_idx, uint64_t frames) finish_stream(sq, stream_idx); } +void sq_frame_samples(SyncQueue *sq, unsigned int stream_idx, + int frame_samples) +{ + SyncQueueStream *st; + + av_assert0(sq->type == SYNC_QUEUE_FRAMES); + av_assert0(stream_idx < sq->nb_streams); + st = &sq->streams[stream_idx]; + + st->frame_samples = frame_samples; +} + SyncQueue *sq_alloc(enum SyncQueueType type, int64_t buf_size_us) { SyncQueue *sq = av_mallocz(sizeof(*sq)); diff --git a/fftools/sync_queue.h b/fftools/sync_queue.h index 9659ee5d50..bc7cd42390 100644 --- a/fftools/sync_queue.h +++ b/fftools/sync_queue.h @@ -71,6 +71,16 @@ int sq_add_stream(SyncQueue *sq, int limiting); void sq_limit_frames(SyncQueue *sq, unsigned int stream_idx, uint64_t max_frames); +/** + * Set a constant output audio frame size, in samples. Can only be used with + * SYNC_QUEUE_FRAMES queues and audio streams. + * + * All output frames will have exactly frame_samples audio samples, except + * possibly for the last one, which may have fewer. + */ +void sq_frame_samples(SyncQueue *sq, unsigned int stream_idx, + int frame_samples); + /** * Submit a frame for the stream with index stream_idx. *