From patchwork Mon Jul 4 16:11:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 36658 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp2924578pzh; Mon, 4 Jul 2022 09:12:01 -0700 (PDT) X-Google-Smtp-Source: AGRyM1uBLqfRKkhtjm9VpGW5kdP+W1Gzar/mkrQAyIVAlVtkUz0/0niZvYuzrvU1yMImppDc9coU X-Received: by 2002:a17:906:cc49:b0:72a:93fe:12a4 with SMTP id mm9-20020a170906cc4900b0072a93fe12a4mr16371965ejb.228.1656951120932; Mon, 04 Jul 2022 09:12:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656951120; cv=none; d=google.com; s=arc-20160816; b=nMW/JuRfweuUABz/oct54vzLOENVD8/iCgk3p5wiBh5eWcCWgJXOcNnKGxTPVv6tgt n2FOm4iumihoEh5tgCTWiI3hhKAd6uct9ovQaveKLpY0dPWTv+PNHwWObenVX8gL470K GhKC/QKe2zQXu7BK+0t57AJGa6pjPVqrY/NV+o7IVuyCv91eRt0OuKA9N+g7saMZwfKi 3GrHDi5u9ej4YqHhT1XpAf91nb+WUpF5K9X2WflOLE+YVKHNF6kE7NOzp4YNDSbxPO7F 4/UOXcFlNWAhsoJBlsAQPnZlF34nupbcxyRo/yPGh3R4cTUqapXtoCg3naTSFnLvaozj +WLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=C6QbQuA5zq3iTNPlHId5JnxD0Rw6QiWoERNhzlfdvCQ=; b=ZuERE19iVqWz3FuNrwyVK7UBHHR4vshxHwmyFW8VlxoxACxRPsEFukS3swfwF7rkpq bU7/Xn4SA8t8E6NAGVMQIqkBqVM2o/8RrptghKKIkVOK6N8f29I3Af0kZZkUdgWcP9w1 k6PFfUgyFC9HxYGTYYbAJGOy906I7F6VZ2kzhWoJwbWnyassjtCmOgvLA0rI9Dhj5AEe IwVXpy056cwOTljJfqckgeO/ZBXQ4YfBLSITYQDiaV8ky29ZAAJOZfH0132GsqDM5o6o +ERcgIhOtjDjzuSKRTv0a+wWnaty5m0dKhIa1a1SKC2c/H2cAWqbAk7hqH2IoYmVAI2q S60A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id h17-20020a056402281100b0043a7075f026si1363127ede.130.2022.07.04.09.11.59; Mon, 04 Jul 2022 09:12:00 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 837D068B8BF; Mon, 4 Jul 2022 19:11:56 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail0.khirnov.net (red.khirnov.net [176.97.15.12]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8339468B89A for ; Mon, 4 Jul 2022 19:11:49 +0300 (EEST) Received: from localhost (localhost [IPv6:::1]) by mail0.khirnov.net (Postfix) with ESMTP id B447B240179 for ; Mon, 4 Jul 2022 18:11:48 +0200 (CEST) Received: from mail0.khirnov.net ([IPv6:::1]) by localhost (mail0.khirnov.net [IPv6:::1]) (amavisd-new, port 10024) with ESMTP id ZN-QqHuM_u_C for ; Mon, 4 Jul 2022 18:11:47 +0200 (CEST) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail0.khirnov.net (Postfix) with ESMTPS id 4496E2400F5 for ; Mon, 4 Jul 2022 18:11:47 +0200 (CEST) Received: by libav.khirnov.net (Postfix, from userid 1000) id 56A673A061D; Mon, 4 Jul 2022 18:11:47 +0200 (CEST) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Mon, 4 Jul 2022 18:11:45 +0200 Message-Id: <20220704161145.25027-1-anton@khirnov.net> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220623221231.GL396728@pb2> References: <20220623221231.GL396728@pb2> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] fftools/ffmpeg: use the sync queues to handle -frames X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: mMjiupzS4V/t Same issues apply to it as to -shortest. Changes the results of the following tests: - matroska-flac-extradata-update The test reencodes two input FLAC streams into three output FLAC streams. The last output stream is limited to 8 frames. The current code results in the first two output streams having 12 frames, after this commit all three streams have 8 frames and are the same length. This new result is better, since it is predictable. - mkv-1242 The test streamcopies one video and one audio stream, video is limited to 11 frames. The new result shortens the audio stream so that it is not longer than the video. --- fftools/ffmpeg.c | 6 ---- fftools/ffmpeg_mux.c | 10 +----- fftools/ffmpeg_opt.c | 24 +++++++++++--- fftools/sync_queue.c | 33 ++++++++++++++++--- fftools/sync_queue.h | 11 ++++++- tests/ref/fate/matroska-flac-extradata-update | 16 +++------ tests/ref/fate/mkv-1242 | 3 -- 7 files changed, 63 insertions(+), 40 deletions(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index 15e350f8fd..503a148103 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -3467,12 +3467,6 @@ static int need_output(void) if (ost->finished || of_finished(of)) continue; - if (ost->frame_number >= ost->max_frames) { - int j; - for (j = 0; j < of->ctx->nb_streams; j++) - close_output_stream(output_streams[of->ost_index + j]); - continue; - } return 1; } diff --git a/fftools/ffmpeg_mux.c b/fftools/ffmpeg_mux.c index 641bdb98b0..56444770bf 100644 --- a/fftools/ffmpeg_mux.c +++ b/fftools/ffmpeg_mux.c @@ -237,19 +237,11 @@ void of_submit_packet(OutputFile *of, AVPacket *pkt, OutputStream *ost) if (pkt) { /* - * Audio encoders may split the packets -- #frames in != #packets out. - * But there is no reordering, so we can limit the number of output packets - * by simply dropping them here. * Counting encoded video frames needs to be done separately because of * reordering, see do_video_out(). */ - if (!(st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO && ost->encoding_needed)) { - if (ost->frame_number >= ost->max_frames) { - av_packet_unref(pkt); - return; - } + if (!(st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO && ost->encoding_needed)) ost->frame_number++; - } } if (of->mux->header_written) { diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index be88743463..12f5d16157 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -2336,6 +2336,7 @@ static int init_complex_filters(void) static int setup_sync_queues(OutputFile *of, AVFormatContext *oc, int64_t buf_size_us) { int nb_av_enc = 0, nb_interleaved = 0; + int limit_frames = 0, limit_frames_av_enc = 0; #define IS_AV_ENC(ost, type) \ (ost->encoding_needed && (type == AVMEDIA_TYPE_VIDEO || type == AVMEDIA_TYPE_AUDIO)) @@ -2350,14 +2351,19 @@ static int setup_sync_queues(OutputFile *of, AVFormatContext *oc, int64_t buf_si nb_interleaved += IS_INTERLEAVED(type); nb_av_enc += IS_AV_ENC(ost, type); + + limit_frames |= ost->max_frames < INT64_MAX; + limit_frames_av_enc |= (ost->max_frames < INT64_MAX) && IS_AV_ENC(ost, type); } - if (!(nb_interleaved > 1 && of->shortest)) + if (!((nb_interleaved > 1 && of->shortest) || + (nb_interleaved > 0 && limit_frames))) return 0; - /* if we have more than one encoded audio/video streams, then we + /* if we have more than one encoded audio/video streams, or at least + * one encoded audio/video stream is frame-limited, then we * synchronize them before encoding */ - if (nb_av_enc > 1) { + if ((of->shortest && nb_av_enc > 1) || limit_frames_av_enc) { of->sq_encode = sq_alloc(SYNC_QUEUE_FRAMES, buf_size_us); if (!of->sq_encode) return AVERROR(ENOMEM); @@ -2369,13 +2375,17 @@ static int setup_sync_queues(OutputFile *of, AVFormatContext *oc, int64_t buf_si if (!IS_AV_ENC(ost, type)) continue; - ost->sq_idx_encode = sq_add_stream(of->sq_encode); + ost->sq_idx_encode = sq_add_stream(of->sq_encode, + of->shortest || ost->max_frames < INT64_MAX); if (ost->sq_idx_encode < 0) return ost->sq_idx_encode; ost->sq_frame = av_frame_alloc(); if (!ost->sq_frame) return AVERROR(ENOMEM); + + if (ost->max_frames != INT64_MAX) + sq_limit_frames(of->sq_encode, ost->sq_idx_encode, ost->max_frames); } } @@ -2393,9 +2403,13 @@ static int setup_sync_queues(OutputFile *of, AVFormatContext *oc, int64_t buf_si if (!IS_INTERLEAVED(type)) continue; - ost->sq_idx_mux = sq_add_stream(of->sq_mux); + ost->sq_idx_mux = sq_add_stream(of->sq_mux, + of->shortest || ost->max_frames < INT64_MAX); if (ost->sq_idx_mux < 0) return ost->sq_idx_mux; + + if (ost->max_frames != INT64_MAX) + sq_limit_frames(of->sq_mux, ost->sq_idx_mux, ost->max_frames); } } diff --git a/fftools/sync_queue.c b/fftools/sync_queue.c index ab654ca790..3143d12cf1 100644 --- a/fftools/sync_queue.c +++ b/fftools/sync_queue.c @@ -34,8 +34,12 @@ typedef struct SyncQueueStream { /* stream head: largest timestamp seen */ int64_t head_ts; + int limiting; /* no more frames will be sent for this stream */ int finished; + + uint64_t frames_sent; + uint64_t frames_max; } SyncQueueStream; struct SyncQueue { @@ -86,7 +90,7 @@ static void finish_stream(SyncQueue *sq, unsigned int stream_idx) st->finished = 1; - if (st->head_ts != AV_NOPTS_VALUE) { + if (st->limiting && st->head_ts != AV_NOPTS_VALUE) { /* check if this stream is the new finished head */ if (sq->head_finished_stream < 0 || av_compare_ts(st->head_ts, st->tb, @@ -121,7 +125,7 @@ static void queue_head_update(SyncQueue *sq) * the queue head */ for (unsigned int i = 0; i < sq->nb_streams; i++) { SyncQueueStream *st = &sq->streams[i]; - if (st->head_ts == AV_NOPTS_VALUE) + if (st->limiting && st->head_ts == AV_NOPTS_VALUE) return; } @@ -132,7 +136,7 @@ static void queue_head_update(SyncQueue *sq) for (unsigned int i = 0; i < sq->nb_streams; i++) { SyncQueueStream *st_head = &sq->streams[sq->head_stream]; SyncQueueStream *st_other = &sq->streams[i]; - if (st_other->head_ts != AV_NOPTS_VALUE && + if (st_other->limiting && st_other->head_ts != AV_NOPTS_VALUE && av_compare_ts(st_other->head_ts, st_other->tb, st_head->head_ts, st_head->tb) < 0) sq->head_stream = i; @@ -159,7 +163,8 @@ static void stream_update_ts(SyncQueue *sq, unsigned int stream_idx, int64_t ts) finish_stream(sq, stream_idx); /* update the overall head timestamp if it could have changed */ - if (sq->head_stream < 0 || sq->head_stream == stream_idx) + if (st->limiting && + (sq->head_stream < 0 || sq->head_stream == stream_idx)) queue_head_update(sq); } @@ -262,6 +267,10 @@ int sq_send(SyncQueue *sq, unsigned int stream_idx, SyncQueueFrame frame) stream_update_ts(sq, stream_idx, ts); + st->frames_sent++; + if (st->frames_sent >= st->frames_max) + finish_stream(sq, stream_idx); + return 0; } @@ -339,7 +348,7 @@ int sq_receive(SyncQueue *sq, int stream_idx, SyncQueueFrame frame) return ret; } -int sq_add_stream(SyncQueue *sq) +int sq_add_stream(SyncQueue *sq, int limiting) { SyncQueueStream *tmp, *st; @@ -360,6 +369,8 @@ int sq_add_stream(SyncQueue *sq) * streams forever; cf. overflow_heartbeat() */ st->tb = (AVRational){ 1, 1 }; st->head_ts = AV_NOPTS_VALUE; + st->frames_max = UINT64_MAX; + st->limiting = limiting; return sq->nb_streams++; } @@ -379,6 +390,18 @@ void sq_set_tb(SyncQueue *sq, unsigned int stream_idx, AVRational tb) st->tb = tb; } +void sq_limit_frames(SyncQueue *sq, unsigned int stream_idx, uint64_t frames) +{ + SyncQueueStream *st; + + av_assert0(stream_idx < sq->nb_streams); + st = &sq->streams[stream_idx]; + + st->frames_max = frames; + if (st->frames_sent >= st->frames_max) + finish_stream(sq, stream_idx); +} + SyncQueue *sq_alloc(enum SyncQueueType type, int64_t buf_size_us) { SyncQueue *sq = av_mallocz(sizeof(*sq)); diff --git a/fftools/sync_queue.h b/fftools/sync_queue.h index e08780b7bf..3f823ff0d9 100644 --- a/fftools/sync_queue.h +++ b/fftools/sync_queue.h @@ -51,11 +51,13 @@ void sq_free(SyncQueue **sq); /** * Add a new stream to the sync queue. * + * @param limiting whether the stream is limiting, i.e. no other stream can be + * longer than this one * @return * - a non-negative stream index on success * - a negative error code on error */ -int sq_add_stream(SyncQueue *sq); +int sq_add_stream(SyncQueue *sq, int limiting); /** * Set the timebase for the stream with index stream_idx. Should be called @@ -63,6 +65,13 @@ int sq_add_stream(SyncQueue *sq); */ void sq_set_tb(SyncQueue *sq, unsigned int stream_idx, AVRational tb); +/** + * Limit the number of output frames for stream with index stream_idx + * to max_frames. + */ +void sq_limit_frames(SyncQueue *sq, unsigned int stream_idx, + uint64_t max_frames); + /** * Submit a frame for the stream with index stream_idx. * diff --git a/tests/ref/fate/matroska-flac-extradata-update b/tests/ref/fate/matroska-flac-extradata-update index e8812f51b5..d5814925f5 100644 --- a/tests/ref/fate/matroska-flac-extradata-update +++ b/tests/ref/fate/matroska-flac-extradata-update @@ -1,8 +1,8 @@ -732446e97bae29037ff0cd9963d4ac08 *tests/data/fate/matroska-flac-extradata-update.matroska -1987 tests/data/fate/matroska-flac-extradata-update.matroska -#extradata 0: 34, 0x7acb09e7 -#extradata 1: 34, 0x7acb09e7 -#extradata 2: 34, 0x443402dd +28bc0ded5dc520d955caf29db80d35da *tests/data/fate/matroska-flac-extradata-update.matroska +1795 tests/data/fate/matroska-flac-extradata-update.matroska +#extradata 0: 34, 0x93650c81 +#extradata 1: 34, 0x93650c81 +#extradata 2: 34, 0x93650c81 #tb 0: 1/1000 #media_type 0: audio #codec_id 0: flac @@ -42,9 +42,3 @@ 0, 672, 672, 96, 26, 0x50dd042e 1, 672, 672, 96, 26, 0x50dd042e 2, 672, 672, 96, 26, 0x50dd042e -0, 768, 768, 96, 26, 0x53de0499 -1, 768, 768, 96, 26, 0x53de0499 -0, 864, 864, 96, 26, 0x53df04b4 -1, 864, 864, 96, 26, 0x53df04b4 -0, 960, 960, 42, 26, 0x5740044b -1, 960, 960, 42, 26, 0x5740044b diff --git a/tests/ref/fate/mkv-1242 b/tests/ref/fate/mkv-1242 index e025701093..1d1a227832 100644 --- a/tests/ref/fate/mkv-1242 +++ b/tests/ref/fate/mkv-1242 @@ -42,6 +42,3 @@ 1, 383, 383, 21, 325, 0xcd7a9fd6 1, 404, 404, 22, 359, 0x6edeb91c 1, 426, 426, 21, 333, 0xb8999fb7 -1, 447, 447, 21, 317, 0xf2589e1a -1, 468, 468, 21, 319, 0x82ed9572 -1, 489, 489, 22, 473, 0xea54e696