From patchwork Mon Jul 4 16:29:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gyan Doshi X-Patchwork-Id: 36659 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp2934030pzh; Mon, 4 Jul 2022 09:29:55 -0700 (PDT) X-Google-Smtp-Source: AGRyM1srkQ8qQFaViYNUMiB8DhIbcFEM4pWfr96h9rL4c1P/UdK5DzZZOxvlnU/jz8fZmmrpvIbH X-Received: by 2002:a17:907:204b:b0:726:34a5:915a with SMTP id pg11-20020a170907204b00b0072634a5915amr29385407ejb.41.1656952195331; Mon, 04 Jul 2022 09:29:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656952195; cv=none; d=google.com; s=arc-20160816; b=aaHVKcMFUJ2inOkEhHGCyHro/1Ww438GzGXT5SofR39cfUYfzn6KgUPXYCm4Lcpkt+ J3fpRVW56BOJa3x4V49lRvmeWMUc0XYIyDm7jdaaHeVkB//NM4verK7vvHp3RFYe/MoW SpNg4aUluefHhjIA0bGJuWkofYzftWqVzCMV/c0x5gEqnhSXiFvTVGGoRrP4P/MRH7Hp 6C3qCZWlXVlP8U+S/NmQf/lxJXIjJxLFPJoM+iJDJnh5r82zTVscIHzUScbVkFB+g/Yj Oi4rux2rOwu9duTwTIxxFH8JsidF1JIQTf5fTMTdKs0HqoRah/Kp8X73zXBQ9mSReVhq YIUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=YXCTDj0hqzrW0cuuUfFNwMhDyaHyIYu9abySpBG3MPk=; b=kviKTgLy74t8h8VrnlB26hJI6p5J+/oBbGoeoig3ZEIBPHeCMlzpjwCPq2HxC2MzDB YfjaVB115f6v3AA6o4Qk/escx05/Y0g0+5MMK6vB4Y8UZQ5IgxoA70sKYnJ0NcQxRYC1 Rir59ulH2IxpqFJCQRJ8M9wh4i4Eqf5Xw0lvpK2p/OCvhOJmtfDBlN6L5bJ0E76Q+SrT Zj5ckIklzUZe5GFJrzqloLdjME79KEk7sjfgR8Rlj1/kx8n2vO/nGzn0B+jb4i5tF7j4 mLHX9AobCqJoC288xmdIlvy1UHcGVodp/B1FhDV8vJFxvR7u3JNQjOZK2c4BDmADTagZ 8Zcw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hq42-20020a1709073f2a00b00707dfcfc53asi10476470ejc.411.2022.07.04.09.29.54; Mon, 04 Jul 2022 09:29:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9888968B887; Mon, 4 Jul 2022 19:29:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mout-p-103.mailbox.org (mout-p-103.mailbox.org [80.241.56.161]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id F2F3E68B887 for ; Mon, 4 Jul 2022 19:29:44 +0300 (EEST) Received: from smtp2.mailbox.org (smtp2.mailbox.org [10.196.197.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-103.mailbox.org (Postfix) with ESMTPS id 4LcB923pmYz9sSj for ; Mon, 4 Jul 2022 18:29:42 +0200 (CEST) From: Gyan Doshi To: ffmpeg-devel@ffmpeg.org Date: Mon, 4 Jul 2022 21:59:12 +0530 Message-Id: <20220704162912.5577-1-ffmpeg@gyani.pro> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3] ffmpeg: add option -isync X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: L/VVy9uGTeYC This is a per-file input option that adjusts an input's timestamps with reference to another input, so that emitted packet timestamps account for the difference between the start times of the two inputs. Typical use case is to sync two or more live inputs such as from capture devices. Both the target and reference input source timestamps should be based on the same clock source. If not all inputs have timestamps, the wallclock times at the time of reception of inputs shall be used. FFmpeg must have been compiled with thread support for this last case. --- This is a single patch with the reception wallclock stored only in fftools. I find this sub-optimal because find_stream_info will have demuxed an indeterminate number of frames by the time the wallclock is stored. But it's better than nothing. doc/ffmpeg.texi | 16 ++++++++++ fftools/ffmpeg.c | 4 +++ fftools/ffmpeg.h | 3 ++ fftools/ffmpeg_opt.c | 69 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 92 insertions(+) diff --git a/doc/ffmpeg.texi b/doc/ffmpeg.texi index 1a534ff1cc..1fe0a559c0 100644 --- a/doc/ffmpeg.texi +++ b/doc/ffmpeg.texi @@ -518,6 +518,22 @@ see @ref{time duration syntax,,the Time duration section in the ffmpeg-utils(1) Like the @code{-ss} option but relative to the "end of file". That is negative values are earlier in the file, 0 is at EOF. +@item -isync @var{input_index} (@emph{input}) +Assign an input as a sync source. + +This will take the difference between the start times of the target and referenced inputs and +offset the timestamps of the target file by that difference. The source timestamps of the two +inputs should derive from the same clock source for expected results. If @code{copyts} is set +then @code{start_at_zero} must also be set. If at least one of the inputs has no starting +timestamp then the wallclock time at time of reception of the inputs is used as a best-effort +sync basis. + +Acceptable values are those that refer to a valid ffmpeg input index. If the sync reference is +the target index itself or @var{-1}, then no adjustment is made to target timestamps. A sync +reference may not itself be synced to any other input. + +Default value is @var{-1}. + @item -itsoffset @var{offset} (@emph{input}) Set the input time offset. diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index e7384f052a..290b8228b8 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -3658,6 +3658,10 @@ static void *input_thread(void *arg) av_thread_message_queue_set_err_recv(f->in_thread_queue, ret); break; } + + if (f->first_pkt_wallclock == AV_NOPTS_VALUE) + f->first_pkt_wallclock = av_gettime(); + queue_pkt = av_packet_alloc(); if (!queue_pkt) { av_packet_unref(pkt); diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index 99d31c346e..760e94d1e5 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -118,6 +118,7 @@ typedef struct OptionsContext { float readrate; int accurate_seek; int thread_queue_size; + int input_sync_ref; SpecifierOpt *ts_scale; int nb_ts_scale; @@ -409,7 +410,9 @@ typedef struct InputFile { int64_t duration; /* actual duration of the longest stream in a file at the moment when looping happens */ AVRational time_base; /* time base of the duration */ + int64_t first_pkt_wallclock; int64_t input_ts_offset; + int input_sync_ref; int64_t ts_offset; int64_t last_ts; diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index e08455478f..b586f3e6b0 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -235,6 +235,7 @@ static void init_options(OptionsContext *o) o->chapters_input_file = INT_MAX; o->accurate_seek = 1; o->thread_queue_size = -1; + o->input_sync_ref = -1; } static int show_hwaccels(void *optctx, const char *opt, const char *arg) @@ -287,6 +288,67 @@ static int parse_and_set_vsync(const char *arg, int *vsync_var, int file_idx, in return 0; } +static int apply_sync_offsets(void) +{ + for (int i = 0; i < nb_input_files; i++) { + InputFile *ref, *self = input_files[i]; + int64_t adjustment; + int64_t self_start_time, ref_start_time, self_seek_start, ref_seek_start; + int sync_fpw = 0, start_times_set = 1; + + if (self->input_sync_ref == -1 || self->input_sync_ref == i) continue; + if (self->input_sync_ref >= nb_input_files || self->input_sync_ref < -1) { + av_log(NULL, AV_LOG_FATAL, "-isync for input %d references non-existent input %d.\n", i, self->input_sync_ref); + exit_program(1); + } + + if (copy_ts && !start_at_zero) { + av_log(NULL, AV_LOG_FATAL, "Use of -isync requires that start_at_zero be set if copyts is set.\n"); + exit_program(1); + } + + ref = input_files[self->input_sync_ref]; + if (ref->input_sync_ref != -1 && ref->input_sync_ref != self->input_sync_ref) { + av_log(NULL, AV_LOG_ERROR, "-isync for input %d references a resynced input %d. Sync not set.\n", i, self->input_sync_ref); + continue; + } + + if (self->ctx->start_time_realtime != AV_NOPTS_VALUE && ref->ctx->start_time_realtime != AV_NOPTS_VALUE) { + self_start_time = self->ctx->start_time_realtime; + ref_start_time = ref->ctx->start_time_realtime; + } else if (self->ctx->start_time != AV_NOPTS_VALUE && ref->ctx->start_time != AV_NOPTS_VALUE) { + self_start_time = self->ctx->start_time; + ref_start_time = ref->ctx->start_time; + } else { +#if HAVE_THREADS + self_start_time = self->first_pkt_wallclock; + ref_start_time = ref->first_pkt_wallclock; + sync_fpw = 1; +#else + start_times_set = 0; +#endif + } + + if (start_times_set) { + self_seek_start = self->start_time == AV_NOPTS_VALUE ? 0 : self->start_time; + ref_seek_start = ref->start_time == AV_NOPTS_VALUE ? 0 : ref->start_time; + + adjustment = (self_start_time - ref_start_time) + !copy_ts*(self_seek_start - ref_seek_start) + ref->input_ts_offset; + + self->ts_offset += adjustment; + + av_log(NULL, AV_LOG_INFO, "Adjusted ts offset for Input #%d by %"PRId64" us to sync with Input #%d", i, adjustment, self->input_sync_ref); + if (sync_fpw) av_log(NULL, AV_LOG_INFO, " using reception wallclock time. Sync may not be obtained"); + av_log(NULL, AV_LOG_INFO, ".\n"); + } else { + av_log(NULL, AV_LOG_INFO, "Unable to identify start times for Inputs #%d and %d both. No sync adjustment made.\n", + i, self->input_sync_ref); + } + } + + return 0; +} + static int opt_filter_threads(void *optctx, const char *opt, const char *arg) { av_free(filter_nbthreads); @@ -1305,6 +1367,8 @@ static int open_input_file(OptionsContext *o, const char *filename) f->ist_index = nb_input_streams - ic->nb_streams; f->start_time = o->start_time; f->recording_time = o->recording_time; + f->first_pkt_wallclock = AV_NOPTS_VALUE; + f->input_sync_ref = o->input_sync_ref; f->input_ts_offset = o->input_ts_offset; f->ts_offset = o->input_ts_offset - (copy_ts ? (start_at_zero && ic->start_time != AV_NOPTS_VALUE ? ic->start_time : 0) : timestamp); f->nb_streams = ic->nb_streams; @@ -3489,6 +3553,8 @@ int ffmpeg_parse_options(int argc, char **argv) goto fail; } + apply_sync_offsets(); + /* create the complex filtergraphs */ ret = init_complex_filters(); if (ret < 0) { @@ -3603,6 +3669,9 @@ const OptionDef options[] = { { "accurate_seek", OPT_BOOL | OPT_OFFSET | OPT_EXPERT | OPT_INPUT, { .off = OFFSET(accurate_seek) }, "enable/disable accurate seeking with -ss" }, + { "isync", HAS_ARG | OPT_INT | OPT_OFFSET | + OPT_EXPERT | OPT_INPUT, { .off = OFFSET(input_sync_ref) }, + "Indicate the input index for sync reference", "sync ref" }, { "itsoffset", HAS_ARG | OPT_TIME | OPT_OFFSET | OPT_EXPERT | OPT_INPUT, { .off = OFFSET(input_ts_offset) }, "set the input ts offset", "time_off" },