From patchwork Wed Jun 22 13:54:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gyan Doshi X-Patchwork-Id: 36382 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1a22:b0:84:42e0:ad30 with SMTP id cj34csp2854986pzb; Wed, 22 Jun 2022 06:55:20 -0700 (PDT) X-Google-Smtp-Source: AGRyM1sHkyoCXC7vMsOiaRWZQU1EaIgj6XQKjhE+hIPamOaGn3O3e12Tihx9xiTwvzoWX4g9wp59 X-Received: by 2002:a05:6402:510a:b0:435:bcb8:756d with SMTP id m10-20020a056402510a00b00435bcb8756dmr1413415edd.277.1655906120028; Wed, 22 Jun 2022 06:55:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1655906120; cv=none; d=google.com; s=arc-20160816; b=Mq8qDXmjqT2jFkMVkiPEIW2hnTyrSlVgyQYm+438rgaW84tWJ9X6cgfxt2ya9CkXCU Ns0JwVci285N15ozkPSAOpbLhfBnWKqmojUBThSY8YQy9XwCpnFj86IwPP8gOWHECNAt pHDojyqyLy37ceHXmdENiPFfP+DQP4AMyk9WGKRMNFJ0LrJeBsH5jxV0flZZebxdB4Oc kxoN447RbO4eF1waXnANlCt3VMbFBPCFtcWIp98wap4BIxRfX1TS+OYTdl9/4aAsWlra igkFl/OcvCPUzF2ITPyCVu8VGVTMZdj5K0AavffnnVfS1cRogiglSF8FPRcnJ0JoUbNt V9Wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=rIOZD12At2RzRYXM2OsbUED4AidQ/tfBQOH4pOnY1ok=; b=mQp50/hE3YH1j9yjZINbsbEeOxUX9QMdLFNjGt+ES+7w6qvf17wqLmBa3+7z9wWkGp Ig7DwyRS1bg3KBOkT46nfZN7Xva2tNwMjlRrm08FtFNzGpNzv9vGSjQDGYsvobgD2RWy tHM7HTH0jmzeFImboYmHExAH56nT1jZKn4chPU4+FtaDFTxMD7dk1mk10dnDd9C4YyU9 iPE+nsBYH8Tnz3+v74f6sWfRLEv7jtBS/+d4yzfdACAh77VT1ZtH7iXHAOG4xcxvfM1o YfbnQAYJkIAgE8AYeZt/3EoXxpG/SrnHNcH07IVd/2tm5FXxxihc1ygWhFq394ulxD+7 O6TA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y7-20020a170906518700b006f3a874f673si21056449ejk.4.2022.06.22.06.55.19; Wed, 22 Jun 2022 06:55:20 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 05AC368B6AA; Wed, 22 Jun 2022 16:55:15 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mout-p-201.mailbox.org (mout-p-201.mailbox.org [80.241.56.171]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BBE2568B3D6 for ; Wed, 22 Jun 2022 16:55:06 +0300 (EEST) Received: from smtp102.mailbox.org (smtp102.mailbox.org [10.196.197.102]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-201.mailbox.org (Postfix) with ESMTPS id 4LSlJ70jrlz9sSd for ; Wed, 22 Jun 2022 15:55:03 +0200 (CEST) From: Gyan Doshi To: ffmpeg-devel@ffmpeg.org Date: Wed, 22 Jun 2022 19:24:36 +0530 Message-Id: <20220622135436.475-1-ffmpeg@gyani.pro> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] ffmpeg: add option -isync X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: lcJiIeZsVUA+ This is a per-file input option that adjusts an input's timestamps with reference to another input, so that emitted packet timestamps account for the difference between the start times of the two inputs. Typical use case is to sync two or more live inputs such as capture devices. Both the target and reference input source timestamps should be based on the same clock source. If not all inputs have timestamps, the wallclock times at the time of reception of inputs shall be used. --- In its current form, the patch depends on first_pts_wallclock being available but that's not integral, just a fallback for crude sync. If f_p_w is not available, then an error log will be inserted in that code path. The f_p_w patch is at https://ffmpeg.org/pipermail/ffmpeg-devel/.html2022-June/297609 The patch has been used to successfully stream a synced audio mix of a karaoke performance and the original audio. It can similarly be used to sync webcam+mic feeds in different inputs or various media feeds received on a LAN. doc/ffmpeg.texi | 16 ++++++++++++++ fftools/ffmpeg.h | 2 ++ fftools/ffmpeg_opt.c | 52 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 70 insertions(+) diff --git a/doc/ffmpeg.texi b/doc/ffmpeg.texi index d943f4d6f5..92dd584c89 100644 --- a/doc/ffmpeg.texi +++ b/doc/ffmpeg.texi @@ -518,6 +518,22 @@ see @ref{time duration syntax,,the Time duration section in the ffmpeg-utils(1) Like the @code{-ss} option but relative to the "end of file". That is negative values are earlier in the file, 0 is at EOF. +@item -isync @var{input_index} (@emph{input}) +Assign an input as a sync source. + +This will take the difference between the start times of the target and referenced inputs and +offset the timestamps of the target file by that difference. The source timestamps of the two +inputs should derive from the same clock source for expected results. If @code{copyts} is set +then @code{start_at_zero} must also be set. If at least one of the inputs has no starting +timestamp then the wallclock time at time of reception of the inputs is used as a best-effort +sync basis. + +Acceptable values are those that refer to a valid ffmpeg input index. If the sync reference is +the target index itself or @var{-1}, then no adjustment is made to target timestamps. A sync +reference may not itself be synced to any other input. + +Default value is var{-1}. + @item -itsoffset @var{offset} (@emph{input}) Set the input time offset. diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index 69a368b8d1..dc74de6684 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -118,6 +118,7 @@ typedef struct OptionsContext { float readrate; int accurate_seek; int thread_queue_size; + int input_sync_ref; SpecifierOpt *ts_scale; int nb_ts_scale; @@ -410,6 +411,7 @@ typedef struct InputFile { at the moment when looping happens */ AVRational time_base; /* time base of the duration */ int64_t input_ts_offset; + int input_sync_ref; int64_t ts_offset; int64_t last_ts; diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index e08455478f..eda0717380 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -235,6 +235,7 @@ static void init_options(OptionsContext *o) o->chapters_input_file = INT_MAX; o->accurate_seek = 1; o->thread_queue_size = -1; + o->input_sync_ref = -1; } static int show_hwaccels(void *optctx, const char *opt, const char *arg) @@ -287,6 +288,51 @@ static int parse_and_set_vsync(const char *arg, int *vsync_var, int file_idx, in return 0; } +static int apply_sync_offsets(void) +{ + for (int i = 0; i < nb_input_files; i++) { + InputFile *ref, *self = input_files[i]; + int64_t adjustment; + int64_t self_start_time, ref_start_time, self_seek_start, ref_seek_start; + + if (self->input_sync_ref == -1 || self->input_sync_ref == i) continue; + if (self->input_sync_ref >= nb_input_files || self->input_sync_ref < -1) { + av_log(NULL, AV_LOG_FATAL, "-isync for input %d references non-existent input %d.\n", i, self->input_sync_ref); + exit_program(1); + } + + if (copy_ts && !start_at_zero) { + av_log(NULL, AV_LOG_FATAL, "Use of -isync requires that start_at_zero be set if copyts is set.\n"); + exit_program(1); + } + + ref = input_files[self->input_sync_ref]; + if (ref->input_sync_ref != -1 && ref->input_sync_ref != self->input_sync_ref) { + av_log(NULL, AV_LOG_ERROR, "-isync for input %d references a resynced input %d. Sync not set.\n", i, self->input_sync_ref); + continue; + } + + if (self->ctx->start_time == AV_NOPTS_VALUE || ref->ctx->start_time == AV_NOPTS_VALUE) { + self_start_time = self->ctx->first_pts_wallclock; + ref_start_time = ref->ctx->first_pts_wallclock; + } else { + self_start_time = self->ctx->start_time; + ref_start_time = ref->ctx->start_time; + } + + self_seek_start = self->start_time == AV_NOPTS_VALUE ? 0 : self->start_time; + ref_seek_start = ref->start_time == AV_NOPTS_VALUE ? 0 : ref->start_time; + + adjustment = (self_start_time - ref_start_time) + !copy_ts*(self_seek_start - ref_seek_start) + ref->input_ts_offset; + + self->ts_offset += adjustment; + + av_log(NULL, AV_LOG_INFO, "adjusted ts offset for Input #%d by %"PRId64"d us to sync with Input #%d\n", i, adjustment, self->input_sync_ref); + } + + return 0; +} + static int opt_filter_threads(void *optctx, const char *opt, const char *arg) { av_free(filter_nbthreads); @@ -1305,6 +1351,7 @@ static int open_input_file(OptionsContext *o, const char *filename) f->ist_index = nb_input_streams - ic->nb_streams; f->start_time = o->start_time; f->recording_time = o->recording_time; + f->input_sync_ref = o->input_sync_ref; f->input_ts_offset = o->input_ts_offset; f->ts_offset = o->input_ts_offset - (copy_ts ? (start_at_zero && ic->start_time != AV_NOPTS_VALUE ? ic->start_time : 0) : timestamp); f->nb_streams = ic->nb_streams; @@ -3489,6 +3536,8 @@ int ffmpeg_parse_options(int argc, char **argv) goto fail; } + apply_sync_offsets(); + /* create the complex filtergraphs */ ret = init_complex_filters(); if (ret < 0) { @@ -3603,6 +3652,9 @@ const OptionDef options[] = { { "accurate_seek", OPT_BOOL | OPT_OFFSET | OPT_EXPERT | OPT_INPUT, { .off = OFFSET(accurate_seek) }, "enable/disable accurate seeking with -ss" }, + { "isync", HAS_ARG | OPT_INT | OPT_OFFSET | + OPT_EXPERT | OPT_INPUT, { .off = OFFSET(input_sync_ref) }, + "Indicate the input index for sync reference", "sync ref" }, { "itsoffset", HAS_ARG | OPT_TIME | OPT_OFFSET | OPT_EXPERT | OPT_INPUT, { .off = OFFSET(input_ts_offset) }, "set the input ts offset", "time_off" },