From patchwork Sat Jun 25 08:29:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gyan Doshi X-Patchwork-Id: 36422 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp793975pzh; Sat, 25 Jun 2022 01:30:19 -0700 (PDT) X-Google-Smtp-Source: AGRyM1vXRFbqb2QOSG0FdZHJcFhTC9+qhD7FEt03APMr7IUQnUexsW8TGoiyCYPwwQ0EaLyYDIfz X-Received: by 2002:a05:6402:4306:b0:435:a1c9:4272 with SMTP id m6-20020a056402430600b00435a1c94272mr3741015edc.205.1656145819305; Sat, 25 Jun 2022 01:30:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656145819; cv=none; d=google.com; s=arc-20160816; b=k66sBBL7012F6jJ1aNl3sTpzWUW6RhDKeQnZjAze+NAIcTzxKZETCpsLsqbSQil/8W nYZvpJTEyr4y5vgbIz7n7TtFq9BgwhqV6A3lqEnnYIq35AMF6I6/kJelHC/q5kAB4WTt LViVPyZat+Z35FTqrsg1lFgUm1AF79+lVIIUwpZJyg7o0pJ9yCBWXV9r8lwEzWyo/2bf N6Kq/tFxZ4IMvqyUKnj2cSUX3+vuHvxyTwf38/C/g/wi24vlCUnR8EGfkyPZLD0nsKpF xKUrq3gKMcBLihO3TX88YMO4njNfevZL8AH7ZIWmZN76bsfGpGLwZHX3O+wrVkfmPrEx Kiag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=LLAFTG1xymGnKe7B5+shOoFHSntaFgn8JUwSmJyYYr4=; b=TVLOvAopZnUfcxX8tHIJqrv8BJdK7K4FjXsBFNxIgob7BpxaZyvmWfvRvBxlqqMdlk 2JqZ3L2VLXL0QXqeEEG9rhGAEqItZUI/HwHXRf8Fg8Uz16uNSgAH4AidfHT/4bLtpYTQ pCshVN7qIrdEGODxU77APhelPyySSEHQtVeNBNdehJCt7pD8YZitsf4M4amu4h0+E9oi 75RxjqdvOfEtJ6JpxJlo6sVZehL8a6WLiSuWSDQuGzf3ny8cJreqPpyvPMU7OGI+vNpA qhbpidGcb+i2a0PJP5gcDm6PzssbHbgGT4U+qmdimedAx1QNWNNu2atahMQf4bF6EApN cpCg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id u25-20020a056402065900b004356afc7003si5375798edx.169.2022.06.25.01.30.18; Sat, 25 Jun 2022 01:30:19 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 19CB368B7E8; Sat, 25 Jun 2022 11:30:15 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mout-p-101.mailbox.org (mout-p-101.mailbox.org [80.241.56.151]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1B91368B63B for ; Sat, 25 Jun 2022 11:30:09 +0300 (EEST) Received: from smtp202.mailbox.org (smtp202.mailbox.org [10.196.197.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-101.mailbox.org (Postfix) with ESMTPS id 4LVRxq5qKkz9sb6 for ; Sat, 25 Jun 2022 10:30:07 +0200 (CEST) From: Gyan Doshi To: ffmpeg-devel@ffmpeg.org Date: Sat, 25 Jun 2022 13:59:50 +0530 Message-Id: <20220625082951.11897-1-ffmpeg@gyani.pro> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 1/2] avformat: add AVFormatContext.first_pkt_wallclock X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Tp3fKCSU3vH8 Stores wallclock time for the first packet received. Used for crude sync offset among inputs. --- doc/APIchanges | 3 +++ libavformat/avformat.h | 10 ++++++++++ libavformat/demux.c | 3 +++ libavformat/options.c | 1 + libavformat/version.h | 2 +- 5 files changed, 18 insertions(+), 1 deletion(-) diff --git a/doc/APIchanges b/doc/APIchanges index 20b944933a..94c53d283f 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -14,6 +14,9 @@ libavutil: 2021-04-27 API changes, most recent first: +2022-06-xx - xxxxxxxxxx - lavf 59.26.100 - avformat.h + Add and set AVFormatContext.first_pkt_wallclock field. + 2022-06-12 - xxxxxxxxxx - lavf 59.25.100 - avio.h Add avio_vprintf(), similar to avio_printf() but allow to use it from within a function taking a variable argument list as input. diff --git a/libavformat/avformat.h b/libavformat/avformat.h index f12fa7d904..a45777ab39 100644 --- a/libavformat/avformat.h +++ b/libavformat/avformat.h @@ -1808,6 +1808,16 @@ typedef struct AVFormatContext { */ int max_probe_packets; + /** + * Wallclock start time of the stream in real world time, in microseconds + * since the Unix epoch (00:00 1st January 1970). That is, the first packet + * was received at this real world time. + * - demuxing: Set by libavformat. Users may want to use start_time_realtime + * if set. + * - muxing: unused. + */ + int64_t first_pkt_wallclock; + /** * A callback for closing the streams opened with AVFormatContext.io_open(). * diff --git a/libavformat/demux.c b/libavformat/demux.c index e121253dfd..7e0540debf 100644 --- a/libavformat/demux.c +++ b/libavformat/demux.c @@ -628,6 +628,9 @@ FF_ENABLE_DEPRECATION_WARNINGS force_codec_ids(s, st); + if (s->first_pkt_wallclock == AV_NOPTS_VALUE) + s->first_pkt_wallclock = av_gettime(); + /* TODO: audio: time filter; video: frame reordering (pts != dts) */ if (s->use_wallclock_as_timestamps) pkt->dts = pkt->pts = av_rescale_q(av_gettime(), AV_TIME_BASE_Q, st->time_base); diff --git a/libavformat/options.c b/libavformat/options.c index 0079a06d9a..77d4dfd1d0 100644 --- a/libavformat/options.c +++ b/libavformat/options.c @@ -184,6 +184,7 @@ AVFormatContext *avformat_alloc_context(void) return NULL; } + s->first_pkt_wallclock = AV_NOPTS_VALUE; si->shortest_end = AV_NOPTS_VALUE; return s; diff --git a/libavformat/version.h b/libavformat/version.h index 966ebb7ed3..0708d619c0 100644 --- a/libavformat/version.h +++ b/libavformat/version.h @@ -31,7 +31,7 @@ #include "version_major.h" -#define LIBAVFORMAT_VERSION_MINOR 25 +#define LIBAVFORMAT_VERSION_MINOR 26 #define LIBAVFORMAT_VERSION_MICRO 100 #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \ From patchwork Sat Jun 25 08:29:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gyan Doshi X-Patchwork-Id: 36423 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp794024pzh; Sat, 25 Jun 2022 01:30:28 -0700 (PDT) X-Google-Smtp-Source: AGRyM1ulrglZGcjWQa8hVBOjy1WlXq+pbHMzAyJoemhJ+0lfRS0sHASseH2BA3A4NMK36K5lc9db X-Received: by 2002:a05:6402:2999:b0:434:edcc:f12c with SMTP id eq25-20020a056402299900b00434edccf12cmr3806503edb.96.1656145828523; Sat, 25 Jun 2022 01:30:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656145828; cv=none; d=google.com; s=arc-20160816; b=vifIi5buSfi9Ey+T6mcBCgvP2KTOcQHBzoX+eoIyPPvn/6b80Har//MkQ3FukixZBk mYWT5WxK7no2MMXh0BawZoRRdwIVIFmAzGF+BgQz1jp3DzOpG2sTCk0eGAaVTMQPeoM5 EfVnIUgehwEywnWWlM6IsCKzzfc8dJdy2v560npWUPECrsLAjuSMWW6R9K+Tl5DnH9ZZ 6iDDZg2qi6tHSG0ywvvf6nMa2itvaA85m98oKkmKmLESwmn0CFWuJXLZBVLe1B5S0j/w k1Of3QE7u3meyAvAWILpm8I9xH7qzNB/OoMUy+RBR3jmBnn+QiSpAlXWn/kcRxbozwBm I/Fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=m/9ShUGyve11Ik6Kv5QbtzgIy6jAHDOHf154vqegFpw=; b=KY6kmA2q36c/oVDXeSrjoqk2fnKYJQRuBCacdL1fTKLhq/dNfK+4tkhjUUAknOpirW hreM3R+ecG3K/gMCv1BN5vX5fO0IZHgMpuWsJ1sVDAsfD0A0kbRSsLotqq19hBOQAx0O nbait5BRvy6tM/XtTOQpOTarj2bMD0uO6EuLP+9OPBtTzfrm16xakjnyPZKcSGBfzTTx 7VqGmH2TMorXobm17KL0R9EkQ/RQHxVNb6hTYVHofhBRyuE2pO4VqnUzayJ8OwvSmmQy mBJpUp4L1wMudsi1IWdn4AJfQ1C8l8giZauFjzUXXwjpf9YO/bcwsbqWSIsruTc15V0m jzyw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b37-20020a509f28000000b0042fadee7b91si5289021edf.406.2022.06.25.01.30.28; Sat, 25 Jun 2022 01:30:28 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3441268B7F1; Sat, 25 Jun 2022 11:30:17 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mout-p-101.mailbox.org (mout-p-101.mailbox.org [80.241.56.151]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 484C468B63B for ; Sat, 25 Jun 2022 11:30:11 +0300 (EEST) Received: from smtp202.mailbox.org (smtp202.mailbox.org [IPv6:2001:67c:2050:b231:465::202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-101.mailbox.org (Postfix) with ESMTPS id 4LVRxs1Fv3z9sTn for ; Sat, 25 Jun 2022 10:30:09 +0200 (CEST) From: Gyan Doshi To: ffmpeg-devel@ffmpeg.org Date: Sat, 25 Jun 2022 13:59:51 +0530 Message-Id: <20220625082951.11897-2-ffmpeg@gyani.pro> In-Reply-To: <20220625082951.11897-1-ffmpeg@gyani.pro> References: <20220625082951.11897-1-ffmpeg@gyani.pro> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4LVRxs1Fv3z9sTn Subject: [FFmpeg-devel] [PATCH v2 2/2] ffmpeg: add option -isync X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: yl/fdoS6GGaK This is a per-file input option that adjusts an input's timestamps with reference to another input, so that emitted packet timestamps account for the difference between the start times of the two inputs. Typical use case is to sync two or more live inputs such as from capture devices. Both the target and reference input source timestamps should be based on the same clock source. If not all inputs have timestamps, the wallclock times at the time of reception of inputs shall be used. --- doc/ffmpeg.texi | 16 ++++++++++++ fftools/ffmpeg.h | 2 ++ fftools/ffmpeg_opt.c | 59 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+) diff --git a/doc/ffmpeg.texi b/doc/ffmpeg.texi index d943f4d6f5..8fc85d3a15 100644 --- a/doc/ffmpeg.texi +++ b/doc/ffmpeg.texi @@ -518,6 +518,22 @@ see @ref{time duration syntax,,the Time duration section in the ffmpeg-utils(1) Like the @code{-ss} option but relative to the "end of file". That is negative values are earlier in the file, 0 is at EOF. +@item -isync @var{input_index} (@emph{input}) +Assign an input as a sync source. + +This will take the difference between the start times of the target and referenced inputs and +offset the timestamps of the target file by that difference. The source timestamps of the two +inputs should derive from the same clock source for expected results. If @code{copyts} is set +then @code{start_at_zero} must also be set. If at least one of the inputs has no starting +timestamp then the wallclock time at time of reception of the inputs is used as a best-effort +sync basis. + +Acceptable values are those that refer to a valid ffmpeg input index. If the sync reference is +the target index itself or @var{-1}, then no adjustment is made to target timestamps. A sync +reference may not itself be synced to any other input. + +Default value is @var{-1}. + @item -itsoffset @var{offset} (@emph{input}) Set the input time offset. diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index 69a368b8d1..dc74de6684 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -118,6 +118,7 @@ typedef struct OptionsContext { float readrate; int accurate_seek; int thread_queue_size; + int input_sync_ref; SpecifierOpt *ts_scale; int nb_ts_scale; @@ -410,6 +411,7 @@ typedef struct InputFile { at the moment when looping happens */ AVRational time_base; /* time base of the duration */ int64_t input_ts_offset; + int input_sync_ref; int64_t ts_offset; int64_t last_ts; diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index e08455478f..de858efbe9 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -235,6 +235,7 @@ static void init_options(OptionsContext *o) o->chapters_input_file = INT_MAX; o->accurate_seek = 1; o->thread_queue_size = -1; + o->input_sync_ref = -1; } static int show_hwaccels(void *optctx, const char *opt, const char *arg) @@ -287,6 +288,58 @@ static int parse_and_set_vsync(const char *arg, int *vsync_var, int file_idx, in return 0; } +static int apply_sync_offsets(void) +{ + for (int i = 0; i < nb_input_files; i++) { + InputFile *ref, *self = input_files[i]; + int64_t adjustment; + int64_t self_start_time, ref_start_time, self_seek_start, ref_seek_start; + int sync_fpw = 0; + + if (self->input_sync_ref == -1 || self->input_sync_ref == i) continue; + if (self->input_sync_ref >= nb_input_files || self->input_sync_ref < -1) { + av_log(NULL, AV_LOG_FATAL, "-isync for input %d references non-existent input %d.\n", i, self->input_sync_ref); + exit_program(1); + } + + if (copy_ts && !start_at_zero) { + av_log(NULL, AV_LOG_FATAL, "Use of -isync requires that start_at_zero be set if copyts is set.\n"); + exit_program(1); + } + + ref = input_files[self->input_sync_ref]; + if (ref->input_sync_ref != -1 && ref->input_sync_ref != self->input_sync_ref) { + av_log(NULL, AV_LOG_ERROR, "-isync for input %d references a resynced input %d. Sync not set.\n", i, self->input_sync_ref); + continue; + } + + if (self->ctx->start_time_realtime != AV_NOPTS_VALUE && ref->ctx->start_time_realtime != AV_NOPTS_VALUE) { + self_start_time = self->ctx->start_time_realtime; + ref_start_time = ref->ctx->start_time_realtime; + } else if (self->ctx->start_time != AV_NOPTS_VALUE && ref->ctx->start_time != AV_NOPTS_VALUE) { + self_start_time = self->ctx->start_time; + ref_start_time = ref->ctx->start_time; + } else { + self_start_time = self->ctx->first_pkt_wallclock; + ref_start_time = ref->ctx->first_pkt_wallclock; + sync_fpw = 1; + } + + self_seek_start = self->start_time == AV_NOPTS_VALUE ? 0 : self->start_time; + ref_seek_start = ref->start_time == AV_NOPTS_VALUE ? 0 : ref->start_time; + + adjustment = (self_start_time - ref_start_time) + !copy_ts*(self_seek_start - ref_seek_start) + ref->input_ts_offset; + + self->ts_offset += adjustment; + + av_log(NULL, AV_LOG_INFO, "Adjusted ts offset for Input #%d by %"PRId64"d us to sync with Input #%d", i, adjustment, self->input_sync_ref); + if (sync_fpw) av_log(NULL, AV_LOG_INFO, " using reception wallclock time. Sync may not be obtained"); + av_log(NULL, AV_LOG_INFO, ".\n"); + } + + return 0; +} + static int opt_filter_threads(void *optctx, const char *opt, const char *arg) { av_free(filter_nbthreads); @@ -1305,6 +1358,7 @@ static int open_input_file(OptionsContext *o, const char *filename) f->ist_index = nb_input_streams - ic->nb_streams; f->start_time = o->start_time; f->recording_time = o->recording_time; + f->input_sync_ref = o->input_sync_ref; f->input_ts_offset = o->input_ts_offset; f->ts_offset = o->input_ts_offset - (copy_ts ? (start_at_zero && ic->start_time != AV_NOPTS_VALUE ? ic->start_time : 0) : timestamp); f->nb_streams = ic->nb_streams; @@ -3489,6 +3543,8 @@ int ffmpeg_parse_options(int argc, char **argv) goto fail; } + apply_sync_offsets(); + /* create the complex filtergraphs */ ret = init_complex_filters(); if (ret < 0) { @@ -3603,6 +3659,9 @@ const OptionDef options[] = { { "accurate_seek", OPT_BOOL | OPT_OFFSET | OPT_EXPERT | OPT_INPUT, { .off = OFFSET(accurate_seek) }, "enable/disable accurate seeking with -ss" }, + { "isync", HAS_ARG | OPT_INT | OPT_OFFSET | + OPT_EXPERT | OPT_INPUT, { .off = OFFSET(input_sync_ref) }, + "Indicate the input index for sync reference", "sync ref" }, { "itsoffset", HAS_ARG | OPT_TIME | OPT_OFFSET | OPT_EXPERT | OPT_INPUT, { .off = OFFSET(input_ts_offset) }, "set the input ts offset", "time_off" },