From patchwork Mon Aug 8 00:01:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 37179 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:92a9:b0:8d:7f3b:94fc with SMTP id q41csp1523993pzg; Sun, 7 Aug 2022 17:01:53 -0700 (PDT) X-Google-Smtp-Source: AA6agR4uaLjHjQA0iTId502DK3tjC7e9vAyKhudY95LM+v7pm5n2KVEDCxtKWyUxldgT2HiDD9bH X-Received: by 2002:a17:907:5ce:b0:730:bae0:deb with SMTP id wg14-20020a17090705ce00b00730bae00debmr12488799ejb.181.1659916913302; Sun, 07 Aug 2022 17:01:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1659916913; cv=none; d=google.com; s=arc-20160816; b=t4cmcXFfVJtVRyTyu5t8V+alzbu6RVlF/0QtuForLaJ03nMAkQfysH2TtvRisgjNoR fj6WRyOY/AMJf8p0OhjBsfZYeGtj8IyrmWLIdKrV52mbKJ9s+SNUwACqZWiuuy8XEEbN XpoZTYTqOqBtwtEcODp7MEqayTRNJ4RnM2br5ELfCqMbS4ztz07eBloPYz8nxFO3cKIV pI/6+Q52DCgw1cBJgvhvOp6qfvAThNXxFYkp+Xlzh+hN6qCIR+Dl3fOp6sLgBWiuE07b nRK0ZG6Q8iL8kewif9fHinUTilWnJ94KuqdEISgoosfy/5MPxBSDdrZwbmsh4J8qdbLT mL9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=kqVc762zynYGh2/3nFF/1IafyPmSjg7PCl7K5eFAwUw=; b=zpazN2f/1YCpnIMLZ1b4GBFQU1zywOAK1HvD917DB5wn5RSyixt47hEX/SpiGU9b2n y740FbhhVTJHfKHztIPHWHofvF8UF3bsLmF2+x0vsO1YNQ1G08d8BiX8AOwciLQPr8nI fKoDDBAwKApEJPRIBltES+qATyqp8j+aptr8I7asTEl4UTPcHQzlfn4DfJ3ygtBWglxI MFSRZ0MDZrRrVglXgGuHGCcIplGgi3XuhMxOF+/VdGxhiXTDs8jogFayM7tOUb/vh2dR XbjdtC28WFf7x+jYSqF5Ro4isNY3IyKwhty8d1lP4qoRKWmg6q3OHpKpxcJZntYJdpLp qODw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=C75kXaLv; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y20-20020a056402441400b0043be7b8aebcsi5790270eda.97.2022.08.07.17.01.52; Sun, 07 Aug 2022 17:01:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=C75kXaLv; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0454F68B70B; Mon, 8 Aug 2022 03:01:49 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-vk1-f177.google.com (mail-vk1-f177.google.com [209.85.221.177]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 44E0568B6B0 for ; Mon, 8 Aug 2022 03:01:43 +0300 (EEST) Received: by mail-vk1-f177.google.com with SMTP id w129so3673041vkg.10 for ; Sun, 07 Aug 2022 17:01:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc; bh=bU3DO25wcDB9RsjmBxU1dAHqiEQt/I3Kw38srP6/kmo=; b=C75kXaLvSlcepbcVzxN0J6VqcKmfNnCTj+nEBbp0pJrfArcZ0nnsjMG1Km+i/JnEKk rotB/Da4wwk23crPrVmDNpLVksHjS7rGx4dgjINA9GCR88/LEfRS7v0Oja+2pLbBfJbr H8dcQiLnBCFgBsjaeCo5gzEx7YmwBBBHYfns44OJoLc+UKTlyqZyO7be1vX40fg2FGPk gGTb1D4WDWlcU+Nmn5iJ2lUI9ahZfuLA1Z4PhlLLprBIQnNv6zBAzbzYpvOnJpzU9KY8 iEUchHlbbI83iFD05ZxF5Y6kDse8YQvg7A0TC83PNV0wZlUn5cqNPwSj1ytZdii0i1pD VEjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc; bh=bU3DO25wcDB9RsjmBxU1dAHqiEQt/I3Kw38srP6/kmo=; b=uCze1+AHjfMEAJbCsWhpvn75DV4m9cM6LZ5GGzYIpZ0GMQwqxx7fUBbaRTgrnUtTEB 1O19hvcguFLEVBUvfYD+mvCU4KTEl7Or63VaLJ4TdAuLj+Bgk2HveEk9QI9kSubfa8F0 CBE9IV19IC6tDZaAzEdSS2Sf3dWmLGXXsMDPKxZrpVx2vxXMadwKa5OdEoKtoLVVGv17 pJ1yBHISHxf9+8YqmKV26nc/aKt7EbCjwy1Ogs71Q+PALVbO76CZnVooN1RZcHyH1DNk xmbozdN0SewphE4CRSa/KHKUvyi1elgyzJoOaTPv3qOLbkDNUW9SOKb2g1vjReKVOYg7 gpbg== X-Gm-Message-State: ACgBeo1xB/OsIWAM/u/Bcq0bUUosEEjgbGZjwKk49yxExZqN0ccNLJRo 1C1Pn14pPFt5ApNnCpOO4LPgGTlTpaA= X-Received: by 2002:a05:6122:643:b0:378:d1a9:b768 with SMTP id h3-20020a056122064300b00378d1a9b768mr2250634vkp.41.1659916901051; Sun, 07 Aug 2022 17:01:41 -0700 (PDT) Received: from localhost.localdomain ([186.136.131.204]) by smtp.gmail.com with ESMTPSA id t2-20020a0561020a0200b00385d88375f4sm6937561vsa.32.2022.08.07.17.01.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 07 Aug 2022 17:01:40 -0700 (PDT) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Sun, 7 Aug 2022 21:01:24 -0300 Message-Id: <20220808000124.12794-1-jamrial@gmail.com> X-Mailer: git-send-email 2.37.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avfilter/framesync: add a new option to set how to sync streams based on secondary input timestamps X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: epPQONtioq8W Include two values for it, a default one that sets/keeps the current behavior, where the frame event generated by the primary input will have a timestamp equal or higher than frames in secondary input, plus a new one where the secondary input frame will be that with the absolute closest timestamp to that of the frame event one. Addresses ticket #9689, where the new optional behavior produces better frame syncronization. Signed-off-by: James Almer --- libavfilter/framesync.c | 19 +++++++++++++++++++ libavfilter/framesync.h | 23 +++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/libavfilter/framesync.c b/libavfilter/framesync.c index 7510550d8e..bd784f6e0b 100644 --- a/libavfilter/framesync.c +++ b/libavfilter/framesync.c @@ -42,6 +42,13 @@ static const AVOption framesync_options[] = { { "pass", "Pass through the main input.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_PASS }, .flags = FLAGS, "eof_action" }, { "shortest", "force termination when the shortest input terminates", OFFSET(opt_shortest), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, { "repeatlast", "extend last frame of secondary streams beyond EOF", OFFSET(opt_repeatlast), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, FLAGS }, + { "ts_sync_mode", "How strictly to sync streams based on secondary input timestamps", + OFFSET(opt_ts_sync_mode), AV_OPT_TYPE_INT, { .i64 = TS_DEFAULT }, + TS_DEFAULT, TS_NEAREST, .flags = FLAGS, "ts_sync_mode" }, + { "default", "Frame from secondary input with the nearest lower or equal timestamp to the primary input frame", + 0, AV_OPT_TYPE_CONST, { .i64 = TS_DEFAULT }, .flags = FLAGS, "ts_sync_mode" }, + { "nearest", "Frame from secondary input with the absolute nearest timestamp to the primary input frame", + 0, AV_OPT_TYPE_CONST, { .i64 = TS_NEAREST }, .flags = FLAGS, "ts_sync_mode" }, { NULL } }; static const AVClass framesync_class = { @@ -110,6 +117,14 @@ static void framesync_sync_level_update(FFFrameSync *fs) av_assert0(level <= fs->sync_level); if (level < fs->sync_level) av_log(fs, AV_LOG_VERBOSE, "Sync level %u\n", level); + if (fs->opt_ts_sync_mode > TS_DEFAULT) { + for (i = 0; i < fs->nb_in; i++) { + if (fs->in[i].sync < level) + fs->in[i].ts_mode = fs->opt_ts_sync_mode; + else + fs->in[i].ts_mode = TS_DEFAULT; + } + } if (level) fs->sync_level = level; else @@ -187,6 +202,10 @@ static int framesync_advance(FFFrameSync *fs) } for (i = 0; i < fs->nb_in; i++) { if (fs->in[i].pts_next == pts || + (fs->in[i].ts_mode == TS_NEAREST && + fs->in[i].have_next && + fs->in[i].pts_next != INT64_MAX && fs->in[i].pts != AV_NOPTS_VALUE && + FFABS(pts - fs->in[i].pts_next) < FFABS(pts - fs->in[i].pts)) || (fs->in[i].before == EXT_INFINITY && fs->in[i].state == STATE_BOF)) { av_frame_free(&fs->in[i].frame); diff --git a/libavfilter/framesync.h b/libavfilter/framesync.h index a246d2d1e5..233f50a0eb 100644 --- a/libavfilter/framesync.h +++ b/libavfilter/framesync.h @@ -75,6 +75,27 @@ enum FFFrameSyncExtMode { EXT_INFINITY, }; +/** + * Timestamp syncronization mode + * + * Describe how the frames of a stream are syncronized based on timestamp + * distance. + */ +enum FFFrameTSSyncMode { + + /** + * Sync to frames from secondary input with the nearest, lower or equal + * timestamp to the frame event one. + */ + TS_DEFAULT, + + /** + * Sync to frames from secondary input with the absolute nearest timestamp + * to the frame event one. + */ + TS_NEAREST, +}; + /** * Input stream structure */ @@ -138,6 +159,7 @@ typedef struct FFFrameSyncIn { */ unsigned sync; + enum FFFrameTSSyncMode ts_mode; } FFFrameSyncIn; /** @@ -205,6 +227,7 @@ typedef struct FFFrameSync { int opt_repeatlast; int opt_shortest; int opt_eof_action; + int opt_ts_sync_mode; } FFFrameSync;