From patchwork Wed Dec 22 16:20:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Walton X-Patchwork-Id: 32843 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp6523341iog; Wed, 22 Dec 2021 08:20:39 -0800 (PST) X-Google-Smtp-Source: ABdhPJz3I9P6XwguL469BCZOM5VbhWy1Svfh0Cn5AK1Ilaw+SU+sQg0JYF8tS82mqAk8Dur/qkfa X-Received: by 2002:a17:907:d29:: with SMTP id gn41mr3217057ejc.124.1640190039816; Wed, 22 Dec 2021 08:20:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1640190039; cv=none; d=google.com; s=arc-20160816; b=zobyId/FA0MkiFs4pQ4XLR8pDlsBdqC2FLjp6RPun82NgXey1b6NLlOjzogrjxCzNV /XBO5rg26ZRBHFkr5G5MeBQo28WXi5aPARx+CyhaLGcQh9suY+37wrV0VTUoMzA79tol uRwqGERPlP2mSW5eZHcKkfEpLPbamC8qWTURuvBtfQdWA/d0c0b/2eVAcMK0CkcUHsV1 5KJZ/gVbrjrdkgdopQneeX8T/RPvGa0ceZbpZ78GjRCzQSlHcyajXrMBIALOfZBLMzPN NaFSc4xFxAWANBgnYB3VbRm/21RtdczTpwQpKyDOKB69wcl0dETZA1e0JRLt41bjlKt5 KtbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:user-agent:date:to:from:message-id :dkim-signature:delivered-to; bh=6XqpLeb6JMk3LyMP4mh8ctm0KmUiq52urx1QaeqEnnw=; b=a1uAzeGRXQb+d2Lpa/8ZkkpGh747OOHHenBBWTQBQM2DGqHYXL0e7VRHdnEnJkkabe Fku77NEBNnPrI6mER/MbSoVt35rTo2nGtIZA9L70qeCtTJnmkP9C5DpkUzT2PLVlZ4HC Lqun+jcMzjU4idS3udBkpV8xJXnjOWvs33yfpfIBJXkORklYsbKMG1epj8cFu7sZ2Ud1 f3rnAVz4B1XijIAqjNdxNhHAMBybSp99ihW6Lz2R1E48HLvo4ZHH5p4w49aDFakdS5Rp SCApeMQK5hTqC3PNvxHYvS14m1IpHQ0T+1DDzlFx7ZWDhCh3N1OWnX84IGTl95om2sSt mm1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@kepstin.ca header.s=google header.b=dkwnJHjc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=REJECT dis=NONE) header.from=kepstin.ca Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a14si1096444edv.226.2021.12.22.08.20.39; Wed, 22 Dec 2021 08:20:39 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@kepstin.ca header.s=google header.b=dkwnJHjc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=REJECT dis=NONE) header.from=kepstin.ca Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A90F568B0D6; Wed, 22 Dec 2021 18:20:36 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7898368B0B9 for ; Wed, 22 Dec 2021 18:20:30 +0200 (EET) Received: by mail-qk1-f169.google.com with SMTP id r139so1900426qke.9 for ; Wed, 22 Dec 2021 08:20:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kepstin.ca; s=google; h=message-id:subject:from:to:date:user-agent:mime-version :content-transfer-encoding; bh=9d+uCONT+Lsjhr9TW1XhEV7e6Vi8O4/tGx96ZP4Eqr0=; b=dkwnJHjcGVwYfmiehBH3MulYpWHogV03xi29iClkHq3JL+uCJ7OyHejmiXcfDxBVEC P5svaYTTgIITm1lbUIG7zUhev4DT2yy6+W/AKbM1Ho17zJ2zcoAnmXp7Co4YrcW+Bo/k yQQqyswZXwsJ7pdLIR+GUxYBbUeFbLuCiILHg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:subject:from:to:date:user-agent :mime-version:content-transfer-encoding; bh=9d+uCONT+Lsjhr9TW1XhEV7e6Vi8O4/tGx96ZP4Eqr0=; b=evyL9RBj7aZ7Sg0gpM9rf7TAEoH5wDSb8vkxyqaC1FYhDJNhHOTfutjwe89SfxRJP5 K/36mCRYP0XgKGM4sLOwUd0nBkvswCUfpLKhUel013yljaV1K8O6DOjb0oYD1+TRYJkF ywwE64QSkc44BjvG3xHEfm0EJXgmdhRJoaPRu5KTyNKovIKc4ZDVstKkx7WF7nyxDfCw YBe9d3vhJm+QHPEDxJSq4npA2Gt13DFxASxXFnhSBITc85i4C0h/1EmQEgG1MH3aqtZh pzgfVWumWuPNSsl4FV4aqk4BirQfuO/qrUZOkYqS2wYYrm8zI3n7uo8YPWW7SfxFssLW ybeg== X-Gm-Message-State: AOAM531ZWh8fnJdnAD6/Ih2apyt3zTfTv9AAeg2nlfYFUfAn0wEG7Q08 Jwn4swyrSItpurS/UFdV+3ggB8iph9P81Uxa X-Received: by 2002:a05:620a:2905:: with SMTP id m5mr2524753qkp.598.1640190028378; Wed, 22 Dec 2021 08:20:28 -0800 (PST) Received: from saya.kepstin.ca (dhcp-108-168-125-232.cable.user.start.ca. [108.168.125.232]) by smtp.gmail.com with ESMTPSA id m1sm2215313qkn.115.2021.12.22.08.20.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Dec 2021 08:20:27 -0800 (PST) Message-ID: <837ff72bd80c351924dc4cf0d96c003f082681f3.camel@kepstin.ca> From: Calvin Walton To: FFmpeg development discussions and patches Date: Wed, 22 Dec 2021 11:20:26 -0500 User-Agent: Evolution 3.40.3 MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC PATCH] vf_fps: Requantize pts of CFR videos X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: acx2tD9scWt6 This is mostly to avoid oddities in small framerate adjustments when you have input videos from containers such as matroska, where the pts values are quantized with a large enough error to cause issues. An example, when converting from 24/1.001 fps to 24 fps with round=down from an mkv source (inlink time base is 1/1000, outlink is 1001/24000): In PTS | Out PTS | Rounded Out PTS 69292 | 1663.008 | 1663 69333 | 1663.992 | 1663 69375 | 1665.000 | 1665 In this example, the fps filter would drop the frame with pts 69292, then duplicate the frame with pts 69333, an undesirable result. By first requantizing the input pts to the inlink frame rate, the result looks much nicer: In PTS | Req. PTS | Out PTS 69292 | 1661 | 1662 69333 | 1662 | 1663 69375 | 1663 | 1664 (Note that the same rounding mode is used for both conversions, resulting in the final out pts being a bit lower in this case. With the normal nearest mode, it would be closer.) I've verified that in conversions of longer mkv files to "close" framerates that previously had issues due to quantization, this significantly reduces the number of incorrectly dropped or duplicated frames. The potential downside of this change is that if an input file is probed as CFR but is actually VFR, then the results will be poor (you'll get unnecessarily dropped frames or added judder). A new option, "requantize", is added to allow disabling this behaviour in those cases. Signed-off-by: Calvin Walton --- libavfilter/vf_fps.c | 48 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/libavfilter/vf_fps.c b/libavfilter/vf_fps.c index 99e679441e..d010083a35 100644 --- a/libavfilter/vf_fps.c +++ b/libavfilter/vf_fps.c @@ -74,6 +74,7 @@ typedef struct FPSContext { char *framerate; ///< expression that defines the target framerate int rounding; ///< AVRounding method for timestamps int eof_action; ///< action performed for last frame in FIFO + int requantize; ///< whether to requantize timestamps of cfr inputs /* Set during outlink configuration */ int64_t in_pts_off; ///< input frame pts offset for start_time handling @@ -111,6 +112,8 @@ static const AVOption fps_options[] = { { "eof_action", "action performed for last frame", OFFSET(eof_action), AV_OPT_TYPE_INT, { .i64 = EOF_ACTION_ROUND }, 0, EOF_ACTION_NB-1, V|F, "eof_action" }, { "round", "round similar to other frames", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_ROUND }, 0, 0, V|F, "eof_action" }, { "pass", "pass through last frame", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_PASS }, 0, 0, V|F, "eof_action" }, + { "requantize", "requantize input timestamps in CFR video based on framerate", + OFFSET(requantize), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, V|F }, { NULL } }; @@ -177,6 +180,7 @@ static int config_props(AVFilterLink* outlink) FPSContext *s = ctx->priv; double var_values[VARS_NB], res; + AVRational in_time_base; int ret; var_values[VAR_SOURCE_FPS] = av_q2d(inlink->frame_rate); @@ -193,6 +197,18 @@ static int config_props(AVFilterLink* outlink) outlink->frame_rate = av_d2q(res, INT_MAX); outlink->time_base = av_inv_q(outlink->frame_rate); + /* Disable requantization if input video is VFR */ + if (s->requantize && inlink->frame_rate.num == 1 && inlink->frame_rate.den == 0) { + av_log(ctx, AV_LOG_INFO, "Not requantizing input timestamps; video is VFR\n"); + s->requantize = 0; + } + + in_time_base = inlink->time_base; + if (s->requantize) { + in_time_base = av_inv_q(inlink->frame_rate); + av_log(ctx, AV_LOG_VERBOSE, "Requantizing input timestamps to time_base=%d/%d\n", in_time_base.num, in_time_base.den); + } + /* Calculate the input and output pts offsets for start_time */ if (s->start_time != DBL_MAX && s->start_time != AV_NOPTS_VALUE) { double first_pts = s->start_time * AV_TIME_BASE; @@ -201,7 +217,7 @@ static int config_props(AVFilterLink* outlink) s->start_time); return AVERROR(EINVAL); } - s->in_pts_off = av_rescale_q_rnd(first_pts, AV_TIME_BASE_Q, inlink->time_base, + s->in_pts_off = av_rescale_q_rnd(first_pts, AV_TIME_BASE_Q, in_time_base, s->rounding | AV_ROUND_PASS_MINMAX); s->out_pts_off = av_rescale_q_rnd(first_pts, AV_TIME_BASE_Q, outlink->time_base, s->rounding | AV_ROUND_PASS_MINMAX); @@ -220,7 +236,8 @@ static int read_frame(AVFilterContext *ctx, FPSContext *s, AVFilterLink *inlink, { AVFrame *frame; int ret; - int64_t in_pts; + AVRational in_time_base; + int64_t orig_pts, in_pts; /* Must only be called when we have buffer room available */ av_assert1(s->frames_count < 2); @@ -231,16 +248,24 @@ static int read_frame(AVFilterContext *ctx, FPSContext *s, AVFilterLink *inlink, if (ret < 0) return ret; + /* Requantize CFR video pts based on frame rate */ + in_time_base = inlink->time_base; + orig_pts = in_pts = frame->pts; + if (s->requantize) { + in_time_base = av_inv_q(inlink->frame_rate); + in_pts = av_rescale_q_rnd(in_pts, inlink->time_base, in_time_base, + s->rounding | AV_ROUND_PASS_MINMAX); + } + /* Convert frame pts to output timebase. * The dance with offsets is required to match the rounding behaviour of the * previous version of the fps filter when using the start_time option. */ - in_pts = frame->pts; frame->pts = s->out_pts_off + av_rescale_q_rnd(in_pts - s->in_pts_off, - inlink->time_base, outlink->time_base, + in_time_base, outlink->time_base, s->rounding | AV_ROUND_PASS_MINMAX); - av_log(ctx, AV_LOG_DEBUG, "Read frame with in pts %"PRId64", out pts %"PRId64"\n", - in_pts, frame->pts); + av_log(ctx, AV_LOG_DEBUG, "Read frame with in pts %"PRId64", req pts %"PRId64", out pts %"PRId64"\n", + orig_pts, in_pts, frame->pts); s->frames[s->frames_count++] = frame; s->frames_in++; @@ -304,7 +329,16 @@ static int write_frame(AVFilterContext *ctx, FPSContext *s, AVFilterLink *outlin static void update_eof_pts(AVFilterContext *ctx, FPSContext *s, AVFilterLink *inlink, AVFilterLink *outlink, int64_t status_pts) { int eof_rounding = (s->eof_action == EOF_ACTION_PASS) ? AV_ROUND_UP : s->rounding; - s->status_pts = av_rescale_q_rnd(status_pts, inlink->time_base, outlink->time_base, + AVRational in_time_base; + + /* Requantize CFR video pts based on frame rate */ + in_time_base = inlink->time_base; + if (s->requantize) { + in_time_base = av_inv_q(inlink->frame_rate); + status_pts = av_rescale_q_rnd(status_pts, inlink->time_base, in_time_base, + eof_rounding | AV_ROUND_PASS_MINMAX); + } + s->status_pts = av_rescale_q_rnd(status_pts, in_time_base, outlink->time_base, eof_rounding | AV_ROUND_PASS_MINMAX); av_log(ctx, AV_LOG_DEBUG, "EOF is at pts %"PRId64"\n", s->status_pts);