From patchwork Sun Oct 6 22:02:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stephen Hutchinson X-Patchwork-Id: 52081 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:938f:0:b0:48e:c0f8:d0de with SMTP id z15csp1595774vqg; Sun, 6 Oct 2024 15:11:04 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWp+48x4/a3rlnQSuUhMWSOj7/zlwOiAVnoa08AccKIvmmIOGo/uLITHuC/WXkM9N2kOtVLJyjj3xczxYVyXfIO@gmail.com X-Google-Smtp-Source: AGHT+IF3B+V4wrkcYcvOr4HeK+GZfOQIu9fCAyFjjIE9euEqoydqB0QsDGbKTKSKrXx4oWWNtziX X-Received: by 2002:a05:651c:154a:b0:2ef:2cdb:5053 with SMTP id 38308e7fff4ca-2faf3d8a8e9mr33445021fa.37.1728252663942; Sun, 06 Oct 2024 15:11:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1728252663; cv=none; d=google.com; s=arc-20240605; b=JczKo+GDujjnC2hTZpn1esE0oR58/oNkKgKomDxdt+e7Mkmhz2BVw+wVHknkdY+Vma nTcORh1AOvU+btA5Rh6nzQQPOCB0W/jlXchnqsBOEsi1WX+f0jQpepT/NVIdV3DYdH9y sZkNtl/4Dc7U430FHmBX2yKQn8H2Ew3zWYVBUxCspti8jm6EzRVe6rpgbqFOiXhI5Vo0 f4dLbY/lTSuRW0O/pJ8dc6pT5NFOw2skApbMpvtMDafaXOlbmvKZ6WS8VY43QiDWT2s1 heYz/vR8cRF3dSqKoImiKZbxfvSyY/h+npa5vxaUFNq7oWsGeif4n74ysd4cxPrdkykF FGcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=BXgT34OCpnCybCogTEdWx8C0yahJEl0z8J+bQf7C8Gc=; fh=dF+SszRLFyubUCP/LDvIY2NwiNB1Gp0Txz6w70WKHpo=; b=XcBGKS0BL1vrNDhslo+69SYNPf814lN4QACpLJbrtYLEpohmKalsHEZ8YTLe+f6TK7 9IJM0KM6/LXXdYZpAVCKcK87Eg45TalWlJ3sm0Mw5mfkFQ1Dg/2rJAMxiV38hbIxdbpe 3mtsyRV25fnx4zCjh/U8VvZ1mTnljVTV/po0klryfD7G+0BnRa20n7VJePiPPmhlWKNd 6WdFA479ZiOMrH7cNjrY3T8UXG7TFj20VLs9WCrtaymA22vfyttDTpWqMcf+0YdlIQML g5/7kG1p+0Y64KzZDUeItDwLgbaJchsQcyoRkdRnia4CcXPj9ermMykGZsnRCjAKR9C6 EFZw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=JDu4SLlH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com; dara=fail header.i=@gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-539afec9169si1564568e87.165.2024.10.06.15.11.03; Sun, 06 Oct 2024 15:11:03 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=JDu4SLlH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com; dara=fail header.i=@gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2887668D9E2; Mon, 7 Oct 2024 01:02:54 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5453268D278 for ; Mon, 7 Oct 2024 01:02:47 +0300 (EEST) Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-6e20a8141c7so32177307b3.0 for ; Sun, 06 Oct 2024 15:02:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728252165; x=1728856965; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=tzu70Vsyeo6bHfzTWGeAH7wUWuKoyeRJsJKqhGOxwUo=; b=JDu4SLlHVnb/Cq0UQde8foZoZe7pBcwp7Swm19jnODod+PLqrMCfiy6KU9qg4Pm630 5PrQzGblluBaxwbZgVFZItQN0/naARUK8HZWm5zels0GArU/UwRTGLjuOVbZ9f7o/5Rp pdAgh/b3zSzwtaHJxOIcaGqI0urrVvYX2cNMG6FOdbB8b7zVIg4dyxMsoXxE9eVMKidW +EAfXXJH9Ye/HH4McxEsS6CbOV9gTE4GKlSeDDSW1lF5CkM8n1f4UkwjVsABBS9rxYHe DBHlxKATY/jW1pCENfuaG7beauRtb1KV3104bsEern282dRIGyQQFk1zKdDTcht30nqj kN8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728252165; x=1728856965; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=tzu70Vsyeo6bHfzTWGeAH7wUWuKoyeRJsJKqhGOxwUo=; b=e3APXYC9X14fKzwb1zOqJ0u3LWzo6Clzx1PK9CsnpocYHZRYdYJpvviTFPetrvJ+g0 67bLh+dZ6wGCUhgL9i59Z/6vGwnjomVK54Wbqoo/vkM7E62tUR93mfEsYVsaAlEWw2h0 +Do9f0MQwWByriBvnRMROsGAsI/mtLXwnwAsdrkiwDQSessCmhc9kdPmUAegfrg75wvX 79D000ysdGvPoanh2VxvH0ga/G4VDEnvpf56tOy58eZa9tf0fierdZuUedL+uSD3WNlh ryJ8BgqkgA/drM2XMWF0Vdu2WxmVEGxrPig7QAG6o/2ZkU7EQ8QMk4aZyR9oHP0lRv8p /n4g== X-Gm-Message-State: AOJu0Yx+2B1I1ANciwqkH4Y2bfuYOyl31ChVwPdHnC9dxwbjasTYKkpc zLQbiVrCR05I13YMIGrI33zAts0zKPG6OciQIBz67GVvcR4YJ8CrD5zJ6Q== X-Received: by 2002:a05:690c:18:b0:6e2:1570:2d4a with SMTP id 00721157ae682-6e2c72f6ab1mr78488297b3.30.1728252165432; Sun, 06 Oct 2024 15:02:45 -0700 (PDT) Received: from cappuccino.. (syn-173-170-140-230.res.spectrum.com. [173.170.140.230]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6e2d927eb29sm8283927b3.39.2024.10.06.15.02.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Oct 2024 15:02:44 -0700 (PDT) From: Stephen Hutchinson To: ffmpeg-devel@ffmpeg.org Date: Sun, 6 Oct 2024 18:02:38 -0400 Message-ID: <20241006220238.16196-1-qyot27@gmail.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH][WIP] avisynth: support variable fps X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Stephen Hutchinson Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: MdeAj77gmcjv Draft #1. For testing reasons, the patch turns on vfr mode by default so it can be tested more quickly. The seeking issues described below are problematic enough that this behavior would be reversed when actually committed, unless it can be satisfactorily resolved before then. Currently has only been tested with GCC 13 (Ubuntu) and 14 (mingw-w64). What works: * Use _DurationNum and _DurationDen frame properties (if present) to set the pkt->[pts|dts|duration]. * If the properties are not present, fall back to the legacy CFR mode, as well as allow users to switch back by choice with the existing -avisynth_flags option ('-avisynth_flags -vfr'). * Transcoding to other formats (ex. ffv1 in mkv) retains the adjusted packet durations correctly. What doesn't work: * As reported by ffprobe, the original mkv I was using as a test file has an offset between the pts and dts, as if the first couple of frames get skipped. While the logic *does* perform a similar enough skew to make the audio sync correct, the original mkv lists the dtses as N/A. * Even though the ptses match that of the original mkv file, transcoding into another mkv reports back 'starting new cluster due to timestamp' warnings, and the ffmpeg-created mkv file has some strange audio seek behavior. Remuxing with mkvmerge fixes it, so I'm not sure if this is really something to worry about with the avisynth demuxer or not. * Seeking is broken. The original file can seek without issues, and instantaneously. Files transcoded from the script act far more like, if not identically, to the original file. Seeking *forward* in the script sometimes works, but is slow, and seeking backward is completely non-functional. The size of the seeks are also seemingly not predictable; a minute or two in one area, or 15 minutes the next. Errata: * There are currently entries added to the AviSynthContext struct to calculate the frame's framerate (avs->curr_fps[num|den]). This was used in some experiments, but didn't lead to any fixes in the short term. I don't know if maybe there is a utility to this in some form that would actually be useful, which is why it's still here in the draft. * It occurs to me that comparing the seek behavior with mkv (or any other container capable of variable frame durations) is potentially like comparing apples and oranges because those containers have additional structures (e.g. an index) that handles some of the load of making sure seeking can be done accurately by their corresponding demuxers. This would mean that such an index would need to be present for the script to actually get functional seeking in vfr mode: whether by it being in the frame properties or - at worst - exposed by the AviSynth+ core itself through an api function or entirely new structure, in which case vfr-mode seeking would have to be shelved until those things appear. * Clearly, if some of the things (all the scaledur/curr_* math) could be achieved with pre-existing functions, that would be better than having to handle those calculations the way I did. That's why this is a draft. --- libavformat/avisynth.c | 95 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 82 insertions(+), 13 deletions(-) diff --git a/libavformat/avisynth.c b/libavformat/avisynth.c index cb2be10925..958e880407 100644 --- a/libavformat/avisynth.c +++ b/libavformat/avisynth.c @@ -53,6 +53,7 @@ #endif #include +#include typedef struct AviSynthLibrary { void *library; @@ -96,6 +97,7 @@ typedef enum AviSynthFlags { AVISYNTH_FRAMEPROP_MATRIX = (1 << 4), AVISYNTH_FRAMEPROP_CHROMA_LOCATION = (1 << 5), AVISYNTH_FRAMEPROP_SAR = (1 << 6), + AVISYNTH_FRAMEPROP_VFR = (1 << 7), } AviSynthFlags; typedef struct AviSynthContext { @@ -115,6 +117,15 @@ typedef struct AviSynthContext { int error; uint32_t flags; + bool is_vfr; + int scaled_durnum; + int scaled_durden; + + int curr_duration; + int total_duration; + int curr_fpsnum; + int curr_fpsden; + struct AviSynthLibrary avs_library; } AviSynthContext; @@ -230,13 +241,6 @@ static int avisynth_create_stream_video(AVFormatContext *s, AVStream *st) st->codecpar->width = avs->vi->width; st->codecpar->height = avs->vi->height; - st->avg_frame_rate = (AVRational) { avs->vi->fps_numerator, - avs->vi->fps_denominator }; - st->start_time = 0; - st->duration = avs->vi->num_frames; - st->nb_frames = avs->vi->num_frames; - avpriv_set_pts_info(st, 32, avs->vi->fps_denominator, avs->vi->fps_numerator); - switch (avs->vi->pixel_type) { /* 10~16-bit YUV pix_fmts (AviSynth+) */ @@ -721,6 +725,21 @@ static int avisynth_create_stream_video(AVFormatContext *s, AVStream *st) st->sample_aspect_ratio = (AVRational){ sar_num, sar_den }; } + /* Variable frame rate */ + if(avs->flags & AVISYNTH_FRAMEPROP_VFR) { + if((avs->avs_library.avs_prop_get_type(avs->env, avsmap, "_DurationDen") == AVS_PROPTYPE_UNSET) || + (avs->avs_library.avs_prop_get_type(avs->env, avsmap, "_DurationNum") == AVS_PROPTYPE_UNSET)) { + avs->is_vfr = false; + avpriv_set_pts_info(st, 32, avs->vi->fps_denominator, avs->vi->fps_numerator); + } else { + avs->is_vfr = true; + avpriv_set_pts_info(st, 64, 1, 1000); + } + } else { + avs->is_vfr = false; + avpriv_set_pts_info(st, 32, avs->vi->fps_denominator, avs->vi->fps_numerator); + } + avs->avs_library.avs_release_video_frame(frame); } else { st->codecpar->field_order = AV_FIELD_UNKNOWN; @@ -737,6 +756,16 @@ static int avisynth_create_stream_video(AVFormatContext *s, AVStream *st) } } + if (avs->is_vfr == false) { + st->avg_frame_rate = (AVRational) { avs->vi->fps_numerator, + avs->vi->fps_denominator }; + st->start_time = 0; + st->duration = avs->vi->num_frames; + st->nb_frames = avs->vi->num_frames; + } else { + st->start_time = AV_NOPTS_VALUE; + } + return 0; } @@ -904,7 +933,9 @@ static int avisynth_read_packet_video(AVFormatContext *s, AVPacket *pkt, unsigned char *dst_p; const unsigned char *src_p; int n, i, plane, rowsize, planeheight, pitch, bits, ret; + float scaledur; const char *error; + AVRational dur; if (avs->curr_frame >= avs->vi->num_frames) return AVERROR_EOF; @@ -926,12 +957,48 @@ static int avisynth_read_packet_video(AVFormatContext *s, AVPacket *pkt, if ((ret = av_new_packet(pkt, pkt->size)) < 0) return ret; - pkt->pts = n; - pkt->dts = n; - pkt->duration = 1; - pkt->stream_index = avs->curr_stream; - frame = avs->avs_library.avs_get_frame(avs->clip, n); + + if (avs->avs_library.avs_get_version(avs->clip) >= 9) { + const AVS_Map *avsmap; + + avsmap = avs->avs_library.avs_get_frame_props_ro(avs->env, frame); + + /* Variable frame rate */ + if (avs->is_vfr == true) { + dur.num = avs->avs_library.avs_prop_get_int(avs->env, avsmap, "_DurationNum", 0, &avs->error); + dur.den = avs->avs_library.avs_prop_get_int(avs->env, avsmap, "_DurationDen", 0, &avs->error); + + if (dur.den < 1000) { + // _Duration[Num/Den] uses simplified numbers rather than always using 1000 as the denominator + scaledur = 1000 / dur.den; + avs->scaled_durnum = dur.num * scaledur; + avs->scaled_durden = dur.den * scaledur; + } else { + avs->scaled_durnum = dur.num; + avs->scaled_durden = dur.den; + } + + avs->curr_fpsnum = (dur.den * 1000) / dur.num; + avs->curr_fpsden = 1000; + } + + avs->curr_duration = avs->scaled_durden * avs->scaled_durnum / 1000; + avs->total_duration += avs->curr_duration; + } + + if (avs->is_vfr == false) { + pkt->pts = n; + pkt->dts = n; + pkt->duration = 1; + pkt->stream_index = avs->curr_stream; + } else { + pkt->pts = avs->total_duration; + pkt->dts = avs->total_duration - avs->curr_duration; + pkt->duration = avs->curr_duration; + pkt->stream_index = avs->curr_stream; + } + error = avs->avs_library.avs_clip_get_error(avs->clip); if (error) { av_log(s, AV_LOG_ERROR, "%s\n", error); @@ -1126,7 +1193,8 @@ static int avisynth_read_seek(AVFormatContext *s, int stream_index, #define AVISYNTH_FRAMEPROP_DEFAULT AVISYNTH_FRAMEPROP_FIELD_ORDER | AVISYNTH_FRAMEPROP_RANGE | \ AVISYNTH_FRAMEPROP_PRIMARIES | AVISYNTH_FRAMEPROP_TRANSFER | \ - AVISYNTH_FRAMEPROP_MATRIX | AVISYNTH_FRAMEPROP_CHROMA_LOCATION + AVISYNTH_FRAMEPROP_MATRIX | AVISYNTH_FRAMEPROP_CHROMA_LOCATION | \ + AVISYNTH_FRAMEPROP_VFR #define OFFSET(x) offsetof(AviSynthContext, x) static const AVOption avisynth_options[] = { { "avisynth_flags", "set flags related to reading frame properties from script (AviSynth+ v3.7.1 or higher)", OFFSET(flags), AV_OPT_TYPE_FLAGS, {.i64 = AVISYNTH_FRAMEPROP_DEFAULT}, 0, INT_MAX, AV_OPT_FLAG_DECODING_PARAM, .unit = "flags" }, @@ -1137,6 +1205,7 @@ static const AVOption avisynth_options[] = { { "matrix", "read matrix coefficients", 0, AV_OPT_TYPE_CONST, {.i64 = AVISYNTH_FRAMEPROP_MATRIX}, 0, 1, AV_OPT_FLAG_DECODING_PARAM, .unit = "flags" }, { "chroma_location", "read chroma location", 0, AV_OPT_TYPE_CONST, {.i64 = AVISYNTH_FRAMEPROP_CHROMA_LOCATION}, 0, 1, AV_OPT_FLAG_DECODING_PARAM, .unit = "flags" }, { "sar", "read sample aspect ratio", 0, AV_OPT_TYPE_CONST, {.i64 = AVISYNTH_FRAMEPROP_SAR}, 0, 1, AV_OPT_FLAG_DECODING_PARAM, .unit = "flags" }, + { "vfr", "read fps per-frame", 0, AV_OPT_TYPE_CONST, {.i64 = AVISYNTH_FRAMEPROP_VFR}, 0, 1, AV_OPT_FLAG_DECODING_PARAM, .unit = "flags" }, { NULL }, };