From patchwork Wed Feb 16 16:54:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pierre-Anthony Lemieux X-Patchwork-Id: 34343 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6838:90eb:0:0:0:0 with SMTP id a11csp1391738nkf; Wed, 16 Feb 2022 08:54:34 -0800 (PST) X-Google-Smtp-Source: ABdhPJwdKcnuIWr6sFqPTyG07p7H9qcPiJjtyPDSjNaeLSmFkdUsjVkoMVLhSw9OJ8l18+5fZ/AH X-Received: by 2002:a05:6402:1d8d:b0:408:4a31:9790 with SMTP id dk13-20020a0564021d8d00b004084a319790mr4084086edb.92.1645030474051; Wed, 16 Feb 2022 08:54:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645030474; cv=none; d=google.com; s=arc-20160816; b=KFLFdhILRonpTHUuGZIAEGBPz3sV0CC1pSwmj5sf9k66LQKYVXgaYqqq6+T8poPKLp e6tHyPvRTRQbrF4jdaqrb60EqOBAW03UIO3bEtcqnumLYKKX3CsKxbSx/fV0l6TqVVIr agtmlZz/yNbbcLCc1zNK2SY8M+/oht73BDc8RNMM4CQXoqaDcq8ZFCQGmsQJnxRLUuas EKQ8zyVxpxNAVhVFkR3aavzuFYsWmthabpR3DWZRl/Qt99imq2g1oFfxLpuVhcJ1f0tX NpzjlmhVSdEdwJjltBDkTT3v/vUBBqRvwvWZHW2Ir0QDd1p0wD5t+B7UBHHvRk83SqmS Bszw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=I5T5Z/6znMFuaLVgp9Hfr30bTMK9KGbwxUzkeHRRgvs=; b=JdoQUjRbRvoQM0s98qxkgm6qVvz9AYBURfaP6R2O0pbEbHbOXVKA38EU9xrwxLDXqm 0oIoTxG1snqBEoyWiy8y7H+FKDl1fJYYQJFIAyvTiumILph308HK/SdxmZCgM9h4GVw1 Bcpbs9JIfP3Et5fPk5mpONX13zCf5CmxAMoBWs/UYymPkeNWRW1M7TsqkVN2W8T2xwbg UkrzQttTPjE/PVKH1Uma65QIvneTSbKn09EvSCQXTs89K8kisdVs+1+sWzKGOl4iv23u wUiyUzy68iKVaCePV73W5Ae5EG5P+OakvMPd5aLIX6B+f7mpE6TYDgj11h3eoFNjwdi1 ArSA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cw16si184664ejc.700.2022.02.16.08.54.33; Wed, 16 Feb 2022 08:54:34 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DB57B68B09C; Wed, 16 Feb 2022 18:54:30 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2507768B09C for ; Wed, 16 Feb 2022 18:54:24 +0200 (EET) Received: by mail-pg1-f177.google.com with SMTP id r76so2639995pgr.10 for ; Wed, 16 Feb 2022 08:54:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=cxtqvGl+p2ZE9tNaWEe2D/86ewtXEJH53t7bH34X75k=; b=Gb8pCEACiSdCz0ibGs6BBAddRCR3pO1kmrYJDMEE3+ll9loe59NDza9hmTNMj2AnHz zAhLF1d8mTqCYjumS1HdJ3GBYAPfJvSNTqUpxrGv40UypkN2TiTUr1XldBJUeEcZknsM hPxcwwu2Mr4shf7PDNf+t9TA7HkfhbdxNMi1JAaWqoxuN3tVy0G7vN3bZUzpbqN8bdn8 IraEpRghE7OJQOXs4RoBbEJnsVa689xszSDc0CDoAxn/JCLLQFFc8cgBZ+q9EMmHypOn wzz/ujrsf5Ak2EwH5rZqvHJ+0RdAJVWBS1vLt0lQ6nIKWwQJWLTphMdVV8IUFZ/SR7h+ 3hDQ== X-Gm-Message-State: AOAM530qqeaLgARcZqdh3S01jCUo+wsYFDTB3LNs16wBmklOi5Oh6cBr SWkusuiRvmHMXQ3HcNQSGcvgfgVsiNQ= X-Received: by 2002:a63:1844:0:b0:36c:6a88:ad72 with SMTP id 4-20020a631844000000b0036c6a88ad72mr2928812pgy.329.1645030461945; Wed, 16 Feb 2022 08:54:21 -0800 (PST) Received: from localhost (76-14-89-2.sf-cable.astound.net. [76.14.89.2]) by smtp.gmail.com with ESMTPSA id gb8sm5920329pjb.21.2022.02.16.08.54.20 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 16 Feb 2022 08:54:21 -0800 (PST) Received: by localhost (sSMTP sendmail emulation); Wed, 16 Feb 2022 08:54:12 -0800 From: pal@sandflow.com To: ffmpeg-devel@ffmpeg.org Date: Wed, 16 Feb 2022 08:54:08 -0800 Message-Id: <20220216165410.17063-1-pal@sandflow.com> X-Mailer: git-send-email 2.35.1.windows.2 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 1/3] avformat/imf: open resources only when first needed X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Pierre-Anthony Lemieux Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: NLUvxczI38zc From: Pierre-Anthony Lemieux IMF CPLs can reference thousands of files, which can result in system limits for the number of open files to be exceeded. The following patch opens and closes files as needed. Addresses https://trac.ffmpeg.org/ticket/9623 --- libavformat/imfdec.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/libavformat/imfdec.c b/libavformat/imfdec.c index 847dead7f0..48bd5c78e8 100644 --- a/libavformat/imfdec.c +++ b/libavformat/imfdec.c @@ -103,10 +103,11 @@ typedef struct IMFVirtualTrackPlaybackCtx { int32_t index; /**< Track index in playlist */ AVRational current_timestamp; /**< Current temporal position */ AVRational duration; /**< Overall duration */ - uint32_t resource_count; /**< Number of resources */ + uint32_t resource_count; /**< Number of resources (<= INT32_MAX) */ unsigned int resources_alloc_sz; /**< Size of the buffer holding the resource */ IMFVirtualTrackResourcePlaybackCtx *resources; /**< Buffer holding the resources */ - uint32_t current_resource_index; /**< Current resource */ + int32_t current_resource_index; /**< Index of the current resource in resources, + or < 0 if a current resource has yet to be selected */ int64_t last_pts; /**< Last timestamp */ } IMFVirtualTrackPlaybackCtx; @@ -435,7 +436,6 @@ static int open_track_file_resource(AVFormatContext *s, IMFContext *c = s->priv_data; IMFAssetLocator *asset_locator; void *tmp; - int ret; asset_locator = find_asset_map_locator(&c->asset_locator_map, track_file_resource->track_file_uuid); if (!asset_locator) { @@ -452,7 +452,7 @@ static int open_track_file_resource(AVFormatContext *s, UID_ARG(asset_locator->uuid), asset_locator->absolute_uri); - if (track->resource_count > UINT32_MAX - track_file_resource->base.repeat_count + if (track->resource_count > INT32_MAX - track_file_resource->base.repeat_count || (track->resource_count + track_file_resource->base.repeat_count) > INT_MAX / sizeof(IMFVirtualTrackResourcePlaybackCtx)) return AVERROR(ENOMEM); @@ -470,8 +470,6 @@ static int open_track_file_resource(AVFormatContext *s, vt_ctx.locator = asset_locator; vt_ctx.resource = track_file_resource; vt_ctx.ctx = NULL; - if ((ret = open_track_resource_context(s, &vt_ctx)) != 0) - return ret; track->resources[track->resource_count++] = vt_ctx; track->duration = av_add_q(track->duration, av_make_q((int)track_file_resource->base.duration @@ -501,6 +499,7 @@ static int open_virtual_track(AVFormatContext *s, if (!(track = av_mallocz(sizeof(IMFVirtualTrackPlaybackCtx)))) return AVERROR(ENOMEM); + track->current_resource_index = -1; track->index = track_index; track->duration = av_make_q(0, 1); @@ -551,6 +550,9 @@ static int set_context_streams_from_tracks(AVFormatContext *s) AVStream *first_resource_stream; /* Open the first resource of the track to get stream information */ + ret = open_track_resource_context(s, &c->tracks[i]->resources[0]); + if (ret) + return ret; first_resource_stream = c->tracks[i]->resources[0].ctx->streams[0]; av_log(s, AV_LOG_DEBUG, "Open the first resource of track %d\n", c->tracks[i]->index); @@ -741,7 +743,8 @@ static IMFVirtualTrackResourcePlaybackCtx *get_resource_context_for_timestamp(AV track->index); if (open_track_resource_context(s, &(track->resources[i])) != 0) return NULL; - avformat_close_input(&(track->resources[track->current_resource_index].ctx)); + if (track->current_resource_index > 0) + avformat_close_input(&track->resources[track->current_resource_index].ctx); track->current_resource_index = i; } From patchwork Wed Feb 16 16:54:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pierre-Anthony Lemieux X-Patchwork-Id: 34344 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6838:90eb:0:0:0:0 with SMTP id a11csp1391867nkf; Wed, 16 Feb 2022 08:54:43 -0800 (PST) X-Google-Smtp-Source: ABdhPJzHkxzd2YuULzPGgKIz0/NvYWLhVQ2L2LIODB5cnNUqoxtSD5shYAjYeHQX+BKWGVNNPgI+ X-Received: by 2002:a50:c29a:0:b0:400:47b6:a928 with SMTP id o26-20020a50c29a000000b0040047b6a928mr3982778edf.144.1645030483167; Wed, 16 Feb 2022 08:54:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645030483; cv=none; d=google.com; s=arc-20160816; b=BXBq8Qg34TkwsQLRBE/2V82dm10dFI9/uduy7Vl0S2diRmQ75zJEP0V0wyxDIcNS3N 8o45Ff7lIw0HV0KAC2MqSkQa5wxYFdLBcdLRg/HCncUC9pCGUUfa1HV7v55Izpxqknqj otCizGHAlYcU1bwSDtRynFAqGojNUM6455LSrhSvUcJI4bMAI+TZPlMd46KWz65udC+y eq6rFP8GhqUuMkg5L0+wM1dS5WfQKnQFX3liPlTdfYfC86ioj5iOODnnIGmhHwodTlD+ qJFu45d4v6PcNIDbzYFNk1qOa8mD8q4RAYbkvXQwM3CdNLhW+ohnOy/d8MuyHjuWqLJU jVIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=8bRyHzAjh0yvSMlQvQriWkx9E8/Qt/ltWfrQto5rbVk=; b=kH15+DrRz+HgBNaMsQXTC/2Q/Leer3nNsfyIelEQJ1CXG++L7lsT0clI3OToRcLPO6 16c0LWyBzB5N2Qu7SySRZbJILFc7x64NJGlDTAXmhiWacXcD08ndMMoOAwx/VynDFyA/ djleLceEG6+CF8d/0OU6F5XpM2VLAiCfypzrWz13nk3VSrX7fmdVb+KBS0hQUc6z8Cyj K5oKaQa+/QXBm9lTmzQUXm7bx1Gd0eH9WKHbG2rKQdSTt/E+A2GLm6wY7pqycLpf1VUG Z4yJmEGguWNh38APYvA/AXxsh6fkZlZvAZNHDCAaH3AvE1v3lzOvbLG8d3CoqEnrzlfp hQ4Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y13si2313536edu.584.2022.02.16.08.54.42; Wed, 16 Feb 2022 08:54:43 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D6CB768B2C6; Wed, 16 Feb 2022 18:54:35 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2763C68B1D0 for ; Wed, 16 Feb 2022 18:54:29 +0200 (EET) Received: by mail-pl1-f180.google.com with SMTP id u5so2456102ple.3 for ; Wed, 16 Feb 2022 08:54:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=at+qUjDHyviFLmkADvYfNpAj+TicGtBwh+4hKywxkt8=; b=Udg9nutI6L4Cxzz0bu+dCdnH6EbM54zFMEAwNK8qAMcMMFfH7vmgR50f4OS6F02neu RF6YIGIPma33/1bzneuXMFGpfUqzZK8+Uz00M/VL9SW42NlrEILaYKseRJ/a89NKxVHo zYDzqg8EwEy0EeBjpxg6VV5s0b+sXLoGnzMEiWs0ArEbnc/UP25opLnGJTOdaZlPhWG5 0gbDhnTwdmTeuS5tJfmLKmc0eimXD4NQyC12FlQa8DUNUJgzVJLdvSJgcLPdCAL42AP7 C5AFHGDEk28/XXsObJ73rH87RgKiDbHKSDWpc0ddA7XfeQ2liIO4eUSorbmzwvGiG8Hy cvyA== X-Gm-Message-State: AOAM533oUU5+qrKy/sN6jLe8fVZeUf0AB4XMs2UeCJz2hYxUNTaipCfp BPTMTVIuvOmfPSDc9QMsR3/In6crX18= X-Received: by 2002:a17:90b:f88:b0:1b8:ad41:e200 with SMTP id ft8-20020a17090b0f8800b001b8ad41e200mr2769939pjb.1.1645030466604; Wed, 16 Feb 2022 08:54:26 -0800 (PST) Received: from localhost (76-14-89-2.sf-cable.astound.net. [76.14.89.2]) by smtp.gmail.com with ESMTPSA id h13sm7128538pfv.198.2022.02.16.08.54.24 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 16 Feb 2022 08:54:26 -0800 (PST) Received: by localhost (sSMTP sendmail emulation); Wed, 16 Feb 2022 08:54:16 -0800 From: pal@sandflow.com To: ffmpeg-devel@ffmpeg.org Date: Wed, 16 Feb 2022 08:54:09 -0800 Message-Id: <20220216165410.17063-2-pal@sandflow.com> X-Mailer: git-send-email 2.35.1.windows.2 In-Reply-To: <20220216165410.17063-1-pal@sandflow.com> References: <20220216165410.17063-1-pal@sandflow.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 2/3] avformat/imf: fix packet pts, dts and muxing X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Pierre-Anthony Lemieux Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 4yPRlWldszEl From: Pierre-Anthony Lemieux The IMF demuxer does not set the DTS and PTS of packets accurately in all scenarios. Moreover, audio packets are not trimmed when they exceed the duration of the underlying resource. imf-cpl-with-repeat FATE ref file is regenerated. Addresses https://trac.ffmpeg.org/ticket/9611 --- libavformat/imfdec.c | 263 +++++++++++++++++++---------- tests/ref/fate/imf-cpl-with-repeat | 20 +-- 2 files changed, 181 insertions(+), 102 deletions(-) diff --git a/libavformat/imfdec.c b/libavformat/imfdec.c index 48bd5c78e8..17a21f5ef9 100644 --- a/libavformat/imfdec.c +++ b/libavformat/imfdec.c @@ -65,8 +65,10 @@ #include "avio_internal.h" #include "imf.h" #include "internal.h" +#include "libavcodec/packet.h" #include "libavutil/avstring.h" #include "libavutil/bprint.h" +#include "libavutil/intreadwrite.h" #include "libavutil/opt.h" #include "mxf.h" #include "url.h" @@ -97,6 +99,9 @@ typedef struct IMFVirtualTrackResourcePlaybackCtx { IMFAssetLocator *locator; FFIMFTrackFileResource *resource; AVFormatContext *ctx; + AVRational start_time; + AVRational end_time; + AVRational ts_offset; } IMFVirtualTrackResourcePlaybackCtx; typedef struct IMFVirtualTrackPlaybackCtx { @@ -108,7 +113,6 @@ typedef struct IMFVirtualTrackPlaybackCtx { IMFVirtualTrackResourcePlaybackCtx *resources; /**< Buffer holding the resources */ int32_t current_resource_index; /**< Index of the current resource in resources, or < 0 if a current resource has yet to be selected */ - int64_t last_pts; /**< Last timestamp */ } IMFVirtualTrackPlaybackCtx; typedef struct IMFContext { @@ -342,6 +346,7 @@ static int open_track_resource_context(AVFormatContext *s, int ret = 0; int64_t entry_point; AVDictionary *opts = NULL; + AVStream *st; if (track_resource->ctx) { av_log(s, @@ -383,23 +388,28 @@ static int open_track_resource_context(AVFormatContext *s, } av_dict_free(&opts); - /* Compare the source timebase to the resource edit rate, - * considering the first stream of the source file - */ - if (av_cmp_q(track_resource->ctx->streams[0]->time_base, - av_inv_q(track_resource->resource->base.edit_rate))) + /* make sure there is only one stream in the file */ + + if (track_resource->ctx->nb_streams != 1) { + ret = AVERROR_INVALIDDATA; + goto cleanup; + } + + st = track_resource->ctx->streams[0]; + + /* Warn if the resource time base does not match the file time base */ + if (av_cmp_q(st->time_base, av_inv_q(track_resource->resource->base.edit_rate))) av_log(s, AV_LOG_WARNING, - "Incoherent source stream timebase %d/%d regarding resource edit rate: %d/%d", - track_resource->ctx->streams[0]->time_base.num, - track_resource->ctx->streams[0]->time_base.den, + "Incoherent source stream timebase " AVRATIONAL_FORMAT + "regarding resource edit rate: " AVRATIONAL_FORMAT, + st->time_base.num, + st->time_base.den, track_resource->resource->base.edit_rate.den, track_resource->resource->base.edit_rate.num); - entry_point = (int64_t)track_resource->resource->base.entry_point - * track_resource->resource->base.edit_rate.den - * AV_TIME_BASE - / track_resource->resource->base.edit_rate.num; + entry_point = av_rescale_q(track_resource->resource->base.entry_point, st->time_base, + av_inv_q(track_resource->resource->base.edit_rate)); if (entry_point) { av_log(s, @@ -407,7 +417,7 @@ static int open_track_resource_context(AVFormatContext *s, "Seek at resource %s entry point: %" PRIu32 "\n", track_resource->locator->absolute_uri, track_resource->resource->base.entry_point); - ret = avformat_seek_file(track_resource->ctx, -1, entry_point, entry_point, entry_point, 0); + ret = avformat_seek_file(track_resource->ctx, 0, entry_point, entry_point, entry_point, 0); if (ret < 0) { av_log(s, AV_LOG_ERROR, @@ -470,11 +480,16 @@ static int open_track_file_resource(AVFormatContext *s, vt_ctx.locator = asset_locator; vt_ctx.resource = track_file_resource; vt_ctx.ctx = NULL; - track->resources[track->resource_count++] = vt_ctx; - track->duration = av_add_q(track->duration, + vt_ctx.start_time = track->duration; + vt_ctx.ts_offset = av_sub_q(vt_ctx.start_time, + av_div_q(av_make_q((int)track_file_resource->base.entry_point, 1), + track_file_resource->base.edit_rate)); + vt_ctx.end_time = av_add_q(track->duration, av_make_q((int)track_file_resource->base.duration * track_file_resource->base.edit_rate.den, track_file_resource->base.edit_rate.num)); + track->resources[track->resource_count++] = vt_ctx; + track->duration = vt_ctx.end_time; } return 0; @@ -701,11 +716,14 @@ static IMFVirtualTrackPlaybackCtx *get_next_track_with_minimum_timestamp(AVForma return track; } -static IMFVirtualTrackResourcePlaybackCtx *get_resource_context_for_timestamp(AVFormatContext *s, - IMFVirtualTrackPlaybackCtx *track) +static int get_resource_context_for_timestamp(AVFormatContext *s, IMFVirtualTrackPlaybackCtx *track, IMFVirtualTrackResourcePlaybackCtx **resource) { - AVRational edit_unit_duration = av_inv_q(track->resources[0].resource->base.edit_rate); - AVRational cumulated_duration = av_make_q(0, edit_unit_duration.den); + *resource = NULL; + + if (av_cmp_q(track->current_timestamp, track->duration) >= 0) { + av_log(s, AV_LOG_DEBUG, "Reached the end of the virtual track\n"); + return AVERROR_EOF; + } av_log(s, AV_LOG_DEBUG, @@ -714,119 +732,180 @@ static IMFVirtualTrackResourcePlaybackCtx *get_resource_context_for_timestamp(AV av_q2d(track->current_timestamp), av_q2d(track->duration)); for (uint32_t i = 0; i < track->resource_count; i++) { - cumulated_duration = av_add_q(cumulated_duration, - av_make_q((int)track->resources[i].resource->base.duration - * edit_unit_duration.num, - edit_unit_duration.den)); - if (av_cmp_q(av_add_q(track->current_timestamp, edit_unit_duration), cumulated_duration) <= 0) { + if (av_cmp_q(track->resources[i].end_time, track->current_timestamp) > 0) { av_log(s, AV_LOG_DEBUG, - "Found resource %d in track %d to read for timestamp %lf " - "(on cumulated=%lf): entry=%" PRIu32 + "Found resource %d in track %d to read at timestamp %lf: " + "entry=%" PRIu32 ", duration=%" PRIu32 - ", editrate=" AVRATIONAL_FORMAT - " | edit_unit_duration=%lf\n", + ", editrate=" AVRATIONAL_FORMAT, i, track->index, av_q2d(track->current_timestamp), - av_q2d(cumulated_duration), track->resources[i].resource->base.entry_point, track->resources[i].resource->base.duration, - AVRATIONAL_ARG(track->resources[i].resource->base.edit_rate), - av_q2d(edit_unit_duration)); + AVRATIONAL_ARG(track->resources[i].resource->base.edit_rate)); if (track->current_resource_index != i) { + int ret; + av_log(s, AV_LOG_DEBUG, "Switch resource on track %d: re-open context\n", track->index); - if (open_track_resource_context(s, &(track->resources[i])) != 0) - return NULL; + + ret = open_track_resource_context(s, &(track->resources[i])); + if (ret != 0) + return ret; if (track->current_resource_index > 0) avformat_close_input(&track->resources[track->current_resource_index].ctx); track->current_resource_index = i; } - return &(track->resources[track->current_resource_index]); + *resource = &(track->resources[track->current_resource_index]); + return 0; } } - return NULL; + + av_log(s, AV_LOG_ERROR, "Could not find IMF track resource to read\n"); + return AVERROR_STREAM_NOT_FOUND; +} + +static int imf_time_to_ts(int64_t *ts, AVRational t, AVRational time_base) +{ + int dst_num; + int dst_den; + AVRational r; + + r = av_div_q(t, time_base); + + if ((av_reduce(&dst_num, &dst_den, r.num, r.den, INT64_MAX) != 1)) + return 1; + + if (dst_den != 1) + return 1; + + *ts = dst_num; + + return 0; } static int imf_read_packet(AVFormatContext *s, AVPacket *pkt) { - IMFContext *c = s->priv_data; - IMFVirtualTrackResourcePlaybackCtx *resource_to_read = NULL; - AVRational edit_unit_duration; + IMFVirtualTrackResourcePlaybackCtx *resource = NULL; int ret = 0; IMFVirtualTrackPlaybackCtx *track; - FFStream *track_stream; + int64_t delta_ts; + AVStream *st; + AVRational next_timestamp; track = get_next_track_with_minimum_timestamp(s); - if (av_cmp_q(track->current_timestamp, track->duration) == 0) - return AVERROR_EOF; + ret = get_resource_context_for_timestamp(s, track, &resource); + if (ret) + return ret; - resource_to_read = get_resource_context_for_timestamp(s, track); + ret = av_read_frame(resource->ctx, pkt); + if (ret) { + av_log(s, AV_LOG_ERROR, "Failed to read frame\n"); + return ret; + } - if (!resource_to_read) { - edit_unit_duration - = av_inv_q(track->resources[track->current_resource_index].resource->base.edit_rate); + av_log(s, AV_LOG_DEBUG, "Got packet: pts=%" PRId64 ", dts=%" PRId64 + ", duration=%" PRId64 ", stream_index=%d, pos=%" PRId64 + ", time_base=" AVRATIONAL_FORMAT "\n", pkt->pts, pkt->dts, pkt->duration, + pkt->stream_index, pkt->pos, pkt->time_base.num, pkt->time_base.den); - if (av_cmp_q(av_add_q(track->current_timestamp, edit_unit_duration), track->duration) > 0) - return AVERROR_EOF; + /* IMF resources contain only one stream */ - av_log(s, AV_LOG_ERROR, "Could not find IMF track resource to read\n"); - return AVERROR_STREAM_NOT_FOUND; + if (pkt->stream_index != 0) + return AVERROR_INVALIDDATA; + st = resource->ctx->streams[0]; + + pkt->stream_index = track->index; + + /* adjust the packet PTS and DTS based on the temporal position of the resource within the timeline */ + + ret = imf_time_to_ts(&delta_ts, resource->ts_offset, st->time_base); + + if (!ret) { + if (pkt->pts != AV_NOPTS_VALUE) + pkt->pts += delta_ts; + if (pkt->dts != AV_NOPTS_VALUE) + pkt->dts += delta_ts; + } else { + av_log(s, AV_LOG_WARNING, "Incoherent time stamp " AVRATIONAL_FORMAT " for time base " AVRATIONAL_FORMAT, + resource->ts_offset.num, resource->ts_offset.den, pkt->time_base.num, + pkt->time_base.den); } - while (!ff_check_interrupt(c->interrupt_callback) && !ret) { - ret = av_read_frame(resource_to_read->ctx, pkt); - av_log(s, - AV_LOG_DEBUG, - "Got packet: pts=%" PRId64 - ", dts=%" PRId64 - ", duration=%" PRId64 - ", stream_index=%d, pos=%" PRId64 - "\n", - pkt->pts, - pkt->dts, - pkt->duration, - pkt->stream_index, - pkt->pos); - - track_stream = ffstream(s->streams[track->index]); - if (ret >= 0) { - /* Update packet info from track */ - if (pkt->dts < track_stream->cur_dts && track->last_pts > 0) - pkt->dts = track_stream->cur_dts; - - pkt->pts = track->last_pts; - pkt->dts = pkt->dts - - (int64_t)track->resources[track->current_resource_index].resource->base.entry_point; - pkt->stream_index = track->index; - - /* Update track cursors */ - track->current_timestamp - = av_add_q(track->current_timestamp, - av_make_q((int)pkt->duration - * resource_to_read->ctx->streams[0]->time_base.num, - resource_to_read->ctx->streams[0]->time_base.den)); - track->last_pts += pkt->duration; + /* advance the track timestamp by the packet duration */ - return 0; - } else if (ret != AVERROR_EOF) { - av_log(s, - AV_LOG_ERROR, - "Could not get packet from track %d: %s\n", - track->index, - av_err2str(ret)); - return ret; + next_timestamp = av_add_q(track->current_timestamp, + av_mul_q(av_make_q((int)pkt->duration, 1), st->time_base)); + + /* if necessary, clamp the next timestamp to the end of the current resource */ + + if (av_cmp_q(next_timestamp, resource->end_time) > 0) { + + int64_t new_pkt_dur; + + /* shrink the packet duration */ + + ret = imf_time_to_ts(&new_pkt_dur, + av_sub_q(resource->end_time, track->current_timestamp), + st->time_base); + + if (!ret) + pkt->duration = new_pkt_dur; + else + av_log(s, AV_LOG_WARNING, "Incoherent time base in packet duration calculation"); + + /* shrink the packet itself for audio essence */ + + if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) { + + if (st->codecpar->codec_id == AV_CODEC_ID_PCM_S24LE) { + /* AV_CODEC_ID_PCM_S24LE is the only PCM format supported in IMF */ + /* in this case, explicitly shrink the packet */ + + int bytes_per_sample = av_get_exact_bits_per_sample(st->codecpar->codec_id) >> 3; + int64_t nbsamples = av_rescale_q(pkt->duration, + st->time_base, + av_make_q(1, st->codecpar->sample_rate)); + av_shrink_packet(pkt, nbsamples * st->codecpar->channels * bytes_per_sample); + + } else { + /* in all other cases, use side data to skip samples */ + int64_t skip_samples; + + ret = imf_time_to_ts(&skip_samples, + av_sub_q(next_timestamp, resource->end_time), + av_make_q(1, st->codecpar->sample_rate)); + + if (ret || skip_samples < 0 || skip_samples > UINT32_MAX) { + av_log(s, AV_LOG_WARNING, "Cannot skip audio samples"); + } else { + uint8_t *side_data = av_packet_new_side_data(pkt, AV_PKT_DATA_SKIP_SAMPLES, 10); + if (!side_data) + return AVERROR(ENOMEM); + + AV_WL32(side_data + 4, skip_samples); /* skip from end of this packet */ + side_data[6] = 1; /* reason for end is convergence */ + } + } + + next_timestamp = resource->end_time; + + } else { + av_log(s, AV_LOG_WARNING, "Non-audio packet duration reduced"); } } - return AVERROR_EOF; + track->current_timestamp = next_timestamp; + + return 0; } static int imf_close(AVFormatContext *s) diff --git a/tests/ref/fate/imf-cpl-with-repeat b/tests/ref/fate/imf-cpl-with-repeat index fe3106d5c6..e5b3992d04 100644 --- a/tests/ref/fate/imf-cpl-with-repeat +++ b/tests/ref/fate/imf-cpl-with-repeat @@ -34,13 +34,13 @@ 0, 28, 28, 1, 1964, 0x106d867a 0, 29, 29, 1, 1964, 0x106d867a 0, 30, 30, 1, 1964, 0x106d867a -0, 30, 31, 1, 2068, 0xd411c582 -0, 30, 32, 1, 2520, 0x7b539dfc -0, 30, 33, 1, 1846, 0x9d184658 -0, 30, 34, 1, 2475, 0x2967a123 -0, 30, 35, 1, 2540, 0xdc61ad87 -0, 30, 36, 1, 2214, 0xa7da105f -0, 30, 37, 1, 2486, 0xa68d89b4 -0, 30, 38, 1, 2615, 0xbc35d5ad -0, 30, 39, 1, 2150, 0x643aea98 -0, 30, 40, 1, 2628, 0x2880c4ad +0, 31, 31, 1, 2068, 0xd411c582 +0, 32, 32, 1, 2520, 0x7b539dfc +0, 33, 33, 1, 1846, 0x9d184658 +0, 34, 34, 1, 2475, 0x2967a123 +0, 35, 35, 1, 2540, 0xdc61ad87 +0, 36, 36, 1, 2214, 0xa7da105f +0, 37, 37, 1, 2486, 0xa68d89b4 +0, 38, 38, 1, 2615, 0xbc35d5ad +0, 39, 39, 1, 2150, 0x643aea98 +0, 40, 40, 1, 2628, 0x2880c4ad From patchwork Wed Feb 16 16:54:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pierre-Anthony Lemieux X-Patchwork-Id: 34345 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6838:90eb:0:0:0:0 with SMTP id a11csp1392028nkf; Wed, 16 Feb 2022 08:54:52 -0800 (PST) X-Google-Smtp-Source: ABdhPJxNOcpMnP3UL+HboHxvFjjW9CpxMwc7uJB5ZjrFekHx9OlZ74mPxTEQLeiYZCm+rhWlF3kf X-Received: by 2002:a17:907:2be6:b0:6ce:70da:12c2 with SMTP id gv38-20020a1709072be600b006ce70da12c2mr3023280ejc.649.1645030492338; Wed, 16 Feb 2022 08:54:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645030492; cv=none; d=google.com; s=arc-20160816; b=SIIhX7tCuEQCpBAFYMzgp+xOEC/fvbItXPrbARu2eM+HKQBqqftQs1Rbs7R23uu3Z1 FqiG/AjjOn8lQ25A9IVGd4L5XsggNz7ac6zKOulyuLQWH+NWkhNUoJCW6cncKJGPz1+h U+tokXM6k3GppZ0z/PQuj4tF9pTY59xz4JRvxfMmJ5w9Q/F3cUBC+04oD58yIEFDwYyJ VN62/Cr0zr1VO6MMyXRtUyRPOle0Ny2iu/JVW7Ltn8Yqqs4DZJaFkhrZuNU5pSFOvkCA Hp41bPsqMDTcCgBbMOjjTajAYteLlZHQf7fb+Q0IExNBb/wnGgT/gkBzkyBVYhlMEmN6 XL7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=ictgUI9igtvLJFRSZxdoLJsfGmlahxWVhbScv+vOFBI=; b=AKUQOnH/NE4WJ38CuHXqlMEcba/op8bHdmyGGAgt9Ab9iPdRJeRMy6LhN7q2rAE+aZ zL7fOmw9zTAUvNAtAIjjhVjb9YfElx43DAHOXcdwaWhHLn2TiVx4QHq2mY3u/GIgg72M C/IXuDwh3A0ntXgdKabsHXeyKCgqO5QzPRrSodMpGr41Se0E6nMCxJVgIna0myZaJyYF DNKIyBKi8tkQP4rvcaIpIZmzEZy92XTB27+As/U/Yq2NXsCZoUfnlhC3Xk9KdUChU53/ 3EzKCsjNEuLwm/kOibc/NRiTej8OhtROYHayniHgEPDPbq3kkWjITXCJ045a14iabhVI UvJA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l21si2173443edw.530.2022.02.16.08.54.52; Wed, 16 Feb 2022 08:54:52 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E037A68B29B; Wed, 16 Feb 2022 18:54:37 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 11A5C68B2CD for ; Wed, 16 Feb 2022 18:54:31 +0200 (EET) Received: by mail-pf1-f171.google.com with SMTP id f6so2580899pfj.11 for ; Wed, 16 Feb 2022 08:54:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eRG3dOdfqz3168AbPZdnR2jm63ki2wHkRdP6bDAegYM=; b=YMtt7HzHAcX20LHtr/vFSL4mA4jWwnTdQp7NLe44qNlwWpUozL631gs+EFl258dqKN PXnRaY8SP/BUm3Ig5wyiEUrCc8D+qC/5RSqg61Ix9VzTVzCK7dA2oHxARszJwnsoINji KY4pFGbO3oMDGDtMOi9WoRDJy5/1mQ17P5vPQB2qDip77LTARHnleBMo8as6mGJ1Vg9k Xc/URY43EF1MrcVaOzQwL2Il19bFPi+NAyZXKu2hjN713qDuvK7xAk3iKK2usw3Ru+l+ zpWDREvRNgnXqORA/1SncqlxhhGRp+qBb2A2l7gF4eRvMIwZRGXOS+KyOjW/YMSwl4GR HM/g== X-Gm-Message-State: AOAM532GroFWKT1rrKr6xF9dnrEyCYiVD09idYZFvr61AUfetgWMim0D SGTcS1Nt8ro8DPxLizUa2vL2IShY4wo= X-Received: by 2002:a05:6a00:2356:b0:4e1:14f2:9fe9 with SMTP id j22-20020a056a00235600b004e114f29fe9mr4153877pfj.38.1645030469032; Wed, 16 Feb 2022 08:54:29 -0800 (PST) Received: from localhost (76-14-89-2.sf-cable.astound.net. [76.14.89.2]) by smtp.gmail.com with ESMTPSA id t22sm4681140pfg.92.2022.02.16.08.54.27 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 16 Feb 2022 08:54:28 -0800 (PST) Received: by localhost (sSMTP sendmail emulation); Wed, 16 Feb 2022 08:54:19 -0800 From: pal@sandflow.com To: ffmpeg-devel@ffmpeg.org Date: Wed, 16 Feb 2022 08:54:10 -0800 Message-Id: <20220216165410.17063-3-pal@sandflow.com> X-Mailer: git-send-email 2.35.1.windows.2 In-Reply-To: <20220216165410.17063-1-pal@sandflow.com> References: <20220216165410.17063-1-pal@sandflow.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 3/3] avformat/imf: document IMFVirtualTrackResourcePlaybackCtx X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Pierre-Anthony Lemieux Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ol+omsNMUnJJ From: Pierre-Anthony Lemieux --- libavformat/imfdec.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/libavformat/imfdec.c b/libavformat/imfdec.c index 17a21f5ef9..143f1086b6 100644 --- a/libavformat/imfdec.c +++ b/libavformat/imfdec.c @@ -96,12 +96,12 @@ typedef struct IMFAssetLocatorMap { } IMFAssetLocatorMap; typedef struct IMFVirtualTrackResourcePlaybackCtx { - IMFAssetLocator *locator; - FFIMFTrackFileResource *resource; - AVFormatContext *ctx; - AVRational start_time; - AVRational end_time; - AVRational ts_offset; + IMFAssetLocator *locator; /**< Location of the resource */ + FFIMFTrackFileResource *resource; /**< Underlying IMF CPL resource */ + AVFormatContext *ctx; /**< Context associated with the resource */ + AVRational start_time; /**< inclusive start time of the resource on the CPL timeline (s) */ + AVRational end_time; /**< exclusive end time of the resource on the CPL timeline (s) */ + AVRational ts_offset; /**< start_time minus the entry point into the resource (s) */ } IMFVirtualTrackResourcePlaybackCtx; typedef struct IMFVirtualTrackPlaybackCtx {