From patchwork Tue Dec 12 12:09:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 45091 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1225:b0:181:818d:5e7f with SMTP id v37csp3544643pzf; Tue, 12 Dec 2023 04:10:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IF+P+OJjTObx3H52UvtICa0nWnueTdQtndy90+VfUqe9kag4R2G7ZOdyx/Apc0HwhiVB39j X-Received: by 2002:a50:9f82:0:b0:54d:7224:bd15 with SMTP id c2-20020a509f82000000b0054d7224bd15mr6806196edf.1.1702383040735; Tue, 12 Dec 2023 04:10:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702383040; cv=none; d=google.com; s=arc-20160816; b=pAjxgz9zteoPx1CYDyAb8SIhvNO7joaSP5+uhRVYq+BQwh1sZvVMjzl+oVJFAbAtZ/ LsqbCQgb78ZOsZ1ekZtb3Bo/Y8G3oAE+3K8YIAEg0rYUDN1DtwqfrsBf/L2LfkfuI1Jy 7IaJ3J2uFICtTztfxWif6PKGCwC4E0UJnzrT/sNz8iMdQJzRf+10w+eG87qgYlTU98Fq fXPkmKZWqnVZl268f37k+NIu3uyxSI9L3/0TlDGSdsgRB4XHwEZ3F6S7PiaW283KdFfs vJ5JrDSHzfBVPoPFoAztmsL9gtmnyvfI4UPugVLkcspnFe9mftjw5ZAwanUm8M/804PT ElTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=H/AvshEuVGb4iRx5p5n8D6UcMEwsKE445OursKK9c5k=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=bNT0AlBerRtgs1oggPPj3ULEvf0V8b/uwpaDiDBpQB5Mgz8LXoRB6ZyUS6mWWwek6F ZsLDvV7CLkSNZAV7K9hTUQ6l5Bj79rjMfPN8/vi3DjVv7L2V2FBJet52PkyqeQMT3JyF +K1RtOIh+/BhQKTQCGqFwb0UKRJDPyVcmuSmK2yczo/qutlc1m3VaWun34Xudhmf/8bj IhKY7ch//ABlQt5IqYh2jNjslGaHogqm9NMiUE2VqVLNN4r2ew5bM6sucbAza0iXqYvk 0w4LZ1UxNkQ4vrXZTelimhVQK6tRHPMzAAKTUiZDLsOdDs5rHIJCZP2HoXWvZ5NIZeJ1 c5Nw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id g6-20020a50d5c6000000b00550dd5e89b7si3294183edj.635.2023.12.12.04.10.38; Tue, 12 Dec 2023 04:10:40 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B383668D17B; Tue, 12 Dec 2023 14:10:17 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail1.khirnov.net (quelana.khirnov.net [94.230.150.81]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5780D68D10B for ; Tue, 12 Dec 2023 14:10:08 +0200 (EET) Received: from localhost (mail1.khirnov.net [IPv6:::1]) by mail1.khirnov.net (Postfix) with ESMTP id 04B95122F for ; Tue, 12 Dec 2023 13:10:08 +0100 (CET) Received: from mail1.khirnov.net ([IPv6:::1]) by localhost (mail1.khirnov.net [IPv6:::1]) (amavis, port 10024) with ESMTP id eFoDX3GAu5f2 for ; Tue, 12 Dec 2023 13:10:07 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail1.khirnov.net (Postfix) with ESMTPS id BF5C212A9 for ; Tue, 12 Dec 2023 13:10:06 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:::1]) by libav.khirnov.net (Postfix) with ESMTP id 941C93A0334 for ; Tue, 12 Dec 2023 13:09:59 +0100 (CET) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Tue, 12 Dec 2023 13:09:56 +0100 Message-ID: <20231212120958.21313-1-anton@khirnov.net> X-Mailer: git-send-email 2.42.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/3] fftools/ffmpeg: merge DemuxPktData into FrameData X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 4kKhwq5n5g4X This way we can propagate arbitrary data from the demuxer all the way into the muxer, using a single struct. --- fftools/ffmpeg.c | 28 ++++++++++++++++++++-------- fftools/ffmpeg.h | 13 +++++++------ fftools/ffmpeg_demux.c | 9 ++++----- fftools/ffmpeg_mux.c | 4 ++-- 4 files changed, 33 insertions(+), 21 deletions(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index 30b594fd97..e7ff9a6d6f 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -404,36 +404,48 @@ InputStream *ist_iter(InputStream *prev) return NULL; } -static int frame_data_ensure(AVFrame *frame, int writable) +static int frame_data_ensure(AVBufferRef **dst, int writable) { - if (!frame->opaque_ref) { + if (!*dst) { FrameData *fd; - frame->opaque_ref = av_buffer_allocz(sizeof(*fd)); - if (!frame->opaque_ref) + *dst = av_buffer_allocz(sizeof(*fd)); + if (!*dst) return AVERROR(ENOMEM); - fd = (FrameData*)frame->opaque_ref->data; + fd = (FrameData*)((*dst)->data); fd->dec.frame_num = UINT64_MAX; fd->dec.pts = AV_NOPTS_VALUE; } else if (writable) - return av_buffer_make_writable(&frame->opaque_ref); + return av_buffer_make_writable(dst); return 0; } FrameData *frame_data(AVFrame *frame) { - int ret = frame_data_ensure(frame, 1); + int ret = frame_data_ensure(&frame->opaque_ref, 1); return ret < 0 ? NULL : (FrameData*)frame->opaque_ref->data; } const FrameData *frame_data_c(AVFrame *frame) { - int ret = frame_data_ensure(frame, 0); + int ret = frame_data_ensure(&frame->opaque_ref, 0); return ret < 0 ? NULL : (const FrameData*)frame->opaque_ref->data; } +FrameData *packet_data(AVPacket *pkt) +{ + int ret = frame_data_ensure(&pkt->opaque_ref, 1); + return ret < 0 ? NULL : (FrameData*)pkt->opaque_ref->data; +} + +const FrameData *packet_data_c(AVPacket *pkt) +{ + int ret = frame_data_ensure(&pkt->opaque_ref, 0); + return ret < 0 ? NULL : (const FrameData*)pkt->opaque_ref->data; +} + void remove_avoptions(AVDictionary **a, AVDictionary *b) { const AVDictionaryEntry *t = NULL; diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index ba82b7490d..aafb35538e 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -115,12 +115,6 @@ typedef struct { } AudioChannelMap; #endif -typedef struct DemuxPktData { - // estimated dts in AV_TIME_BASE_Q, - // to be used when real dts is missing - int64_t dts_est; -} DemuxPktData; - typedef struct OptionsContext { OptionGroup *g; @@ -622,6 +616,10 @@ typedef struct OutputFile { // optionally attached as opaque_ref to decoded AVFrames typedef struct FrameData { + // demuxer-estimated dts in AV_TIME_BASE_Q, + // to be used when real dts is missing + int64_t dts_est; + // properties that come from the decoder struct { uint64_t frame_num; @@ -723,6 +721,9 @@ FrameData *frame_data(AVFrame *frame); const FrameData *frame_data_c(AVFrame *frame); +FrameData *packet_data (AVPacket *pkt); +const FrameData *packet_data_c(AVPacket *pkt); + /** * Set up fallback filtering parameters from a decoder context. They will only * be used if no frames are ever sent on this input, otherwise the actual diff --git a/fftools/ffmpeg_demux.c b/fftools/ffmpeg_demux.c index 91cd7a1125..392b447338 100644 --- a/fftools/ffmpeg_demux.c +++ b/fftools/ffmpeg_demux.c @@ -328,14 +328,13 @@ static int ist_dts_update(DemuxStream *ds, AVPacket *pkt) av_assert0(!pkt->opaque_ref); if (ds->streamcopy_needed) { - DemuxPktData *pd; + FrameData *fd; - pkt->opaque_ref = av_buffer_allocz(sizeof(*pd)); - if (!pkt->opaque_ref) + fd = packet_data(pkt); + if (!fd) return AVERROR(ENOMEM); - pd = (DemuxPktData*)pkt->opaque_ref->data; - pd->dts_est = ds->dts; + fd->dts_est = ds->dts; } return 0; diff --git a/fftools/ffmpeg_mux.c b/fftools/ffmpeg_mux.c index 067dc65d4e..62925be8d0 100644 --- a/fftools/ffmpeg_mux.c +++ b/fftools/ffmpeg_mux.c @@ -381,8 +381,8 @@ static int of_streamcopy(OutputStream *ost, AVPacket *pkt) { OutputFile *of = output_files[ost->file_index]; MuxStream *ms = ms_from_ost(ost); - DemuxPktData *pd = pkt->opaque_ref ? (DemuxPktData*)pkt->opaque_ref->data : NULL; - int64_t dts = pd ? pd->dts_est : AV_NOPTS_VALUE; + FrameData *fd = pkt->opaque_ref ? (FrameData*)pkt->opaque_ref->data : NULL; + int64_t dts = fd ? fd->dts_est : AV_NOPTS_VALUE; int64_t start_time = (of->start_time == AV_NOPTS_VALUE) ? 0 : of->start_time; int64_t ts_offset; From patchwork Tue Dec 12 12:09:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 45090 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1225:b0:181:818d:5e7f with SMTP id v37csp3544519pzf; Tue, 12 Dec 2023 04:10:30 -0800 (PST) X-Google-Smtp-Source: AGHT+IF6Khzt/N4r08Iku3CKE8+zoV9MtDdQQnZz0j8t64e9zsTki5IWKGt9LmcYIu37/9ytvuv/ X-Received: by 2002:a2e:a805:0:b0:2cc:1e83:65ee with SMTP id l5-20020a2ea805000000b002cc1e8365eemr2924546ljq.56.1702383029933; Tue, 12 Dec 2023 04:10:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702383029; cv=none; d=google.com; s=arc-20160816; b=e7BiZMNo1vqPUoc49uzKzL1kie9+Ez220AI5nRh3ZucyzOgciH+CbIZtAz0lO72XgV aAlImGQQTiy343fyzoZcLaYKXAsYv9P0wvYKv3VA8y1U7Ziv9t2/0a8Tl9PxA9OrjYuR 9aNLH0DdKe9n4KwyiGDnNbPqXJI7YgAPE1R6H10/3JvtYC1/+dLldwGgpZmsy8IqAMqa lTxgMZb3jtwxi17xw0kU2PML1srOLl2Id/+a/38iBYtIDqxYnoVeknx8ylWqHy9Bf8Ed 5hRcwhHK1ndkaeD9y3AsizpLwxEEKMiwEhTyyKLHcojTGtxcV2Eyk1pDff1hQ/pTn4NT cU+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=LrJUfFIhRy1H2U/D0hJ1AooEXoF1myJ1jLSuj+QllZc=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=AKeAzWDRgilbMGYX4KVBY+ThO9tCFO7RRzI1LjMUA2iuAvwYjpzkt0yOuS2TKG6rbq 4ZearWIWW8AdS7B2rfe/MNQzt7CScXch7yT4RLlVXo99v+CA2vFQPlJvAA7dlBpYFgAs YHFmyf8AUHclE2rfjd8kaJ8KrzNnqp8TmolrQpxwEGgh4HWtokksciSXuMKkZJYtKKnu YdL+puVGj5oGMJ/LzbUX+aulaEhVHhIuW73w5dJOEqI/KNYujHL6sFYKdAH6JtlPbtAB evbxntSsJVAph4Kh2EduPtSFclTsT0UNTF5sqGlUHTsKLuAhXk/0TZUTt0Ovby+vVdAs GS1g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f11-20020aa7d84b000000b0054ce03c4027si4700250eds.646.2023.12.12.04.10.29; Tue, 12 Dec 2023 04:10:29 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 991E068D16E; Tue, 12 Dec 2023 14:10:16 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail1.khirnov.net (quelana.khirnov.net [94.230.150.81]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2FFE368D102 for ; Tue, 12 Dec 2023 14:10:08 +0200 (EET) Received: from localhost (mail1.khirnov.net [IPv6:::1]) by mail1.khirnov.net (Postfix) with ESMTP id B657A1BA1 for ; Tue, 12 Dec 2023 13:10:07 +0100 (CET) Received: from mail1.khirnov.net ([IPv6:::1]) by localhost (mail1.khirnov.net [IPv6:::1]) (amavis, port 10024) with ESMTP id oFHbWecRcWST for ; Tue, 12 Dec 2023 13:10:06 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail1.khirnov.net (Postfix) with ESMTPS id BD066122F for ; Tue, 12 Dec 2023 13:10:06 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:::1]) by libav.khirnov.net (Postfix) with ESMTP id A1CBB3A0585 for ; Tue, 12 Dec 2023 13:09:59 +0100 (CET) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Tue, 12 Dec 2023 13:09:57 +0100 Message-ID: <20231212120958.21313-2-anton@khirnov.net> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231212120958.21313-1-anton@khirnov.net> References: <20231212120958.21313-1-anton@khirnov.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/3] fftools/ffmpeg: attach wallclock timing information to packets and frames X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: VFf3eqwf4M5v Will become useful in following commits. --- fftools/ffmpeg.c | 3 +++ fftools/ffmpeg.h | 13 +++++++++++++ fftools/ffmpeg_dec.c | 14 ++++++++++++-- fftools/ffmpeg_demux.c | 26 ++++++++++++-------------- fftools/ffmpeg_enc.c | 16 ++++++++++++++++ fftools/ffmpeg_filter.c | 9 +++++++++ 6 files changed, 65 insertions(+), 16 deletions(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index e7ff9a6d6f..42e675dcb9 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -416,6 +416,9 @@ static int frame_data_ensure(AVBufferRef **dst, int writable) fd->dec.frame_num = UINT64_MAX; fd->dec.pts = AV_NOPTS_VALUE; + + for (unsigned i = 0; i < FF_ARRAY_ELEMS(fd->wallclock); i++) + fd->wallclock[i] = INT64_MIN; } else if (writable) return av_buffer_make_writable(dst); diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index aafb35538e..7f84f3266d 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -94,6 +94,17 @@ enum PacketOpaque { PKT_OPAQUE_FIX_SUB_DURATION, }; +enum LatencyProbe { + LATENCY_PROBE_DEMUX, + LATENCY_PROBE_DEC_PRE, + LATENCY_PROBE_DEC_POST, + LATENCY_PROBE_FILTER_PRE, + LATENCY_PROBE_FILTER_POST, + LATENCY_PROBE_ENC_PRE, + LATENCY_PROBE_ENC_POST, + LATENCY_PROBE_NB, +}; + typedef struct HWDevice { const char *name; enum AVHWDeviceType type; @@ -631,6 +642,8 @@ typedef struct FrameData { AVRational frame_rate_filter; int bits_per_raw_sample; + + int64_t wallclock[LATENCY_PROBE_NB]; } FrameData; extern InputFile **input_files; diff --git a/fftools/ffmpeg_dec.c b/fftools/ffmpeg_dec.c index 5dde82a276..ce5aa33f73 100644 --- a/fftools/ffmpeg_dec.c +++ b/fftools/ffmpeg_dec.c @@ -22,6 +22,7 @@ #include "libavutil/log.h" #include "libavutil/pixdesc.h" #include "libavutil/pixfmt.h" +#include "libavutil/time.h" #include "libavutil/timestamp.h" #include "libavcodec/avcodec.h" @@ -475,6 +476,13 @@ static int packet_decode(InputStream *ist, AVPacket *pkt, AVFrame *frame) pkt->dts = AV_NOPTS_VALUE; } + if (pkt) { + FrameData *fd = packet_data(pkt); + if (!fd) + return AVERROR(ENOMEM); + fd->wallclock[LATENCY_PROBE_DEC_PRE] = av_gettime_relative(); + } + ret = avcodec_send_packet(dec, pkt); if (ret < 0 && !(ret == AVERROR_EOF && !pkt)) { // In particular, we don't expect AVERROR(EAGAIN), because we read all @@ -528,8 +536,6 @@ static int packet_decode(InputStream *ist, AVPacket *pkt, AVFrame *frame) return AVERROR_INVALIDDATA; } - - av_assert0(!frame->opaque_ref); fd = frame_data(frame); if (!fd) { av_frame_unref(frame); @@ -540,6 +546,8 @@ static int packet_decode(InputStream *ist, AVPacket *pkt, AVFrame *frame) fd->dec.frame_num = dec->frame_num - 1; fd->bits_per_raw_sample = dec->bits_per_raw_sample; + fd->wallclock[LATENCY_PROBE_DEC_POST] = av_gettime_relative(); + frame->time_base = dec->pkt_timebase; if (dec->codec_type == AVMEDIA_TYPE_AUDIO) { @@ -932,6 +940,8 @@ int dec_open(InputStream *ist, Scheduler *sch, unsigned sch_idx) if (ist->st->disposition & AV_DISPOSITION_ATTACHED_PIC) av_dict_set(&ist->decoder_opts, "threads", "1", 0); + av_dict_set(&ist->decoder_opts, "flags", "+copy_opaque", AV_DICT_MULTIKEY); + ret = hw_device_setup_for_decode(ist); if (ret < 0) { av_log(ist, AV_LOG_ERROR, diff --git a/fftools/ffmpeg_demux.c b/fftools/ffmpeg_demux.c index 392b447338..d2466e885d 100644 --- a/fftools/ffmpeg_demux.c +++ b/fftools/ffmpeg_demux.c @@ -271,7 +271,7 @@ static void ts_discontinuity_process(Demuxer *d, InputStream *ist, ts_discontinuity_detect(d, ist, pkt); } -static int ist_dts_update(DemuxStream *ds, AVPacket *pkt) +static int ist_dts_update(DemuxStream *ds, AVPacket *pkt, FrameData *fd) { InputStream *ist = &ds->ist; const AVCodecParameters *par = ist->par; @@ -326,21 +326,12 @@ static int ist_dts_update(DemuxStream *ds, AVPacket *pkt) break; } - av_assert0(!pkt->opaque_ref); - if (ds->streamcopy_needed) { - FrameData *fd; - - fd = packet_data(pkt); - if (!fd) - return AVERROR(ENOMEM); - - fd->dts_est = ds->dts; - } + fd->dts_est = ds->dts; return 0; } -static int ts_fixup(Demuxer *d, AVPacket *pkt) +static int ts_fixup(Demuxer *d, AVPacket *pkt, FrameData *fd) { InputFile *ifile = &d->f; InputStream *ist = ifile->streams[pkt->stream_index]; @@ -424,7 +415,7 @@ static int ts_fixup(Demuxer *d, AVPacket *pkt) ts_discontinuity_process(d, ist, pkt); // update estimated/predicted dts - ret = ist_dts_update(ds, pkt); + ret = ist_dts_update(ds, pkt, fd); if (ret < 0) return ret; @@ -436,9 +427,14 @@ static int input_packet_process(Demuxer *d, AVPacket *pkt, unsigned *send_flags) InputFile *f = &d->f; InputStream *ist = f->streams[pkt->stream_index]; DemuxStream *ds = ds_from_ist(ist); + FrameData *fd; int ret = 0; - ret = ts_fixup(d, pkt); + fd = packet_data(pkt); + if (!fd) + return AVERROR(ENOMEM); + + ret = ts_fixup(d, pkt, fd); if (ret < 0) return ret; @@ -455,6 +451,8 @@ static int input_packet_process(Demuxer *d, AVPacket *pkt, unsigned *send_flags) ds->data_size += pkt->size; ds->nb_packets++; + fd->wallclock[LATENCY_PROBE_DEMUX] = av_gettime_relative(); + if (debug_ts) { av_log(NULL, AV_LOG_INFO, "demuxer+ffmpeg -> ist_index:%d:%d type:%s pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s duration:%s duration_time:%s off:%s off_time:%s\n", f->index, pkt->stream_index, diff --git a/fftools/ffmpeg_enc.c b/fftools/ffmpeg_enc.c index 320df2dee7..43e8b077eb 100644 --- a/fftools/ffmpeg_enc.c +++ b/fftools/ffmpeg_enc.c @@ -34,6 +34,7 @@ #include "libavutil/log.h" #include "libavutil/pixdesc.h" #include "libavutil/rational.h" +#include "libavutil/time.h" #include "libavutil/timestamp.h" #include "libavcodec/avcodec.h" @@ -615,6 +616,14 @@ static int encode_frame(OutputFile *of, OutputStream *ost, AVFrame *frame, int ret; if (frame) { + FrameData *fd = frame_data(frame); + + fd = frame_data(frame); + if (!fd) + return AVERROR(ENOMEM); + + fd->wallclock[LATENCY_PROBE_ENC_PRE] = av_gettime_relative(); + if (ost->enc_stats_pre.io) enc_stats_write(ost, &ost->enc_stats_pre, frame, NULL, ost->frames_encoded); @@ -644,6 +653,8 @@ static int encode_frame(OutputFile *of, OutputStream *ost, AVFrame *frame, } while (1) { + FrameData *fd; + av_packet_unref(pkt); ret = avcodec_receive_packet(enc, pkt); @@ -665,6 +676,11 @@ static int encode_frame(OutputFile *of, OutputStream *ost, AVFrame *frame, return ret; } + fd = packet_data(pkt); + if (!fd) + return AVERROR(ENOMEM); + fd->wallclock[LATENCY_PROBE_ENC_POST] = av_gettime_relative(); + if (enc->codec_type == AVMEDIA_TYPE_VIDEO) { ret = update_video_stats(ost, pkt, !!vstats_filename); if (ret < 0) diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c index ada235b084..21f823be12 100644 --- a/fftools/ffmpeg_filter.c +++ b/fftools/ffmpeg_filter.c @@ -36,6 +36,7 @@ #include "libavutil/pixfmt.h" #include "libavutil/imgutils.h" #include "libavutil/samplefmt.h" +#include "libavutil/time.h" #include "libavutil/timestamp.h" // FIXME private header, used for mid_pred() @@ -2364,6 +2365,8 @@ static int fg_output_step(OutputFilterPriv *ofp, FilterGraphThread *fgt, return AVERROR(ENOMEM); } + fd->wallclock[LATENCY_PROBE_FILTER_POST] = av_gettime_relative(); + // only use bits_per_raw_sample passed through from the decoder // if the filtergraph did not touch the frame data if (!fgp->is_meta) @@ -2576,6 +2579,7 @@ static int send_frame(FilterGraph *fg, FilterGraphThread *fgt, InputFilter *ifilter, AVFrame *frame) { InputFilterPriv *ifp = ifp_from_ifilter(ifilter); + FrameData *fd; AVFrameSideData *sd; int need_reinit, ret; @@ -2651,6 +2655,11 @@ static int send_frame(FilterGraph *fg, FilterGraphThread *fgt, ) #endif + fd = frame_data(frame); + if (!fd) + return AVERROR(ENOMEM); + fd->wallclock[LATENCY_PROBE_FILTER_PRE] = av_gettime_relative(); + ret = av_buffersrc_add_frame_flags(ifp->filter, frame, AV_BUFFERSRC_FLAG_PUSH); if (ret < 0) { From patchwork Tue Dec 12 12:09:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 45089 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1225:b0:181:818d:5e7f with SMTP id v37csp3544416pzf; Tue, 12 Dec 2023 04:10:21 -0800 (PST) X-Google-Smtp-Source: AGHT+IEWFtKWlOtR8duJ9z8zb09fCIfLtOWFAaO9rM/ILjf4ddLWfNjtbxwYt5LMHmNbpvJgRijt X-Received: by 2002:a05:651c:150:b0:2ca:361c:e135 with SMTP id c16-20020a05651c015000b002ca361ce135mr2510417ljd.25.1702383020987; Tue, 12 Dec 2023 04:10:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702383020; cv=none; d=google.com; s=arc-20160816; b=zYruHwBIjar15SZv9A29B0NvtbQQ3lRYaH9baFUduzU7FZZD9TR/hBIi3AS5giZ+ft 9JIGrx5oLTK8ujpxoRXr1rK7mmYeR6Z0PuYATdoVU5oxrm7JuqarHymvj9PDEWEIUQp7 D1Qz1If7d5lg8IGqpxqBfazLp7bfuqi+a4wCfslNYTVM8j4i6Ns3zJd6gLxUcDm+yl45 F+clvXxz8azwiR/E9ANVMHX/lFU6yvlHTTYlIOnO3AvrFfYuzWRCAVuZ3vZ5AOn4ZFix 7wNLjb/E39GyUfNnM7kjguqj0X99+6wJe4EpmrUcVd+kokDY2+7VGHnl/AZY5vVJ1uB1 nkpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=4r2KgsX7QlrzQjCXsEVquQKXzQQsBXdcJNBWaMROlvo=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=zKf9NPQZq1fdhd7Ba6UpW1/gyZBhn3vux8qntnFkecqJCte6NSU7AyrqSQQhdHbbI/ GjwH10HqbWz140gNdroG+MNCg1cp5a962S7s3qzFxFHAYLpL1xVr7iTjAJK03OzW62FV CfKX5ZP0uFODOnV2rR2ssECB5yZe/2LfvpTOz6F03JSyYgPB6U01wX0r9+shg79Azjmg PXczWJM6qFuQ12Gl/w9tOXK++XgxMdamSyx2o0P5fxJIDDcA00kv7L4LgeYaXvw+2A+1 X5eRLWBIEj/tnQdSUXnhHcoyOiXsq04/AymoID+WPNE47NDV/XWbrdCgwHKLq0CwzRk3 yoYA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r24-20020a50c018000000b0054c5ef5de9esi4321636edb.579.2023.12.12.04.10.19; Tue, 12 Dec 2023 04:10:20 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 81B8A68D102; Tue, 12 Dec 2023 14:10:15 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail1.khirnov.net (quelana.khirnov.net [94.230.150.81]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2DF2068D008 for ; Tue, 12 Dec 2023 14:10:08 +0200 (EET) Received: from localhost (mail1.khirnov.net [IPv6:::1]) by mail1.khirnov.net (Postfix) with ESMTP id 8772E1BA2 for ; Tue, 12 Dec 2023 13:10:07 +0100 (CET) Received: from mail1.khirnov.net ([IPv6:::1]) by localhost (mail1.khirnov.net [IPv6:::1]) (amavis, port 10024) with ESMTP id Co9lraV-S3jf for ; Tue, 12 Dec 2023 13:10:06 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail1.khirnov.net (Postfix) with ESMTPS id C156612B6 for ; Tue, 12 Dec 2023 13:10:06 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:::1]) by libav.khirnov.net (Postfix) with ESMTP id AD8233A05D0 for ; Tue, 12 Dec 2023 13:09:59 +0100 (CET) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Tue, 12 Dec 2023 13:09:58 +0100 Message-ID: <20231212120958.21313-3-anton@khirnov.net> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231212120958.21313-1-anton@khirnov.net> References: <20231212120958.21313-1-anton@khirnov.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/3] fftools/ffmpeg_mux: print latency information in -debug_ts muxing output X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: IxbI4Lm+WmLO --- fftools/ffmpeg_mux.c | 87 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 77 insertions(+), 10 deletions(-) diff --git a/fftools/ffmpeg_mux.c b/fftools/ffmpeg_mux.c index 62925be8d0..ca1996622d 100644 --- a/fftools/ffmpeg_mux.c +++ b/fftools/ffmpeg_mux.c @@ -25,10 +25,12 @@ #include "ffmpeg_utils.h" #include "sync_queue.h" +#include "libavutil/avstring.h" #include "libavutil/fifo.h" #include "libavutil/intreadwrite.h" #include "libavutil/log.h" #include "libavutil/mem.h" +#include "libavutil/time.h" #include "libavutil/timestamp.h" #include "libavcodec/packet.h" @@ -59,6 +61,79 @@ static int64_t filesize(AVIOContext *pb) return ret; } +static void mux_log_debug_ts(OutputStream *ost, const AVPacket *pkt) +{ + static const char *desc[] = { + [LATENCY_PROBE_DEMUX] = "demux", + [LATENCY_PROBE_DEC_PRE] = "decode", + [LATENCY_PROBE_DEC_POST] = "decode", + [LATENCY_PROBE_FILTER_PRE] = "filter", + [LATENCY_PROBE_FILTER_POST] = "filter", + [LATENCY_PROBE_ENC_PRE] = "encode", + [LATENCY_PROBE_ENC_POST] = "encode", + [LATENCY_PROBE_NB] = "mux", + }; + + char latency[512] = { 0 }; + + if (pkt->opaque_ref) { + const FrameData *fd = (FrameData*)pkt->opaque_ref->data; + int64_t now = av_gettime_relative(); + int64_t total = INT64_MIN; + + int next; + + for (unsigned i = 0; i < FF_ARRAY_ELEMS(fd->wallclock); i = next) { + int64_t val = fd->wallclock[i]; + + next = i + 1; + + if (val == INT64_MIN) + continue; + + if (total == INT64_MIN) { + total = now - val; + snprintf(latency, sizeof(latency), "total:%gms", total / 1e3); + } + + // find the next valid entry + for (; next <= FF_ARRAY_ELEMS(fd->wallclock); next++) { + int64_t val_next = (next == FF_ARRAY_ELEMS(fd->wallclock)) ? + now : fd->wallclock[next]; + int64_t diff; + + if (val_next == INT64_MIN) + continue; + diff = val_next - val; + + // print those stages that take at least 5% of total + if (100. * diff > 5. * total) { + av_strlcat(latency, ", ", sizeof(latency)); + + if (!strcmp(desc[i], desc[next])) + av_strlcat(latency, desc[i], sizeof(latency)); + else + av_strlcatf(latency, sizeof(latency), "%s-%s:", + desc[i], desc[next]); + + av_strlcatf(latency, sizeof(latency), " %gms/%d%%", + diff / 1e3, (int)(100. * diff / total)); + } + + break; + } + + } + } + + av_log(ost, AV_LOG_INFO, "muxer <- pts:%s pts_time:%s dts:%s dts_time:%s " + "duration:%s duration_time:%s size:%d latency(%s)\n", + av_ts2str(pkt->pts), av_ts2timestr(pkt->pts, &ost->st->time_base), + av_ts2str(pkt->dts), av_ts2timestr(pkt->dts, &ost->st->time_base), + av_ts2str(pkt->duration), av_ts2timestr(pkt->duration, &ost->st->time_base), + pkt->size, *latency ? latency : "N/A"); +} + static int write_packet(Muxer *mux, OutputStream *ost, AVPacket *pkt) { MuxStream *ms = ms_from_ost(ost); @@ -140,16 +215,8 @@ static int write_packet(Muxer *mux, OutputStream *ost, AVPacket *pkt) pkt->stream_index = ost->index; - if (debug_ts) { - av_log(ost, AV_LOG_INFO, "muxer <- type:%s " - "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s duration:%s duration_time:%s size:%d\n", - av_get_media_type_string(ost->type), - av_ts2str(pkt->pts), av_ts2timestr(pkt->pts, &ost->st->time_base), - av_ts2str(pkt->dts), av_ts2timestr(pkt->dts, &ost->st->time_base), - av_ts2str(pkt->duration), av_ts2timestr(pkt->duration, &ost->st->time_base), - pkt->size - ); - } + if (debug_ts) + mux_log_debug_ts(ost, pkt); if (ms->stats.io) enc_stats_write(ost, &ms->stats, NULL, pkt, frame_num);