From patchwork Fri Dec 8 17:31:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Izen X-Patchwork-Id: 44980 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1225:b0:181:818d:5e7f with SMTP id v37csp1447917pzf; Fri, 8 Dec 2023 09:31:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IHt9lmzbOGWnKTgMzKLNXvpIJe7dboO9Xacv1uVqUKVrfLG3HiKkQb7Fgz8NgyMw3tlF3ix X-Received: by 2002:a17:906:cb94:b0:a18:91d0:c58e with SMTP id mf20-20020a170906cb9400b00a1891d0c58emr118498ejb.52.1702056680207; Fri, 08 Dec 2023 09:31:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702056680; cv=none; d=google.com; s=arc-20160816; b=ENyQ6YY/Z+2uaLaopaEZRKfMDGor+A617V4msCQXZ5Q9XWgoqtmm1LlW+EMp8B8MIs CYkqpnQfxycCOEgtcG9cBDAwEveG6GBbeeAdgk2GKRpxZ7oLkEaxvSXhMwk9345JOmGr Yff9vEJ4loYcSzRU1hqyIY5DUaIbj2TTuTreeNLnzqDROI6VMnLvbBpLrNJi//K9gl1W M7HTGBW61mzaUyJ9lXc5A3cRHgTON1c0GxSeMWvXV0AAb07IaofRcOKiATzvYDvUY1Z4 sC7cg2M/HBjclJQIorJrC5SlmIunNqcmcMoa6lWotLGlU9c5TKo54zcXJ57KUlrzOs9r h8BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=yC1GG3UIBVmVfh6Rp5BPCLoL2cgayUESjbZkd8T/r+E=; fh=+bdjGe20eEUjtjncwA1dnEEVYNfJL4vyhV+sIRR4l+g=; b=BQpphtKR8ZOVUK1jtJFlBHNoFdvrS7pibp/sPtOrbakv3+Euid42L99ML30Qpldz6S rH5O8WG7FS7HN5I2IoXjNQ38I/1xEeHQd1WCfusmP/RQxml5HN7UiEDBdmDu76Zln/0G 1ZUpeYhjOgDdWlHL2vegvZIagarGhckY0n6K2Hbf4yNnF3stNJyPpV0Pe7mBMwQyWRX+ 6DZAMC/w5OGOwaWl6uRkdBP3LhPxZ8kx87MDRR805NOjjeAqQtkFK9X/fBh7uyOeeQfe zaCRIMYI6yMpOm97tkag5m1J4fa8qc/tBSeifyRI3suOPh+WeW64W9zyBLzLAElfLERy uRnA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=ZDexLxDc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id p12-20020a1709060e8c00b00a1d30e0d22csi965435ejf.123.2023.12.08.09.31.19; Fri, 08 Dec 2023 09:31:20 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=ZDexLxDc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 67FE168CBBE; Fri, 8 Dec 2023 19:31:16 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A42A368CBBE for ; Fri, 8 Dec 2023 19:31:09 +0200 (EET) Received: by mail-qt1-f180.google.com with SMTP id d75a77b69052e-4258ca2ee9dso4555231cf.1 for ; Fri, 08 Dec 2023 09:31:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702056668; x=1702661468; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=hKOgtScGeqBU6l6auqDFqRfILkmCihHG6IJSDERDeIQ=; b=ZDexLxDc0yKoo1BX4rBFwAIjVf9KZk9i4uzrJccHXzeRAa3xAHXM7TGNZTnfHvHJiM ifaBftydReBOpEQKZmjcHeX5CzkFuwq7GE6JehoLD2LC8m7twbTg5KG6Tbzh7SO25yeG gQgCLqTDx3rgZtvmjNuCtDVsLQmxjlHmbL7tT0TJAmsGYnS9iZ1jPDw3xpDWqJDCn4Sp 6HZZqigcUMmsmi/qDcZwr3wNMmr/jREcdatAbLts8MjqFmP/5Fd/yL3K4wqBwo29uDJe MtvoReh0vzhmGwL/Ut4+pYJm/3ovWZEW3Xi1uLWLuK9Z5F5s72hEe6Ut+K1CL8D/lSD0 4fMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702056668; x=1702661468; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hKOgtScGeqBU6l6auqDFqRfILkmCihHG6IJSDERDeIQ=; b=H1onTXqIw2ErEJEC8XoI0D8/t+3oz3TlcEJvo0hctpGSV5IbW/ZZQRRF2svOYpM0TI 9y4h2usgBuhREKVSi8viybafKpF3n5mciFLms+k9q5um/U/RQ+xGX70aNChCOXoa/7HP nbbxfm/ZVJjsORbQlcy6ZBsycvszV4BBMyh1XtqH5NybqXX2k/YHVUbO0xspv/A+xCbi rcneieoR4LFWrU3KkqLIab+eFsC1sjCD5PJpWU8by4aWR0OZIdhQloTFbKSdm8kv5b30 fxVrN5VdVgKlGKqQZkVU+BfEONeBvLMVEre4NeDWx2ZLSIzI1HxlN5aw3y30ghW9yhdL Cm9w== X-Gm-Message-State: AOJu0Yw5r32Um2ISSTO1pUdBriOHZSoJBvG73uexrcTfnBiUALfoZdSZ uqzihFvARzkLzYRrQunv+2A6aJqSIzM= X-Received: by 2002:ac8:7284:0:b0:41e:a62b:3d28 with SMTP id v4-20020ac87284000000b0041ea62b3d28mr680399qto.4.1702056668172; Fri, 08 Dec 2023 09:31:08 -0800 (PST) Received: from gauss.local (c-68-56-149-176.hsd1.mi.comcast.net. [68.56.149.176]) by smtp.gmail.com with ESMTPSA id d11-20020ac8668b000000b0042545c7beccsm956802qtp.59.2023.12.08.09.31.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 09:31:07 -0800 (PST) From: Leo Izen To: ffmpeg-devel@ffmpeg.org Date: Fri, 8 Dec 2023 12:31:06 -0500 Message-ID: <20231208173106.165084-1-leo.izen@gmail.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avcodec/libjxldec: emit proper PTS to decoded AVFrame X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Leo Izen Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 3nd23mp6SDWc If a sequence of JXL images is encapsulated in a container that has PTS information, we should use the PTS information from the container. At this time there is no container that does this, but if JPEG XL support is ever added to NUT, AVTransport, or some other container, this commit should allow the PTS information those containers provide to work as expected. Signed-off-by: Leo Izen --- libavcodec/libjxldec.c | 77 +++++++++++++++++++++++++++++++----------- 1 file changed, 57 insertions(+), 20 deletions(-) diff --git a/libavcodec/libjxldec.c b/libavcodec/libjxldec.c index 002740d9c1..494060ac8c 100644 --- a/libavcodec/libjxldec.c +++ b/libavcodec/libjxldec.c @@ -370,6 +370,7 @@ static int libjxl_receive_frame(AVCodecContext *avctx, AVFrame *frame) while (1) { size_t remaining; + JxlFrameHeader header; if (!pkt->size) { av_packet_unref(pkt); @@ -428,13 +429,16 @@ static int libjxl_receive_frame(AVCodecContext *avctx, AVFrame *frame) } if ((ret = ff_set_dimensions(avctx, ctx->basic_info.xsize, ctx->basic_info.ysize)) < 0) return ret; + /* + * If animation is present, we use the timebase provided by + * the animated image itself. + * If the image is not animated, we use ctx->pts + * to refer to the frame number, not an actual + * PTS value, thus we may leave ctx->timebase unset. + */ if (ctx->basic_info.have_animation) ctx->timebase = av_make_q(ctx->basic_info.animation.tps_denominator, ctx->basic_info.animation.tps_numerator); - else if (avctx->pkt_timebase.num) - ctx->timebase = avctx->pkt_timebase; - else - ctx->timebase = AV_TIME_BASE_Q; continue; case JXL_DEC_COLOR_ENCODING: av_log(avctx, AV_LOG_DEBUG, "COLOR_ENCODING event emitted\n"); @@ -462,23 +466,24 @@ static int libjxl_receive_frame(AVCodecContext *avctx, AVFrame *frame) #endif continue; case JXL_DEC_FRAME: + /* Frame here refers to the Frame bundle, not a decoded picture */ av_log(avctx, AV_LOG_DEBUG, "FRAME event emitted\n"); - if (!ctx->basic_info.have_animation || ctx->prev_is_last) { + if (ctx->prev_is_last) { + /* + * The last frame sent was tagged as "is_last" which + * means this is a new image file altogether. + */ ctx->frame->pict_type = AV_PICTURE_TYPE_I; ctx->frame->flags |= AV_FRAME_FLAG_KEY; } - if (ctx->basic_info.have_animation) { - JxlFrameHeader header; - if (JxlDecoderGetFrameHeader(ctx->decoder, &header) != JXL_DEC_SUCCESS) { - av_log(avctx, AV_LOG_ERROR, "Bad libjxl dec frame event\n"); - return AVERROR_EXTERNAL; - } - ctx->prev_is_last = header.is_last; - ctx->frame_duration = header.duration; - } else { - ctx->prev_is_last = 1; - ctx->frame_duration = 1; + if (JxlDecoderGetFrameHeader(ctx->decoder, &header) != JXL_DEC_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Bad libjxl dec frame event\n"); + return AVERROR_EXTERNAL; } + ctx->prev_is_last = header.is_last; + /* zero duration for animation means the frame is not presented */ + if (ctx->basic_info.have_animation && header.duration) + ctx->frame_duration = header.duration; continue; case JXL_DEC_FULL_IMAGE: /* full image is one frame, even if animated */ @@ -490,12 +495,44 @@ static int libjxl_receive_frame(AVCodecContext *avctx, AVFrame *frame) /* ownership is transfered, and it is not ref-ed */ ctx->iccp = NULL; } - if (avctx->pkt_timebase.num) { - ctx->frame->pts = av_rescale_q(ctx->pts, ctx->timebase, avctx->pkt_timebase); - ctx->frame->duration = av_rescale_q(ctx->frame_duration, ctx->timebase, avctx->pkt_timebase); + if (ctx->basic_info.have_animation) { + if (avctx->pkt_timebase.num) { + /* + * ideally, the demuxer set avctx->pkt_timebase to equal the animation's timebase + * or something strictly finer. This is true about the jpegxl_anim demuxer. + */ + ctx->frame->pts = av_rescale_q(ctx->pts, ctx->timebase, avctx->pkt_timebase); + ctx->frame->duration = av_rescale_q(ctx->frame_duration, ctx->timebase, avctx->pkt_timebase); + } else { + /* + * If we don't know the container timebase, we have to set the frame->timebase, + * even if it is currently ignored by most users. We don't have permission + * to set avctx->pkt_timebase. + */ + ctx->frame->time_base = ctx->timebase; + ctx->frame->pts = ctx->pts; + ctx->frame->duration = ctx->frame_duration; + } + } else if (avctx->pkt_timebase.num) { + if (pkt->pts != AV_NOPTS_VALUE) { + /* The container has provided the PTS for us, so we don't need to count frames. */ + ctx->frame->pts = pkt->pts; + } else { + /* + * The demuxer has provided us with a timebase, but not with PTS information. + * We use 1/1 as a dummy timebase, for 1fps as a dummy framerate, and set the + * PTS based on frame count. + */ + const AVRational dummy = {.num = 1, .den = 1}; + ctx->frame->pts = av_rescale_q(ctx->pts, dummy, avctx->pkt_timebase); + } + ctx->frame->duration = pkt->duration; } else { + /* + * There is no timing information. Set the frame PTS to frame counter. + */ ctx->frame->pts = ctx->pts; - ctx->frame->duration = ctx->frame_duration; + ctx->frame->duration = 0; } ctx->pts += ctx->frame_duration; av_frame_move_ref(frame, ctx->frame);