From patchwork Thu Apr 27 13:09:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Izen X-Patchwork-Id: 41359 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp501598pzb; Thu, 27 Apr 2023 06:10:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5PEk1p0iBB84sJARGogkLnJGnWfQPIU2rFVl2pTBPXp5nrVS2Q5nJXXqDDjtuf0xbP/KCJ X-Received: by 2002:a17:907:60cc:b0:94d:69e0:6098 with SMTP id hv12-20020a17090760cc00b0094d69e06098mr1952340ejc.45.1682601014456; Thu, 27 Apr 2023 06:10:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682601014; cv=none; d=google.com; s=arc-20160816; b=q6BUG2+JSPSAWZVv4yepS3JiOhm65nnBi1s69IAGTE1q8xrIBQvhQEa2rARsWZa6Br qL182gvS7QFDHxHZ7zF3P8Ygv/ByiNVc9n5CI8t2G5u9a6SHMbFF/6AWAuqsVJ6Da34t fPweQGHlqpBY3pJRaQUpWLdnXT2tTjhfC5Wkic59ud+dX7sbKVqD/4qvdlyn0AFlyY2D WIATcHt/lKVmjDueOo/LYDXQadKrvXZUxqlReor4DEILs+QlgvuQTHiLE85EXFc9n5/5 yGstHQZz507DgPKL5cPTLPoduqXhX5w0abJHK4PtXCE2Nwo3kkbuR2TYJ06qOgu/MwEq HtJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=FJzTsysYQqxMTkC8LXWQWBapMyGwMM1/lPbLbcMuoDQ=; b=n8ENhMFMxrg1yqaci91Gg9pdEct+fUwTCBviSDFx8EBvnHq6bLEO/C0ka5QwxTyRj4 IgjoV9Ke/sof49vFekqx9GqDcbXlLn2H0ah70lJ/C92mhXICYLMp/LsVeAnJ+stvcvzy i7AmGQM3GQW4C4hfvoK5KALSUufDhMnTVXlc8d0wGjM0ZQf6fjLDdksp7ltnwoQUeZf9 Cz0Q0oDDYOs1/d0uleTy4zbiZYnzHQH8YJorH/jf/Ry57e70C/tpnvP+nR/IP/hcsQZN pYt4Y9RXOCubRootTRWabnqLF/Rluq1XnMQESaj85cAppmLMjlm8IhAHtaS8uXTS7JH9 BDjw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=gS0fiKqH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bm4-20020a0564020b0400b005047e14ecc4si13717633edb.224.2023.04.27.06.10.03; Thu, 27 Apr 2023 06:10:14 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=gS0fiKqH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4BEB268BF53; Thu, 27 Apr 2023 16:09:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D6F3668B5DA for ; Thu, 27 Apr 2023 16:09:40 +0300 (EEST) Received: by mail-yb1-f176.google.com with SMTP id 3f1490d57ef6-b991bac987dso1111461276.1 for ; Thu, 27 Apr 2023 06:09:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682600979; x=1685192979; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sfXckP6cBTOpv4Vqp/gAgs3qitU4GKRCtk2C0NEsC3s=; b=gS0fiKqHAmLNwfJG/YmzYPphTL4cbQcwGSzGhE7ICfKKPTN/hw2mg/q+DbI1ulqQNR HbsLc+Hcot50EvFwp+rQeabJ0JARlzSoXfwlycgUjwt0jG+biQ/kcb2wRdx/AXF9aNgi D6anwBb3RSB+0b497qxaSjvJQUlwb3yNsqOMUglfamZCRYOinMOKT9DMrB/lim1sfYvT hONTYiiZdDnZOvYg3lrTEUJxADtMheSVp1BetAL5kvuK5JPN04vYqrn4YES5xjv3D1qP mqnKaZ2+kImTavq7Ud+Xg46DDIyxt0gd1WzCiLNg5Gb50ydReq0LTXLXfuGBaiD75oYN Hz5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682600979; x=1685192979; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sfXckP6cBTOpv4Vqp/gAgs3qitU4GKRCtk2C0NEsC3s=; b=T7TQPpDCb9pesEnsVCFNkvO0jP4mUgbKZ92hCs6MWblplg0LczwodnDnRtAyV+Hder VrnZBjK7m/4jOflLTIK3jdEte4UzLEjRnMAz39PqSzpGvEuHjZV3HKuVqDhQeeuTEXtB /4LtjrQmhRJv0qV+YMZelsC+9ZqOAgdB19wkZxM9Yr4BBZ/k0ceJYK/2sYy1VA55tQXj KbzfpMVtiF+O4jdEW8Jpzy0qHIDQpFeAEY7ycLXPYO1wAus53QKFlnmHaNOnd/Od7u2o gjH+3O0zSgR37Ip2qdh6Sg4qEVUvzs0GAvnL5Hqdd0MoIj0BN7V0qLD05IdWvZEHKDmb h27Q== X-Gm-Message-State: AC+VfDx3bk2kB/Co/WAYxKnKecf7W4cKPH/tuN/sjnZ25X9K0pelpLcv 5f3BART4xbCZTbG0a4I4zZAKdveniOI= X-Received: by 2002:a81:1209:0:b0:54c:a77:cc7a with SMTP id 9-20020a811209000000b0054c0a77cc7amr1123323yws.1.1682600979290; Thu, 27 Apr 2023 06:09:39 -0700 (PDT) Received: from gauss.local (c-98-224-219-15.hsd1.mi.comcast.net. [98.224.219.15]) by smtp.gmail.com with ESMTPSA id h7-20020a0dde07000000b0054662f7b42dsm4736321ywe.63.2023.04.27.06.09.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Apr 2023 06:09:39 -0700 (PDT) From: Leo Izen To: ffmpeg-devel@ffmpeg.org Date: Thu, 27 Apr 2023 09:09:29 -0400 Message-Id: <20230427130930.45638-2-leo.izen@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230427130930.45638-1-leo.izen@gmail.com> References: <20230427130930.45638-1-leo.izen@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 1/2] avcodec/libjxldec: add animated decode support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Leo Izen Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: EddwQs2d46Yq Migrate the libjxl decoder wrapper from the decode_frame method to the receive_frame method, which allows sending more than one frame from a single packet. This allows the libjxl decoder to decode JPEG XL files that are animated, and emit every frame of the animation. Now, clients that feed the libjxl decoder with an animated JPEG XL file will be able to receieve the full animation. Signed-off-by: Leo Izen --- libavcodec/libjxldec.c | 109 ++++++++++++++++++++++++++++++----------- libavcodec/version.h | 2 +- 2 files changed, 82 insertions(+), 29 deletions(-) diff --git a/libavcodec/libjxldec.c b/libavcodec/libjxldec.c index 045a1535f9..5940d0f407 100644 --- a/libavcodec/libjxldec.c +++ b/libavcodec/libjxldec.c @@ -52,13 +52,19 @@ typedef struct LibJxlDecodeContext { #endif JxlDecoderStatus events; AVBufferRef *iccp; + AVPacket *avpkt; + int64_t pts; + int64_t frame_duration; + int prev_is_last; + AVRational timebase; } LibJxlDecodeContext; static int libjxl_init_jxl_decoder(AVCodecContext *avctx) { LibJxlDecodeContext *ctx = avctx->priv_data; - ctx->events = JXL_DEC_BASIC_INFO | JXL_DEC_FULL_IMAGE | JXL_DEC_COLOR_ENCODING; + ctx->events = JXL_DEC_BASIC_INFO | JXL_DEC_FULL_IMAGE + | JXL_DEC_COLOR_ENCODING | JXL_DEC_FRAME; if (JxlDecoderSubscribeEvents(ctx->decoder, ctx->events) != JXL_DEC_SUCCESS) { av_log(avctx, AV_LOG_ERROR, "Error subscribing to JXL events\n"); return AVERROR_EXTERNAL; @@ -71,6 +77,8 @@ static int libjxl_init_jxl_decoder(AVCodecContext *avctx) memset(&ctx->basic_info, 0, sizeof(JxlBasicInfo)); memset(&ctx->jxl_pixfmt, 0, sizeof(JxlPixelFormat)); + ctx->prev_is_last = 1; + ctx->frame_duration = 1; return 0; } @@ -93,6 +101,11 @@ static av_cold int libjxl_decode_init(AVCodecContext *avctx) return AVERROR_EXTERNAL; } + ctx->avpkt = av_packet_alloc(); + if (!ctx->avpkt) + return AVERROR(ENOMEM); + ctx->pts = 0; + return libjxl_init_jxl_decoder(avctx); } @@ -328,19 +341,33 @@ static int libjxl_color_encoding_event(AVCodecContext *avctx, AVFrame *frame) return 0; } -static int libjxl_decode_frame(AVCodecContext *avctx, AVFrame *frame, int *got_frame, AVPacket *avpkt) +static int libjxl_receive_frame(AVCodecContext *avctx, AVFrame *frame) { LibJxlDecodeContext *ctx = avctx->priv_data; - const uint8_t *buf = avpkt->data; - size_t remaining = avpkt->size; - JxlDecoderStatus jret; + JxlDecoderStatus jret = JXL_DEC_SUCCESS; int ret; - *got_frame = 0; + AVPacket *pkt = ctx->avpkt; while (1) { + size_t remaining; - jret = JxlDecoderSetInput(ctx->decoder, buf, remaining); + if (!pkt->size) { + av_packet_unref(pkt); + ret = ff_decode_get_packet(avctx, pkt); + if (ret < 0 && ret != AVERROR_EOF) + return ret; + if (!pkt->size) { + /* jret set by the last iteration of the loop */ + if (jret == JXL_DEC_NEED_MORE_INPUT) { + av_log(avctx, AV_LOG_ERROR, "Unexpected end of JXL codestream\n"); + return AVERROR_INVALIDDATA; + } else { + return AVERROR_EOF; + } + } + } + jret = JxlDecoderSetInput(ctx->decoder, pkt->data, pkt->size); if (jret == JXL_DEC_ERROR) { /* this should never happen here unless there's a bug in libjxl */ av_log(avctx, AV_LOG_ERROR, "Unknown libjxl decode error\n"); @@ -354,18 +381,19 @@ static int libjxl_decode_frame(AVCodecContext *avctx, AVFrame *frame, int *got_f * the number of bytes that it did read */ remaining = JxlDecoderReleaseInput(ctx->decoder); - buf = avpkt->data + avpkt->size - remaining; + pkt->data += pkt->size - remaining; + pkt->size = remaining; switch(jret) { case JXL_DEC_ERROR: av_log(avctx, AV_LOG_ERROR, "Unknown libjxl decode error\n"); return AVERROR_INVALIDDATA; case JXL_DEC_NEED_MORE_INPUT: - if (remaining == 0) { - av_log(avctx, AV_LOG_ERROR, "Unexpected end of JXL codestream\n"); - return AVERROR_INVALIDDATA; - } av_log(avctx, AV_LOG_DEBUG, "NEED_MORE_INPUT event emitted\n"); + if (!pkt->size) { + av_packet_unref(pkt); + return AVERROR(EAGAIN); + } continue; case JXL_DEC_BASIC_INFO: av_log(avctx, AV_LOG_DEBUG, "BASIC_INFO event emitted\n"); @@ -384,6 +412,13 @@ static int libjxl_decode_frame(AVCodecContext *avctx, AVFrame *frame, int *got_f } if ((ret = ff_set_dimensions(avctx, ctx->basic_info.xsize, ctx->basic_info.ysize)) < 0) return ret; + if (ctx->basic_info.have_animation) + ctx->timebase = av_make_q(ctx->basic_info.animation.tps_denominator, + ctx->basic_info.animation.tps_numerator); + else if (avctx->pkt_timebase.num) + ctx->timebase = avctx->pkt_timebase; + else + ctx->timebase = AV_TIME_BASE_Q; continue; case JXL_DEC_COLOR_ENCODING: av_log(avctx, AV_LOG_DEBUG, "COLOR_ENCODING event emitted\n"); @@ -407,11 +442,28 @@ static int libjxl_decode_frame(AVCodecContext *avctx, AVFrame *frame, int *got_f } #endif continue; + case JXL_DEC_FRAME: + av_log(avctx, AV_LOG_DEBUG, "FRAME event emitted\n"); + if (!ctx->basic_info.have_animation || ctx->prev_is_last) { + frame->pict_type = AV_PICTURE_TYPE_I; + frame->key_frame = 1; + } + if (ctx->basic_info.have_animation) { + JxlFrameHeader header; + if (JxlDecoderGetFrameHeader(ctx->decoder, &header) != JXL_DEC_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Bad libjxl dec frame event\n"); + return AVERROR_EXTERNAL; + } + ctx->prev_is_last = header.is_last; + ctx->frame_duration = header.duration; + } else { + ctx->prev_is_last = 1; + ctx->frame_duration = 1; + } + continue; case JXL_DEC_FULL_IMAGE: /* full image is one frame, even if animated */ av_log(avctx, AV_LOG_DEBUG, "FULL_IMAGE event emitted\n"); - frame->pict_type = AV_PICTURE_TYPE_I; - frame->key_frame = 1; if (ctx->iccp) { AVFrameSideData *sd = av_frame_new_side_data_from_buf(frame, AV_FRAME_DATA_ICC_PROFILE, ctx->iccp); if (!sd) @@ -419,25 +471,25 @@ static int libjxl_decode_frame(AVCodecContext *avctx, AVFrame *frame, int *got_f /* ownership is transfered, and it is not ref-ed */ ctx->iccp = NULL; } - *got_frame = 1; - return avpkt->size - remaining; + if (avctx->pkt_timebase.num) { + frame->pts = av_rescale_q(ctx->pts, ctx->timebase, avctx->pkt_timebase); + frame->duration = av_rescale_q(ctx->frame_duration, ctx->timebase, avctx->pkt_timebase); + } else { + frame->pts = ctx->pts; + frame->duration = ctx->frame_duration; + } + ctx->pts += ctx->frame_duration; + return 0; case JXL_DEC_SUCCESS: av_log(avctx, AV_LOG_DEBUG, "SUCCESS event emitted\n"); /* - * The SUCCESS event isn't fired until after JXL_DEC_FULL_IMAGE. If this - * stream only contains one JXL image then JXL_DEC_SUCCESS will never fire. - * If the image2 sequence being decoded contains several JXL files, then - * libjxl will fire this event after the next AVPacket has been passed, - * which means the current packet is actually the next image in the sequence. - * This is why we reset the decoder and populate the packet data now, since - * this is the next packet and it has not been decoded yet. The decoder does - * have to be reset to allow us to use it for the next image, or libjxl - * will become very confused if the header information is not identical. + * this event will be fired when the zero-length EOF + * packet is sent to the decoder by the client, + * but it will also be fired when the next image of + * an image2pipe sequence is loaded up */ JxlDecoderReset(ctx->decoder); libjxl_init_jxl_decoder(avctx); - buf = avpkt->data; - remaining = avpkt->size; continue; default: av_log(avctx, AV_LOG_ERROR, "Bad libjxl event: %d\n", jret); @@ -457,6 +509,7 @@ static av_cold int libjxl_decode_close(AVCodecContext *avctx) JxlDecoderDestroy(ctx->decoder); ctx->decoder = NULL; av_buffer_unref(&ctx->iccp); + av_packet_free(&ctx->avpkt); return 0; } @@ -468,7 +521,7 @@ const FFCodec ff_libjxl_decoder = { .p.id = AV_CODEC_ID_JPEGXL, .priv_data_size = sizeof(LibJxlDecodeContext), .init = libjxl_decode_init, - FF_CODEC_DECODE_CB(libjxl_decode_frame), + FF_CODEC_RECEIVE_FRAME_CB(libjxl_receive_frame), .close = libjxl_decode_close, .p.capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_OTHER_THREADS, .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE | diff --git a/libavcodec/version.h b/libavcodec/version.h index 80e2ae630d..c576ee1520 100644 --- a/libavcodec/version.h +++ b/libavcodec/version.h @@ -30,7 +30,7 @@ #include "version_major.h" #define LIBAVCODEC_VERSION_MINOR 10 -#define LIBAVCODEC_VERSION_MICRO 100 +#define LIBAVCODEC_VERSION_MICRO 101 #define LIBAVCODEC_VERSION_INT AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \ LIBAVCODEC_VERSION_MINOR, \ From patchwork Thu Apr 27 13:09:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Izen X-Patchwork-Id: 41360 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp502018pzb; Thu, 27 Apr 2023 06:10:39 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6Cg7OBqMURkTGQs0GNWC3TzgQbw15uaqIc25TTK/1lKBMVJgkG6E0aSTl3tyFM+ut516rP X-Received: by 2002:a17:907:704:b0:95d:e076:5033 with SMTP id xb4-20020a170907070400b0095de0765033mr1563833ejb.16.1682601038850; Thu, 27 Apr 2023 06:10:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682601038; cv=none; d=google.com; s=arc-20160816; b=0aeyGJx6umoOMkrvmBaS+wRH/ELxe/jNsv2B/o1c9Y2/L3JnrlTCvEfOztBnlGL311 xuw/gN87EAqMCaRkl/FOAySJdgTa3wJxI2EniqZPlZbx9AsL8Dewg2g2kBK0xZ6bX/VJ 5oXPa069RrVQr0HnO8Oz9pPvkO4B42hsyKfvYCGQAde4imaq1j7qXDgUj76FlCUHq2na qhtTJkcfzFNo4DkcfY7xR/QLz9yvhvOH42gEDLwZs3cad8RG0MuTcIQByBC7YIQzGJMx JfRUkQArw2fljA3tsr9Z8VvV4agxCe/9jIfOa6eFNEsDcfKezNletSu81r78IGKfmCKX A2qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=jYSD+BU/HCSfo7PWTsGLLa89jh1AC77yfAnA0APC2ig=; b=Mo+iL070ShPFLfbf1nnmBF01e8/kgGWQgN9BG+K4My1K0KdNeOvTTJ9GfLXBab+iGk C/OInzehPeRMv8qCcx5hfUgKEr3/xlt7DYMXUo5fl3jRWXlrA5SK1kvIZlpYt6BzgksR x6IitGcIJeKj2rwSmjhgfQrYCDMmmWXmvh87Qn5bP1bjlBnry1rpp7uP2oXfd6vXyJyO Qa38/YUImfqczMn6J6Nzl/J7SmEToIIm0mc9VKcwWaKJF4mAsRHkz0+7DnPHKxekXP1o P2Q3SjX12egTLzCO2MHy6ldgm4cmAdfd/20sUMqcwXY6m2DyuViKrhqe/xKVz9tmMkNi r/Dg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=qSGuGYUd; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id fm15-20020a1709072acf00b00957179f1f33si10822301ejc.156.2023.04.27.06.10.14; Thu, 27 Apr 2023 06:10:38 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=qSGuGYUd; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6390E68BF5C; Thu, 27 Apr 2023 16:09:49 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9CF4668BEF8 for ; Thu, 27 Apr 2023 16:09:41 +0300 (EEST) Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-5560fba3cacso10076777b3.1 for ; Thu, 27 Apr 2023 06:09:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682600980; x=1685192980; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u43yOUVy0de7NpPsAUmkCpgN+80gkCAa23hCo/4VAFM=; b=qSGuGYUdP8jBWVjTCMct2gv+88OGi0vqW+KUFmfz23/nVX23ms8/W4okJIPzyUn3F9 G6fvn87HtnmF6VqnaIUu7W6/DWfhV+hYjB4WrkzlA4/KgbFvqpWsF3P4DUwCB8gsYcnn dN8/UdhjMvesu7D6VyagSUexnsSt3Ff0xcM3oQJ1eQI2ACTEi2zYiOZeyIVTlTbYiV6Z u6PvyjgNXvGXKt0k0l8LpmEgVtQhh6NtUwbWw2az557SHt8ojGppnyO+NjqevjuQHg9D dQNRnqgh7+3WYvl8zl6PgkpAhL84mehOU9ORWzf28WNjz1yVbD5DS1kGdGRO9gZQwXy1 ueuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682600980; x=1685192980; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=u43yOUVy0de7NpPsAUmkCpgN+80gkCAa23hCo/4VAFM=; b=ktB6nvUMRyqRDHHKDEs1bGeFa6YrVgcj3ytMMWWOrXhEZYLIk0Z12hhS90uIPGwDMm lb3ZqiONtiq6UT8xUZolXvXa2IqQas/IwO7L/tHzON3KI3Sunax8eTG7GSymVgVd12B0 UNEqSRJcTwY4UBoLkT4CGfQmfGGE5Wz3fce7EKJiQdwwnRwB/wJr8thuEuRJWGUQEXmg 177dWPOh6Gcv3zW9crfJxMBXEQ6KA+MXkZAx++SnFBkrOzSM3AWwfoTnUv196+m0R+u6 1KSLGP3zo+fwVuBnQ5Ois7qJ2eqVk7X0dAx/0P0/FRfv2X2Jg2mTjaNKXw3pq+TAvrB3 86lQ== X-Gm-Message-State: AC+VfDyeNBmEXlHDI6/WpPLhdioV66gCVBLpIyB9FhlaWM1l+/e8VPTe LFdSzvlE1XvN4vCpB1J8LDWNyDB7UjQ= X-Received: by 2002:a81:fd5:0:b0:556:e93c:fbe with SMTP id 204-20020a810fd5000000b00556e93c0fbemr1058885ywp.5.1682600980160; Thu, 27 Apr 2023 06:09:40 -0700 (PDT) Received: from gauss.local (c-98-224-219-15.hsd1.mi.comcast.net. [98.224.219.15]) by smtp.gmail.com with ESMTPSA id h7-20020a0dde07000000b0054662f7b42dsm4736321ywe.63.2023.04.27.06.09.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Apr 2023 06:09:39 -0700 (PDT) From: Leo Izen To: ffmpeg-devel@ffmpeg.org Date: Thu, 27 Apr 2023 09:09:30 -0400 Message-Id: <20230427130930.45638-3-leo.izen@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230427130930.45638-1-leo.izen@gmail.com> References: <20230427130930.45638-1-leo.izen@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 2/2] avformat/jpegxl_anim_dec: add animated JPEG XL demuxer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Leo Izen Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: OKEO06CR5Ki0 Animated JPEG XL files requires a separate demuxer than image2, because the timebase information is set by the demuxer. Should the timebase of an animated JPEG XL file be incompatible with the timebase set by the image2pipe demuxer (usually 1/25 unless set otherwise), rescaling will fail. Adding a separate demuxer for animated JPEG XL files allows the timebase to be set correctly. Signed-off-by: Leo Izen --- MAINTAINERS | 1 + libavformat/Makefile | 1 + libavformat/allformats.c | 1 + libavformat/img2dec.c | 2 +- libavformat/jpegxl_anim_dec.c | 266 ++++++++++++++++++++++++++++++++++ libavformat/jpegxl_probe.c | 19 +-- libavformat/jpegxl_probe.h | 7 +- libavformat/version.h | 2 +- 8 files changed, 287 insertions(+), 12 deletions(-) create mode 100644 libavformat/jpegxl_anim_dec.c diff --git a/MAINTAINERS b/MAINTAINERS index 854ccc3fa4..d57e4a8ed3 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -432,6 +432,7 @@ Muxers/Demuxers: ipmovie.c Mike Melanson ircam* Paul B Mahol iss.c Stefan Gehrer + jpegxl_anim_dec.c Leo Izen jpegxl_probe.* Leo Izen jvdec.c Peter Ross kvag.c Zane van Iperen diff --git a/libavformat/Makefile b/libavformat/Makefile index f8ad7c6a11..05434a0f82 100644 --- a/libavformat/Makefile +++ b/libavformat/Makefile @@ -316,6 +316,7 @@ OBJS-$(CONFIG_IVF_MUXER) += ivfenc.o OBJS-$(CONFIG_IVR_DEMUXER) += rmdec.o rm.o rmsipr.o OBJS-$(CONFIG_JACOSUB_DEMUXER) += jacosubdec.o subtitles.o OBJS-$(CONFIG_JACOSUB_MUXER) += jacosubenc.o rawenc.o +OBJS-$(CONFIG_JPEGXL_ANIM_DEMUXER) += jpegxl_anim_dec.o jpegxl_probe.o OBJS-$(CONFIG_JV_DEMUXER) += jvdec.o OBJS-$(CONFIG_KUX_DEMUXER) += flvdec.o OBJS-$(CONFIG_KVAG_DEMUXER) += kvag.o diff --git a/libavformat/allformats.c b/libavformat/allformats.c index efdb34e29d..96443a7272 100644 --- a/libavformat/allformats.c +++ b/libavformat/allformats.c @@ -238,6 +238,7 @@ extern const AVInputFormat ff_ivr_demuxer; extern const AVInputFormat ff_jacosub_demuxer; extern const FFOutputFormat ff_jacosub_muxer; extern const AVInputFormat ff_jv_demuxer; +extern const AVInputFormat ff_jpegxl_anim_demuxer; extern const AVInputFormat ff_kux_demuxer; extern const AVInputFormat ff_kvag_demuxer; extern const FFOutputFormat ff_kvag_muxer; diff --git a/libavformat/img2dec.c b/libavformat/img2dec.c index c037b6aa88..b986d3a502 100644 --- a/libavformat/img2dec.c +++ b/libavformat/img2dec.c @@ -850,7 +850,7 @@ static int jpegxl_probe(const AVProbeData *p) if (AV_RL16(b) != FF_JPEGXL_CODESTREAM_SIGNATURE_LE) return 0; #if CONFIG_IMAGE_JPEGXL_PIPE_DEMUXER - if (ff_jpegxl_verify_codestream_header(p->buf, p->buf_size) >= 0) + if (ff_jpegxl_verify_codestream_header(p->buf, p->buf_size, 1) >= 0) return AVPROBE_SCORE_MAX - 2; #endif return 0; diff --git a/libavformat/jpegxl_anim_dec.c b/libavformat/jpegxl_anim_dec.c new file mode 100644 index 0000000000..6ea6c46d8f --- /dev/null +++ b/libavformat/jpegxl_anim_dec.c @@ -0,0 +1,266 @@ +/* + * Animated JPEG XL Demuxer + * Copyright (c) 2023 Leo Izen (thebombzen) + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * Animated JPEG XL Demuxer + * @see ISO/IEC 18181-1 and 18181-2 + */ + +#include +#include + +#define BITSTREAM_READER_LE +#include "libavcodec/get_bits.h" + +#include "libavutil/intreadwrite.h" +#include "libavutil/opt.h" + +#include "avformat.h" +#include "internal.h" +#include "jpegxl_probe.h" + +typedef struct JXLAnimDemuxContext { + AVBufferRef *initial; +} JXLAnimDemuxContext; + +/* + * copies as much of the codestream into the buffer as possible + * pass a shorter buflen to request less + * returns the number of bytes consumed from input, may be greater than input_len + * if the input doesn't end on an ISOBMFF-box boundary + */ +static int jpegxl_collect_codestream_header(const uint8_t *input_buffer, int input_len, uint8_t *buffer, int buflen, int *copied) { + const uint8_t *b = input_buffer; + *copied = 0; + + while (1) { + uint64_t size; + uint32_t tag; + int head_size = 8; + + if (b - input_buffer >= input_len - 16) + break; + + size = AV_RB32(b); + b += 4; + if (size == 1) { + size = AV_RB64(b); + b += 8; + head_size = 16; + } + /* invalid ISOBMFF size */ + if (size > 0 && size <= head_size) + return AVERROR_INVALIDDATA; + if (size > 0) + size -= head_size; + + tag = AV_RL32(b); + b += 4; + if (tag == MKTAG('j', 'x', 'l', 'p')) { + b += 4; + size -= 4; + } + + if (tag == MKTAG('j', 'x', 'l', 'c') || tag == MKTAG('j', 'x', 'l', 'p')) { + /* + * size = 0 means "until EOF". this is legal but uncommon + * here we just set it to the remaining size of the probe buffer + * which at this point should always be nonnegative + */ + if (size == 0 || size > input_len - (b - input_buffer)) + size = input_len - (b - input_buffer); + + if (size > buflen - *copied) + size = buflen - *copied; + /* + * arbitrary chunking of the payload makes this memcpy hard to avoid + * in practice this will only be performed one or two times at most + */ + memcpy(buffer + *copied, b, size); + *copied += size; + } + b += size; + if (b >= input_buffer + input_len || *copied >= buflen) + break; + } + + return b - input_buffer; +} + +static int jpegxl_anim_probe(const AVProbeData *p) +{ + uint8_t buffer[4096]; + int copied; + + /* this is a raw codestream */ + if (AV_RL16(p->buf) == FF_JPEGXL_CODESTREAM_SIGNATURE_LE) { + if (ff_jpegxl_verify_codestream_header(p->buf, p->buf_size, 1) >= 1) + return AVPROBE_SCORE_MAX; + + return 0; + } + + /* not a JPEG XL file at all */ + if (AV_RL64(p->buf) != FF_JPEGXL_CONTAINER_SIGNATURE_LE) + return 0; + + if (jpegxl_collect_codestream_header(p->buf, p->buf_size, buffer, sizeof(buffer), &copied) <= 0 || copied <= 0) + return 0; + + if (ff_jpegxl_verify_codestream_header(buffer, copied, 0) >= 1) + return AVPROBE_SCORE_MAX; + + return 0; +} + +static int jpegxl_anim_read_header(AVFormatContext *s) +{ + JXLAnimDemuxContext *ctx = s->priv_data; + AVIOContext *pb = s->pb; + AVStream *st; + int offset = 0; + uint8_t head[256]; + int headsize = 0; + int ctrl; + AVRational tb; + GetBitContext gbi, *gb = &gbi; + + uint64_t sig16 = avio_rl16(pb); + if (sig16 == FF_JPEGXL_CODESTREAM_SIGNATURE_LE) { + AV_WL16(head, sig16); + headsize = avio_read(s->pb, head + 2, sizeof(head) - 2); + if (headsize < 0) + return headsize; + headsize += 2; + ctx->initial = av_buffer_alloc(headsize); + if (!ctx->initial) + return AVERROR(ENOMEM); + memcpy(ctx->initial->data, head, headsize); + } else { + uint64_t sig64 = avio_rl64(pb); + sig64 = (sig64 << 16) | sig16; + if (sig64 != FF_JPEGXL_CONTAINER_SIGNATURE_LE) + return AVERROR_INVALIDDATA; + avio_skip(pb, 2); // first box always 12 bytes + while (1) { + int copied; + uint8_t buf[4096]; + int read = avio_read(pb, buf, sizeof(buf)); + if (read < 0) + return read; + if (!ctx->initial) { + ctx->initial = av_buffer_alloc(read + 12); + if (!ctx->initial) + return AVERROR(ENOMEM); + AV_WL64(ctx->initial->data, FF_JPEGXL_CONTAINER_SIGNATURE_LE); + AV_WL32(ctx->initial->data + 8, 0x0a870a0d); + } else { + /* this only should be happening zero or one times in practice */ + if (av_buffer_realloc(&ctx->initial, ctx->initial->size + read) < 0) + return AVERROR(ENOMEM); + } + jpegxl_collect_codestream_header(buf, read, head + headsize, sizeof(head) - headsize, &copied); + memcpy(ctx->initial->data + (ctx->initial->size - read), buf, read); + headsize += copied; + if (headsize >= sizeof(head) || read < sizeof(buf)) + break; + } + } + /* offset in bits of the animation header */ + offset = ff_jpegxl_verify_codestream_header(head, headsize, 0); + if (offset <= 0) + return AVERROR_INVALIDDATA; + if (init_get_bits8(gb, head, headsize) < 0) + return AVERROR_INVALIDDATA; + skip_bits_long(gb, offset); + + st = avformat_new_stream(s, NULL); + if (!st) + return AVERROR(ENOMEM); + + st->codecpar->codec_type = AVMEDIA_TYPE_VIDEO; + st->codecpar->codec_id = AV_CODEC_ID_JPEGXL; + ctrl = get_bits(gb, 2); + tb.den = (const uint32_t[]){100, 1000, 1, 1}[ctrl] + get_bits_long(gb, (const uint32_t[]){0, 0, 10, 30}[ctrl]); + ctrl = get_bits(gb, 2); + tb.num = (const uint32_t[]){1, 1001, 1, 1}[ctrl] + get_bits_long(gb, (const uint32_t[]){0, 0, 8, 10}[ctrl]); + avpriv_set_pts_info(st, 1, tb.num, tb.den); + + return 0; +} + +/* the decoder requires the full input file as a single packet */ +static int jpegxl_anim_read_packet(AVFormatContext *s, AVPacket *pkt) +{ + JXLAnimDemuxContext *ctx = s->priv_data; + AVIOContext *pb = s->pb; + int ret; + int64_t size; + size_t offset = 0; + + if ((size = avio_size(pb)) < 0) + return size; + + /* animated JXL this big should not exist */ + if (size > INT_MAX) + return AVERROR_INVALIDDATA; + + if (ctx->initial && size < ctx->initial->size) + size = ctx->initial->size; + + if ((ret = av_new_packet(pkt, size) < 0)) + return ret; + + if (ctx->initial) { + offset = ctx->initial->size; + memcpy(pkt->data, ctx->initial->data, offset); + av_buffer_unref(&ctx->initial); + } + + if ((ret = avio_read(pb, pkt->data + offset, size - offset)) < 0) + return ret; + + return 0; +} + +static int jpegxl_anim_close(AVFormatContext *s) +{ + JXLAnimDemuxContext *ctx = s->priv_data; + if (ctx->initial) + av_buffer_unref(&ctx->initial); + + return 0; +} + +const AVInputFormat ff_jpegxl_anim_demuxer = { + .name = "jpegxl_anim", + .long_name = NULL_IF_CONFIG_SMALL("Animated JPEG XL"), + .priv_data_size = sizeof(JXLAnimDemuxContext), + .read_probe = jpegxl_anim_probe, + .read_header = jpegxl_anim_read_header, + .read_packet = jpegxl_anim_read_packet, + .read_close = jpegxl_anim_close, + .flags_internal = FF_FMT_INIT_CLEANUP, + .flags = AVFMT_GENERIC_INDEX, + .mime_type = "image/jxl", + .extensions = "jxl", +}; diff --git a/libavformat/jpegxl_probe.c b/libavformat/jpegxl_probe.c index 3de002f004..a3845b037d 100644 --- a/libavformat/jpegxl_probe.c +++ b/libavformat/jpegxl_probe.c @@ -208,7 +208,7 @@ static void jpegxl_skip_bit_depth(GetBitContext *gb) * validate a Jpeg XL Extra Channel Info bundle * @return >= 0 upon valid, < 0 upon invalid */ -static int jpegxl_read_extra_channel_info(GetBitContext *gb) +static int jpegxl_read_extra_channel_info(GetBitContext *gb, int validate_level) { int all_default = jxl_bits(1); uint32_t type, name_len = 0; @@ -217,7 +217,7 @@ static int jpegxl_read_extra_channel_info(GetBitContext *gb) type = jxl_enum(); if (type > 63) return -1; /* enum types cannot be 64+ */ - if (type == FF_JPEGXL_CT_BLACK) + if (type == FF_JPEGXL_CT_BLACK && validate_level) return -1; jpegxl_skip_bit_depth(gb); jxl_u32(0, 3, 4, 1, 0, 0, 0, 3); /* dim-shift */ @@ -242,12 +242,12 @@ static int jpegxl_read_extra_channel_info(GetBitContext *gb) return 0; } -/* verify that a codestream header is valid */ -int ff_jpegxl_verify_codestream_header(const uint8_t *buf, int buflen) +int ff_jpegxl_verify_codestream_header(const uint8_t *buf, int buflen, int validate_level) { GetBitContext gbi, *gb = &gbi; int all_default, extra_fields = 0; int xyb_encoded = 1, have_icc_profile = 0; + int animation_offset = 0; uint32_t num_extra_channels; uint64_t extensions; int ret; @@ -259,7 +259,7 @@ int ff_jpegxl_verify_codestream_header(const uint8_t *buf, int buflen) if (jxl_bits(16) != FF_JPEGXL_CODESTREAM_SIGNATURE_LE) return -1; - if (jpegxl_read_size_header(gb) < 0) + if (jpegxl_read_size_header(gb) < 0 && validate_level) return -1; all_default = jxl_bits(1); @@ -285,6 +285,7 @@ int ff_jpegxl_verify_codestream_header(const uint8_t *buf, int buflen) /* animation header */ if (jxl_bits(1)) { + animation_offset = get_bits_count(gb); jxl_u32(100, 1000, 1, 1, 0, 0, 10, 30); jxl_u32(1, 1001, 1, 1, 0, 0, 8, 10); jxl_u32(0, 0, 0, 0, 0, 3, 16, 32); @@ -296,14 +297,14 @@ int ff_jpegxl_verify_codestream_header(const uint8_t *buf, int buflen) jpegxl_skip_bit_depth(gb); /* modular_16bit_buffers must equal 1 */ - if (!jxl_bits(1)) + if (!jxl_bits(1) && validate_level) return -1; num_extra_channels = jxl_u32(0, 1, 2, 1, 0, 0, 4, 12); - if (num_extra_channels > 4) + if (num_extra_channels > 4 && validate_level) return -1; for (uint32_t i = 0; i < num_extra_channels; i++) { - if (jpegxl_read_extra_channel_info(gb) < 0) + if (jpegxl_read_extra_channel_info(gb, validate_level) < 0) return -1; } @@ -392,5 +393,5 @@ int ff_jpegxl_verify_codestream_header(const uint8_t *buf, int buflen) if (get_bits_left(gb) < 0) return -1; - return 0; + return animation_offset; } diff --git a/libavformat/jpegxl_probe.h b/libavformat/jpegxl_probe.h index 2960e81e11..496445fbce 100644 --- a/libavformat/jpegxl_probe.h +++ b/libavformat/jpegxl_probe.h @@ -27,6 +27,11 @@ #define FF_JPEGXL_CODESTREAM_SIGNATURE_LE 0x0aff #define FF_JPEGXL_CONTAINER_SIGNATURE_LE 0x204c584a0c000000 -int ff_jpegxl_verify_codestream_header(const uint8_t *buf, int buflen); +/** + * @brief verify that a codestream header is valid + * @return Negative upon error, 0 upon verifying that the codestream is not animated, + * and 1 upon verifying that it is animated + */ +int ff_jpegxl_verify_codestream_header(const uint8_t *buf, int buflen, int validate_level); #endif /* AVFORMAT_JPEGXL_PROBE_H */ diff --git a/libavformat/version.h b/libavformat/version.h index e2634b85ae..4bde82abb4 100644 --- a/libavformat/version.h +++ b/libavformat/version.h @@ -31,7 +31,7 @@ #include "version_major.h" -#define LIBAVFORMAT_VERSION_MINOR 5 +#define LIBAVFORMAT_VERSION_MINOR 6 #define LIBAVFORMAT_VERSION_MICRO 100 #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \