From patchwork Thu Dec 21 21:22:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Zsolt_Vad=C3=A1sz?= X-Patchwork-Id: 45272 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6623:b0:194:e134:edd4 with SMTP id n35csp694907pzh; Thu, 21 Dec 2023 13:22:49 -0800 (PST) X-Google-Smtp-Source: AGHT+IEHpvxqS2LHOGkoLlcsSQ2JQuXqtOogu4kqGaxbaDXQ1RCoCGiGmYcp5jY7zovVhu1WmXEq X-Received: by 2002:a5d:5145:0:b0:336:7f03:4af with SMTP id u5-20020a5d5145000000b003367f0304afmr213490wrt.123.1703193769145; Thu, 21 Dec 2023 13:22:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703193769; cv=none; d=google.com; s=arc-20160816; b=sKnCTSyKbSyhezz0qC9BSymgA7eRcXzmPupZkLxGuNM/6wV1sgoj3Q2FoZZn1qTwxa DzGXL1Pxnyic7rE1QBNoM1jCYKeCVprysQJYuo11gHml6TjgSuyPv91WzJUlRU6KK+DV xiS+h+qfOjtHjGe+tBq1/GBJM8Y0fU0qWoQ6TmfJLfpgagWJgIu629LgXcaHrq8bMt4G 7hOSt8FzvWuRfx7htzyOhmsfjOGuYE7j9rhSmrok6EJvDyVGBURhYsnV4AHkwFxRZ/bJ UEHTt/JnIWAhA+EDbCSniMm62Z8drFj4apIWDh5kw7mHkasOA2Mxse1h1BwGIlqtRpeK Bhdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:feedback-id:message-id:to :date:delivered-to; bh=1PKStCYfFYZgUt+rtZPqpacxg8mJWrszFxWqjQxw/5w=; fh=lvOXre93Kc+LM49rLl0yJcoSym9484cIB4O1jbJgoXk=; b=xbOks0jUEsOtMKjTxjscohG2QRbrtlch6OJ2YF75QdLTBVS9x7HyazyGFvlekFnDzP OCi7uXoWnU+3RU9m3bIg31uHWybQrOJgb86zjQJtwXjcoxmbzaY5ckJilwHxlszdgIKy uy+dGO+L+7gZVZ6ou/VTnGtj5+rhLTj9VWdEIyzuFLw6N/RdeO3WBbq8MPtv3GSPyrHW POC5NK6b8jbaGYBfVotaP7bnVp5B6Az+f1621ZQrXuJBS3OyHQ7C2+wQTmz//7LFFjd8 TDg0geYbjr4aQ73vROtHRbJzS7xht+u3dltSwFYMu0g/HzF8QCMToZVU7EN2x2s2V+wK ajFQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id i19-20020a170906115300b00a2365ea58a6si1243496eja.665.2023.12.21.13.22.48; Thu, 21 Dec 2023 13:22:49 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BD0D968D146; Thu, 21 Dec 2023 23:22:44 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-40130.protonmail.ch (mail-40130.protonmail.ch [185.70.40.130]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E9F7168CEE2 for ; Thu, 21 Dec 2023 23:22:37 +0200 (EET) Date: Thu, 21 Dec 2023 21:22:23 +0000 To: FFmpeg development discussions and patches Message-ID: Feedback-ID: 28710920:user:proton MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 1/1] avcodec/libjxlenc: Add libjxl_animated encoder X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: =?utf-8?q?Zsolt_Vad=C3=A1sz_via_ffmpeg-devel?= From: =?utf-8?q?Zsolt_Vad=C3=A1sz?= Reply-To: FFmpeg development discussions and patches Cc: =?utf-8?q?Zsolt_Vad=C3=A1sz?= Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: bwAjwTCimLe/ --- configure | 1 + libavcodec/allcodecs.c | 1 + libavcodec/libjxlenc.c | 226 ++++++++++++++++++++++++++++++++++------- 3 files changed, 189 insertions(+), 39 deletions(-) diff --git a/configure b/configure index 7d2ee66000..2d27d8015a 100755 --- a/configure +++ b/configure @@ -3418,6 +3418,7 @@ libilbc_decoder_deps="libilbc" libilbc_encoder_deps="libilbc" libjxl_decoder_deps="libjxl libjxl_threads" libjxl_encoder_deps="libjxl libjxl_threads" +libjxl_anim_encoder_deps="libjxl libjxl_threads" libkvazaar_encoder_deps="libkvazaar" libmodplug_demuxer_deps="libmodplug" libmp3lame_encoder_deps="libmp3lame" diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c index b0f004e15c..22c7227946 100644 --- a/libavcodec/allcodecs.c +++ b/libavcodec/allcodecs.c @@ -784,6 +784,7 @@ extern const FFCodec ff_libilbc_encoder; extern const FFCodec ff_libilbc_decoder; extern const FFCodec ff_libjxl_decoder; extern const FFCodec ff_libjxl_encoder; +extern const FFCodec ff_libjxl_anim_encoder; extern const FFCodec ff_libmp3lame_encoder; extern const FFCodec ff_libopencore_amrnb_encoder; extern const FFCodec ff_libopencore_amrnb_decoder; diff --git a/libavcodec/libjxlenc.c b/libavcodec/libjxlenc.c index d707f3a61b..f23e86939e 100644 --- a/libavcodec/libjxlenc.c +++ b/libavcodec/libjxlenc.c @@ -34,6 +34,7 @@ #include "libavutil/opt.h" #include "libavutil/pixdesc.h" #include "libavutil/pixfmt.h" +#include "libavutil/rational.h" #include "libavutil/version.h" #include "avcodec.h" @@ -49,11 +50,18 @@ typedef struct LibJxlEncodeContext { void *runner; JxlEncoder *encoder; JxlEncoderFrameSettings *options; + JxlPixelFormat jxl_fmt; int effort; float distance; int modular; uint8_t *buffer; size_t buffer_size; + int animated; + /* Only used by libjxl_animated */ + AVRational libjxl_time_base; + int is_first_frame; + int64_t duration; + AVFrame *prev; } LibJxlEncodeContext; /** @@ -183,6 +191,8 @@ static av_cold int libjxl_encode_init(AVCodecContext *avctx) return AVERROR(ENOMEM); } + ctx->animated = 0; + return 0; } @@ -237,28 +247,19 @@ static int libjxl_populate_primaries(void *avctx, JxlColorEncoding *jxl_color, e return 0; } -/** - * Encode an entire frame. Currently animation, is not supported by - * this encoder, so this will always reinitialize a new still image - * and encode a one-frame image (for image2 and image2pipe). - */ -static int libjxl_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFrame *frame, int *got_packet) +static int libjxl_encode_init_image(AVCodecContext *avctx, const AVFrame *frame) { LibJxlEncodeContext *ctx = avctx->priv_data; AVFrameSideData *sd; const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(frame->format); JxlBasicInfo info; JxlColorEncoding jxl_color; - JxlPixelFormat jxl_fmt; + JxlPixelFormat *jxl_fmt = &ctx->jxl_fmt; int bits_per_sample; #if JPEGXL_NUMERIC_VERSION >= JPEGXL_COMPUTE_NUMERIC_VERSION(0, 8, 0) JxlBitDepth jxl_bit_depth; #endif - JxlEncoderStatus jret; int ret; - size_t available = ctx->buffer_size; - size_t bytes_written = 0; - uint8_t *next_out = ctx->buffer; ret = libjxl_init_jxl_encoder(avctx); if (ret) { @@ -268,23 +269,30 @@ static int libjxl_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFra /* populate the basic info settings */ JxlEncoderInitBasicInfo(&info); - jxl_fmt.num_channels = pix_desc->nb_components; + jxl_fmt->num_channels = pix_desc->nb_components; info.xsize = frame->width; info.ysize = frame->height; - info.num_extra_channels = (jxl_fmt.num_channels + 1) % 2; - info.num_color_channels = jxl_fmt.num_channels - info.num_extra_channels; - bits_per_sample = av_get_bits_per_pixel(pix_desc) / jxl_fmt.num_channels; + info.num_extra_channels = (jxl_fmt->num_channels + 1) % 2; + info.num_color_channels = jxl_fmt->num_channels - info.num_extra_channels; + bits_per_sample = av_get_bits_per_pixel(pix_desc) / jxl_fmt->num_channels; info.bits_per_sample = avctx->bits_per_raw_sample > 0 && !(pix_desc->flags & AV_PIX_FMT_FLAG_FLOAT) ? avctx->bits_per_raw_sample : bits_per_sample; info.alpha_bits = (info.num_extra_channels > 0) * info.bits_per_sample; if (pix_desc->flags & AV_PIX_FMT_FLAG_FLOAT) { info.exponent_bits_per_sample = info.bits_per_sample > 16 ? 8 : 5; info.alpha_exponent_bits = info.alpha_bits ? info.exponent_bits_per_sample : 0; - jxl_fmt.data_type = info.bits_per_sample > 16 ? JXL_TYPE_FLOAT : JXL_TYPE_FLOAT16; + jxl_fmt->data_type = info.bits_per_sample > 16 ? JXL_TYPE_FLOAT : JXL_TYPE_FLOAT16; } else { info.exponent_bits_per_sample = 0; info.alpha_exponent_bits = 0; - jxl_fmt.data_type = info.bits_per_sample <= 8 ? JXL_TYPE_UINT8 : JXL_TYPE_UINT16; + jxl_fmt->data_type = info.bits_per_sample <= 8 ? JXL_TYPE_UINT8 : JXL_TYPE_UINT16; + } + if(ctx->animated) { + info.have_animation = 1; + info.animation.have_timecodes = 0; + info.animation.num_loops = 0; + info.animation.tps_numerator = frame->time_base.den; + info.animation.tps_denominator = frame->time_base.num; } #if JPEGXL_NUMERIC_VERSION >= JPEGXL_COMPUTE_NUMERIC_VERSION(0, 8, 0) @@ -382,19 +390,19 @@ static int libjxl_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFra av_log(avctx, AV_LOG_WARNING, "Could not increase codestream level\n"); } - jxl_fmt.endianness = JXL_NATIVE_ENDIAN; - jxl_fmt.align = frame->linesize[0]; + jxl_fmt->endianness = JXL_NATIVE_ENDIAN; + jxl_fmt->align = frame->linesize[0]; - if (JxlEncoderAddImageFrame(ctx->options, &jxl_fmt, frame->data[0], jxl_fmt.align * info.ysize) != JXL_ENC_SUCCESS) { - av_log(avctx, AV_LOG_ERROR, "Failed to add Image Frame\n"); - return AVERROR_EXTERNAL; - } - /* - * Run this after the last frame in the image has been passed. - * TODO support animation - */ - JxlEncoderCloseInput(ctx->encoder); + return 0; +} + +static int libjxl_encode_process_output(AVCodecContext *avctx, size_t *bytes_written) +{ + LibJxlEncodeContext *ctx = avctx->priv_data; + JxlEncoderStatus jret; + size_t available = ctx->buffer_size; + uint8_t *next_out = ctx->buffer; while (1) { jret = JxlEncoderProcessOutput(ctx->encoder, &next_out, &available); @@ -402,7 +410,7 @@ static int libjxl_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFra av_log(avctx, AV_LOG_ERROR, "Unspecified libjxl error occurred\n"); return AVERROR_EXTERNAL; } - bytes_written = ctx->buffer_size - available; + *bytes_written = ctx->buffer_size - available; /* all data passed has been encoded */ if (jret == JXL_ENC_SUCCESS) break; @@ -419,14 +427,44 @@ static int libjxl_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFra return AVERROR(ENOMEM); ctx->buffer = temp; ctx->buffer_size = new_size; - next_out = ctx->buffer + bytes_written; - available = new_size - bytes_written; + next_out = ctx->buffer + *bytes_written; + available = new_size - *bytes_written; continue; } av_log(avctx, AV_LOG_ERROR, "Bad libjxl event: %d\n", jret); return AVERROR_EXTERNAL; } + return 0; +} + +/** + * Encode an entire frame. this will always reinitialize a new still image + * and encode a one-frame image (for image2 and image2pipe). + */ +static int libjxl_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFrame *frame, int *got_packet) +{ + LibJxlEncodeContext *ctx = avctx->priv_data; + JxlPixelFormat *jxl_fmt = &ctx->jxl_fmt; + int ret; + size_t bytes_written = 0; + + if((ret = libjxl_encode_init_image(avctx, frame)) < 0) + return ret; + + if (JxlEncoderAddImageFrame(ctx->options, + jxl_fmt, + frame->data[0], + jxl_fmt->align * frame->height) != JXL_ENC_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Failed to add Image Frame\n"); + return AVERROR_EXTERNAL; + } + + JxlEncoderCloseInput(ctx->encoder); + + if((ret = libjxl_encode_process_output(avctx, &bytes_written)) < 0) + return ret; + ret = ff_get_encode_buffer(avctx, pkt, bytes_written, 0); if (ret < 0) return ret; @@ -437,6 +475,76 @@ static int libjxl_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFra return 0; } +static int libjxl_anim_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFrame *frame, int *got_packet) +{ + LibJxlEncodeContext *ctx = avctx->priv_data; + int ret = 0; + + if(ctx->is_first_frame) { + if((ret = libjxl_encode_init_image(avctx, frame)) < 0) + return ret; + ctx->is_first_frame = 0; + ctx->prev = av_frame_clone(frame); + *got_packet = 0; + return AVERROR(EAGAIN); + } + + if(!ctx->prev && !frame) { + *got_packet = 0; + } else { + JxlPixelFormat *jxl_fmt = &ctx->jxl_fmt; + JxlFrameHeader frame_header; + size_t bytes_written = 0; + + if(frame && !ctx->duration) + ctx->duration = av_rescale_q(frame->pts - ctx->prev->pts, + avctx->time_base, + ctx->libjxl_time_base); + + JxlEncoderInitFrameHeader(&frame_header); + frame_header.duration = ctx->prev->duration; + pkt->duration = ctx->duration; + pkt->pts = av_rescale_q(ctx->prev->pts, ctx->prev->time_base, avctx->time_base); + pkt->dts = pkt->pts; + + if(JxlEncoderSetFrameHeader(ctx->options, &frame_header) != JXL_ENC_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Failed to set frame header\n"); + return AVERROR_EXTERNAL; + } + if (JxlEncoderAddImageFrame(ctx->options, + jxl_fmt, + ctx->prev->data[0], + jxl_fmt->align * ctx->prev->height) != JXL_ENC_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Failed to add Image Frame\n"); + return AVERROR_EXTERNAL; + } + + /* + * Run this after the last frame in the image has been passed. + */ + if(!frame) + JxlEncoderCloseInput(ctx->encoder); + + if((ret = libjxl_encode_process_output(avctx, &bytes_written)) < 0) + return ret; + + ret = ff_get_encode_buffer(avctx, pkt, bytes_written, 0); + if (ret < 0) + return ret; + + memcpy(pkt->data, ctx->buffer, bytes_written); + *got_packet = 1; + + if(frame) { + av_frame_replace(ctx->prev, frame); + } else { + av_frame_free(&ctx->prev); + } + } + + return 0; +} + static av_cold int libjxl_encode_close(AVCodecContext *avctx) { LibJxlEncodeContext *ctx = avctx->priv_data; @@ -458,6 +566,24 @@ static av_cold int libjxl_encode_close(AVCodecContext *avctx) return 0; } +static av_cold int libjxl_anim_encode_init(AVCodecContext *avctx) +{ + int ret; + LibJxlEncodeContext *ctx = avctx->priv_data; + if((ret = libjxl_encode_init(avctx)) < 0) + return ret; + if(JxlEncoderUseContainer(ctx->encoder, 0) != JXL_ENC_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Failed to disable container format\n"); + return AVERROR_EXTERNAL; + } + ctx->animated = 1; + ctx->is_first_frame = 1; + ctx->libjxl_time_base = av_inv_q(avctx->time_base); + ctx->duration = 0; + ret = libjxl_init_jxl_encoder(avctx); + return ret; +} + #define OFFSET(x) offsetof(LibJxlEncodeContext, x) #define VE AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM @@ -476,6 +602,15 @@ static const AVClass libjxl_encode_class = { .version = LIBAVUTIL_VERSION_INT, }; +static const enum AVPixelFormat libjxl_pix_fmts[] = { + AV_PIX_FMT_RGB24, AV_PIX_FMT_RGBA, + AV_PIX_FMT_RGB48, AV_PIX_FMT_RGBA64, + AV_PIX_FMT_GRAY8, AV_PIX_FMT_YA8, + AV_PIX_FMT_GRAY16, AV_PIX_FMT_YA16, + AV_PIX_FMT_GRAYF32, + AV_PIX_FMT_NONE +}; + const FFCodec ff_libjxl_encoder = { .p.name = "libjxl", CODEC_LONG_NAME("libjxl JPEG XL"), @@ -490,14 +625,27 @@ const FFCodec ff_libjxl_encoder = { .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE | FF_CODEC_CAP_AUTO_THREADS | FF_CODEC_CAP_INIT_CLEANUP | FF_CODEC_CAP_ICC_PROFILES, - .p.pix_fmts = (const enum AVPixelFormat[]) { - AV_PIX_FMT_RGB24, AV_PIX_FMT_RGBA, - AV_PIX_FMT_RGB48, AV_PIX_FMT_RGBA64, - AV_PIX_FMT_GRAY8, AV_PIX_FMT_YA8, - AV_PIX_FMT_GRAY16, AV_PIX_FMT_YA16, - AV_PIX_FMT_GRAYF32, - AV_PIX_FMT_NONE - }, + .p.pix_fmts = libjxl_pix_fmts, + .p.priv_class = &libjxl_encode_class, + .p.wrapper_name = "libjxl", +}; + +const FFCodec ff_libjxl_anim_encoder = { + .p.name = "libjxl_anim", + CODEC_LONG_NAME("libjxl Animated JPEG XL"), + .p.type = AVMEDIA_TYPE_VIDEO, + .p.id = AV_CODEC_ID_JPEGXL, + .priv_data_size = sizeof(LibJxlEncodeContext), + .init = libjxl_anim_encode_init, + FF_CODEC_ENCODE_CB(libjxl_anim_encode_frame), + .close = libjxl_encode_close, + .p.capabilities = AV_CODEC_CAP_OTHER_THREADS | + AV_CODEC_CAP_ENCODER_REORDERED_OPAQUE | + AV_CODEC_CAP_DELAY, + .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE | + FF_CODEC_CAP_AUTO_THREADS | FF_CODEC_CAP_INIT_CLEANUP | + FF_CODEC_CAP_ICC_PROFILES | FF_CODEC_CAP_EOF_FLUSH, + .p.pix_fmts = libjxl_pix_fmts, .p.priv_class = &libjxl_encode_class, .p.wrapper_name = "libjxl", };