From patchwork Sun Mar 5 11:17:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 40583 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:d046:b0:cd:afd7:272c with SMTP id hv6csp2117982pzb; Sun, 5 Mar 2023 03:17:40 -0800 (PST) X-Google-Smtp-Source: AK7set/RUN1oZuDVu263ngERXsrgR3MkwT1T1dnh2lmCfL1pl/CA4bXPBZ+fhm5Zc3awm0PbHas3 X-Received: by 2002:a17:906:4716:b0:89c:d072:e33e with SMTP id y22-20020a170906471600b0089cd072e33emr7004769ejq.49.1678015060572; Sun, 05 Mar 2023 03:17:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678015060; cv=none; d=google.com; s=arc-20160816; b=OrO3CYMnKq2NT5/oe/1JRhqDUqZrrVUrccffvNvYBMVL33aWTsrHX80AmWJW2WnbDy 0vRnfg/dAt7D9XQ1DNF5OzNUzodYExqkHR75p02RyhYBMv/bAKpCkJh4bnrMyp4poAZM /dHBxlBJ97hOlvaWkSUISoCQ3FNBTMHgJX7uYwq47L9XluL43e1sG0btwCi3AIsYcKm0 haXG50MfwtGJIGaV4yEw16xVORzuMBGKnUeLN262XE1lbt8fTfrWI1gyMVZQ47zvHkgh ZdtkZiXMAR8JtiUGQrtlRiDzOPQYtf/Gx0ejyeTCqFx/jQ5VMquzDq4MGhNy8JnPXsgE Wimg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=iqgHHV4lGBknl+/xcbS1FHsQ6GNyEgxZVytZdsBtHfQ=; b=UDtVJambfTirdhhZSB0Fox70AhqHUT7UuUyEl8muZlLKpaR8Fvg6kjRnd/AyLzBNsw E1/LE9De/bKkZxj6wv7Iay+Z73nKL45I1cikfgCkkUNnsvqxIoZzqIc3fFxDDs0hszsF /GjGNzzji0y6LPaH4EKMvtWec8lhoQa6XmmdDhkAbOuaG42OFq0p3J9+gKEbOG2lQ0P9 RdQU7d89RoEARNpA0YcVpGKSlMioxwkSWpQafSiLYTOPuI4YeKAXCpyYmsCAu/JBrxKL OsFV0/+EenyxnCDSpVgtUMPHQCSKR1ExBa5nUoXSR/18UrDXiCb3BOjEc5G+MMy2bQPU Iaig== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=Xf67UHRx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id m26-20020a170906849a00b008beb8136584si7327812ejx.491.2023.03.05.03.17.38; Sun, 05 Mar 2023 03:17:40 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=Xf67UHRx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C6C3A68B187; Sun, 5 Mar 2023 13:17:33 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf1-f50.google.com (mail-lf1-f50.google.com [209.85.167.50]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0732B68AA32 for ; Sun, 5 Mar 2023 13:17:28 +0200 (EET) Received: by mail-lf1-f50.google.com with SMTP id r27so9107447lfe.10 for ; Sun, 05 Mar 2023 03:17:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678015047; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=cnOg6PNHHn3Nsz4QXbyeyfFZ+k4HQwZJg98Py5iGkg0=; b=Xf67UHRx+8x8iPDjVIbqkAtIDT4jrL4msW27PypHN/iBnUmt5a3K3WcIa3/x6QI23M 8MyVK2Y+vHWKYtFdFFyb6O7CVEI2f7EWN8SUltXX3PzL4YnjhDTsAphD9jf9fIH9TQA7 lMo5XYs9vE5GqtxAaiGPkmuF/XdskNPVtYe1HFDWzZey1KnS9a3J9qDj2mfh7lMFszd7 1Bg0J0m7mW5eRLIZLzZLT8945INGb9VTTgZ26zBHzj1A+qehRQMJElNLy1OHpcs3pwKp dhgGUI4E8o4e6L9CQjkNgaEQOiP9uxpUn30PoRRFJL4LBtN15rhjD4ObRHla5Z01qWox iSEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678015047; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cnOg6PNHHn3Nsz4QXbyeyfFZ+k4HQwZJg98Py5iGkg0=; b=8Ha7VMPAYIK4m5OdNjpmZl9bxvZ+YbyM4xrKNmuBVV5XHFpmP1/G4dwUGNwehirjoE PBomderIsyIPG8s1j6AICVvAFGMv2weYjz8Iq94R8ANUkdqGTYPA+UK5OiQbs5jcRQAD HyOgZjjrBcdARlRk3EQ9G5jIPIoMdEi72B+J/KnjxN1bLTTASPP+Cu4H87h4gXa6LETp exBrqIwh+IcwDMgNpie91nXXlP/3sCyjMTWr23tyYtH4vUx0f8FURxNMuqtCGACI46z7 vF4XTcdT2D/ranCfACIYqtHfoUzELxv2kcq47/NKtqQmLI7kxKRNdgNnhz0PZT2xUu+T d7FQ== X-Gm-Message-State: AO0yUKUdyzI3NFr0wsAv7pZTH13RkNxutlTtf0bi/rJ+P021Vwu1GxVf pp67pUBCUMbaP0hnCdO6yWPav43mqdY= X-Received: by 2002:ac2:53a8:0:b0:4dd:9a38:1b9c with SMTP id j8-20020ac253a8000000b004dd9a381b9cmr1945228lfh.20.1678015046871; Sun, 05 Mar 2023 03:17:26 -0800 (PST) Received: from localhost.localdomain (91-153-198-187.elisa-laajakaista.fi. [91.153.198.187]) by smtp.gmail.com with ESMTPSA id l26-20020a19c21a000000b004dd7f285614sm1160420lfc.233.2023.03.05.03.17.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Mar 2023 03:17:26 -0800 (PST) From: =?utf-8?q?Jan_Ekstr=C3=B6m?= To: ffmpeg-devel@ffmpeg.org Date: Sun, 5 Mar 2023 13:17:20 +0200 Message-Id: <20230305111720.6019-1-jeebjp@gmail.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] ffmpeg: pass encoder init AVFrame side data to output AVStream X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: eyiu5jJbsNze This enables passing through various side data during encoding, which is not yet in AVCodecContext/AVCodecParameters, but is read from AVStream side data in muxers. Additionally, add a FATE test that demonstrates PNG->J2K MP4 transcoding with the ICC profile getting passed through. --- fftools/ffmpeg.c | 6 ++ fftools/ffmpeg.h | 1 + fftools/ffmpeg_mux.c | 47 +++++++++++++ tests/fate/ffmpeg.mak | 4 ++ tests/ref/fate/ffmpeg-side-data-to-avstreams | 72 ++++++++++++++++++++ 5 files changed, 130 insertions(+) create mode 100644 tests/ref/fate/ffmpeg-side-data-to-avstreams diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index d721a5e721..134258b825 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -3113,6 +3113,12 @@ static int init_output_stream_encode(OutputStream *ost, AVFrame *frame) av_pix_fmt_desc_get(enc_ctx->pix_fmt)->comp[0].depth); if (frame) { + if (!(ost->side_data_frame = av_frame_alloc())) + return AVERROR(ENOMEM); + + if ((ret = av_frame_copy_props(ost->side_data_frame, frame)) < 0) + return ret; + enc_ctx->color_range = frame->color_range; enc_ctx->color_primaries = frame->color_primaries; enc_ctx->color_trc = frame->color_trc; diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index 4d4433f5ba..21dd0f3a9e 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -593,6 +593,7 @@ typedef struct OutputStream { AVFrame *filtered_frame; AVFrame *last_frame; AVFrame *sq_frame; + AVFrame *side_data_frame; AVPacket *pkt; int64_t last_dropped; int64_t last_nb0_frames[3]; diff --git a/fftools/ffmpeg_mux.c b/fftools/ffmpeg_mux.c index cf58051949..13da9ab8da 100644 --- a/fftools/ffmpeg_mux.c +++ b/fftools/ffmpeg_mux.c @@ -580,6 +580,49 @@ static int bsf_init(MuxStream *ms) return 0; } +static int avframe_side_data_to_avstream(AVStream *stream, const AVFrame *frame) +{ + static const struct sd_mapping { + enum AVPacketSideDataType packet; + enum AVFrameSideDataType frame; + } sd_list[] = { + { AV_PKT_DATA_REPLAYGAIN , AV_FRAME_DATA_REPLAYGAIN }, + { AV_PKT_DATA_DISPLAYMATRIX, AV_FRAME_DATA_DISPLAYMATRIX }, + { AV_PKT_DATA_SPHERICAL, AV_FRAME_DATA_SPHERICAL }, + { AV_PKT_DATA_STEREO3D, AV_FRAME_DATA_STEREO3D }, + { AV_PKT_DATA_AUDIO_SERVICE_TYPE, AV_FRAME_DATA_AUDIO_SERVICE_TYPE }, + { AV_PKT_DATA_MASTERING_DISPLAY_METADATA, AV_FRAME_DATA_MASTERING_DISPLAY_METADATA }, + { AV_PKT_DATA_CONTENT_LIGHT_LEVEL, AV_FRAME_DATA_CONTENT_LIGHT_LEVEL }, + { AV_PKT_DATA_A53_CC, AV_FRAME_DATA_A53_CC }, + { AV_PKT_DATA_ICC_PROFILE, AV_FRAME_DATA_ICC_PROFILE }, + { AV_PKT_DATA_S12M_TIMECODE, AV_FRAME_DATA_S12M_TIMECODE }, + { AV_PKT_DATA_DYNAMIC_HDR10_PLUS, AV_FRAME_DATA_DYNAMIC_HDR_PLUS }, + }; + + if (!frame || !frame->nb_side_data) + return 0; + + if (!stream) + return AVERROR(EINVAL); + + for (int i = 0; i < FF_ARRAY_ELEMS(sd_list); i++) { + const struct sd_mapping mapping = sd_list[i]; + uint8_t *packet_sd = NULL; + AVFrameSideData *sd = av_frame_get_side_data(frame, mapping.frame); + if (!sd) + continue; + + packet_sd = av_stream_new_side_data(stream, mapping.packet, + sd->size); + if (!packet_sd) + return AVERROR(ENOMEM); + + memcpy(packet_sd, sd->data, sd->size); + } + + return 0; +} + int of_stream_init(OutputFile *of, OutputStream *ost) { Muxer *mux = mux_from_of(of); @@ -589,6 +632,9 @@ int of_stream_init(OutputFile *of, OutputStream *ost) if (ost->sq_idx_mux >= 0) sq_set_tb(mux->sq_mux, ost->sq_idx_mux, ost->mux_timebase); + if ((ret = avframe_side_data_to_avstream(ost->st, ost->side_data_frame)) < 0) + return ret; + /* initialize bitstream filters for the output stream * needs to be done here, because the codec id for streamcopy is not * known until now */ @@ -666,6 +712,7 @@ static void ost_free(OutputStream **post) av_frame_free(&ost->filtered_frame); av_frame_free(&ost->sq_frame); av_frame_free(&ost->last_frame); + av_frame_free(&ost->side_data_frame); av_packet_free(&ost->pkt); av_dict_free(&ost->encoder_opts); diff --git a/tests/fate/ffmpeg.mak b/tests/fate/ffmpeg.mak index 0f33c2a0ed..3d6ef2a8fb 100644 --- a/tests/fate/ffmpeg.mak +++ b/tests/fate/ffmpeg.mak @@ -224,3 +224,7 @@ FATE_TIME_BASE-$(call PARSERDEMDEC, MPEGVIDEO, MPEGPS, MPEG2VIDEO, MPEGVIDEO_DEM fate-time_base: CMD = md5 -i $(TARGET_SAMPLES)/mpeg2/dvd_single_frame.vob -an -sn -c:v copy -r 25 -time_base 1001:30000 -fflags +bitexact -f mxf FATE_SAMPLES_FFMPEG-yes += $(FATE_TIME_BASE-yes) + +FATE_SAMPLES_FFMPEG-$(call TRANSCODE, JPEG2000 PNG, MP4 IMAGE_PNG_PIPE) += fate-ffmpeg-side-data-to-avstreams +fate-ffmpeg-side-data-to-avstreams: CMD = transcode png_pipe $(TARGET_SAMPLES)/png1/lena-int_rgb24.png\ + mp4 "-c jpeg2000" "" "-show_streams" diff --git a/tests/ref/fate/ffmpeg-side-data-to-avstreams b/tests/ref/fate/ffmpeg-side-data-to-avstreams new file mode 100644 index 0000000000..7681861c94 --- /dev/null +++ b/tests/ref/fate/ffmpeg-side-data-to-avstreams @@ -0,0 +1,72 @@ +95be5b4c97ba8989fb738df35cb8b7f2 *tests/data/fate/ffmpeg-side-data-to-avstreams.mp4 +36472 tests/data/fate/ffmpeg-side-data-to-avstreams.mp4 +#tb 0: 1/25 +#media_type 0: video +#codec_id 0: rawvideo +#dimensions 0: 128x128 +#sar 0: 1/1 +0, 0, 0, 1, 49152, 0xf6fe3b30 +[STREAM] +index=0 +codec_name=jpeg2000 +profile=0 +codec_type=video +codec_tag_string=mp4v +codec_tag=0x7634706d +width=128 +height=128 +coded_width=128 +coded_height=128 +closed_captions=0 +film_grain=0 +has_b_frames=0 +sample_aspect_ratio=1:1 +display_aspect_ratio=1:1 +pix_fmt=rgb24 +level=-99 +color_range=unknown +color_space=unknown +color_transfer=unknown +color_primaries=unknown +chroma_location=unspecified +field_order=bt +refs=1 +id=0x1 +r_frame_rate=25/1 +avg_frame_rate=25/1 +time_base=1/12800 +start_pts=0 +start_time=0.000000 +duration_ts=512 +duration=0.040000 +bit_rate=6503600 +max_bit_rate=N/A +bits_per_raw_sample=8 +nb_frames=1 +nb_read_frames=N/A +nb_read_packets=N/A +DISPOSITION:default=1 +DISPOSITION:dub=0 +DISPOSITION:original=0 +DISPOSITION:comment=0 +DISPOSITION:lyrics=0 +DISPOSITION:karaoke=0 +DISPOSITION:forced=0 +DISPOSITION:hearing_impaired=0 +DISPOSITION:visual_impaired=0 +DISPOSITION:clean_effects=0 +DISPOSITION:attached_pic=0 +DISPOSITION:timed_thumbnails=0 +DISPOSITION:captions=0 +DISPOSITION:descriptions=0 +DISPOSITION:metadata=0 +DISPOSITION:dependent=0 +DISPOSITION:still_image=0 +TAG:language=und +TAG:handler_name=VideoHandler +TAG:vendor_id=[0][0][0][0] +TAG:encoder=Lavc jpeg2000 +[SIDE_DATA] +side_data_type=ICC Profile +[/SIDE_DATA] +[/STREAM]