From patchwork Mon Jan 3 00:27:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 33022 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp15720097iog; Sun, 2 Jan 2022 16:28:34 -0800 (PST) X-Google-Smtp-Source: ABdhPJxCdZl72YQNi8kWlQ0blRRcJejo5o7ozFvwQCcJnM2SwoaE2Rvn5zxutXaQFqhDz51ULeuz X-Received: by 2002:a17:906:ccd0:: with SMTP id ot16mr34505485ejb.396.1641169714396; Sun, 02 Jan 2022 16:28:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1641169714; cv=none; d=google.com; s=arc-20160816; b=gHlctzZ4pGcDQTdkJ9zRIC1xOhQBj00XrieS1xrc5Z0jmKKN7WAhvgcFnV3PbYhgDs cjFfwHAtLCZ+Ynml5ctxzz5RgBhPZIxRsPUUt/rNVC99FaKXvo18YLJ1qK81vbg4lOfP gsJw3XjtwancQDwGJjGRmdnWDXV08h8y1lBSxbhQuToxrqC3ENQAp8ELolqR4K/glF7g Qo+HdT3YVcljuYbqv0q59woj/d7ALUj5fPnb9FvE9wRiEo/tY+pK+A5oFFdo7176EQwP ZvcWV4tbsFlIbiAlNliMkjKHhJxV06BK5QO0v+m040WWgUIdD8Mw9BGDl89WliXcVD65 XFcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=b3ofbiDxSdQ6Nbe/adxQ5e3kTclkssAbDNsfNaR0kfc=; b=bXQ7wWth5D6Y+lmRWwkLw5QuuKHYYm4KW8M/OPUpczykZwzCKRa734S5J4EPzIj4T2 jyXJkAf9L5bsRbm8tvGi9Y6rLQreDp73vvuQ13HN/II9mb5pFrzbW0xIpLHW6JSfEHsZ 83fGF/DiXZbqYbWvKJkttImX6Mmw8YdibXqiNtvt3gkGV2f35cL60JpwOUP4OVi4BbNg DiXm0/ssJSSqh4WBqjPfLbf5lg88r9EHGNrNyeMKyRb30gDP/gnqJyqm3tcP6mQiw1o1 ak+lLJ5w1Par7Wb4DuOr+hJVuIJ8fcF7u2pKN6Z4oqRiOrvqv61kGdbrzTnxVH3bAdwF 3Fyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=Gwn2tHFM; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s1si14515255edd.28.2022.01.02.16.28.33; Sun, 02 Jan 2022 16:28:34 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=Gwn2tHFM; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E61E668B1B1; Mon, 3 Jan 2022 02:27:41 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 440A768B160 for ; Mon, 3 Jan 2022 02:27:35 +0200 (EET) Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id 1C2BD4949D; Mon, 3 Jan 2022 01:27:29 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1641169649; bh=wEFNXTHKgp1wX//1G2PKaCpUQ1k2a9NpB6j9yiHADaM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Gwn2tHFM7PlwcEk7L3rMS3lUDMOmKlnNyftdEyGtB6e1FjUh++Y7R5RuusB74X3Lm 1o75mW6PEmBKSi+AOavPCzKKuKDt1BoRpI6eN0+4eUgQlf7CMF9ZIGXoBxYvzu4xAt TJayaBPO/BiP4UG668ilBCoc7IpTi8CNF75+D21E= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Mon, 3 Jan 2022 01:27:27 +0100 Message-Id: <20220103002727.38840-6-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220103002727.38840-1-ffmpeg@haasn.xyz> References: <20220103002727.38840-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v10 6/6] lavc/hevcdec: Parse DOVI RPU NALs X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: WIpVgOS607Or From: Niklas Haas And expose the parsed values as frame side data. Update FATE results to match. It's worth documenting that this relies on the dovi configuration record being present on the first AVPacket fed to the decoder, which in practice is the case if if the API user has called something like av_format_inject_global_side_data, which is unfortunately not the default. This commit is not the time and place to change that behavior, though. Signed-off-by: Niklas Haas --- configure | 2 +- libavcodec/hevcdec.c | 37 ++++-- libavcodec/hevcdec.h | 2 + tests/ref/fate/hevc-dv-rpu | 224 +++++++++++++++++++++++++++++++++++++ 4 files changed, 256 insertions(+), 9 deletions(-) diff --git a/configure b/configure index 8303e1329e..3f71af77db 100755 --- a/configure +++ b/configure @@ -2826,7 +2826,7 @@ h264_decoder_suggest="error_resilience" hap_decoder_select="snappy texturedsp" hap_encoder_deps="libsnappy" hap_encoder_select="texturedspenc" -hevc_decoder_select="atsc_a53 bswapdsp cabac golomb hevcparse videodsp" +hevc_decoder_select="atsc_a53 bswapdsp cabac dovi_rpu golomb hevcparse videodsp" huffyuv_decoder_select="bswapdsp huffyuvdsp llviddsp" huffyuv_encoder_select="bswapdsp huffman huffyuvencdsp llvidencdsp" hymt_decoder_select="huffyuv_decoder" diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index 3aa70e2245..8d7a4f7147 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -2723,6 +2723,7 @@ error: static int set_side_data(HEVCContext *s) { AVFrame *out = s->ref->frame; + int ret; if (s->sei.frame_packing.present && s->sei.frame_packing.arrangement_type >= 3 && @@ -2977,6 +2978,9 @@ static int set_side_data(HEVCContext *s) s->rpu_buf = NULL; } + if ((ret = ff_dovi_attach_side_data(&s->dovi_ctx, out)) < 0) + return ret; + return 0; } @@ -3308,16 +3312,23 @@ static int decode_nal_units(HEVCContext *s, const uint8_t *buf, int length) if (s->pkt.nb_nals > 1 && s->pkt.nals[s->pkt.nb_nals - 1].type == HEVC_NAL_UNSPEC62 && s->pkt.nals[s->pkt.nb_nals - 1].size > 2 && !s->pkt.nals[s->pkt.nb_nals - 1].nuh_layer_id && !s->pkt.nals[s->pkt.nb_nals - 1].temporal_id) { + H2645NAL *nal = &s->pkt.nals[s->pkt.nb_nals - 1]; if (s->rpu_buf) { av_buffer_unref(&s->rpu_buf); av_log(s->avctx, AV_LOG_WARNING, "Multiple Dolby Vision RPUs found in one AU. Skipping previous.\n"); } - s->rpu_buf = av_buffer_alloc(s->pkt.nals[s->pkt.nb_nals - 1].raw_size - 2); + s->rpu_buf = av_buffer_alloc(nal->raw_size - 2); if (!s->rpu_buf) return AVERROR(ENOMEM); + memcpy(s->rpu_buf->data, nal->raw_data + 2, nal->raw_size - 2); - memcpy(s->rpu_buf->data, s->pkt.nals[s->pkt.nb_nals - 1].raw_data + 2, s->pkt.nals[s->pkt.nb_nals - 1].raw_size - 2); + ret = ff_dovi_rpu_parse(&s->dovi_ctx, nal->data + 2, nal->size - 2); + if (ret < 0) { + av_buffer_unref(&s->rpu_buf); + av_log(s->avctx, AV_LOG_WARNING, "Error parsing DOVI NAL unit.\n"); + /* ignore */ + } } /* decode the NAL units */ @@ -3450,8 +3461,8 @@ static int hevc_decode_frame(AVCodecContext *avctx, void *data, int *got_output, AVPacket *avpkt) { int ret; - size_t new_extradata_size; - uint8_t *new_extradata; + uint8_t *sd; + size_t sd_size; HEVCContext *s = avctx->priv_data; if (!avpkt->size) { @@ -3463,14 +3474,17 @@ static int hevc_decode_frame(AVCodecContext *avctx, void *data, int *got_output, return 0; } - new_extradata = av_packet_get_side_data(avpkt, AV_PKT_DATA_NEW_EXTRADATA, - &new_extradata_size); - if (new_extradata && new_extradata_size > 0) { - ret = hevc_decode_extradata(s, new_extradata, new_extradata_size, 0); + sd = av_packet_get_side_data(avpkt, AV_PKT_DATA_NEW_EXTRADATA, &sd_size); + if (sd && sd_size > 0) { + ret = hevc_decode_extradata(s, sd, sd_size, 0); if (ret < 0) return ret; } + sd = av_packet_get_side_data(avpkt, AV_PKT_DATA_DOVI_CONF, &sd_size); + if (sd && sd_size > 0) + ff_dovi_update_cfg(&s->dovi_ctx, (AVDOVIDecoderConfigurationRecord *) sd); + s->ref = NULL; ret = decode_nal_units(s, avpkt->data, avpkt->size); if (ret < 0) @@ -3563,6 +3577,7 @@ static av_cold int hevc_decode_free(AVCodecContext *avctx) pic_arrays_free(s); + ff_dovi_ctx_unref(&s->dovi_ctx); av_buffer_unref(&s->rpu_buf); av_freep(&s->md5_ctx); @@ -3647,6 +3662,7 @@ static av_cold int hevc_init_context(AVCodecContext *avctx) ff_bswapdsp_init(&s->bdsp); + s->dovi_ctx.logctx = avctx; s->context_initialized = 1; s->eos = 0; @@ -3755,6 +3771,10 @@ static int hevc_update_thread_context(AVCodecContext *dst, if (ret < 0) return ret; + ret = ff_dovi_ctx_replace(&s->dovi_ctx, &s0->dovi_ctx); + if (ret < 0) + return ret; + s->sei.frame_packing = s0->sei.frame_packing; s->sei.display_orientation = s0->sei.display_orientation; s->sei.mastering_display = s0->sei.mastering_display; @@ -3811,6 +3831,7 @@ static void hevc_decode_flush(AVCodecContext *avctx) HEVCContext *s = avctx->priv_data; ff_hevc_flush_dpb(s); ff_hevc_reset_sei(&s->sei); + ff_dovi_ctx_flush(&s->dovi_ctx); av_buffer_unref(&s->rpu_buf); s->max_ra = INT_MAX; s->eos = 1; diff --git a/libavcodec/hevcdec.h b/libavcodec/hevcdec.h index 870ff178d4..157bc6926f 100644 --- a/libavcodec/hevcdec.h +++ b/libavcodec/hevcdec.h @@ -32,6 +32,7 @@ #include "avcodec.h" #include "bswapdsp.h" #include "cabac.h" +#include "dovi_rpu.h" #include "get_bits.h" #include "hevcpred.h" #include "h2645_parse.h" @@ -574,6 +575,7 @@ typedef struct HEVCContext { int nuh_layer_id; AVBufferRef *rpu_buf; ///< 0 or 1 Dolby Vision RPUs. + DOVIContext dovi_ctx; ///< Dolby Vision decoding context } HEVCContext; /** diff --git a/tests/ref/fate/hevc-dv-rpu b/tests/ref/fate/hevc-dv-rpu index 416d9c51a6..1980ab13ea 100644 --- a/tests/ref/fate/hevc-dv-rpu +++ b/tests/ref/fate/hevc-dv-rpu @@ -5,9 +5,233 @@ side_data_type=H.26[45] User Data Unregistered SEI message [SIDE_DATA] side_data_type=Dolby Vision RPU Data [/SIDE_DATA] +[SIDE_DATA] +side_data_type=Dolby Vision Metadata +rpu_type=2 +rpu_format=18 +vdr_rpu_profile=1 +vdr_rpu_level=0 +chroma_resampling_explicit_filter_flag=0 +coef_data_type=0 +coef_log2_denom=23 +vdr_rpu_normalized_idc=1 +bl_video_full_range_flag=0 +bl_bit_depth=10 +el_bit_depth=10 +vdr_bit_depth=12 +spatial_resampling_filter_flag=0 +el_spatial_resampling_filter_flag=0 +disable_residual_flag=1 +vdr_rpu_id=0 +mapping_color_space=0 +mapping_chroma_format_idc=0 +nlq_method_idc=-1 +nlq_method_idc_name=none +num_x_partitions=1 +num_y_partitions=1 +[COMPONENT] +pivots=63 132 362 618 874 911 927 935 942 +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-409680 16721463 -20276640 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-119056 13575212 -12867889 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=1317527 5338528 -948122 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=2119979 2065496 2288524 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=7982780 -11367226 9973944 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=53792084 -114243184 67724328 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=112973872 -244837568 139764480 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=236828416 -518849056 291306944 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-4200453 +mmr_coef=9094176 31944476 739652 -27103344 -3431599 -11015088 22758042 -2028643 -30021218 -906885 26272992 7291404 16488745 -78087176 -1487777 12496543 762460 -4281948 -5768035 -7843163 103636960 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-10387889 +mmr_coef=29604202 3214133 46206184 -8564340 -53383996 1628407 5426045 -21634202 -5251953 -50812292 19221972 76726656 -425892 -35041240 5917361 2863974 25030554 -14404292 -41230120 1229046 53575140 +[/SECTION] +[/COMPONENT] +dm_metadata_id=0 +scene_refresh_flag=1 +ycc_to_rgb_matrix=9574/8192 0/8192 13802/8192 9574/8192 -1540/8192 -5348/8192 9574/8192 17610/8192 0/8192 +ycc_to_rgb_offset=16777216/268435456 134217728/268435456 134217728/268435456 +rgb_to_lms_matrix=7222/16384 8771/16384 390/16384 2654/16384 12430/16384 1300/16384 0/16384 422/16384 15962/16384 +signal_eotf=65535 +signal_eotf_param0=0 +signal_eotf_param1=0 +signal_eotf_param2=0 +signal_bit_depth=12 +signal_color_space=0 +signal_chroma_format=0 +signal_full_range_flag=1 +source_min_pq=0 +source_max_pq=3079 +source_diagonal=42 +[/SIDE_DATA] [/FRAME] [FRAME] [SIDE_DATA] side_data_type=Dolby Vision RPU Data [/SIDE_DATA] +[SIDE_DATA] +side_data_type=Dolby Vision Metadata +rpu_type=2 +rpu_format=18 +vdr_rpu_profile=1 +vdr_rpu_level=0 +chroma_resampling_explicit_filter_flag=0 +coef_data_type=0 +coef_log2_denom=23 +vdr_rpu_normalized_idc=1 +bl_video_full_range_flag=0 +bl_bit_depth=10 +el_bit_depth=10 +vdr_bit_depth=12 +spatial_resampling_filter_flag=0 +el_spatial_resampling_filter_flag=0 +disable_residual_flag=1 +vdr_rpu_id=0 +mapping_color_space=0 +mapping_chroma_format_idc=0 +nlq_method_idc=-1 +nlq_method_idc_name=none +num_x_partitions=1 +num_y_partitions=1 +[COMPONENT] +pivots=63 132 362 618 874 911 927 935 942 +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-409680 16721463 -20276640 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-119056 13575212 -12867889 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=1317527 5338528 -948122 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=2119979 2065496 2288524 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=7982780 -11367226 9973944 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=53792084 -114243184 67724328 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=112973872 -244837568 139764480 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=236828416 -518849056 291306944 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-4200453 +mmr_coef=9094176 31944476 739652 -27103344 -3431599 -11015088 22758042 -2028643 -30021218 -906885 26272992 7291404 16488745 -78087176 -1487777 12496543 762460 -4281948 -5768035 -7843163 103636960 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-10387889 +mmr_coef=29604202 3214133 46206184 -8564340 -53383996 1628407 5426045 -21634202 -5251953 -50812292 19221972 76726656 -425892 -35041240 5917361 2863974 25030554 -14404292 -41230120 1229046 53575140 +[/SECTION] +[/COMPONENT] +dm_metadata_id=0 +scene_refresh_flag=1 +ycc_to_rgb_matrix=9574/8192 0/8192 13802/8192 9574/8192 -1540/8192 -5348/8192 9574/8192 17610/8192 0/8192 +ycc_to_rgb_offset=16777216/268435456 134217728/268435456 134217728/268435456 +rgb_to_lms_matrix=7222/16384 8771/16384 390/16384 2654/16384 12430/16384 1300/16384 0/16384 422/16384 15962/16384 +signal_eotf=65535 +signal_eotf_param0=0 +signal_eotf_param1=0 +signal_eotf_param2=0 +signal_bit_depth=12 +signal_color_space=0 +signal_chroma_format=0 +signal_full_range_flag=1 +source_min_pq=0 +source_max_pq=3079 +source_diagonal=42 +[/SIDE_DATA] [/FRAME]