From patchwork Sun Dec 12 15:53:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 32335 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp4322379iog; Sun, 12 Dec 2021 07:53:50 -0800 (PST) X-Google-Smtp-Source: ABdhPJxUnHY6JCj3HI70gIL1aWDteMw6SssZbp/XxkvAW4sqjQWXHSHcJ1NcWn1X6SHa3eY7b46p X-Received: by 2002:a50:f189:: with SMTP id x9mr56886872edl.95.1639324430463; Sun, 12 Dec 2021 07:53:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639324430; cv=none; d=google.com; s=arc-20160816; b=CmJNVNYhuBFK42aNov6VjWK7k9Z5L8YkYurIugh0/I2WD8N8HxbhHjvPbq9HfatiMn QXOi6meuKTyOehnV4DGLO9Dt0vaSZMCtz0aHUC3Db4QwWUvybVEgA7OeSEfguYz5Wi+Z nsZAGVy3gfkVEpwpmNHSn8soiJlFLYrT5TFE1FzS4SHYONgzYasA0tUPlHky6rbdadve lsmujw9CzY9TYTALqOsUDHu5MvkMS1zckrEDZ4T5drir62Sw/oXYEB+UgcE9SCZghWVC Zhwupm7JSuzlsDFS2Biv9FUGDpuMWRbb8weK6h9Ua1o0yVq3UwL10kY//j41sNJK8Y+Q wrqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=g/zZPQmm3G1fd3/BHT/mGsAqBZr++hXbVjJn7uruZqA=; b=nbprIIICRJTIFNw8IK9HmtSYMDIIZJFR05f6YB9tIGWDvHYwtNrAt1Btaq/DQ1FBz7 BLO/SPKs2s0dc90f00wGwCb0/Zx6UODHXNUU5Qbe6GXbWBPighXI34lYuBzTJdn1DxOj wmFdIEacajt2ob5bKJVTwvH3id2jWXgPGZ+LR54DcFlnmIDl4ylGBIHKC8TQlqTdbvdE X2Dg+isoZnk2vV/miWR3MTP0LoZqjgRIi9elOssiedH+FuLvBddbSLrSDrKijDmA0WMZ yJ2KbPzL/DI/C7CS/SzuWPTzd9qH84iC0WGnDjb+PDOaEI0s8t+utnL4oOJMjhR7tD6i sayQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=u+mZRJyd; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 8si13378519edv.30.2021.12.12.07.53.50; Sun, 12 Dec 2021 07:53:50 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=u+mZRJyd; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BCEC368AD39; Sun, 12 Dec 2021 17:53:46 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 54B8A68AAA0 for ; Sun, 12 Dec 2021 17:53:40 +0200 (EET) Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id 0F0904924E; Sun, 12 Dec 2021 16:53:40 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1639324420; bh=xlIqfCObMr5R3oAlfc2ftoaRoE4P8UEyjes6KFz/X70=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u+mZRJydskt65bOrPc0pgKCeYM7kskaoID97x5w3Psyraf7OwNsNsF+J7GOvEtpJV wUqyKWNdeuoAzdb4OJJJ63jJDncfQaNWUN6LfGfVX3ojKxOrM5XxI2cnpbofN4Sgcd wIEO7xp/GdSTA/MialbrUzQOcGpWtV3mh4c31YcY= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Sun, 12 Dec 2021 16:53:38 +0100 Message-Id: <20211212155338.25298-1-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211212115546.119662-6-ffmpeg@haasn.xyz> References: <20211212115546.119662-6-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v6] lavc/hevcdec: Parse DOVI RPU NALs X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: SXjVk3j/8ymz From: Niklas Haas Oops. The previous version of this patch forgot to actually forward the dovi configuration record to the DOVIContext. --- And expose the parsed values as frame side data. Update FATE results to match. It's worth documenting that this relies on the dovi configuration record being present on the first AVPacket fed to the decoder, which in practice is the case if if the API user has called something like av_format_inject_global_side_data, which is unfortunately not the default. This commit is not the time and place to change that behavior, though. Signed-off-by: Niklas Haas --- configure | 2 +- libavcodec/hevcdec.c | 63 +++++++++-- libavcodec/hevcdec.h | 3 + tests/ref/fate/hevc-dv-rpu | 224 +++++++++++++++++++++++++++++++++++++ 4 files changed, 283 insertions(+), 9 deletions(-) diff --git a/configure b/configure index b8c9c5d2ae..d4336d20f5 100755 --- a/configure +++ b/configure @@ -2813,7 +2813,7 @@ h264_decoder_suggest="error_resilience" hap_decoder_select="snappy texturedsp" hap_encoder_deps="libsnappy" hap_encoder_select="texturedspenc" -hevc_decoder_select="atsc_a53 bswapdsp cabac golomb hevcparse videodsp" +hevc_decoder_select="atsc_a53 bswapdsp cabac dovi_rpu golomb hevcparse videodsp" huffyuv_decoder_select="bswapdsp huffyuvdsp llviddsp" huffyuv_encoder_select="bswapdsp huffman huffyuvencdsp llvidencdsp" hymt_decoder_select="huffyuv_decoder" diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index 46d9edf8eb..9ca2fa224d 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -2723,6 +2723,7 @@ error: static int set_side_data(HEVCContext *s) { AVFrame *out = s->ref->frame; + int ret; if (s->sei.frame_packing.present && s->sei.frame_packing.arrangement_type >= 3 && @@ -2967,6 +2968,9 @@ static int set_side_data(HEVCContext *s) s->rpu_buf = NULL; } + if ((ret = ff_dovi_attach_side_data(&s->dovi_ctx, out)) < 0) + return ret; + return 0; } @@ -3298,16 +3302,24 @@ static int decode_nal_units(HEVCContext *s, const uint8_t *buf, int length) if (s->pkt.nb_nals > 1 && s->pkt.nals[s->pkt.nb_nals - 1].type == HEVC_NAL_UNSPEC62 && s->pkt.nals[s->pkt.nb_nals - 1].size > 2 && !s->pkt.nals[s->pkt.nb_nals - 1].nuh_layer_id && !s->pkt.nals[s->pkt.nb_nals - 1].temporal_id) { + H2645NAL *nal = &s->pkt.nals[s->pkt.nb_nals - 1]; if (s->rpu_buf) { av_buffer_unref(&s->rpu_buf); av_log(s->avctx, AV_LOG_WARNING, "Multiple Dolby Vision RPUs found in one AU. Skipping previous.\n"); } - s->rpu_buf = av_buffer_alloc(s->pkt.nals[s->pkt.nb_nals - 1].raw_size - 2); + s->rpu_buf = av_buffer_alloc(nal->raw_size - 2); if (!s->rpu_buf) return AVERROR(ENOMEM); + memcpy(s->rpu_buf->data, nal->raw_data + 2, nal->raw_size - 2); - memcpy(s->rpu_buf->data, s->pkt.nals[s->pkt.nb_nals - 1].raw_data + 2, s->pkt.nals[s->pkt.nb_nals - 1].raw_size - 2); + s->dovi_ctx.config = s->dovi_cfg ? (void *) s->dovi_cfg->data : NULL; + ret = ff_dovi_rpu_parse(&s->dovi_ctx, nal->data + 2, nal->size - 2); + if (ret < 0) { + av_buffer_unref(&s->rpu_buf); + av_log(s->avctx, AV_LOG_WARNING, "Error parsing DOVI NAL unit.\n"); + /* ignore */ + } } /* decode the NAL units */ @@ -3440,8 +3452,8 @@ static int hevc_decode_frame(AVCodecContext *avctx, void *data, int *got_output, AVPacket *avpkt) { int ret; - size_t new_extradata_size; - uint8_t *new_extradata; + uint8_t *sd; + size_t sd_size; HEVCContext *s = avctx->priv_data; if (!avpkt->size) { @@ -3453,14 +3465,37 @@ static int hevc_decode_frame(AVCodecContext *avctx, void *data, int *got_output, return 0; } - new_extradata = av_packet_get_side_data(avpkt, AV_PKT_DATA_NEW_EXTRADATA, - &new_extradata_size); - if (new_extradata && new_extradata_size > 0) { - ret = hevc_decode_extradata(s, new_extradata, new_extradata_size, 0); + sd = av_packet_get_side_data(avpkt, AV_PKT_DATA_NEW_EXTRADATA, &sd_size); + if (sd && sd_size > 0) { + ret = hevc_decode_extradata(s, sd, sd_size, 0); if (ret < 0) return ret; } + sd = av_packet_get_side_data(avpkt, AV_PKT_DATA_DOVI_CONF, &sd_size); + if (sd && sd_size > 0) { + if (s->dovi_cfg) { + /* Reuse existing buffer */ + if ((ret = av_buffer_make_writable(&s->dovi_cfg)) < 0) + return ret; + } else { + /* Allocate new buffer */ + AVDOVIDecoderConfigurationRecord *cfg; + size_t cfg_size; + cfg = av_dovi_alloc(&cfg_size); + if (!cfg) + return AVERROR(ENOMEM); + s->dovi_cfg = av_buffer_create((uint8_t *) cfg, cfg_size, NULL, NULL, 0); + if (!s->dovi_cfg) { + av_free(cfg); + return AVERROR(ENOMEM); + } + } + + av_assert0(sd_size >= s->dovi_cfg->size); + memcpy(s->dovi_cfg->data, sd, s->dovi_cfg->size); + } + s->ref = NULL; ret = decode_nal_units(s, avpkt->data, avpkt->size); if (ret < 0) @@ -3553,6 +3588,8 @@ static av_cold int hevc_decode_free(AVCodecContext *avctx) pic_arrays_free(s); + ff_dovi_ctx_unref(&s->dovi_ctx); + av_buffer_unref(&s->dovi_cfg); av_buffer_unref(&s->rpu_buf); av_freep(&s->md5_ctx); @@ -3637,6 +3674,7 @@ static av_cold int hevc_init_context(AVCodecContext *avctx) ff_bswapdsp_init(&s->bdsp); + s->dovi_ctx.avctx = avctx; s->context_initialized = 1; s->eos = 0; @@ -3745,6 +3783,14 @@ static int hevc_update_thread_context(AVCodecContext *dst, if (ret < 0) return ret; + ret = av_buffer_replace(&s->dovi_cfg, s0->dovi_cfg); + if (ret < 0) + return ret; + + ret = ff_dovi_ctx_replace(&s->dovi_ctx, &s0->dovi_ctx); + if (ret < 0) + return ret; + s->sei.frame_packing = s0->sei.frame_packing; s->sei.display_orientation = s0->sei.display_orientation; s->sei.mastering_display = s0->sei.mastering_display; @@ -3801,6 +3847,7 @@ static void hevc_decode_flush(AVCodecContext *avctx) HEVCContext *s = avctx->priv_data; ff_hevc_flush_dpb(s); ff_hevc_reset_sei(&s->sei); + ff_dovi_ctx_unref(&s->dovi_ctx); av_buffer_unref(&s->rpu_buf); s->max_ra = INT_MAX; s->eos = 1; diff --git a/libavcodec/hevcdec.h b/libavcodec/hevcdec.h index 870ff178d4..c8dde6fd17 100644 --- a/libavcodec/hevcdec.h +++ b/libavcodec/hevcdec.h @@ -32,6 +32,7 @@ #include "avcodec.h" #include "bswapdsp.h" #include "cabac.h" +#include "dovi_rpu.h" #include "get_bits.h" #include "hevcpred.h" #include "h2645_parse.h" @@ -574,6 +575,8 @@ typedef struct HEVCContext { int nuh_layer_id; AVBufferRef *rpu_buf; ///< 0 or 1 Dolby Vision RPUs. + AVBufferRef *dovi_cfg; ///< contains AVDOVIDecoderConfigurationRecord + DOVIContext dovi_ctx; ///< Dolby Vision decoding context } HEVCContext; /** diff --git a/tests/ref/fate/hevc-dv-rpu b/tests/ref/fate/hevc-dv-rpu index 416d9c51a6..1980ab13ea 100644 --- a/tests/ref/fate/hevc-dv-rpu +++ b/tests/ref/fate/hevc-dv-rpu @@ -5,9 +5,233 @@ side_data_type=H.26[45] User Data Unregistered SEI message [SIDE_DATA] side_data_type=Dolby Vision RPU Data [/SIDE_DATA] +[SIDE_DATA] +side_data_type=Dolby Vision Metadata +rpu_type=2 +rpu_format=18 +vdr_rpu_profile=1 +vdr_rpu_level=0 +chroma_resampling_explicit_filter_flag=0 +coef_data_type=0 +coef_log2_denom=23 +vdr_rpu_normalized_idc=1 +bl_video_full_range_flag=0 +bl_bit_depth=10 +el_bit_depth=10 +vdr_bit_depth=12 +spatial_resampling_filter_flag=0 +el_spatial_resampling_filter_flag=0 +disable_residual_flag=1 +vdr_rpu_id=0 +mapping_color_space=0 +mapping_chroma_format_idc=0 +nlq_method_idc=-1 +nlq_method_idc_name=none +num_x_partitions=1 +num_y_partitions=1 +[COMPONENT] +pivots=63 132 362 618 874 911 927 935 942 +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-409680 16721463 -20276640 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-119056 13575212 -12867889 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=1317527 5338528 -948122 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=2119979 2065496 2288524 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=7982780 -11367226 9973944 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=53792084 -114243184 67724328 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=112973872 -244837568 139764480 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=236828416 -518849056 291306944 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-4200453 +mmr_coef=9094176 31944476 739652 -27103344 -3431599 -11015088 22758042 -2028643 -30021218 -906885 26272992 7291404 16488745 -78087176 -1487777 12496543 762460 -4281948 -5768035 -7843163 103636960 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-10387889 +mmr_coef=29604202 3214133 46206184 -8564340 -53383996 1628407 5426045 -21634202 -5251953 -50812292 19221972 76726656 -425892 -35041240 5917361 2863974 25030554 -14404292 -41230120 1229046 53575140 +[/SECTION] +[/COMPONENT] +dm_metadata_id=0 +scene_refresh_flag=1 +ycc_to_rgb_matrix=9574/8192 0/8192 13802/8192 9574/8192 -1540/8192 -5348/8192 9574/8192 17610/8192 0/8192 +ycc_to_rgb_offset=16777216/268435456 134217728/268435456 134217728/268435456 +rgb_to_lms_matrix=7222/16384 8771/16384 390/16384 2654/16384 12430/16384 1300/16384 0/16384 422/16384 15962/16384 +signal_eotf=65535 +signal_eotf_param0=0 +signal_eotf_param1=0 +signal_eotf_param2=0 +signal_bit_depth=12 +signal_color_space=0 +signal_chroma_format=0 +signal_full_range_flag=1 +source_min_pq=0 +source_max_pq=3079 +source_diagonal=42 +[/SIDE_DATA] [/FRAME] [FRAME] [SIDE_DATA] side_data_type=Dolby Vision RPU Data [/SIDE_DATA] +[SIDE_DATA] +side_data_type=Dolby Vision Metadata +rpu_type=2 +rpu_format=18 +vdr_rpu_profile=1 +vdr_rpu_level=0 +chroma_resampling_explicit_filter_flag=0 +coef_data_type=0 +coef_log2_denom=23 +vdr_rpu_normalized_idc=1 +bl_video_full_range_flag=0 +bl_bit_depth=10 +el_bit_depth=10 +vdr_bit_depth=12 +spatial_resampling_filter_flag=0 +el_spatial_resampling_filter_flag=0 +disable_residual_flag=1 +vdr_rpu_id=0 +mapping_color_space=0 +mapping_chroma_format_idc=0 +nlq_method_idc=-1 +nlq_method_idc_name=none +num_x_partitions=1 +num_y_partitions=1 +[COMPONENT] +pivots=63 132 362 618 874 911 927 935 942 +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-409680 16721463 -20276640 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-119056 13575212 -12867889 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=1317527 5338528 -948122 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=2119979 2065496 2288524 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=7982780 -11367226 9973944 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=53792084 -114243184 67724328 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=112973872 -244837568 139764480 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=236828416 -518849056 291306944 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-4200453 +mmr_coef=9094176 31944476 739652 -27103344 -3431599 -11015088 22758042 -2028643 -30021218 -906885 26272992 7291404 16488745 -78087176 -1487777 12496543 762460 -4281948 -5768035 -7843163 103636960 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-10387889 +mmr_coef=29604202 3214133 46206184 -8564340 -53383996 1628407 5426045 -21634202 -5251953 -50812292 19221972 76726656 -425892 -35041240 5917361 2863974 25030554 -14404292 -41230120 1229046 53575140 +[/SECTION] +[/COMPONENT] +dm_metadata_id=0 +scene_refresh_flag=1 +ycc_to_rgb_matrix=9574/8192 0/8192 13802/8192 9574/8192 -1540/8192 -5348/8192 9574/8192 17610/8192 0/8192 +ycc_to_rgb_offset=16777216/268435456 134217728/268435456 134217728/268435456 +rgb_to_lms_matrix=7222/16384 8771/16384 390/16384 2654/16384 12430/16384 1300/16384 0/16384 422/16384 15962/16384 +signal_eotf=65535 +signal_eotf_param0=0 +signal_eotf_param1=0 +signal_eotf_param2=0 +signal_bit_depth=12 +signal_color_space=0 +signal_chroma_format=0 +signal_full_range_flag=1 +source_min_pq=0 +source_max_pq=3079 +source_diagonal=42 +[/SIDE_DATA] [/FRAME]