From patchwork Sun Dec 12 11:55:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 32306 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp4110657iog; Sun, 12 Dec 2021 03:56:55 -0800 (PST) X-Google-Smtp-Source: ABdhPJyjxtwdjPprPQT2LcjWqEq3ns/iVf2473sFtKChW98xt0LqaHZ0FqpowylTmTCP7SKCutHp X-Received: by 2002:a05:6402:b26:: with SMTP id bo6mr57045842edb.207.1639310215273; Sun, 12 Dec 2021 03:56:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639310215; cv=none; d=google.com; s=arc-20160816; b=OUuAtshSQBTXb0bwp4bukn9Dj8f0IlASVHMNYCXLSeYFxMX3I4+xtvtmLjNhWuCj+0 aG4O2gyEW/y12CTpjvzp54ka5YPsxW/AbOnLHAnzMrfzQE8caipmDZ1U3zIGXWCkUAIM g9sZ+1fSDRrOwipwg0OTvoImwQCvJwaXwNtICDWVYcDOXQb/XBN80nWBQU8PaPfS8NHL 63KV15LyzKKHIg5a63O/fMDnYPBjn0ct9SIzMas1zgSXWtAJDs4FFhbyDyk9lgPozvAy OCIW7cKUAXt84USG9z8Q0Y2SfmH/5YPzDMGM0RQJm7tbnMkhfM52j4cOWZ/nYK6vWTTI qpOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=64gHO0sIsESkC9/AkodhelC1ZEOawyeIufwzlvOWnX8=; b=v5qNyOUUhDDBb8u1/Xi9VM2dw8dCjx7kwIohDcsRtenlcf7nI/abNYqImGCW5GDCcq fqHFdW5XyD7mqy24UXId2KGFb48i/Lm8FKSs9ppJlmUDtCHnDaVVXhfipmXss7wfW5Cw DOQfEfxJtkJ1sP8wkXtQp52/QtEVM5pnyjTYjTlMbNRzOUnGIFI+KHxNb45gF8aXxCEM Okh809OKJ1F7/Y+FH0ltKqP/arR5FTq6m8IBbiqJ89dzQY4rjEObHuEII1M9cUsp+mX6 m+wOUxzWoX0OFHjNd42VyKb4md2c/ZisuHcS6X8KTkTws4C5XDH95Y4EJH4/U19LQ5aB ZfuQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=nYDVUdjI; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hc16si11651317ejc.187.2021.12.12.03.56.54; Sun, 12 Dec 2021 03:56:55 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=nYDVUdjI; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BEFF468AEA0; Sun, 12 Dec 2021 13:56:00 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AE113680836 for ; Sun, 12 Dec 2021 13:55:53 +0200 (EET) Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id AE62349286; Sun, 12 Dec 2021 12:55:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1639310148; bh=nxZUkNQ1422lsUUmqmKaOOujYfXc5maEzNt42I/ZPUQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nYDVUdjI0YizJyA0sz5BgR6JjsryMx2VnaYr2FK9HbdbhjVXrHiU4qP9uU+hzu65R MP7v4epIU5o1x+nuGUa2jR3SOF9QwtWPLkE6Nuzr8nz4ub7QqQ+kfQPX7hEodzcagC DFB6too3RmxWZX8pzJAayRDdP92VBYU7jfG6Z6RM= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Sun, 12 Dec 2021 12:55:46 +0100 Message-Id: <20211212115546.119662-6-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211212115546.119662-1-ffmpeg@haasn.xyz> References: <20211212115546.119662-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v5 6/6] lavc/hevcdec: Parse DOVI RPU NALs X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: KzQcUGUhHkmH From: Niklas Haas And expose the parsed values as frame side data. Update FATE results to match. It's worth documenting that this relies on the dovi configuration record being present on the first AVPacket fed to the decoder, which in practice is the case if if the API user has called something like av_format_inject_global_side_data, which is unfortunately not the default. This commit is not the time and place to change that behavior, though. Signed-off-by: Niklas Haas --- configure | 2 +- libavcodec/hevcdec.c | 62 ++++++++-- libavcodec/hevcdec.h | 3 + tests/ref/fate/hevc-dv-rpu | 224 +++++++++++++++++++++++++++++++++++++ 4 files changed, 282 insertions(+), 9 deletions(-) diff --git a/configure b/configure index b8c9c5d2ae..d4336d20f5 100755 --- a/configure +++ b/configure @@ -2813,7 +2813,7 @@ h264_decoder_suggest="error_resilience" hap_decoder_select="snappy texturedsp" hap_encoder_deps="libsnappy" hap_encoder_select="texturedspenc" -hevc_decoder_select="atsc_a53 bswapdsp cabac golomb hevcparse videodsp" +hevc_decoder_select="atsc_a53 bswapdsp cabac dovi_rpu golomb hevcparse videodsp" huffyuv_decoder_select="bswapdsp huffyuvdsp llviddsp" huffyuv_encoder_select="bswapdsp huffman huffyuvencdsp llvidencdsp" hymt_decoder_select="huffyuv_decoder" diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index 46d9edf8eb..82410ae5cc 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -2723,6 +2723,7 @@ error: static int set_side_data(HEVCContext *s) { AVFrame *out = s->ref->frame; + int ret; if (s->sei.frame_packing.present && s->sei.frame_packing.arrangement_type >= 3 && @@ -2967,6 +2968,9 @@ static int set_side_data(HEVCContext *s) s->rpu_buf = NULL; } + if ((ret = ff_dovi_attach_side_data(&s->dovi_ctx, out)) < 0) + return ret; + return 0; } @@ -3298,16 +3302,23 @@ static int decode_nal_units(HEVCContext *s, const uint8_t *buf, int length) if (s->pkt.nb_nals > 1 && s->pkt.nals[s->pkt.nb_nals - 1].type == HEVC_NAL_UNSPEC62 && s->pkt.nals[s->pkt.nb_nals - 1].size > 2 && !s->pkt.nals[s->pkt.nb_nals - 1].nuh_layer_id && !s->pkt.nals[s->pkt.nb_nals - 1].temporal_id) { + H2645NAL *nal = &s->pkt.nals[s->pkt.nb_nals - 1]; if (s->rpu_buf) { av_buffer_unref(&s->rpu_buf); av_log(s->avctx, AV_LOG_WARNING, "Multiple Dolby Vision RPUs found in one AU. Skipping previous.\n"); } - s->rpu_buf = av_buffer_alloc(s->pkt.nals[s->pkt.nb_nals - 1].raw_size - 2); + s->rpu_buf = av_buffer_alloc(nal->raw_size - 2); if (!s->rpu_buf) return AVERROR(ENOMEM); + memcpy(s->rpu_buf->data, nal->raw_data + 2, nal->raw_size - 2); - memcpy(s->rpu_buf->data, s->pkt.nals[s->pkt.nb_nals - 1].raw_data + 2, s->pkt.nals[s->pkt.nb_nals - 1].raw_size - 2); + ret = ff_dovi_rpu_parse(&s->dovi_ctx, nal->data + 2, nal->size - 2); + if (ret < 0) { + av_buffer_unref(&s->rpu_buf); + av_log(s->avctx, AV_LOG_WARNING, "Error parsing DOVI NAL unit.\n"); + /* ignore */ + } } /* decode the NAL units */ @@ -3440,8 +3451,8 @@ static int hevc_decode_frame(AVCodecContext *avctx, void *data, int *got_output, AVPacket *avpkt) { int ret; - size_t new_extradata_size; - uint8_t *new_extradata; + uint8_t *sd; + size_t sd_size; HEVCContext *s = avctx->priv_data; if (!avpkt->size) { @@ -3453,14 +3464,37 @@ static int hevc_decode_frame(AVCodecContext *avctx, void *data, int *got_output, return 0; } - new_extradata = av_packet_get_side_data(avpkt, AV_PKT_DATA_NEW_EXTRADATA, - &new_extradata_size); - if (new_extradata && new_extradata_size > 0) { - ret = hevc_decode_extradata(s, new_extradata, new_extradata_size, 0); + sd = av_packet_get_side_data(avpkt, AV_PKT_DATA_NEW_EXTRADATA, &sd_size); + if (sd && sd_size > 0) { + ret = hevc_decode_extradata(s, sd, sd_size, 0); if (ret < 0) return ret; } + sd = av_packet_get_side_data(avpkt, AV_PKT_DATA_DOVI_CONF, &sd_size); + if (sd && sd_size > 0) { + if (s->dovi_cfg) { + /* Reuse existing buffer */ + if ((ret = av_buffer_make_writable(&s->dovi_cfg)) < 0) + return ret; + } else { + /* Allocate new buffer */ + AVDOVIDecoderConfigurationRecord *cfg; + size_t cfg_size; + cfg = av_dovi_alloc(&cfg_size); + if (!cfg) + return AVERROR(ENOMEM); + s->dovi_cfg = av_buffer_create((uint8_t *) cfg, cfg_size, NULL, NULL, 0); + if (!s->dovi_cfg) { + av_free(cfg); + return AVERROR(ENOMEM); + } + } + + av_assert0(sd_size >= s->dovi_cfg->size); + memcpy(s->dovi_cfg->data, sd, s->dovi_cfg->size); + } + s->ref = NULL; ret = decode_nal_units(s, avpkt->data, avpkt->size); if (ret < 0) @@ -3553,6 +3587,8 @@ static av_cold int hevc_decode_free(AVCodecContext *avctx) pic_arrays_free(s); + ff_dovi_ctx_unref(&s->dovi_ctx); + av_buffer_unref(&s->dovi_cfg); av_buffer_unref(&s->rpu_buf); av_freep(&s->md5_ctx); @@ -3637,6 +3673,7 @@ static av_cold int hevc_init_context(AVCodecContext *avctx) ff_bswapdsp_init(&s->bdsp); + s->dovi_ctx.avctx = avctx; s->context_initialized = 1; s->eos = 0; @@ -3745,6 +3782,14 @@ static int hevc_update_thread_context(AVCodecContext *dst, if (ret < 0) return ret; + ret = av_buffer_replace(&s->dovi_cfg, s0->dovi_cfg); + if (ret < 0) + return ret; + + ret = ff_dovi_ctx_replace(&s->dovi_ctx, &s0->dovi_ctx); + if (ret < 0) + return ret; + s->sei.frame_packing = s0->sei.frame_packing; s->sei.display_orientation = s0->sei.display_orientation; s->sei.mastering_display = s0->sei.mastering_display; @@ -3801,6 +3846,7 @@ static void hevc_decode_flush(AVCodecContext *avctx) HEVCContext *s = avctx->priv_data; ff_hevc_flush_dpb(s); ff_hevc_reset_sei(&s->sei); + ff_dovi_ctx_unref(&s->dovi_ctx); av_buffer_unref(&s->rpu_buf); s->max_ra = INT_MAX; s->eos = 1; diff --git a/libavcodec/hevcdec.h b/libavcodec/hevcdec.h index 870ff178d4..c8dde6fd17 100644 --- a/libavcodec/hevcdec.h +++ b/libavcodec/hevcdec.h @@ -32,6 +32,7 @@ #include "avcodec.h" #include "bswapdsp.h" #include "cabac.h" +#include "dovi_rpu.h" #include "get_bits.h" #include "hevcpred.h" #include "h2645_parse.h" @@ -574,6 +575,8 @@ typedef struct HEVCContext { int nuh_layer_id; AVBufferRef *rpu_buf; ///< 0 or 1 Dolby Vision RPUs. + AVBufferRef *dovi_cfg; ///< contains AVDOVIDecoderConfigurationRecord + DOVIContext dovi_ctx; ///< Dolby Vision decoding context } HEVCContext; /** diff --git a/tests/ref/fate/hevc-dv-rpu b/tests/ref/fate/hevc-dv-rpu index 416d9c51a6..1980ab13ea 100644 --- a/tests/ref/fate/hevc-dv-rpu +++ b/tests/ref/fate/hevc-dv-rpu @@ -5,9 +5,233 @@ side_data_type=H.26[45] User Data Unregistered SEI message [SIDE_DATA] side_data_type=Dolby Vision RPU Data [/SIDE_DATA] +[SIDE_DATA] +side_data_type=Dolby Vision Metadata +rpu_type=2 +rpu_format=18 +vdr_rpu_profile=1 +vdr_rpu_level=0 +chroma_resampling_explicit_filter_flag=0 +coef_data_type=0 +coef_log2_denom=23 +vdr_rpu_normalized_idc=1 +bl_video_full_range_flag=0 +bl_bit_depth=10 +el_bit_depth=10 +vdr_bit_depth=12 +spatial_resampling_filter_flag=0 +el_spatial_resampling_filter_flag=0 +disable_residual_flag=1 +vdr_rpu_id=0 +mapping_color_space=0 +mapping_chroma_format_idc=0 +nlq_method_idc=-1 +nlq_method_idc_name=none +num_x_partitions=1 +num_y_partitions=1 +[COMPONENT] +pivots=63 132 362 618 874 911 927 935 942 +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-409680 16721463 -20276640 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-119056 13575212 -12867889 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=1317527 5338528 -948122 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=2119979 2065496 2288524 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=7982780 -11367226 9973944 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=53792084 -114243184 67724328 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=112973872 -244837568 139764480 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=236828416 -518849056 291306944 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-4200453 +mmr_coef=9094176 31944476 739652 -27103344 -3431599 -11015088 22758042 -2028643 -30021218 -906885 26272992 7291404 16488745 -78087176 -1487777 12496543 762460 -4281948 -5768035 -7843163 103636960 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-10387889 +mmr_coef=29604202 3214133 46206184 -8564340 -53383996 1628407 5426045 -21634202 -5251953 -50812292 19221972 76726656 -425892 -35041240 5917361 2863974 25030554 -14404292 -41230120 1229046 53575140 +[/SECTION] +[/COMPONENT] +dm_metadata_id=0 +scene_refresh_flag=1 +ycc_to_rgb_matrix=9574/8192 0/8192 13802/8192 9574/8192 -1540/8192 -5348/8192 9574/8192 17610/8192 0/8192 +ycc_to_rgb_offset=16777216/268435456 134217728/268435456 134217728/268435456 +rgb_to_lms_matrix=7222/16384 8771/16384 390/16384 2654/16384 12430/16384 1300/16384 0/16384 422/16384 15962/16384 +signal_eotf=65535 +signal_eotf_param0=0 +signal_eotf_param1=0 +signal_eotf_param2=0 +signal_bit_depth=12 +signal_color_space=0 +signal_chroma_format=0 +signal_full_range_flag=1 +source_min_pq=0 +source_max_pq=3079 +source_diagonal=42 +[/SIDE_DATA] [/FRAME] [FRAME] [SIDE_DATA] side_data_type=Dolby Vision RPU Data [/SIDE_DATA] +[SIDE_DATA] +side_data_type=Dolby Vision Metadata +rpu_type=2 +rpu_format=18 +vdr_rpu_profile=1 +vdr_rpu_level=0 +chroma_resampling_explicit_filter_flag=0 +coef_data_type=0 +coef_log2_denom=23 +vdr_rpu_normalized_idc=1 +bl_video_full_range_flag=0 +bl_bit_depth=10 +el_bit_depth=10 +vdr_bit_depth=12 +spatial_resampling_filter_flag=0 +el_spatial_resampling_filter_flag=0 +disable_residual_flag=1 +vdr_rpu_id=0 +mapping_color_space=0 +mapping_chroma_format_idc=0 +nlq_method_idc=-1 +nlq_method_idc_name=none +num_x_partitions=1 +num_y_partitions=1 +[COMPONENT] +pivots=63 132 362 618 874 911 927 935 942 +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-409680 16721463 -20276640 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=-119056 13575212 -12867889 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=1317527 5338528 -948122 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=2119979 2065496 2288524 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=7982780 -11367226 9973944 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=53792084 -114243184 67724328 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=112973872 -244837568 139764480 +[/SECTION] +[SECTION] +mapping_idc=0 +mapping_idc_name=polynomial +poly_order=2 +poly_coef=236828416 -518849056 291306944 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-4200453 +mmr_coef=9094176 31944476 739652 -27103344 -3431599 -11015088 22758042 -2028643 -30021218 -906885 26272992 7291404 16488745 -78087176 -1487777 12496543 762460 -4281948 -5768035 -7843163 103636960 +[/SECTION] +[/COMPONENT] +[COMPONENT] +pivots=0 1023 +[SECTION] +mapping_idc=1 +mapping_idc_name=mmr +mmr_order=3 +mmr_constant=-10387889 +mmr_coef=29604202 3214133 46206184 -8564340 -53383996 1628407 5426045 -21634202 -5251953 -50812292 19221972 76726656 -425892 -35041240 5917361 2863974 25030554 -14404292 -41230120 1229046 53575140 +[/SECTION] +[/COMPONENT] +dm_metadata_id=0 +scene_refresh_flag=1 +ycc_to_rgb_matrix=9574/8192 0/8192 13802/8192 9574/8192 -1540/8192 -5348/8192 9574/8192 17610/8192 0/8192 +ycc_to_rgb_offset=16777216/268435456 134217728/268435456 134217728/268435456 +rgb_to_lms_matrix=7222/16384 8771/16384 390/16384 2654/16384 12430/16384 1300/16384 0/16384 422/16384 15962/16384 +signal_eotf=65535 +signal_eotf_param0=0 +signal_eotf_param1=0 +signal_eotf_param2=0 +signal_bit_depth=12 +signal_color_space=0 +signal_chroma_format=0 +signal_full_range_flag=1 +source_min_pq=0 +source_max_pq=3079 +source_diagonal=42 +[/SIDE_DATA] [/FRAME]