From patchwork Wed Dec 8 10:12:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 32173 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp150005iog; Wed, 8 Dec 2021 02:13:35 -0800 (PST) X-Google-Smtp-Source: ABdhPJwTRCeB4fkflVSNnm6DPWDZOTp44cr0v3Le0qzgob8fr8ilfB66hQ2KP8MraWjNm5pYVAer X-Received: by 2002:a05:6402:908:: with SMTP id g8mr18105246edz.59.1638958415168; Wed, 08 Dec 2021 02:13:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638958415; cv=none; d=google.com; s=arc-20160816; b=LRgpgEgjUEl3kLTrivsYpqZdTmBrFm25GSCSdXzTxSCKBo9qKkRrpGFNMigSmx8OIC KkZBWlTQgbqBN6Yde1lUJRnaDjNTXYKIbCQmWDe3ScrzuNH5OLa1uwSNM/+XhGsRD3+R fCbTSmeZ6qQVnRIaHEq9E97NUmmysovqlPvEUaqaIbzJyyILlkKsRaSkHZbXFVchu8LF jm/O+WlgoG20kg9l4XdZnYKPUb6puZfluoVIS7iM2OujH4zboKu8Q/FZK57e1Diol3yK /xFuqLt1H6qrvpKObieELIsQPX7pLvnuo7N+MBGf6OB4Atv7nLhNZazmnofxYjf6GwCz Vp7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=nWVI1IANT98YemPI89FZy8Njmkbmelza7M6kIJjf7Bc=; b=SUSMj91baGGkZnWj4YWBTcqIZrLP5Jc7IoPvyL78C3N0MHzIUKc8+E49yYw5FApZpj /dqTSio66HK+9Q2aFn5XGIMG2JZjL7LF0y0d4fbWk8d9bwglxcPjw1I8E12K7rwNsvtz 6jTX0WibE8kaXxkhDzUctxV4OxmVs15uSkeJIQW56/n20Hg2IjLsXs3a9dfQH9eL6WKY Zc244xdzJr3CRFeHXiadCUj+Z7ozfW4PG+tzXZRbCXXrVUU8RixD71dWHA5qPL1WBGwB od9mqDHmZkN4tI8XtIq8PCN6HziDT+H5O1xdsFtYVlA5TH77bSkqBH86pV+JoYwdxkn1 1fWQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b="LLO/9Gya"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a5si3793574edy.585.2021.12.08.02.13.11; Wed, 08 Dec 2021 02:13:35 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b="LLO/9Gya"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B30C16801A0; Wed, 8 Dec 2021 12:12:49 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9C05D68AAE6 for ; Wed, 8 Dec 2021 12:12:42 +0200 (EET) Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id 4E9B04910E; Wed, 8 Dec 2021 11:12:40 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1638958360; bh=xhcWLy7Qy7ZQFcQ734My2Yfbi5cymg7TMMKYIBgUsp4=; h=From:To:Cc:Subject:Date:From; b=LLO/9Gya4dHJiSw997kvL/N1Uzu7BYvRkg9ckXQX+xPzI/ikAv+Ph2acErhFg9st5 En+/W6z+h9xubUcI2fVzP9D9a0hmkWTS4hqtr8bVB3Bkmp+ZM+YdSVnVVBrlejjym0 0/fz/VBUtvcWi/FZ8y5Sidhwr19eSUtQEXI6hOyc= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 8 Dec 2021 11:12:34 +0100 Message-Id: <20211208101237.18407-1-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 1/4] lavu/frame: Add Dolby Vision metadata side data type X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: IHf81TFZxEwI From: Niklas Haas Signed-off-by: Niklas Haas --- doc/APIchanges | 3 ++ libavutil/dovi_meta.c | 23 ++++++++ libavutil/dovi_meta.h | 122 ++++++++++++++++++++++++++++++++++++++++++ libavutil/frame.c | 1 + libavutil/frame.h | 9 +++- libavutil/version.h | 2 +- 6 files changed, 158 insertions(+), 2 deletions(-) diff --git a/doc/APIchanges b/doc/APIchanges index 2914ad6734..422874e3b9 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -14,6 +14,9 @@ libavutil: 2021-04-27 API changes, most recent first: +2021-12-06 - xxxxxxxxxx - lavu 57.11.100 - frame.h + Add AV_FRAME_DATA_DOVI_RESHAPING. + 2021-11-xx - xxxxxxxxxx - lavfi 8.19.100 - avfilter.h Add AVFILTER_FLAG_METADATA_ONLY. diff --git a/libavutil/dovi_meta.c b/libavutil/dovi_meta.c index 7bd08f6c54..e2ef8ac3a4 100644 --- a/libavutil/dovi_meta.c +++ b/libavutil/dovi_meta.c @@ -33,3 +33,26 @@ AVDOVIDecoderConfigurationRecord *av_dovi_alloc(size_t *size) return dovi; } + +/* based on guesswork, see mkvtoolnix and dovi_tool */ +int av_dovi_profile(const AVDOVIRpuDataHeader *hdr) +{ + switch (hdr->vdr_rpu_profile) { + case 0: + if (hdr->bl_video_full_range_flag) + return 5; + break; + case 1: + if (hdr->el_spatial_resampling_filter_flag && !hdr->disable_residual_flag) { + if (hdr->vdr_bit_depth == 12) { + return 7; + } else { + return 4; + } + } else { + return 8; + } + } + + return 0; /* unknown */ +} diff --git a/libavutil/dovi_meta.h b/libavutil/dovi_meta.h index 299911d434..a3a97463ac 100644 --- a/libavutil/dovi_meta.h +++ b/libavutil/dovi_meta.h @@ -29,6 +29,7 @@ #include #include +#include "rational.h" /* * DOVI configuration @@ -67,4 +68,125 @@ typedef struct AVDOVIDecoderConfigurationRecord { */ AVDOVIDecoderConfigurationRecord *av_dovi_alloc(size_t *size); +/** + * Dolby Vision RPU data header. + */ +typedef struct AVDOVIRpuDataHeader { + uint8_t rpu_type; + uint16_t rpu_format; + uint8_t vdr_rpu_profile; + uint8_t vdr_rpu_level; + int chroma_resampling_explicit_filter_flag; + uint8_t coef_data_type; /* informative, lavc always converts to fixed */ + uint8_t coef_log2_denom; + uint8_t vdr_rpu_normalized_idc; + int bl_video_full_range_flag; + uint8_t bl_bit_depth; /* [8, 16] */ + uint8_t el_bit_depth; /* [8, 16] */ + uint8_t vdr_bit_depth; /* [8, 16] */ + int spatial_resampling_filter_flag; + int el_spatial_resampling_filter_flag; + int disable_residual_flag; +} AVDOVIRpuDataHeader; + +/** + * Return the Dolby Vision profile number derived from a given RPU data header, + * or 0 for unknown/unrecognized profiles. + */ +int av_dovi_profile(const AVDOVIRpuDataHeader *hdr); + +enum AVDOVIMappingMethod { + AV_DOVI_MAPPING_POLYNOMIAL = 0, + AV_DOVI_MAPPING_MMR = 1, +}; + +/** + * Coefficients of a piece-wise function. The pieces of the function span the + * value ranges between two adjacent pivot values. + */ +#define FF_DOVI_MAX_PIECES 8 +typedef struct AVDOVIReshapingCurve { + uint8_t num_pivots; /* [2, 9] */ + uint16_t pivots[FF_DOVI_MAX_PIECES + 1]; /* sorted ascending */ + enum AVDOVIMappingMethod mapping_idc[FF_DOVI_MAX_PIECES]; + /* AV_DOVI_MAPPING_POLYNOMIAL */ + uint8_t poly_order[FF_DOVI_MAX_PIECES]; /* [1, 2] */ + int64_t poly_coef[FF_DOVI_MAX_PIECES][3]; /* x^0, x^1, x^2 */ + /* AV_DOVI_MAPPING_MMR */ + uint8_t mmr_order[FF_DOVI_MAX_PIECES]; /* [1, 3] */ + int64_t mmr_constant[FF_DOVI_MAX_PIECES]; + int64_t mmr_coef[FF_DOVI_MAX_PIECES][3/* order - 1 */][7]; +} AVDOVIReshapingCurve; + +enum AVDOVINLQMethod { + AV_DOVI_NLQ_NONE = -1, + AV_DOVI_NLQ_LINEAR_DZ = 0, +}; + +/** + * Coefficients of the non-linear inverse quantization. For the interpretation + * of these, see ETSI GS CCM 001. + */ +typedef struct AVDOVINLQParams { + uint64_t nlq_offset; + uint64_t vdr_in_max; + /* AV_DOVI_NLQ_LINEAR_DZ */ + uint64_t linear_deadzone_slope; + uint64_t linear_deadzone_threshold; +} AVDOVINLQParams; + +/** + * Dolby Vision RPU data mapping parameters. + */ +typedef struct AVDOVIDataMapping { + uint8_t vdr_rpu_id; + uint8_t mapping_color_space; + uint8_t mapping_chroma_format_idc; + AVDOVIReshapingCurve curves[3]; /* per component */ + + /* Non-linear inverse quantization */ + enum AVDOVINLQMethod nlq_method_idc; + uint32_t num_x_partitions; + uint32_t num_y_partitions; + AVDOVINLQParams nlq[3]; /* per component */ +} AVDOVIDataMapping; + +typedef struct AVDOVIColorMetadata { + uint8_t dm_metadata_id; + int scene_refresh_flag; + + /** + * Coefficients of the custom Dolby Vision IPT-PQ matrices. These are to be + * used instead of the matrices indicated by the frame's colorspace tags. + * The output of rgb_to_lms_matrix is to be fed into a BT.2020 LMS->RGB + * matrix based on a Hunt-Pointer-Estevez transform, but without any + * crosstalk. (See the definition of the ICtCp colorspace for more + * information.) + */ + AVRational ycc_to_rgb_matrix[9]; /* before PQ linearization */ + AVRational ycc_to_rgb_offset[3]; /* input offset of neutral value */ + AVRational rgb_to_lms_matrix[9]; /* after PQ linearization */ + + /** + * Extra signal metadata (see Dolby patents for more info). + */ + uint16_t signal_eotf; + uint16_t signal_eotf_param0; + uint16_t signal_eotf_param1; + uint32_t signal_eotf_param2; + uint8_t signal_bit_depth; + uint8_t signal_color_space; + uint8_t signal_chroma_format; + uint8_t signal_full_range_flag; /* [0, 3] */ + uint16_t source_min_pq; + uint16_t source_max_pq; + uint16_t source_diagonal; +} AVDOVIColorMetadata; + +typedef struct AVDOVIMetadata { + AVDOVIRpuDataHeader header; + AVDOVIDataMapping mapping; + AVDOVIColorMetadata color; +} AVDOVIMetadata; + #endif /* AVUTIL_DOVI_META_H */ diff --git a/libavutil/frame.c b/libavutil/frame.c index 0912ad9131..8997c85e35 100644 --- a/libavutil/frame.c +++ b/libavutil/frame.c @@ -729,6 +729,7 @@ const char *av_frame_side_data_name(enum AVFrameSideDataType type) case AV_FRAME_DATA_FILM_GRAIN_PARAMS: return "Film grain parameters"; case AV_FRAME_DATA_DETECTION_BBOXES: return "Bounding boxes for object detection and classification"; case AV_FRAME_DATA_DOVI_RPU_BUFFER: return "Dolby Vision RPU Data"; + case AV_FRAME_DATA_DOVI_METADATA: return "Dolby Vision Metadata"; } return NULL; } diff --git a/libavutil/frame.h b/libavutil/frame.h index 3f295f6b9e..18e239f870 100644 --- a/libavutil/frame.h +++ b/libavutil/frame.h @@ -189,11 +189,18 @@ enum AVFrameSideDataType { AV_FRAME_DATA_DETECTION_BBOXES, /** - * Dolby Vision RPU data, suitable for passing to x265 + * Dolby Vision RPU raw data, suitable for passing to x265 * or other libraries. Array of uint8_t, with NAL emulation * bytes intact. */ AV_FRAME_DATA_DOVI_RPU_BUFFER, + + /** + * Parsed Dolby Vision metadata, suitable for passing to a software + * implementation. The payload is the AVDOVIMetadata struct defined in + * libavutil/dovi_meta.h. + */ + AV_FRAME_DATA_DOVI_METADATA, }; enum AVActiveFormatDescription { diff --git a/libavutil/version.h b/libavutil/version.h index 017fc277a6..678401fcf5 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -79,7 +79,7 @@ */ #define LIBAVUTIL_VERSION_MAJOR 57 -#define LIBAVUTIL_VERSION_MINOR 10 +#define LIBAVUTIL_VERSION_MINOR 11 #define LIBAVUTIL_VERSION_MICRO 101 #define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \ From patchwork Wed Dec 8 10:12:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 32175 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp150126iog; Wed, 8 Dec 2021 02:13:42 -0800 (PST) X-Google-Smtp-Source: ABdhPJxnBk09ZsvrXi4VXNW4nXCYnj33iNOHd4IOJoRXRrkkOgBeh9lPjSiUQxW9ZyZqpR1HzfZi X-Received: by 2002:a17:906:4fc8:: with SMTP id i8mr6344481ejw.427.1638958422374; Wed, 08 Dec 2021 02:13:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638958422; cv=none; d=google.com; s=arc-20160816; b=sogcLWicOr/VRohkQ0e9tBebHvu2+n5y0qeKNJwinNHqjL4+XKZt/s2L1v3WPi+esH 5HgDgTRn5D54VECKanmfOCeLqYrOXCyfBl1e/M8I12AlfAR8i7fzqInSuC7IFGA4rI/K dUXCfWW7rwMVNLO/0qjvDjGzDYHR0zNcKbeE8HzoZ13ic2w3QZFbTXHayGRtc+ofs+3w K8Y/1kc6Xj/m8J4wpACMbo4zyMo3NY7YZs0gbgKAtpSV6Gijd5PVeJfp6uWpWjHHgzrP t87d4jNTNpofKxoKP093P25wg3J5Oy7d2pcLzldTaiEs8IMSqoNT5X5wZkbDGp4mL70q KOOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=t5If4v6hdUg1iRMtX3lWakarFSWYoauVCdEWMEvKwAI=; b=fUGTp4VWYcbxmG/XV4HSI6rHd0AuTLWOHhqTUJ73qVSVLGg4IYrMCxuLv2JUeT5zZQ VeZ0yCJsoSlFt5J8R1rSmdjaQEa4MaxygbOEZzx3hjVPq6TwgkQHF6iQrD5jcfqDK5jf ELcaV6HhS8AFewCJY/InKpRGqs1zA9a1Adye7e0pByvZsfZdJyrYN2wMg5xcSHOj507g byPdE3bhuebHeryuQBCI6JuedLLvQVPwHOdBLIFdE9CMd6zBYwPJuI6IzFiQwKjub8vM p09AogF57xd5QXVF6X8yGqq+Nq5Z8DiHY0oo12dLrzPiJRK+J0d9xN1/Ph1DBHKt8siy n5ug== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=mtbSipoi; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hq4si4004409ejc.66.2021.12.08.02.13.19; Wed, 08 Dec 2021 02:13:42 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=mtbSipoi; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9D49768AE3C; Wed, 8 Dec 2021 12:12:50 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9E99368AB35 for ; Wed, 8 Dec 2021 12:12:42 +0200 (EET) Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id 95226491A1; Wed, 8 Dec 2021 11:12:40 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1638958360; bh=XRfw3O0N8+r9GzE8MtuecKFyvUrVhaUoHe1nvhz3Teo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mtbSipoiDnQxth20khGdbylFVE3dkh0h2fCmdVyliwyS+mwzuYJzskak1DHFQIykk tUMGY4K8ADVtW6QSQIsO98OyXz15JKPFclg/8o+qH2j/fgtQGZDZtqY05XP84/w9SQ Zz8mcyOY0zFI3kZfaJKTGbRIanQcO3Q6CnTp/xZs= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 8 Dec 2021 11:12:35 +0100 Message-Id: <20211208101237.18407-2-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211208101237.18407-1-ffmpeg@haasn.xyz> References: <20211208101237.18407-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 2/4] lavfi/showinfo: Support AV_FRAME_DATA_DOVI_METADATA X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: jFZvwEXT2SKR From: Niklas Haas Signed-off-by: Niklas Haas --- libavfilter/vf_showinfo.c | 108 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) diff --git a/libavfilter/vf_showinfo.c b/libavfilter/vf_showinfo.c index 62c7833247..79294f6891 100644 --- a/libavfilter/vf_showinfo.c +++ b/libavfilter/vf_showinfo.c @@ -27,6 +27,7 @@ #include "libavutil/bswap.h" #include "libavutil/adler32.h" #include "libavutil/display.h" +#include "libavutil/dovi_meta.h" #include "libavutil/imgutils.h" #include "libavutil/internal.h" #include "libavutil/film_grain_params.h" @@ -429,6 +430,110 @@ static void dump_sei_film_grain_params_metadata(AVFilterContext *ctx, const AVFr } } +static void dump_dovi_metadata(AVFilterContext *ctx, const AVFrameSideData *sd) +{ + const AVDOVIMetadata *dovi = (const AVDOVIMetadata *) sd->data; + const AVDOVIRpuDataHeader *hdr = &dovi->header; + const AVDOVIDataMapping *mapping = &dovi->mapping; + const AVDOVIColorMetadata *color = &dovi->color; + + av_log(ctx, AV_LOG_INFO, "Dolby Vision RPU metadata:\n"); + av_log(ctx, AV_LOG_INFO, " rpu_type=%"PRIu8"; ", hdr->rpu_type); + av_log(ctx, AV_LOG_INFO, "rpu_format=%"PRIu16"; ", hdr->rpu_format); + av_log(ctx, AV_LOG_INFO, "vdr_rpu_profile=%"PRIu8"; ", hdr->vdr_rpu_profile); + av_log(ctx, AV_LOG_INFO, "vdr_rpu_level=%"PRIu8"; ", hdr->vdr_rpu_level); + av_log(ctx, AV_LOG_INFO, "chroma_resampling_explicit_filter_flag=%d; ", hdr->chroma_resampling_explicit_filter_flag); + av_log(ctx, AV_LOG_INFO, "coef_data_type=%"PRIu8"; ", hdr->coef_data_type); + av_log(ctx, AV_LOG_INFO, "coef_log2_denom=%"PRIu8"; ", hdr->coef_log2_denom); + av_log(ctx, AV_LOG_INFO, "vdr_rpu_normalized_idc=%"PRIu8"; ", hdr->vdr_rpu_normalized_idc); + av_log(ctx, AV_LOG_INFO, "bl_video_full_range_flag=%d; ", hdr->bl_video_full_range_flag); + av_log(ctx, AV_LOG_INFO, "bl_bit_depth=%"PRIu8"; ", hdr->bl_bit_depth); + av_log(ctx, AV_LOG_INFO, "el_bit_depth=%"PRIu8"; ", hdr->el_bit_depth); + av_log(ctx, AV_LOG_INFO, "vdr_bit_depth=%"PRIu8"; ", hdr->vdr_bit_depth); + av_log(ctx, AV_LOG_INFO, "spatial_resampling_filter_flag=%d; ", hdr->spatial_resampling_filter_flag); + av_log(ctx, AV_LOG_INFO, "el_spatial_resampling_filter_flag=%d; ", hdr->el_spatial_resampling_filter_flag); + av_log(ctx, AV_LOG_INFO, "disable_residual_flag=%d\n", hdr->disable_residual_flag); + + av_log(ctx, AV_LOG_INFO, " data mapping: "); + av_log(ctx, AV_LOG_INFO, "vdr_rpu_id=%"PRIu8"; ", mapping->vdr_rpu_id); + av_log(ctx, AV_LOG_INFO, "mapping_color_space=%"PRIu8"; ", mapping->mapping_color_space); + av_log(ctx, AV_LOG_INFO, "mapping_chroma_format_idc=%"PRIu8"; ", mapping->mapping_chroma_format_idc); + av_log(ctx, AV_LOG_INFO, "nlq_method_idc=%d; ", (int) mapping->nlq_method_idc); + av_log(ctx, AV_LOG_INFO, "num_x_partitions=%"PRIu32"; ", mapping->num_x_partitions); + av_log(ctx, AV_LOG_INFO, "num_y_partitions=%"PRIu32"\n", mapping->num_y_partitions); + + for (int c = 0; c < 3; c++) { + const AVDOVIReshapingCurve *curve = &mapping->curves[c]; + const AVDOVINLQParams *nlq = &mapping->nlq[c]; + av_log(ctx, AV_LOG_INFO, " channel %d: ", c); + av_log(ctx, AV_LOG_INFO, "pivots={ "); + for (int i = 0; i < curve->num_pivots; i++) + av_log(ctx, AV_LOG_INFO, "%"PRIu16" ", curve->pivots[i]); + av_log(ctx, AV_LOG_INFO, "}; mapping_idc={ "); + for (int i = 0; i < curve->num_pivots - 1; i++) + av_log(ctx, AV_LOG_INFO, "%d ", (int) curve->mapping_idc[i]); + av_log(ctx, AV_LOG_INFO, "}; poly_order={ "); + for (int i = 0; i < curve->num_pivots - 1; i++) + av_log(ctx, AV_LOG_INFO, "%"PRIu8" ", curve->poly_order[i]); + av_log(ctx, AV_LOG_INFO, "}; poly_coef={ "); + for (int i = 0; i < curve->num_pivots - 1; i++) { + av_log(ctx, AV_LOG_INFO, "{%"PRIi64", %"PRIi64", %"PRIi64"} ", + curve->poly_coef[i][0], + curve->poly_coef[i][1], + curve->poly_coef[i][2]); + } + + av_log(ctx, AV_LOG_INFO, "}; mmr_order={ "); + for (int i = 0; i < curve->num_pivots - 1; i++) + av_log(ctx, AV_LOG_INFO, "%"PRIu8" ", curve->mmr_order[i]); + av_log(ctx, AV_LOG_INFO, "}; mmr_constant={ "); + for (int i = 0; i < curve->num_pivots - 1; i++) + av_log(ctx, AV_LOG_INFO, "%"PRIi64" ", curve->mmr_constant[i]); + av_log(ctx, AV_LOG_INFO, "}; mmr_coef={ "); + for (int i = 0; i < curve->num_pivots - 1; i++) { + av_log(ctx, AV_LOG_INFO, "{"); + for (int j = 0; j < curve->mmr_order[i]; j++) { + for (int k = 0; k < 7; k++) + av_log(ctx, AV_LOG_INFO, "%"PRIi64" ", curve->mmr_coef[i][j][k]); + } + av_log(ctx, AV_LOG_INFO, "} "); + } + + av_log(ctx, AV_LOG_INFO, "}; nlq_offset=%"PRIu64"; ", nlq->nlq_offset); + av_log(ctx, AV_LOG_INFO, "vdr_in_max=%"PRIu64"; ", nlq->vdr_in_max); + switch (mapping->nlq_method_idc) { + case AV_DOVI_NLQ_LINEAR_DZ: + av_log(ctx, AV_LOG_INFO, "linear_deadzone_slope=%"PRIu64"; ", nlq->linear_deadzone_slope); + av_log(ctx, AV_LOG_INFO, "linear_deadzone_threshold=%"PRIu64"\n", nlq->linear_deadzone_threshold); + break; + } + } + + av_log(ctx, AV_LOG_INFO, " color metadata: "); + av_log(ctx, AV_LOG_INFO, "dm_metadata_id=%"PRIu8"; ", color->dm_metadata_id); + av_log(ctx, AV_LOG_INFO, "scene_refresh_flag=%d; ", color->scene_refresh_flag); + av_log(ctx, AV_LOG_INFO, "ycc_to_rgb_matrix={ "); + for (int i = 0; i < 9; i++) + av_log(ctx, AV_LOG_INFO, "%f ", av_q2d(color->ycc_to_rgb_matrix[i])); + av_log(ctx, AV_LOG_INFO, "}; ycc_to_rgb_offset={ "); + for (int i = 0; i < 3; i++) + av_log(ctx, AV_LOG_INFO, "%f ", av_q2d(color->ycc_to_rgb_offset[i])); + av_log(ctx, AV_LOG_INFO, "}; rgb_to_lms_matrix={ "); + for (int i = 0; i < 9; i++) + av_log(ctx, AV_LOG_INFO, "%f ", av_q2d(color->rgb_to_lms_matrix[i])); + av_log(ctx, AV_LOG_INFO, "}; signal_eotf=%"PRIu16"; ", color->signal_eotf); + av_log(ctx, AV_LOG_INFO, "signal_eotf_param0=%"PRIu16"; ", color->signal_eotf_param0); + av_log(ctx, AV_LOG_INFO, "signal_eotf_param1=%"PRIu16"; ", color->signal_eotf_param1); + av_log(ctx, AV_LOG_INFO, "signal_eotf_param2=%"PRIu32"; ", color->signal_eotf_param2); + av_log(ctx, AV_LOG_INFO, "signal_bit_depth=%"PRIu8"; ", color->signal_bit_depth); + av_log(ctx, AV_LOG_INFO, "signal_color_space=%"PRIu8"; ", color->signal_color_space); + av_log(ctx, AV_LOG_INFO, "signal_chroma_format=%"PRIu8"; ", color->signal_chroma_format); + av_log(ctx, AV_LOG_INFO, "signal_full_range_flag=%"PRIu8"; ", color->signal_full_range_flag); + av_log(ctx, AV_LOG_INFO, "source_min_pq=%"PRIu16"; ", color->source_min_pq); + av_log(ctx, AV_LOG_INFO, "source_max_pq=%"PRIu16"; ", color->source_max_pq); + av_log(ctx, AV_LOG_INFO, "source_diagonal=%"PRIu16"; ", color->source_diagonal); +} + static void dump_color_property(AVFilterContext *ctx, AVFrame *frame) { const char *color_range_str = av_color_range_name(frame->color_range); @@ -617,6 +722,9 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame) case AV_FRAME_DATA_FILM_GRAIN_PARAMS: dump_sei_film_grain_params_metadata(ctx, sd); break; + case AV_FRAME_DATA_DOVI_METADATA: + dump_dovi_metadata(ctx, sd); + break; default: av_log(ctx, AV_LOG_WARNING, "unknown side data type %d " "(%"SIZE_SPECIFIER" bytes)\n", sd->type, sd->size); From patchwork Wed Dec 8 10:12:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 32174 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp149945iog; Wed, 8 Dec 2021 02:13:32 -0800 (PST) X-Google-Smtp-Source: ABdhPJylCwq73HwelO+aSM8/DHn8ZL9yMFp9HdKF/Iy0F2onghBHOlVX9DJfZRytlB5OFmbAz9l5 X-Received: by 2002:a05:6402:1e90:: with SMTP id f16mr17856652edf.91.1638958412156; Wed, 08 Dec 2021 02:13:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638958412; cv=none; d=google.com; s=arc-20160816; b=UWobKnypYgvcEpYRYu2e2pmepra7xTPIKxOA7VsOM7L98SEVeaXONRbXcJQQRWU8Ks Gt+YAW9x5LDfBa6kDxRDRm+CxoEFqPGhyIPs+W1cx0kLRg7yBuzSht3QW5dkoo940PCD i1jQ1wfLvNv6mnhchkI0TAUgCYQqTxGqAMtQBuo69lqoecBTw4ubJVf6D1hZlcMI8zzw 6+cO9N96cO/Qjlxxqvhp2DEvtG0kcUJVQ+Kn90idG4ZT7/WDpTmrCnQPj5dFyaUw0Ewo Ep8s1BRWtXIA6r23xbnkySepEeaKMVybDewUoG9j6l6+I+2b1Ecx3f+b0/kKBIU3eAz4 DkOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=JqYd1CcB/5q0KEXUl3Xw32XRSyiPkVn5yxJSb/TIl14=; b=mhny6qna5dEYHA2euv3CuwaYuVhUb342vc+7rjxdqGXv4Q/seFgQEAiLIm5k++nHup clyuUzoGh151d44CCoyKU0TuT2hXR2EvdGW52fo7DoV70WDwvlp7BOkVTpF3inEVLqf4 o2DheQsHo4PZBSVucrqDA01p/PhzlYQQ7SWv5KlpgYdgxt6SqB/jM7aSnvIYHb7wzeNf Dn7dXIhBbD71zYcYGPaLuQ2v6jhs9vz2W/r4ihUtwVo6eJZ1wnZVOqfHFCr9XeDqIzns MrRyuGL5Y3ufgYNjC32KsbUtooSn5YwCYGtXbdW+Uu78iArpJQeGQZ8fui3bgPyPaP4r C0dg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=bx4FizQq; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b6si3450221eja.133.2021.12.08.02.13.31; Wed, 08 Dec 2021 02:13:32 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=bx4FizQq; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8AD9568AB35; Wed, 8 Dec 2021 12:12:51 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9717C6801E6 for ; Wed, 8 Dec 2021 12:12:42 +0200 (EET) Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id D1D3B491C0; Wed, 8 Dec 2021 11:12:40 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1638958360; bh=2GtbF2nO9rEH4M54nv/JNgVobDE5h25aFpiqbKdGMTE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bx4FizQqUK7u0iSiHk72HExgBuoMBnz46yE2Tv72TQs3I1IhUdlZ0kqmBEIj5KQSc I8nVAivoP9/7ovWhSNFZxe8pIKsh43lGL9IbfQP3vC9vCJkzDqs3M0uKiJkZT+ho3L RXlvTD8/rTGaFBJBpzuLmja+xMMPDpEl520BfFWY= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 8 Dec 2021 11:12:36 +0100 Message-Id: <20211208101237.18407-3-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211208101237.18407-1-ffmpeg@haasn.xyz> References: <20211208101237.18407-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 3/4] lavc: Implement Dolby Vision RPU parsing X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: wL2E3ER4uDq1 From: Niklas Haas Based on a mixture of guesswork, partial documentation in patents, and reverse engineering of real-world samples. Confirmed working for all the samples I've thrown at it. Contains some annoying machinery to persist these values in between frames, which is needed in theory even though I've never actually seen a sample that relies on it in practice. May or may not work. Notable omissions: - CRC32 verificaiton. This is based on the MPEG2 CRC32 type, which does not seem to be implemented in lavc. (And I don't care enough to do so) - Linear interpolation support. Nothing documents this (beyond its existence) and no samples use it, so impossible to implement. - All of the extension metadata blocks, but these contain values that are basically congruent with ST2094, HDR10, or other existing forms of side data, so I will defer parsing/attaching them to a future commit. Heavily influenced by https://github.com/quietvoid/dovi_tool Documentation drawn from US Patent 10,701,399 B2 and ETSI GS CCM 001 Signed-off-by: Niklas Haas --- libavcodec/Makefile | 1 + libavcodec/dovi.c | 369 ++++++++++++++++++++++++++++++++++++++++++++ libavcodec/dovi.h | 62 ++++++++ 3 files changed, 432 insertions(+) create mode 100644 libavcodec/dovi.c create mode 100644 libavcodec/dovi.h diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 4122a9b144..c327e0d764 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -40,6 +40,7 @@ OBJS = ac3_parser.o \ d3d11va.o \ decode.o \ dirac.o \ + dovi.o \ dv_profile.o \ encode.o \ imgconvert.o \ diff --git a/libavcodec/dovi.c b/libavcodec/dovi.c new file mode 100644 index 0000000000..7e90092f0d --- /dev/null +++ b/libavcodec/dovi.c @@ -0,0 +1,369 @@ +/* + * Dolby Vision RPU decoder + * + * Copyright (C) 2021 Jan Ekström + * Copyright (C) 2021 Niklas Haas + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "dovi.h" +#include "golomb.h" +#include "get_bits.h" + +enum { + RPU_COEFF_FIXED = 0, + RPU_COEFF_FLOAT = 1, +}; + +/** + * Private contents of vdr_ref. + */ +typedef struct DOVIVdrRef { + AVDOVIDataMapping mapping; + AVDOVIColorMetadata color; +} DOVIVdrRef; + +void ff_dovi_ctx_unref(DOVIContext *s) +{ + for (int i = 0; i < DOVI_MAX_DM_ID; i++) + av_buffer_unref(&s->vdr_ref[i]); + + /* Preserve the AVCodecContext explicitly, reset everything else */ + *s = (DOVIContext) { + .avctx = s->avctx, + }; +} + +int ff_dovi_ctx_replace(DOVIContext *s, const DOVIContext *s0) +{ + int ret; + for (int i = 0; i < DOVI_MAX_DM_ID; i++) { + if ((ret = av_buffer_replace(&s->vdr_ref[i], s0->vdr_ref[i])) < 0) { + s->mapping = NULL; /* sanity */ + s->color = NULL; + return ret; + } + } + + s->mapping = s0->mapping; + s->color = s0->color; + return 0; +} + +static inline uint64_t get_ue_coef(GetBitContext *gb, const AVDOVIRpuDataHeader *hdr) +{ + uint64_t ipart; + union { uint32_t u32; float f32; } fpart; + + switch (hdr->coef_data_type) { + case RPU_COEFF_FIXED: + ipart = get_ue_golomb_long(gb); + fpart.u32 = get_bits_long(gb, hdr->coef_log2_denom); + return (ipart << hdr->coef_log2_denom) + fpart.u32; + + case RPU_COEFF_FLOAT: + fpart.u32 = get_bits_long(gb, 32); + return fpart.f32 * (1 << hdr->coef_log2_denom); + } + + return 0; /* unreachable */ +} + +static inline int64_t get_se_coef(GetBitContext *gb, const AVDOVIRpuDataHeader *hdr) +{ + int64_t ipart; + union { uint32_t u32; float f32; } fpart; + + switch (hdr->coef_data_type) { + case RPU_COEFF_FIXED: + ipart = get_se_golomb_long(gb); + fpart.u32 = get_bits_long(gb, hdr->coef_log2_denom); + return (ipart << hdr->coef_log2_denom) + fpart.u32; + + case RPU_COEFF_FLOAT: + fpart.u32 = get_bits_long(gb, 32); + return fpart.f32 * (1 << hdr->coef_log2_denom); + } + + return 0; /* unreachable */ +} + +#define VALIDATE(VAR, MIN, MAX) \ + do { \ + if (VAR < MIN || VAR > MAX) { \ + av_log(s->avctx, AV_LOG_ERROR, "RPU validation failed: " \ + #MIN" <= "#VAR" = %d <= "#MAX"\n", (int) VAR); \ + goto fail; \ + } \ + } while (0) + +int ff_dovi_rpu_parse(DOVIContext *s, const uint8_t *rpu, size_t rpu_size) +{ + AVDOVIRpuDataHeader *hdr = &s->header; + GetBitContext *gb = &(GetBitContext){0}; + DOVIVdrRef *vdr; + int ret; + + uint8_t nal_prefix; + uint8_t rpu_type; + uint8_t vdr_seq_info_present; + uint8_t vdr_dm_metadata_present; + uint8_t use_prev_vdr_rpu; + uint8_t use_nlq; + if ((ret = init_get_bits8(gb, rpu, rpu_size)) < 0) + return ret; + + /* RPU header, common values */ + nal_prefix = get_bits(gb, 8); + VALIDATE(nal_prefix, 25, 25); + rpu_type = get_bits(gb, 6); + if (rpu_type != 2) { + av_log(s->avctx, AV_LOG_WARNING, "Unrecognized RPU type " + "%"PRIu8", ignoring\n", rpu_type); + return 0; + } + + hdr->rpu_type = rpu_type; + hdr->rpu_format = get_bits(gb, 11); + + /* Values specific to RPU type 2 */ + hdr->vdr_rpu_profile = get_bits(gb, 4); + hdr->vdr_rpu_level = get_bits(gb, 4); + + vdr_seq_info_present = get_bits1(gb); + if (vdr_seq_info_present) { + hdr->chroma_resampling_explicit_filter_flag = get_bits1(gb); + hdr->coef_data_type = get_bits(gb, 2); + VALIDATE(hdr->coef_data_type, RPU_COEFF_FIXED, RPU_COEFF_FLOAT); + switch (hdr->coef_data_type) { + case RPU_COEFF_FIXED: + hdr->coef_log2_denom = get_ue_golomb(gb); + VALIDATE(hdr->coef_log2_denom, 13, 32); + break; + case RPU_COEFF_FLOAT: + hdr->coef_log2_denom = 32; /* arbitrary, choose maximum precision */ + break; + } + + hdr->vdr_rpu_normalized_idc = get_bits(gb, 2); + hdr->bl_video_full_range_flag = get_bits1(gb); + + if ((hdr->rpu_format & 0x700) == 0) { + int bl_bit_depth_minus8 = get_ue_golomb_31(gb); + int el_bit_depth_minus8 = get_ue_golomb_31(gb); + int vdr_bit_depth_minus8 = get_ue_golomb_31(gb); + VALIDATE(bl_bit_depth_minus8, 0, 8); + VALIDATE(el_bit_depth_minus8, 0, 8); + VALIDATE(vdr_bit_depth_minus8, 0, 8); + hdr->bl_bit_depth = bl_bit_depth_minus8 + 8; + hdr->el_bit_depth = el_bit_depth_minus8 + 8; + hdr->vdr_bit_depth = vdr_bit_depth_minus8 + 8; + hdr->spatial_resampling_filter_flag = get_bits1(gb); + skip_bits(gb, 3); /* reserved_zero_3bits */ + hdr->el_spatial_resampling_filter_flag = get_bits1(gb); + hdr->disable_residual_flag = get_bits1(gb); + } + } + + if (!hdr->bl_bit_depth) { + av_log(s->avctx, AV_LOG_ERROR, "Missing RPU VDR sequence info?\n"); + goto fail; + } + + vdr_dm_metadata_present = get_bits1(gb); + use_prev_vdr_rpu = get_bits1(gb); + use_nlq = (hdr->rpu_format & 0x700) == 0 && !hdr->disable_residual_flag; + if (av_dovi_profile(hdr) == 5 && use_nlq) { + av_log(s->avctx, AV_LOG_ERROR, "Profile 5 RPUs should not use NLQ\n"); + goto fail; + } + + if (use_prev_vdr_rpu) { + int prev_vdr_rpu_id = get_ue_golomb_31(gb); + VALIDATE(prev_vdr_rpu_id, 0, DOVI_MAX_DM_ID); + if (!s->vdr_ref[prev_vdr_rpu_id]) { + av_log(s->avctx, AV_LOG_ERROR, "Unknown previous RPU ID: %u\n", + prev_vdr_rpu_id); + goto fail; + } + vdr = (DOVIVdrRef *) s->vdr_ref[prev_vdr_rpu_id]->data; + s->mapping = &vdr->mapping; + } else { + int vdr_rpu_id = get_ue_golomb_31(gb); + VALIDATE(vdr_rpu_id, 0, DOVI_MAX_DM_ID); + if (!s->vdr_ref[vdr_rpu_id]) { + s->vdr_ref[vdr_rpu_id] = av_buffer_allocz(sizeof(DOVIVdrRef)); + if (!s->vdr_ref[vdr_rpu_id]) + return AVERROR(ENOMEM); + } + + vdr = (DOVIVdrRef *) s->vdr_ref[vdr_rpu_id]->data; + s->mapping = &vdr->mapping; + vdr->mapping = (AVDOVIDataMapping) { + .vdr_rpu_id = vdr_rpu_id, + .mapping_color_space = get_ue_golomb_31(gb), + .mapping_chroma_format_idc = get_ue_golomb_31(gb), + }; + + for (int c = 0; c < 3; c++) { + AVDOVIReshapingCurve *curve = &vdr->mapping.curves[c]; + int num_pivots_minus_2 = get_ue_golomb_31(gb); + int pivot = 0; + + VALIDATE(num_pivots_minus_2, 0, FF_DOVI_MAX_PIECES - 1); + curve->num_pivots = num_pivots_minus_2 + 2; + for (int i = 0; i < curve->num_pivots; i++) { + pivot += get_bits(gb, hdr->bl_bit_depth); + curve->pivots[i] = av_clip_uint16(pivot); + } + } + + if (use_nlq) { + vdr->mapping.nlq_method_idc = get_bits(gb, 3); + /** + * The patent mentions another legal value, NLQ_MU_LAW, but it's + * not documented anywhere how to parse or apply that type of NLQ. + */ + VALIDATE(vdr->mapping.nlq_method_idc, 0, AV_DOVI_NLQ_LINEAR_DZ); + } else { + vdr->mapping.nlq_method_idc = AV_DOVI_NLQ_NONE; + } + + vdr->mapping.num_x_partitions = get_ue_golomb_long(gb) + 1; + vdr->mapping.num_y_partitions = get_ue_golomb_long(gb) + 1; + /* End of rpu_data_header(), start of vdr_rpu_data_payload() */ + + for (int c = 0; c < 3; c++) { + AVDOVIReshapingCurve *curve = &vdr->mapping.curves[c]; + for (int i = 0; i < curve->num_pivots - 1; i++) { + int mapping_idc = get_ue_golomb_31(gb); + VALIDATE(mapping_idc, 0, 1); + curve->mapping_idc[i] = mapping_idc; + switch (mapping_idc) { + case AV_DOVI_MAPPING_POLYNOMIAL: { + int poly_order_minus1 = get_ue_golomb_31(gb); + VALIDATE(poly_order_minus1, 0, 1); + curve->poly_order[i] = poly_order_minus1 + 1; + if (poly_order_minus1 == 0) { + int linear_interp_flag = get_bits1(gb); + if (linear_interp_flag) { + /* lack of documentation/samples */ + av_log(s->avctx, AV_LOG_WARNING, "Linear " + "interpolation is unimplemented, please " + "open a bug report with the sample.\n"); + ff_dovi_ctx_unref(s); + return AVERROR_PATCHWELCOME; + } + } + for (int k = 0; k <= curve->poly_order[i]; k++) + curve->poly_coef[i][k] = get_se_coef(gb, hdr); + break; + } + case AV_DOVI_MAPPING_MMR: { + int mmr_order_minus1 = get_bits(gb, 2); + VALIDATE(mmr_order_minus1, 0, 2); + curve->mmr_order[i] = mmr_order_minus1 + 1; + curve->mmr_constant[i] = get_se_coef(gb, hdr); + for (int j = 0; j < curve->mmr_order[i]; j++) { + for (int k = 0; k < 7; k++) + curve->mmr_coef[i][j][k] = get_se_coef(gb, hdr); + } + break; + } + } + } + } + + if (use_nlq) { + for (int c = 0; c < 3; c++) { + AVDOVINLQParams *nlq = &vdr->mapping.nlq[c]; + nlq->nlq_offset = get_bits(gb, hdr->el_bit_depth); + nlq->vdr_in_max = get_ue_coef(gb, hdr); + switch (vdr->mapping.nlq_method_idc) { + case AV_DOVI_NLQ_LINEAR_DZ: + nlq->linear_deadzone_slope = get_ue_coef(gb, hdr); + nlq->linear_deadzone_threshold = get_ue_coef(gb, hdr); + break; + } + } + } + } + + if (vdr_dm_metadata_present) { + AVDOVIColorMetadata *color; + int affected_dm_id = get_ue_golomb_31(gb); + int current_dm_id = get_ue_golomb_31(gb); + VALIDATE(affected_dm_id, 0, DOVI_MAX_DM_ID); + VALIDATE(current_dm_id, 0, DOVI_MAX_DM_ID); + if (!s->vdr_ref[affected_dm_id]) { + s->vdr_ref[affected_dm_id] = av_buffer_allocz(sizeof(DOVIVdrRef)); + if (!s->vdr_ref[affected_dm_id]) + return AVERROR(ENOMEM); + } + + if (!s->vdr_ref[current_dm_id]) { + av_log(s->avctx, AV_LOG_ERROR, "Unknown previous RPU DM ID: %u\n", + current_dm_id); + goto fail; + } + + /* Update current pointer based on current_dm_id */ + vdr = (DOVIVdrRef *) s->vdr_ref[current_dm_id]->data; + s->color = &vdr->color; + + /* Update values of affected_dm_id */ + vdr = (DOVIVdrRef *) s->vdr_ref[affected_dm_id]->data; + color = &vdr->color; + color->dm_metadata_id = affected_dm_id; + color->scene_refresh_flag = get_ue_golomb_31(gb); + for (int i = 0; i < 9; i++) + color->ycc_to_rgb_matrix[i] = av_make_q(get_sbits(gb, 16), 1 << 13); + for (int i = 0; i < 3; i++) { + int denom = av_dovi_profile(hdr) == 4 ? (1 << 30) : (1 << 28); + unsigned offset = get_bits_long(gb, 32); + while (offset > INT_MAX) { + /* Ensure the result fits inside AVRational */ + offset >>= 1; + denom >>= 1; + } + color->ycc_to_rgb_offset[i] = av_make_q(offset, denom); + } + for (int i = 0; i < 9; i++) + color->rgb_to_lms_matrix[i] = av_make_q(get_sbits(gb, 16), 1 << 14); + + color->signal_eotf = get_bits(gb, 16); + color->signal_eotf_param0 = get_bits(gb, 16); + color->signal_eotf_param1 = get_bits(gb, 16); + color->signal_eotf_param2 = get_bits_long(gb, 32); + color->signal_bit_depth = get_bits(gb, 5); + VALIDATE(color->signal_bit_depth, 8, 16); + color->signal_color_space = get_bits(gb, 2); + color->signal_chroma_format = get_bits(gb, 2); + color->signal_full_range_flag = get_bits(gb, 2); + color->source_min_pq = get_bits(gb, 12); + color->source_max_pq = get_bits(gb, 12); + color->source_diagonal = get_bits(gb, 10); + } + + /* FIXME: verify CRC32, requires implementation of AV_CRC_32_MPEG_2 */ + return 0; + +fail: + ff_dovi_ctx_unref(s); /* don't leak potentially invalid state */ + return AVERROR(EINVAL); +} diff --git a/libavcodec/dovi.h b/libavcodec/dovi.h new file mode 100644 index 0000000000..f6fc7be066 --- /dev/null +++ b/libavcodec/dovi.h @@ -0,0 +1,62 @@ +/* + * Dolby Vision RPU decoder + * + * Copyright (C) 2021 Jan Ekström + * Copyright (C) 2021 Niklas Haas + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_DOVI_H +#define AVCODEC_DOVI_H + +#include "libavutil/buffer.h" +#include "libavutil/dovi_meta.h" + +#include "avcodec.h" + +#define DOVI_MAX_DM_ID 15 +typedef struct DOVIContext { + AVCodecContext *avctx; + AVBufferRef *vdr_ref[DOVI_MAX_DM_ID+1]; ///< decoded VDR data mappings + + /** + * Currently active RPU data header, updates on every dovi_rpu_parse(). + */ + AVDOVIRpuDataHeader header; + + /** + * Currently active data mappings, or NULL. Points into memory owned by the + * corresponding rpu/vdr_ref, which becomes invalid on the next call to + * dovi_rpu_parse. + */ + const AVDOVIDataMapping *mapping; + const AVDOVIColorMetadata *color; +} DOVIContext; + +void ff_dovi_ctx_unref(DOVIContext *s); +int ff_dovi_ctx_replace(DOVIContext *s, const DOVIContext *s0); + +/** + * Parse the contents of a Dovi RPU NAL and update the parsed values in the + * DOVIContext struct. + * + * Returns 0 or an error code. + */ +int ff_dovi_rpu_parse(DOVIContext *s, const uint8_t *rpu, size_t rpu_size); + +#endif /* AVCODEC_DOVI_H */ From patchwork Wed Dec 8 10:12:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 32172 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp149694iog; Wed, 8 Dec 2021 02:13:17 -0800 (PST) X-Google-Smtp-Source: ABdhPJzG9J2jKzVgCoyMgFlUMEc7SIuTVSHt6sc/pHq75e80/NAte17zbbMgb+TrhIZy5PF1XyJy X-Received: by 2002:a05:6402:2789:: with SMTP id b9mr17562259ede.28.1638958396862; Wed, 08 Dec 2021 02:13:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638958396; cv=none; d=google.com; s=arc-20160816; b=I3Wp+9GpT8BFsc92KIzxn55kbEV6FC3P24/3hD2fCvfnGsBKLDtlTePSvLJ3/8Nh97 Ttls0/NpSfaSBw3E1PsE7xcQzngCBVAMzLJS//midhPxm8+DbiNh/cMg1KYDljoT3sUh uXlpPd8npOUJjgX2p+XeQ0aIb89cU4EB013LzhNv2hggRGF73yOKHEQTkDdBkxgMSRTT yRR+XVI6Nxfj57bI9D3NCzxxiL9CJeEkE6R2AyKq15dDJAbPN24rSLiJOZrEDdSCh46B bv9fz+v85oTVsV40JRX49UbPbTlthnxXhJB+E9B7DW6jOGcvTQt0KfptdOadIZAP83qB A2FQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=heFtEgofXmQ+x0PGlHGYo1ysnQYZp+5YJLfq0a5Edqo=; b=b2ajHutdCqcq3cYxBTwFjlQI3P5M6pzw1qTuqnEidPk1bR3A3sSkaU9ISqLfX587ZQ xxYfyGr2NZV2LaFXOUxTd+etIFQcyhycTxXPlqVGPzLZe9BK1B1lEvJ60URk+4QWmxMo N1cJVWNVRLMPGnYJYfFIaKXETsg5dx05rfgue/8aKYYB3isBGEiQBvYhjWabniNkWFgT oQd6lQjiyiVKEeEDy0GrRx8+7PA2b3vX5NuTHHGVyqDyeqOkLGYSBznMvwhj1QLButA2 PTNHz1UTt3m1XFW+a8JtSAIYSTdJbpZ3WXFPHfV48LO1pjyrK5P+w83OTWLNQG1z5Y9N xK+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=h0Hxexzz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id m15si4780244edd.538.2021.12.08.02.13.16; Wed, 08 Dec 2021 02:13:16 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=h0Hxexzz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B0AAE68ADF6; Wed, 8 Dec 2021 12:12:48 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 995E268A92E for ; Wed, 8 Dec 2021 12:12:42 +0200 (EET) Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id 20CC9491C2; Wed, 8 Dec 2021 11:12:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1638958361; bh=AV8DgsWUeCgBf8zXYlL20LnvHGHsz7MXKtpPKmUpSpU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=h0Hxexzzt9qFyRkbfkjkUSHeYe/wP5n9YU3OhfAbJ/bzfddFm3NOLJFONM8wM53FP U93wnIE+piP0IZ3g2/4m1tvr3Sow0onJrfduI+FzsLaS6zGuhYUu5Zww6P4+PB+8TG M5eYCDeYc+WBGu+XMn43a7K8NkYAyu39Cj8WSqz8= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 8 Dec 2021 11:12:37 +0100 Message-Id: <20211208101237.18407-4-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211208101237.18407-1-ffmpeg@haasn.xyz> References: <20211208101237.18407-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 4/4] lavc/hevcdec: Parse DOVI RPU NALs X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: upFadd6gB5QX From: Niklas Haas And expose the parsed values as frame side data. Signed-off-by: Niklas Haas --- libavcodec/hevcdec.c | 34 +++++++++++++++++++++++++++++++--- libavcodec/hevcdec.h | 2 ++ 2 files changed, 33 insertions(+), 3 deletions(-) diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index 46d9edf8eb..926764bdf4 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -38,6 +38,7 @@ #include "bswapdsp.h" #include "bytestream.h" #include "cabac_functions.h" +#include "dovi.h" #include "golomb.h" #include "hevc.h" #include "hevc_data.h" @@ -2967,6 +2968,19 @@ static int set_side_data(HEVCContext *s) s->rpu_buf = NULL; } + if (s->dovi_ctx.mapping && s->dovi_ctx.color) { + AVDOVIMetadata *dovi; + AVFrameSideData *sd = av_frame_new_side_data(out, AV_FRAME_DATA_DOVI_METADATA, + sizeof(AVDOVIMetadata)); + if (!sd) + return AVERROR(ENOMEM); + + dovi = (AVDOVIMetadata *) sd->data; + memcpy(&dovi->header, &s->dovi_ctx.header, sizeof(dovi->header)); + memcpy(&dovi->mapping, s->dovi_ctx.mapping, sizeof(dovi->mapping)); + memcpy(&dovi->color, s->dovi_ctx.color, sizeof(dovi->color)); + } + return 0; } @@ -3298,16 +3312,23 @@ static int decode_nal_units(HEVCContext *s, const uint8_t *buf, int length) if (s->pkt.nb_nals > 1 && s->pkt.nals[s->pkt.nb_nals - 1].type == HEVC_NAL_UNSPEC62 && s->pkt.nals[s->pkt.nb_nals - 1].size > 2 && !s->pkt.nals[s->pkt.nb_nals - 1].nuh_layer_id && !s->pkt.nals[s->pkt.nb_nals - 1].temporal_id) { + H2645NAL *nal = &s->pkt.nals[s->pkt.nb_nals - 1]; if (s->rpu_buf) { av_buffer_unref(&s->rpu_buf); av_log(s->avctx, AV_LOG_WARNING, "Multiple Dolby Vision RPUs found in one AU. Skipping previous.\n"); } - s->rpu_buf = av_buffer_alloc(s->pkt.nals[s->pkt.nb_nals - 1].raw_size - 2); + s->rpu_buf = av_buffer_alloc(nal->raw_size - 2); if (!s->rpu_buf) return AVERROR(ENOMEM); - memcpy(s->rpu_buf->data, s->pkt.nals[s->pkt.nb_nals - 1].raw_data + 2, s->pkt.nals[s->pkt.nb_nals - 1].raw_size - 2); + memcpy(s->rpu_buf->data, nal->raw_data + 2, nal->raw_size - 2); + ret = ff_dovi_rpu_parse(&s->dovi_ctx, nal->data + 2, nal->size - 2); + if (ret < 0) { + av_buffer_unref(&s->rpu_buf); + av_log(s->avctx, AV_LOG_WARNING, "Error parsing DOVI NAL unit.\n"); + goto fail; + } } /* decode the NAL units */ @@ -3325,7 +3346,7 @@ static int decode_nal_units(HEVCContext *s, const uint8_t *buf, int length) if (ret < 0) { av_log(s->avctx, AV_LOG_WARNING, "Error parsing NAL unit #%d.\n", i); - goto fail; + /* ignore */ } } @@ -3553,6 +3574,7 @@ static av_cold int hevc_decode_free(AVCodecContext *avctx) pic_arrays_free(s); + ff_dovi_ctx_unref(&s->dovi_ctx); av_buffer_unref(&s->rpu_buf); av_freep(&s->md5_ctx); @@ -3637,6 +3659,7 @@ static av_cold int hevc_init_context(AVCodecContext *avctx) ff_bswapdsp_init(&s->bdsp); + s->dovi_ctx.avctx = avctx; s->context_initialized = 1; s->eos = 0; @@ -3745,6 +3768,10 @@ static int hevc_update_thread_context(AVCodecContext *dst, if (ret < 0) return ret; + ret = ff_dovi_ctx_replace(&s->dovi_ctx, &s0->dovi_ctx); + if (ret < 0) + return ret; + s->sei.frame_packing = s0->sei.frame_packing; s->sei.display_orientation = s0->sei.display_orientation; s->sei.mastering_display = s0->sei.mastering_display; @@ -3801,6 +3828,7 @@ static void hevc_decode_flush(AVCodecContext *avctx) HEVCContext *s = avctx->priv_data; ff_hevc_flush_dpb(s); ff_hevc_reset_sei(&s->sei); + ff_dovi_ctx_unref(&s->dovi_ctx); av_buffer_unref(&s->rpu_buf); s->max_ra = INT_MAX; s->eos = 1; diff --git a/libavcodec/hevcdec.h b/libavcodec/hevcdec.h index 870ff178d4..5b04a8ad83 100644 --- a/libavcodec/hevcdec.h +++ b/libavcodec/hevcdec.h @@ -32,6 +32,7 @@ #include "avcodec.h" #include "bswapdsp.h" #include "cabac.h" +#include "dovi.h" #include "get_bits.h" #include "hevcpred.h" #include "h2645_parse.h" @@ -574,6 +575,7 @@ typedef struct HEVCContext { int nuh_layer_id; AVBufferRef *rpu_buf; ///< 0 or 1 Dolby Vision RPUs. + DOVIContext dovi_ctx; ///< Dolby Vision decoding context } HEVCContext; /**