From patchwork Wed Jan 16 19:54:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mohammad Izadi X-Patchwork-Id: 11773 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 93DBC44D461 for ; Wed, 16 Jan 2019 22:00:27 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AB99068AA82; Wed, 16 Jan 2019 22:00:15 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 885AC68AA47 for ; Wed, 16 Jan 2019 22:00:09 +0200 (EET) Received: by mail-pf1-f194.google.com with SMTP id r136so3576638pfc.6 for ; Wed, 16 Jan 2019 12:00:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RS4H/0+IDFDMq4f8nr0/oCp022VuVNtbcXljqeCpjhk=; b=qeEGYLEu8wwgRJ6vhRneD7jecVNTugJ8QZDzq8PFvDkDayrAxgkxo55fK0V3ChduVz 944PVz0oK4jh4kZEPEIKGQDx8hsHvMPGuufcfA/f3H4QkbeWkpUPASRivA+7K4ZlH9wI JtUh5j8XGbdKHuDznLM61fCPD0F18Oat+90fe7GqYGKgspn6x71PSpw038mGvwBNqhgd 7nxesOGRfU/kiY3nwPTLtFd1I0stq6HtZL8Zlg95Ex3Bqm1O9jVGrst4BdAP4edx2MZP PTQbSoegBIldvsLTgqeJ02De/grYjFenL8luIzccKsCm/I1QEWehQm+GsagGg7ytMBSO 0xTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RS4H/0+IDFDMq4f8nr0/oCp022VuVNtbcXljqeCpjhk=; b=mVYX33v92ntN+AyXPjO5kegB0hc21QC3ij0ywKPM9uZyiPtle3xCcSadI6y3kWb03C YjhImxz3L4IZ2sZK2igg3RDvFQghBngGZxX9uzVxiOZqHjG1LmZ5ArWRmZ9Y+VRapeyc bQg+yvImfMwics8Skpajcgk44WtdmFXHhAHQgktczKIEKSZ5FeR8I9KQxGseyxaV766f sQmaFVhdTd/HgIklDT4HRmdevQiKhtwHx2yeIAMehn9erjvxsZaRqrSlyc4HsE0IsONr yxiVhmD7vzItt6slk5ibmG74hf4HkPeFjpEmXIFl/nzurCeO8xXiJul2qBVydV1UcdeY kfKw== X-Gm-Message-State: AJcUuke1/RCXymLB+f4p4Wc8BwdQwlsyYaIN4I1RaQ1S8KrxUBqxZrgd 9I7SaIyriXx073FLw0bFoNvPhIE= X-Google-Smtp-Source: ALg8bN6NOYX13EptZI/MLhHl3nlsY3HrnWxgnHZJVp+dL/UZchRb2qc6nE7gUWuBwgc0unHzHQ1YTQ== X-Received: by 2002:a63:6103:: with SMTP id v3mr10135320pgb.75.1547668495082; Wed, 16 Jan 2019 11:54:55 -0800 (PST) Received: from izadi.mtv.corp.google.com ([2620:0:1000:4011:3bcf:9bbf:ad2c:87b2]) by smtp.gmail.com with ESMTPSA id x3sm11863858pgt.45.2019.01.16.11.54.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 11:54:53 -0800 (PST) From: Mohammad Izadi To: ffmpeg-devel@ffmpeg.org Date: Wed, 16 Jan 2019 11:54:42 -0800 Message-Id: <20190116195442.104284-1-moh.izadi@gmail.com> X-Mailer: git-send-email 2.20.1.97.g81188d93c3-goog In-Reply-To: References: MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] Support HDR dynamic metdata (HDR10+) in HEVC decoder. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Mohammad Izadi Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" --- libavcodec/hevc_sei.c | 211 ++++++++++++++++++++++++++++++++++++++++-- libavcodec/hevc_sei.h | 6 ++ libavcodec/hevcdec.c | 22 +++++ 3 files changed, 233 insertions(+), 6 deletions(-) diff --git a/libavcodec/hevc_sei.c b/libavcodec/hevc_sei.c index c59bd4321e..265e3f4dd1 100644 --- a/libavcodec/hevc_sei.c +++ b/libavcodec/hevc_sei.c @@ -25,6 +25,7 @@ #include "golomb.h" #include "hevc_ps.h" #include "hevc_sei.h" +#include "libavutil/hdr_dynamic_metadata.h" static int decode_nal_sei_decoded_picture_hash(HEVCSEIPictureHash *s, GetBitContext *gb) { @@ -206,10 +207,179 @@ static int decode_registered_user_data_closed_caption(HEVCSEIA53Caption *s, GetB return 0; } -static int decode_nal_sei_user_data_registered_itu_t_t35(HEVCSEI *s, GetBitContext *gb, +static int decode_registered_user_data_dynamic_hdr_plus(AVDynamicHDRPlus *s, GetBitContext *gb, + void *logctx, int size) +{ + const int luminance_den = 10000; + const int peak_luminance_den = 15; + const int rgb_den = 100000; + const int fraction_pixel_den = 1000; + const int knee_point_den = 4095; + const int bezier_anchor_den = 1023; + const int saturation_weight_den = 8; + + int w, i, j; + + if (get_bits_left(gb) < size * 8) + return AVERROR_INVALIDDATA; + + if (get_bits_left(gb) < 2) + return AVERROR_INVALIDDATA; + s->num_windows = get_bits(gb, 2); + if (s->num_windows < 1 || s->num_windows > 3) { + av_log(logctx, AV_LOG_ERROR, "num_windows=%d, must be in [1, 3]\n", + s->num_windows); + return AVERROR_INVALIDDATA; + } + + if (get_bits_left(gb) < ((19 * 8 + 1) * (s->num_windows - 1))) + return AVERROR_INVALIDDATA; + for (w = 1; w < s->num_windows; w++) { + s->params[w].window_upper_left_corner_x.num = get_bits(gb, 16); + s->params[w].window_upper_left_corner_y.num = get_bits(gb, 16); + s->params[w].window_lower_right_corner_x.num = get_bits(gb, 16); + s->params[w].window_lower_right_corner_y.num = get_bits(gb, 16); + // The corners are set to absolute coordinates here. They should be + // converted to the relative coordinates (in [0, 1]) in the decoder. + s->params[w].window_upper_left_corner_x.den = 1; + s->params[w].window_upper_left_corner_y.den = 1; + s->params[w].window_lower_right_corner_x.den = 1; + s->params[w].window_lower_right_corner_y.den = 1; + + s->params[w].center_of_ellipse_x = get_bits(gb, 16); + s->params[w].center_of_ellipse_y = get_bits(gb, 16); + s->params[w].rotation_angle = get_bits(gb, 8); + s->params[w].semimajor_axis_internal_ellipse = get_bits(gb, 16); + s->params[w].semimajor_axis_external_ellipse = get_bits(gb, 16); + s->params[w].semiminor_axis_external_ellipse = get_bits(gb, 16); + s->params[w].overlap_process_option = get_bits(gb, 1); + } + + if (get_bits_left(gb) < 28) + return AVERROR(EINVAL); + s->targeted_system_display_maximum_luminance.num = get_bits(gb, 27); + s->targeted_system_display_maximum_luminance.den = luminance_den; + s->targeted_system_display_actual_peak_luminance_flag = get_bits(gb, 1); + + if (s->targeted_system_display_actual_peak_luminance_flag) { + int rows, cols; + if (get_bits_left(gb) < 10) + return AVERROR(EINVAL); + rows = get_bits(gb, 5); + cols = get_bits(gb, 5); + if (((rows < 2) && (rows > 25)) || ((cols < 2) && (cols > 25))) { + av_log(logctx, AV_LOG_ERROR, "num_rows=%d, num_cols=%d, they must [2, 25] for targeted_system_display_actual_peak_luminance\n", rows, cols); + return AVERROR_INVALIDDATA; + } + s->num_rows_targeted_system_display_actual_peak_luminance = rows; + s->num_cols_targeted_system_display_actual_peak_luminance = cols; + + if (get_bits_left(gb) < (rows * cols * 4)) + return AVERROR(EINVAL); + + for (i = 0; i < rows; i++) { + for (j = 0; j < cols; j++) { + s->targeted_system_display_actual_peak_luminance[i][j].num = get_bits(gb, 4); + s->targeted_system_display_actual_peak_luminance[i][j].den = peak_luminance_den; + } + } + } + for (w = 0; w < s->num_windows; w++) { + if (get_bits_left(gb) < (3 * 17 + 17 + 4)) + return AVERROR(EINVAL); + for (i = 0; i < 3; i++) { + s->params[w].maxscl[i].num = get_bits(gb, 17); + s->params[w].maxscl[i].den = rgb_den; + } + s->params[w].average_maxrgb.num = get_bits(gb, 17); + s->params[w].average_maxrgb.den = rgb_den; + s->params[w].num_distribution_maxrgb_percentiles = get_bits(gb, 4); + + if (get_bits_left(gb) < + (s->params[w].num_distribution_maxrgb_percentiles * 24)) + return AVERROR(EINVAL); + for (i = 0; i < s->params[w].num_distribution_maxrgb_percentiles; i++) { + s->params[w].distribution_maxrgb[i].percentage = get_bits(gb, 7); + s->params[w].distribution_maxrgb[i].percentile.num = get_bits(gb, 17); + s->params[w].distribution_maxrgb[i].percentile.den = rgb_den; + } + + if (get_bits_left(gb) < 10) + return AVERROR(EINVAL); + s->params[w].fraction_bright_pixels.num = get_bits(gb, 10); + s->params[w].fraction_bright_pixels.den = fraction_pixel_den; + } + if (get_bits_left(gb) < 1) + return AVERROR(EINVAL); + s->mastering_display_actual_peak_luminance_flag = get_bits(gb, 1); + if (s->mastering_display_actual_peak_luminance_flag) { + int rows, cols; + if (get_bits_left(gb) < 10) + return AVERROR(EINVAL); + rows = get_bits(gb, 5); + cols = get_bits(gb, 5); + if (((rows < 2) && (rows > 25)) || ((cols < 2) && (cols > 25))) { + av_log(logctx, AV_LOG_ERROR, "num_rows=%d, num_cols=%d, they must be in [2, 25] for mastering_display_actual_peak_luminance\n", rows, cols); + return AVERROR_INVALIDDATA; + } + s->num_rows_mastering_display_actual_peak_luminance = rows; + s->num_cols_mastering_display_actual_peak_luminance = cols; + + if (get_bits_left(gb) < (rows * cols * 4)) + return AVERROR(EINVAL); + + for (i = 0; i < rows; i++) { + for (j = 0; j < cols; j++) { + s->mastering_display_actual_peak_luminance[i][j].num = get_bits(gb, 4); + s->mastering_display_actual_peak_luminance[i][j].den = peak_luminance_den; + } + } + } + + for (w = 0; w < s->num_windows; w++) { + if (get_bits_left(gb) < 1) + return AVERROR(EINVAL); + s->params[w].tone_mapping_flag = get_bits(gb, 1); + if (s->params[w].tone_mapping_flag) { + if (get_bits_left(gb) < 28) + return AVERROR(EINVAL); + s->params[w].knee_point_x.num = get_bits(gb, 12); + s->params[w].knee_point_x.den = knee_point_den; + s->params[w].knee_point_y.num = get_bits(gb, 12); + s->params[w].knee_point_y.den = knee_point_den; + s->params[w].num_bezier_curve_anchors = get_bits(gb, 4); + + if (get_bits_left(gb) < (s->params[w].num_bezier_curve_anchors * 10)) + return AVERROR(EINVAL); + for (i = 0; i < s->params[w].num_bezier_curve_anchors; i++) { + s->params[w].bezier_curve_anchors[i].num = get_bits(gb, 10); + s->params[w].bezier_curve_anchors[i].den = bezier_anchor_den; + } + } + + if (get_bits_left(gb) < 1) + return AVERROR(EINVAL); + s->params[w].color_saturation_mapping_flag = get_bits(gb, 1); + if (s->params[w].color_saturation_mapping_flag) { + if (get_bits_left(gb) < 6) + return AVERROR(EINVAL); + s->params[w].color_saturation_weight.num = get_bits(gb, 6); + s->params[w].color_saturation_weight.den = saturation_weight_den; + } + } + + skip_bits(gb, get_bits_left(gb)); + + return 0; +} + +static int decode_nal_sei_user_data_registered_itu_t_t35(HEVCSEI *s, + GetBitContext *gb, + void *logctx, int size) { - uint32_t country_code; + uint8_t country_code; + uint16_t provider_code; uint32_t user_identifier; if (size < 7) @@ -222,11 +392,39 @@ static int decode_nal_sei_user_data_registered_itu_t_t35(HEVCSEI *s, GetBitConte size--; } - skip_bits(gb, 8); - skip_bits(gb, 8); - + provider_code = get_bits(gb, 16); user_identifier = get_bits_long(gb, 32); + // Check for dynamic metadata - HDR10+(SMPTE 2094-40). + if ((provider_code == 0x003C) && + ((user_identifier & 0xFFFFFF00) == 0x00010400)) { + int err; + size_t size; + AVDynamicHDRPlus *hdr_plus = av_dynamic_hdr_plus_alloc(&size); + if (!hdr_plus) { + return AVERROR(ENOMEM); + } + if (s->dynamic_hdr_plus.info){ + av_buffer_unref(&s->dynamic_hdr_plus.info); + } + s->dynamic_hdr_plus.info = + av_buffer_create((uint8_t*)hdr_plus, size, + av_buffer_default_free, NULL, 0); + if (!s->dynamic_hdr_plus.info) { + av_freep(&hdr_plus); + return AVERROR(ENOMEM); + } + + hdr_plus->itu_t_t35_country_code = country_code; + hdr_plus->application_version = + (uint8_t)((user_identifier & 0x000000FF)); + + err = decode_registered_user_data_dynamic_hdr_plus(hdr_plus, gb, logctx, size); + if (!err) + av_buffer_unref(&s->dynamic_hdr_plus.info); + return err; + } + switch (user_identifier) { case MKBETAG('G', 'A', '9', '4'): return decode_registered_user_data_closed_caption(&s->a53_caption, gb, size); @@ -292,7 +490,7 @@ static int decode_nal_sei_prefix(GetBitContext *gb, void *logctx, HEVCSEI *s, case HEVC_SEI_TYPE_ACTIVE_PARAMETER_SETS: return decode_nal_sei_active_parameter_sets(s, gb, logctx); case HEVC_SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35: - return decode_nal_sei_user_data_registered_itu_t_t35(s, gb, size); + return decode_nal_sei_user_data_registered_itu_t_t35(s, gb, logctx, size); case HEVC_SEI_TYPE_ALTERNATIVE_TRANSFER_CHARACTERISTICS: return decode_nal_sei_alternative_transfer(&s->alternative_transfer, gb); default: @@ -365,4 +563,5 @@ void ff_hevc_reset_sei(HEVCSEI *s) { s->a53_caption.a53_caption_size = 0; av_freep(&s->a53_caption.a53_caption); + av_buffer_unref(&s->dynamic_hdr_plus.info); } diff --git a/libavcodec/hevc_sei.h b/libavcodec/hevc_sei.h index 2fec00ace0..80c56b10bb 100644 --- a/libavcodec/hevc_sei.h +++ b/libavcodec/hevc_sei.h @@ -23,6 +23,7 @@ #include +#include "libavutil/buffer.h" #include "get_bits.h" /** @@ -94,6 +95,10 @@ typedef struct HEVCSEIMasteringDisplay { uint32_t min_luminance; } HEVCSEIMasteringDisplay; +typedef struct HEVCSEIDynamicHDRPlus{ + AVBufferRef *info; +} HEVCSEIDynamicHDRPlus; + typedef struct HEVCSEIContentLight { int present; uint16_t max_content_light_level; @@ -109,6 +114,7 @@ typedef struct HEVCSEI { HEVCSEIPictureHash picture_hash; HEVCSEIFramePacking frame_packing; HEVCSEIDisplayOrientation display_orientation; + HEVCSEIDynamicHDRPlus dynamic_hdr_plus; HEVCSEIPictureTiming picture_timing; HEVCSEIA53Caption a53_caption; HEVCSEIMasteringDisplay mastering_display; diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index 10bf2563c0..a7ed26b9d9 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -28,6 +28,7 @@ #include "libavutil/display.h" #include "libavutil/internal.h" #include "libavutil/mastering_display_metadata.h" +#include "libavutil/hdr_dynamic_metadata.h" #include "libavutil/md5.h" #include "libavutil/opt.h" #include "libavutil/pixdesc.h" @@ -2769,6 +2770,26 @@ static int set_side_data(HEVCContext *s) s->avctx->color_trc = out->color_trc = s->sei.alternative_transfer.preferred_transfer_characteristics; } + if (s->sei.dynamic_hdr_plus.info){ + AVDynamicHDRPlus *metadata = (AVDynamicHDRPlus*)s->sei.dynamic_hdr_plus.info->data; + // Convert coordinates to relative coordinate in [0, 1]. + metadata->params[0].window_upper_left_corner_x.num = 0; + metadata->params[0].window_upper_left_corner_y.num = 0; + metadata->params[0].window_lower_right_corner_x.num = out->width-1; + metadata->params[0].window_lower_right_corner_y.num = out->height-1; + for (int w = 0; w < metadata->num_windows; w++) { + metadata->params[w].window_upper_left_corner_x.den = out->width-1; + metadata->params[w].window_upper_left_corner_y.den = out->height-1; + metadata->params[w].window_lower_right_corner_x.den = out->width-1; + metadata->params[w].window_lower_right_corner_y.den = out->height-1; + } + if (!av_frame_new_side_data_from_buf(out, AV_FRAME_DATA_DYNAMIC_HDR_PLUS, s->sei.dynamic_hdr_plus.info)) { + av_buffer_unref(&s->sei.dynamic_hdr_plus.info); + return AVERROR(ENOMEM); + } + s->sei.dynamic_hdr_plus.info = NULL; + } + return 0; } @@ -3309,6 +3330,7 @@ static av_cold int hevc_decode_free(AVCodecContext *avctx) s->HEVClc = NULL; av_freep(&s->HEVClcList[0]); + ff_hevc_reset_sei(&s->sei); ff_h2645_packet_uninit(&s->pkt); return 0;