From patchwork Thu May 10 05:47:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xiang, Haihao" X-Patchwork-Id: 8905 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:155:0:0:0:0:0 with SMTP id c82-v6csp662116jad; Wed, 9 May 2018 22:48:25 -0700 (PDT) X-Google-Smtp-Source: AB8JxZqAhhiPW7pMPAYv+1LjwIn1DMPtepZZQUkboW/PtJ7J71Wsmdm978KsOyIoLDGLEA4pC+Xk X-Received: by 2002:a1c:8b0b:: with SMTP id n11-v6mr234090wmd.12.1525931305556; Wed, 09 May 2018 22:48:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525931305; cv=none; d=google.com; s=arc-20160816; b=OLFkDsYQoOwu0dpyn2wAHMFHsOlloQV3wuH4DJxbwcKnEc/LVpXZ1nMx1bFiPnOAtS ttDMs9QcV7fU5afvqhe47ASEApHFEu9siSDEqY28VMnK5L0mk4S+JsXPJYp4hpF9sszI 0ivFOfMaezFBnrrgm8rxRYd902I4m4nlkoyI4xM1M/POLE9umOzSUdOcGy/o+BBC3WSW +I6xuiDiZqvd9HTMwFPcELifzd33ykPzouD5FSlvvzMysrC04qeqj2g+kIGEX74gS+d4 lurUrTSw1ohQbCRdlwNbtaJyw0BZ2SFBvbxKB9ewo/Pv/+nFd6p3t28i5jkldmaMHy/H ZfyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:delivered-to:arc-authentication-results; bh=EASRLgRBhwxf8FoUqu7D9bIehdrqOo5lkhOhhQLgN2Y=; b=K0ureAebb/uflwPgXHI+kdwaDlpnxMGJxMudqP3eAAT8RMeN+6xuXTPyfAcP0A8qBm 6PAGFMROW9Hvkk/m5EfFwilbnGCGTxJGSLf+GnIL8IlAFN0KvtVLVnfIhifZqBQqVvSk koxvfWUsx20uuwRbi5X0EDZA9Au3zU6p3eMFdocK8W9DxuPtCqOY7bF34Xpz0gQWtjln KiSMhPzbTgLvUvGu9h8tHTyhUiUnNdsgI7smPySe3HqvEJIBfdW/4gCUhNpGx9ng1qXu a6svVPEB+TM5azYQrPqP3mmg+DNx51TUfdAnP08AMhQtHqNyzx1v4023f5AvmNo7Pmo0 i6gA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z11-v6si36720wrl.42.2018.05.09.22.48.25; Wed, 09 May 2018 22:48:25 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5A5D368A7D7; Thu, 10 May 2018 08:47:42 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8A9B568A671 for ; Thu, 10 May 2018 08:47:35 +0300 (EEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 May 2018 22:48:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,384,1520924400"; d="scan'208";a="227284023" Received: from xhh-cfl64.sh.intel.com ([10.239.13.24]) by fmsmga005.fm.intel.com with ESMTP; 09 May 2018 22:48:08 -0700 From: Haihao Xiang To: ffmpeg-devel@ffmpeg.org Date: Thu, 10 May 2018 13:47:45 +0800 Message-Id: <20180510054747.7967-2-haihao.xiang@intel.com> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180510054747.7967-1-haihao.xiang@intel.com> References: <20180510054747.7967-1-haihao.xiang@intel.com> Subject: [FFmpeg-devel] [PATCH v3 2/4] vaapi_encode_h265: Insert mastering display colour volume if needed X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Haihao Xiang , Mark Thompson MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" '-sei xxx' is added to control SEI insertion, so far only mastering display colour volume is available for testing. v2: use the mastering display parameters from AVMasteringDisplayMetadata, set SEI_MASTERING_DISPLAY to 8 to match the H.264 part and take VAAPIEncodeH265Context::sei_needed as a ORed value so that we needn't check the correspoding SEI message again when writting the header. v3: fix typo in commit log, change the option to '-sei hdr' and add some comments about the medtadata update Signed-off-by: Haihao Xiang --- libavcodec/vaapi_encode_h265.c | 132 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 131 insertions(+), 1 deletion(-) diff --git a/libavcodec/vaapi_encode_h265.c b/libavcodec/vaapi_encode_h265.c index 5203c6871d..4ff569c21f 100644 --- a/libavcodec/vaapi_encode_h265.c +++ b/libavcodec/vaapi_encode_h265.c @@ -24,15 +24,20 @@ #include "libavutil/avassert.h" #include "libavutil/common.h" #include "libavutil/opt.h" +#include "libavutil/mastering_display_metadata.h" #include "avcodec.h" #include "cbs.h" #include "cbs_h265.h" #include "hevc.h" +#include "hevc_sei.h" #include "internal.h" #include "put_bits.h" #include "vaapi_encode.h" +enum { + SEI_MASTERING_DISPLAY = 0x08, +}; typedef struct VAAPIEncodeH265Context { unsigned int ctu_width; @@ -47,6 +52,9 @@ typedef struct VAAPIEncodeH265Context { H265RawSPS sps; H265RawPPS pps; H265RawSlice slice; + H265RawSEI sei; + + H265RawSEIMasteringDiplayColourVolume mastering_display; int64_t last_idr_frame; int pic_order_cnt; @@ -58,6 +66,7 @@ typedef struct VAAPIEncodeH265Context { CodedBitstreamContext *cbc; CodedBitstreamFragment current_access_unit; int aud_needed; + int sei_needed; } VAAPIEncodeH265Context; typedef struct VAAPIEncodeH265Options { @@ -65,6 +74,7 @@ typedef struct VAAPIEncodeH265Options { int aud; int profile; int level; + int sei; } VAAPIEncodeH265Options; @@ -175,6 +185,64 @@ fail: return err; } +static int vaapi_encode_h265_write_extra_header(AVCodecContext *avctx, + VAAPIEncodePicture *pic, + int index, int *type, + char *data, size_t *data_len) +{ + VAAPIEncodeContext *ctx = avctx->priv_data; + VAAPIEncodeH265Context *priv = ctx->priv_data; + CodedBitstreamFragment *au = &priv->current_access_unit; + int err, i; + + if (priv->sei_needed) { + if (priv->aud_needed) { + err = vaapi_encode_h265_add_nal(avctx, au, &priv->aud); + if (err < 0) + goto fail; + priv->aud_needed = 0; + } + + memset(&priv->sei, 0, sizeof(priv->sei)); + priv->sei.nal_unit_header = (H265RawNALUnitHeader) { + .nal_unit_type = HEVC_NAL_SEI_PREFIX, + .nuh_layer_id = 0, + .nuh_temporal_id_plus1 = 1, + }; + + i = 0; + + if (priv->sei_needed & SEI_MASTERING_DISPLAY) { + priv->sei.payload[i].payload_type = HEVC_SEI_TYPE_MASTERING_DISPLAY_INFO; + priv->sei.payload[i].payload.mastering_display = priv->mastering_display; + ++i; + } + + priv->sei.payload_count = i; + av_assert0(priv->sei.payload_count > 0); + + err = vaapi_encode_h265_add_nal(avctx, au, &priv->sei); + if (err < 0) + goto fail; + priv->sei_needed = 0; + + err = vaapi_encode_h265_write_access_unit(avctx, data, data_len, au); + if (err < 0) + goto fail; + + ff_cbs_fragment_uninit(priv->cbc, au); + + *type = VAEncPackedHeaderRawData; + return 0; + } else { + return AVERROR_EOF; + } + +fail: + ff_cbs_fragment_uninit(priv->cbc, au); + return err; +} + static int vaapi_encode_h265_init_sequence_params(AVCodecContext *avctx) { VAAPIEncodeContext *ctx = avctx->priv_data; @@ -587,6 +655,57 @@ static int vaapi_encode_h265_init_picture_params(AVCodecContext *avctx, priv->aud_needed = 0; } + priv->sei_needed = 0; + + // Only look for the metadata on I/IDR frame on the output. We + // may force an IDR frame on the output where the medadata gets + // changed on the input frame. + if ((opt->sei & SEI_MASTERING_DISPLAY) && + (pic->type == PICTURE_TYPE_I || pic->type == PICTURE_TYPE_IDR)) { + AVFrameSideData *sd = + av_frame_get_side_data(pic->input_image, + AV_FRAME_DATA_MASTERING_DISPLAY_METADATA); + + if (sd) { + AVMasteringDisplayMetadata *mdm = + (AVMasteringDisplayMetadata *)sd->data; + + // SEI is needed when both the primaries and luminance are set + if (mdm->has_primaries && mdm->has_luminance) { + const int mapping[3] = {1, 2, 0}; + const int chroma_den = 50000; + const int luma_den = 10000; + + for (i = 0; i < 3; i++) { + const int j = mapping[i]; + priv->mastering_display.display_primaries_x[i] = + FFMIN(lrint(chroma_den * + av_q2d(mdm->display_primaries[j][0])), + chroma_den); + priv->mastering_display.display_primaries_y[i] = + FFMIN(lrint(chroma_den * + av_q2d(mdm->display_primaries[j][1])), + chroma_den); + } + + priv->mastering_display.white_point_x = + FFMIN(lrint(chroma_den * av_q2d(mdm->white_point[0])), + chroma_den); + priv->mastering_display.white_point_y = + FFMIN(lrint(chroma_den * av_q2d(mdm->white_point[1])), + chroma_den); + + priv->mastering_display.max_display_mastering_luminance = + lrint(luma_den * av_q2d(mdm->max_luminance)); + priv->mastering_display.min_display_mastering_luminance = + FFMIN(lrint(luma_den * av_q2d(mdm->min_luminance)), + priv->mastering_display.max_display_mastering_luminance); + + priv->sei_needed |= SEI_MASTERING_DISPLAY; + } + } + } + vpic->decoded_curr_pic = (VAPictureHEVC) { .picture_id = pic->recon_surface, .pic_order_cnt = priv->pic_order_cnt, @@ -895,6 +1014,8 @@ static const VAAPIEncodeType vaapi_encode_type_h265 = { .slice_header_type = VAEncPackedHeaderHEVC_Slice, .write_slice_header = &vaapi_encode_h265_write_slice_header, + + .write_extra_header = &vaapi_encode_h265_write_extra_header, }; static av_cold int vaapi_encode_h265_init(AVCodecContext *avctx) @@ -943,7 +1064,8 @@ static av_cold int vaapi_encode_h265_init(AVCodecContext *avctx) ctx->va_packed_headers = VA_ENC_PACKED_HEADER_SEQUENCE | // VPS, SPS and PPS. - VA_ENC_PACKED_HEADER_SLICE; // Slice headers. + VA_ENC_PACKED_HEADER_SLICE | // Slice headers. + VA_ENC_PACKED_HEADER_MISC; // SEI ctx->surface_width = FFALIGN(avctx->width, 16); ctx->surface_height = FFALIGN(avctx->height, 16); @@ -1003,6 +1125,14 @@ static const AVOption vaapi_encode_h265_options[] = { { LEVEL("6.2", 186) }, #undef LEVEL + { "sei", "Set SEI to include", + OFFSET(sei), AV_OPT_TYPE_FLAGS, + { .i64 = SEI_MASTERING_DISPLAY }, + 0, INT_MAX, FLAGS, "sei" }, + { "hdr", "Include HDR metadata for mastering display colour volume", + 0, AV_OPT_TYPE_CONST, { .i64 = SEI_MASTERING_DISPLAY }, + INT_MIN, INT_MAX, FLAGS, "sei" }, + { NULL }, };