From patchwork Tue May 28 03:29:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Wang X-Patchwork-Id: 13309 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id E3161448923 for ; Tue, 28 May 2019 06:30:45 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C833B680903; Tue, 28 May 2019 06:30:45 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7E9536801B1 for ; Tue, 28 May 2019 06:30:38 +0300 (EEST) Received: by mail-pg1-f196.google.com with SMTP id n2so10055848pgp.11 for ; Mon, 27 May 2019 20:30:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+/6gUP4E9Dk/p8bQmFrHnTQrMTtUcmhUm/uNdmDKevg=; b=PxrVgbfmv27QEge5IHmK2RR8+HMXfLgqlH/V9Z1mX0Ei2+FDazpLcDDiuxZ+xReGRK 0KZYVRtratud5mP20PdDVSTyLjpL6sTFMOKjXOaz7MgGxLTLXS34XD6JMF5mw507y28v 4sZCcsIu77hQRvBOdkxCgxkr36CLsV/WgTgLS+/tAh0jRkc+2NJRHfOZzxAyr81g8j9o F9UPwidVraXU1aE0u8jwZxYIYk1965evlKsgcQnU14a4DkRNOyZFefGndO/0eWBb/XKk OYKQwjnk9M0tX7VgI/8wAON9CGQOC1FtwHDOCuMYL+2vBaofReCoNZmHWkz0bQvnNAJ2 jWsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+/6gUP4E9Dk/p8bQmFrHnTQrMTtUcmhUm/uNdmDKevg=; b=TWbnnpkX7OU8+jRXa+twvAI2eI8tdYXJnBiva8QCsFvMR+jSYkR/+WGy+XrqwT/8ra 1HfH2s1BQ7fDOOV5Mikl6Jg5XuW/i1qgT3T127w/F5JawWRKTiixSBqHS9cSZOH2Tf7l P1zxHdHVx5LRXVRPBpUdN4SaHptwx1t1KRD60q87/P+gK4V/evFaCdKDqPRg9W1qnpvJ v/mQaT9wmywVMJUi1Wh5yKoIZZ8AIDGhsNTTt52UYqaWmBPmy0cOrEvgFbR/NjHaQPA4 a9c+Poh4UwvqOpPR5BB2FmE28tMUqSQVqVKC/W0yXPQuE2s17ZE6Oh8pslW2r8uO29Xx oCcg== X-Gm-Message-State: APjAAAU2vzUlNjfGxDsHB70N1cIN5a/lepUBHEcrTGd4AS2vxiiyEfUw C1Ep68W3rrS4TwNrV3ge8gE2ZZfuIWI= X-Google-Smtp-Source: APXvYqzATYyG9YB0UxE1L/GB2oFBmrdG3Y09hMB99yiJSFnLi/kKPkj5bwyYnUjsQKPN4DC4fqgZOA== X-Received: by 2002:a62:65c1:: with SMTP id z184mr122446521pfb.130.1559014236214; Mon, 27 May 2019 20:30:36 -0700 (PDT) Received: from localhost.localdomain ([47.90.99.151]) by smtp.gmail.com with ESMTPSA id v4sm14095774pff.45.2019.05.27.20.30.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 27 May 2019 20:30:35 -0700 (PDT) From: lance.lmwang@gmail.com To: ffmpeg-devel@ffmpeg.org Date: Tue, 28 May 2019 11:29:52 +0800 Message-Id: <20190528032953.3864-2-lance.lmwang@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190528032953.3864-1-lance.lmwang@gmail.com> References: <20190522065954.62328-2-lance.lmwang@gmail.com> <20190528032953.3864-1-lance.lmwang@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 2/3] avcodec/nvenc: add master display and light level sei for HDR10 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Limin Wang , jamrial@gmail.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Limin Wang The testing command for the HDR10 output with nvenc: $ ./ffmpeg_g -y -i 4K.mp4 -c:v hevc_nvenc -color_primaries bt2020 -colorspace bt2020_ncl -color_trc smpte2084 -sei hdr10 \ -master_display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,50)" -max_cll "0, 0" test.ts Please notice it is preferable to use the frame sei side data than master_display and max_cll paramters config Signed-off-by: Limin Wang --- libavcodec/nvenc.c | 114 ++++++++++++++++++++++++++++++++++++++++ libavcodec/nvenc.h | 18 +++++++ libavcodec/nvenc_hevc.c | 11 ++++ 3 files changed, 143 insertions(+) diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c index 75dda6d689..30cf4df928 100644 --- a/libavcodec/nvenc.c +++ b/libavcodec/nvenc.c @@ -22,6 +22,8 @@ #include "config.h" #include "nvenc.h" +#include "hevc_sei.h" +#include "put_bits.h" #include "libavutil/hwcontext_cuda.h" #include "libavutil/hwcontext.h" @@ -30,6 +32,7 @@ #include "libavutil/avassert.h" #include "libavutil/mem.h" #include "libavutil/pixdesc.h" +#include "libavutil/mastering_display_metadata.h" #include "internal.h" #define CHECK_CU(x) FF_CUDA_CHECK_DL(avctx, dl_fn->cuda_dl, x) @@ -1491,6 +1494,46 @@ av_cold int ff_nvenc_encode_init(AVCodecContext *avctx) ctx->data_pix_fmt = avctx->pix_fmt; } + ctx->display_primaries_x[0] = 13250; + ctx->display_primaries_y[0] = 34500; + ctx->display_primaries_x[1] = 7500; + ctx->display_primaries_y[1] = 3000; + ctx->display_primaries_x[2] = 34000; + ctx->display_primaries_y[2] = 16000; + ctx->white_point_x = 15635; + ctx->white_point_y = 16450; + ctx->max_display_mastering_luminance = 10000000; + ctx->min_display_mastering_luminance = 500; + ctx->max_content_light_level = 0; + ctx->max_pic_average_light_level = 0; + if (ctx->master_display) { + ret = sscanf(ctx->master_display, "G(%hu,%hu)B(%hu,%hu)R(%hu,%hu)WP(%hu,%hu)L(%u,%u)", + &ctx->display_primaries_x[0], &ctx->display_primaries_y[0], + &ctx->display_primaries_x[1], &ctx->display_primaries_y[1], + &ctx->display_primaries_x[2], &ctx->display_primaries_y[2], + &ctx->white_point_x, &ctx->white_point_y, + &ctx->max_display_mastering_luminance, &ctx->min_display_mastering_luminance); + if (ret != 10) { + ret = sscanf(ctx->master_display, "G[%hu,%hu]B[%hu,%hu]R[%hu,%hu]WP[%hu,%hu]L[%u,%u]", + &ctx->display_primaries_x[0], &ctx->display_primaries_y[0], + &ctx->display_primaries_x[1], &ctx->display_primaries_y[1], + &ctx->display_primaries_x[2], &ctx->display_primaries_y[2], + &ctx->white_point_x, &ctx->white_point_y, + &ctx->max_display_mastering_luminance, &ctx->min_display_mastering_luminance); + } + + if (ret != 10) { + av_log(avctx, AV_LOG_INFO, "Failed to parse master display(%s)\n", ctx->master_display); + } + } + + if (ctx->max_cll) { + ret = sscanf(ctx->max_cll, "%hu,%hu", &ctx->max_content_light_level, &ctx->max_pic_average_light_level); + if (ret != 2) { + av_log(avctx, AV_LOG_INFO, "Failed to parse max cll(%s)\n", ctx->max_cll); + } + } + if ((ret = nvenc_load_libraries(avctx)) < 0) return ret; @@ -2110,6 +2153,77 @@ int ff_nvenc_send_frame(AVCodecContext *avctx, const AVFrame *frame) } } + if (ctx->sei & SEI_MASTERING_DISPLAY) { + AVFrameSideData *sd = av_frame_get_side_data(frame, AV_FRAME_DATA_MASTERING_DISPLAY_METADATA); + + if (sd) { + AVMasteringDisplayMetadata *mdm = (AVMasteringDisplayMetadata *)sd->data; + // HEVC uses a g,b,r ordering, which we convert from a more natural r,g,b + const int mapping[3] = {2, 0, 1}; + const int chroma_den = 50000; + const int luma_den = 10000; + + if (mdm->has_primaries && mdm->has_luminance) { + + for (i = 0; i < 3; i++) { + const int j = mapping[i]; + ctx->display_primaries_x[i] = chroma_den * av_q2d(mdm->display_primaries[j][0]); + ctx->display_primaries_y[i] = chroma_den * av_q2d( mdm->display_primaries[j][1]); + } + + ctx->white_point_x = chroma_den * av_q2d(mdm->white_point[0]); + ctx->white_point_y = chroma_den * av_q2d(mdm->white_point[1]); + ctx->max_display_mastering_luminance = luma_den * av_q2d(mdm->max_luminance); + ctx->min_display_mastering_luminance = luma_den * av_q2d(mdm->min_luminance); + } + } + + sei_data[sei_count].payloadSize = 24; + sei_data[sei_count].payloadType = HEVC_SEI_TYPE_MASTERING_DISPLAY_INFO; + sei_data[sei_count].payload = av_mallocz(sei_data[sei_count].payloadSize); + if (sei_data[sei_count].payload) { + PutBitContext pb; + + init_put_bits(&pb, sei_data[sei_count].payload, sei_data[sei_count].payloadSize); + for (i = 0; i < 3; i++) { + put_bits(&pb, 16, ctx->display_primaries_x[i]); + put_bits(&pb, 16, ctx->display_primaries_y[i]); + } + put_bits(&pb, 16, ctx->white_point_x); + put_bits(&pb, 16, ctx->white_point_y); + put_bits(&pb, 32, ctx->max_display_mastering_luminance); + put_bits(&pb, 32, ctx->min_display_mastering_luminance); + flush_put_bits(&pb); + + sei_count ++; + } + } + + if (ctx->sei & SEI_CONTENT_LIGHT_LEVEL) { + AVFrameSideData *sd = av_frame_get_side_data(frame, AV_FRAME_DATA_CONTENT_LIGHT_LEVEL); + + if (sd) { + AVContentLightMetadata *clm = (AVContentLightMetadata *)sd->data; + + ctx->max_content_light_level = FFMIN(clm->MaxCLL, 65535); + ctx->max_pic_average_light_level = FFMIN(clm->MaxFALL, 65535); + } + + sei_data[sei_count].payloadSize = 4; + sei_data[sei_count].payloadType = HEVC_SEI_TYPE_CONTENT_LIGHT_LEVEL_INFO; + sei_data[sei_count].payload = av_mallocz(sei_data[sei_count].payloadSize); + if (sei_data[sei_count].payload) { + PutBitContext pb; + + init_put_bits(&pb, sei_data[sei_count].payload, sei_data[sei_count].payloadSize); + put_bits(&pb, 16, ctx->max_content_light_level); + put_bits(&pb, 16, ctx->max_pic_average_light_level); + flush_put_bits(&pb); + + sei_count ++; + } + } + nvenc_codec_specific_pic_params(avctx, &pic_params, sei_data, sei_count); } else { pic_params.encodePicFlags = NV_ENC_PIC_FLAG_EOS; diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h index ddd6168409..583c48d090 100644 --- a/libavcodec/nvenc.h +++ b/libavcodec/nvenc.h @@ -54,6 +54,11 @@ typedef void ID3D11Device; #define NVENC_HAVE_HEVC_BFRAME_REF_MODE #endif +enum { + SEI_MASTERING_DISPLAY = 0x08, + SEI_CONTENT_LIGHT_LEVEL = 0x10, +}; + typedef struct NvencSurface { NV_ENC_INPUT_PTR input_surface; @@ -192,6 +197,19 @@ typedef struct NvencContext int coder; int b_ref_mode; int a53_cc; + uint64_t sei; + + char *master_display; + char *max_cll; + uint16_t display_primaries_x[3]; + uint16_t display_primaries_y[3]; + uint16_t white_point_x; + uint16_t white_point_y; + uint32_t max_display_mastering_luminance; + uint32_t min_display_mastering_luminance; + + uint16_t max_content_light_level; + uint16_t max_pic_average_light_level; } NvencContext; int ff_nvenc_encode_init(AVCodecContext *avctx); diff --git a/libavcodec/nvenc_hevc.c b/libavcodec/nvenc_hevc.c index d567d960ba..74ebd03d8e 100644 --- a/libavcodec/nvenc_hevc.c +++ b/libavcodec/nvenc_hevc.c @@ -127,6 +127,17 @@ static const AVOption options[] = { { "each", "", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, VE, "b_ref_mode" }, { "middle", "", 0, AV_OPT_TYPE_CONST, { .i64 = 2 }, 0, 0, VE, "b_ref_mode" }, #endif + { "sei", "Set SEI to include", + OFFSET(sei), AV_OPT_TYPE_FLAGS, + { .i64 = SEI_MASTERING_DISPLAY | SEI_CONTENT_LIGHT_LEVEL }, + 0, INT_MAX, VE, "sei" }, + { "hdr10","Include HDR metadata for mastering display colour volume and content light level information", + 0, AV_OPT_TYPE_CONST, { .i64 = SEI_MASTERING_DISPLAY | SEI_CONTENT_LIGHT_LEVEL }, + 0, 0, VE, "sei" }, + { "master_display", "SMPTE ST 2086 master display color volume info SEI (HDR), the string format is: \"G(x,y)B(x,y)R(x,y)WP(x,y)L(max,min)\"", + OFFSET(master_display), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, VE }, + { "max_cll", "content light level info, the string format is: \"cll, fall\"", + OFFSET(max_cll), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, VE }, { NULL } };