From patchwork Mon Nov 21 02:31:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xiang, Haihao" X-Patchwork-Id: 39373 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp1864886pzb; Sun, 20 Nov 2022 18:35:37 -0800 (PST) X-Google-Smtp-Source: AA0mqf6LQ4t4JL5b/rCCbcrZna4DKCHyfgyEzPiAMpafji9HQJbcuOrgkQhConISRZtMSBFDupdQ X-Received: by 2002:a17:906:350e:b0:7ae:f441:6ade with SMTP id r14-20020a170906350e00b007aef4416ademr1846774eja.436.1668998137201; Sun, 20 Nov 2022 18:35:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668998137; cv=none; d=google.com; s=arc-20160816; b=cI10h5rI3CBhpHHBxZeBbi3E2eYYSGlfaSu2/R5EmPKAjgJQ8rJA1c5XlPiTBdmIvZ lO/PC5btu/ym+IH1deiKsRWDJCIqvXrO9RErW/2K3nSy+kah/ckxf6gryrR8T9ZnPGLf XnpmXLiaMv1q8ciQ/Nj4WFYoQaZCHZ9VF1G3MVOBDe1pcCtEm7dkamMv/I+Fp2rJ6WXy FDgI0EkktagMxOr3i4/mNfN4Th3C9sqO9x1Zxd1YU98dIp30/SRPb8VLwoNGuIUPrnoO kvQJIHbzgdA5Sd8CeZXmKqUgPKNKxtJnLhNww/xdPZnuqe2lUq95CNrhQ6UpjOrsGHJ6 p6RQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=PmakWF6If15fbIuH/9tTQR1eupyf3ZEQZxfMmbHbzSg=; b=gAkfigTalWP87HaJUCXYzNQnqUYkDPwjxO+IVnT1uLwrbeMpa+Q7VFqlNHpDnW1A+S rT/6uuEK3NVg9kaQxiraqFknKIcWr5loHjl0ObzBfgPAnRS4lx9nT9AXTvIMTuSkE0XZ r2/5nnKbLbzUQyOlAjzBm+ymLrXs9x1rlfrIY2ZTjOCf4PReTDURLhuw2ua4ywrjp5rl ZjTHm0cj8LuWvhByHXQbqBBvC+euwmoZDycs7/Kef4gbPTw68uTc1laoej3AIRKM9S3G TUAZsnx6dbuCUexBbZEbZYlROf8HQWMXX1xQr7PYymLuHLcZkLWmwR4Gei9UwyVhpPqd BHmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=FYBJu49J; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b6-20020a056402350600b00459f9c3d02bsi8804493edd.22.2022.11.20.18.35.36; Sun, 20 Nov 2022 18:35:37 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=FYBJu49J; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C540F68BCBD; Mon, 21 Nov 2022 04:35:32 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2F5AE68BC90 for ; Mon, 21 Nov 2022 04:35:24 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668998130; x=1700534130; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=BcVzI06hebjBewXrksVRmvjaWmqpc/ARkEXcqaAKhOY=; b=FYBJu49Jiel28q+1E4lu84XvgIz+DQlBw6ANN/50WabPIeZpicuIeAEq wzkbjFJloPflREh4QPoSpTEexxlZSt8K42AnXE8rpMpzIe6gBvHWDtT+9 hFgW18OKS+w80Ts1Dyjb7nZFnIXHCwingqVMSn8ISwyxmNQiDmfFbw/Qb 8IJAzqvBWxKJsN0vcmgvPes+mG82RpICKGk/x1aTJXo+qQJuNQzLVZ1S3 5SYfPV+mElKkwfj/FHYKCUDKme0/URgIeHmMBqNhBo6zEX5aQY/Xsvk1z kdlSuoAHG+8rhuUVn3xecfBHhMvmHMx+1jNmVNCQ7Pv69OYlNz6x6yXks w==; X-IronPort-AV: E=McAfee;i="6500,9779,10537"; a="314606392" X-IronPort-AV: E=Sophos;i="5.96,180,1665471600"; d="scan'208";a="314606392" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2022 18:35:22 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10537"; a="671939409" X-IronPort-AV: E=Sophos;i="5.96,180,1665471600"; d="scan'208";a="671939409" Received: from xhh-tgl64.sh.intel.com ([10.238.2.19]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2022 18:35:21 -0800 From: "Xiang, Haihao" To: ffmpeg-devel@ffmpeg.org Date: Mon, 21 Nov 2022 10:31:24 +0800 Message-Id: <20221121023124.203786-1-haihao.xiang@intel.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] lavc/qsvdec: update HDR side data on output AVFrame X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Haihao Xiang Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 5XwbxqwKsQne From: Haihao Xiang The SDK may provides HDR metadata for HDR streams via mfxExtBuffer attached on output mfxFrameSurface1 Signed-off-by: Haihao Xiang --- libavcodec/qsv_internal.h | 6 ++++ libavcodec/qsvdec.c | 73 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+) diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h index f6e739a686..5119ef4dff 100644 --- a/libavcodec/qsv_internal.h +++ b/libavcodec/qsv_internal.h @@ -85,6 +85,12 @@ typedef struct QSVFrame { #if QSV_VERSION_ATLEAST(1, 34) mfxExtAV1FilmGrainParam av1_film_grain_param; #endif + +#if QSV_VERSION_ATLEAST(1, 35) + mfxExtMasteringDisplayColourVolume mdcv; + mfxExtContentLightLevelInfo clli; +#endif + mfxExtBuffer *ext_param[QSV_MAX_FRAME_EXT_PARAMS]; int num_ext_params; diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c index 73405b5747..92bfea196e 100644 --- a/libavcodec/qsvdec.c +++ b/libavcodec/qsvdec.c @@ -41,6 +41,7 @@ #include "libavutil/time.h" #include "libavutil/imgutils.h" #include "libavutil/film_grain_params.h" +#include "libavutil/mastering_display_metadata.h" #include "avcodec.h" #include "codec_internal.h" @@ -492,6 +493,22 @@ static int alloc_frame(AVCodecContext *avctx, QSVContext *q, QSVFrame *frame) } #endif +#if QSV_VERSION_ATLEAST(1, 35) + if (QSV_RUNTIME_VERSION_ATLEAST(q->ver, 1, 35) && avctx->codec_id == AV_CODEC_ID_HEVC) { + frame->mdcv.Header.BufferId = MFX_EXTBUFF_MASTERING_DISPLAY_COLOUR_VOLUME; + frame->mdcv.Header.BufferSz = sizeof(frame->mdcv); + // The data in mdcv is valid when this flag is 1 + frame->mdcv.InsertPayloadToggle = 0; + ff_qsv_frame_add_ext_param(avctx, frame, (mfxExtBuffer *)&frame->mdcv); + + frame->clli.Header.BufferId = MFX_EXTBUFF_CONTENT_LIGHT_LEVEL_INFO; + frame->clli.Header.BufferSz = sizeof(frame->clli); + // The data in clli is valid when this flag is 1 + frame->clli.InsertPayloadToggle = 0; + ff_qsv_frame_add_ext_param(avctx, frame, (mfxExtBuffer *)&frame->clli); + } +#endif + frame->used = 1; return 0; @@ -628,6 +645,53 @@ static int qsv_export_film_grain(AVCodecContext *avctx, mfxExtAV1FilmGrainParam } #endif +#if QSV_VERSION_ATLEAST(1, 35) +static int qsv_export_hdr_side_data(AVCodecContext *avctx, mfxExtMasteringDisplayColourVolume *mdcv, + mfxExtContentLightLevelInfo *clli, AVFrame *frame) +{ + // The SDK re-uses this flag for HDR SEI parsing + if (mdcv->InsertPayloadToggle) { + AVMasteringDisplayMetadata *mastering = av_mastering_display_metadata_create_side_data(frame); + const int mapping[3] = {2, 0, 1}; + const int chroma_den = 50000; + const int luma_den = 10000; + int i; + + if (!mastering) + return AVERROR(ENOMEM); + + for (i = 0; i < 3; i++) { + const int j = mapping[i]; + mastering->display_primaries[i][0] = av_make_q(mdcv->DisplayPrimariesX[j], chroma_den); + mastering->display_primaries[i][1] = av_make_q(mdcv->DisplayPrimariesY[j], chroma_den); + } + + mastering->white_point[0] = av_make_q(mdcv->WhitePointX, chroma_den); + mastering->white_point[1] = av_make_q(mdcv->WhitePointY, chroma_den); + + mastering->max_luminance = av_make_q(mdcv->MaxDisplayMasteringLuminance, luma_den); + mastering->min_luminance = av_make_q(mdcv->MinDisplayMasteringLuminance, luma_den); + + mastering->has_luminance = 1; + mastering->has_primaries = 1; + } + + // The SDK re-uses this flag for HDR SEI parsing + if (clli->InsertPayloadToggle) { + AVContentLightMetadata *light = av_content_light_metadata_create_side_data(frame); + + if (!light) + return AVERROR(ENOMEM); + + light->MaxCLL = clli->MaxContentLightLevel; + light->MaxFALL = clli->MaxPicAverageLightLevel; + } + + return 0; +} + +#endif + static int qsv_decode(AVCodecContext *avctx, QSVContext *q, AVFrame *frame, int *got_frame, const AVPacket *avpkt) @@ -749,6 +813,15 @@ static int qsv_decode(AVCodecContext *avctx, QSVContext *q, } #endif +#if QSV_VERSION_ATLEAST(1, 35) + if (QSV_RUNTIME_VERSION_ATLEAST(q->ver, 1, 35) && avctx->codec_id == AV_CODEC_ID_HEVC) { + ret = qsv_export_hdr_side_data(avctx, &aframe.frame->mdcv, &aframe.frame->clli, frame); + + if (ret < 0) + return ret; + } +#endif + frame->repeat_pict = outsurf->Info.PicStruct & MFX_PICSTRUCT_FRAME_TRIPLING ? 4 : outsurf->Info.PicStruct & MFX_PICSTRUCT_FRAME_DOUBLING ? 2 :