From patchwork Fri Jul 21 21:30:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devin Heitmueller X-Patchwork-Id: 42897 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:be1d:b0:130:ccc6:6c4b with SMTP id ge29csp504454pzb; Fri, 21 Jul 2023 14:31:40 -0700 (PDT) X-Google-Smtp-Source: APBJJlEW+0rtmA7whnwY6a3pH+7kR1wvPh6x6RTfY3dt9tPTyj0XBU25G56BpVKsiiMsM+autF2G X-Received: by 2002:a17:906:11a:b0:991:fef4:bb9 with SMTP id 26-20020a170906011a00b00991fef40bb9mr2520162eje.58.1689975100353; Fri, 21 Jul 2023 14:31:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689975100; cv=none; d=google.com; s=arc-20160816; b=PA10X1TON2CAQuuBf+zJXDJU+mHt1athpDuPLmNght8zFeDDxrUIgvXl5M2CVmSTri s5qyzU0T8iU6KD93vUvmsYFjFMostwxK2OHxrR15wluCGb57R1RyJLBvS7hOhZ6KqSia D0X6wEMfs8V93gSAprjSHz7THK8jD3Cu+Vpuc8wal53UVOnoDQfZFkF+5axxUjRIuWjK RZQ9tG6icVQyaPxS+zOq2ptKOBSX+jzJ/PjWz9NKvjuduCMWK/DzfJY97ndILYxQiUE1 PVfX6kXbHcO+XH0Uelmqnokr6UtQSYnaVUDBu7zt2W7tj/LvDAQBduYaHYSrGa6wBXKt FQ9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=0fBnlJPNOQDzSwEZYSw0e4voIc5GbGgj6DtkYKY7o3w=; fh=DKzdJpyueDOox6ShoNYUi750QZs6R2naOS9+hcsrM90=; b=ndqr8UwyzQDbjAdg5q1MAAmqWlW9/p2h5CyVI26LdEmqeUBFlvczl0Kws3c96kW254 PbG96cyqxyHFp7GM72oerLca6Qmy/FvnoN7tJ/snzAMJHAjSXliSSlhWg/UCIrFtXRRg FTzSIR4u5m+tl+L3KjNXz0whQyHTjh9/NDrVYB1anBoTtfQ/p07PG9TPwgsQSiVMVnhi xwFYQoQToRz8IGBhfFFRsKoolHtty3qYRjOIdb9pKLwxi7qVStwEdBQcYrIFXzJILDX+ T5I6OEcvzkZmkB7vbBmNRk7N0dfV3zG02J61y9961fWs4jEMWKrMcEBCJh/nbivHuy/y fSCA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20221208.gappssmtp.com header.s=20221208 header.b=YXUS9B+f; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id mc21-20020a170906eb5500b00982a59bf247si2625140ejb.1027.2023.07.21.14.31.39; Fri, 21 Jul 2023 14:31:40 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20221208.gappssmtp.com header.s=20221208 header.b=YXUS9B+f; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E6D9868C621; Sat, 22 Jul 2023 00:31:11 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2B57268BA8D for ; Sat, 22 Jul 2023 00:31:04 +0300 (EEST) Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-76728ae3162so199661085a.3 for ; Fri, 21 Jul 2023 14:31:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ltnglobal-com.20221208.gappssmtp.com; s=20221208; t=1689975063; x=1690579863; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=k27HggGtyh/7SQlalRdyM0KZT2OoJTKokQsrvPp4dcE=; b=YXUS9B+fw33LEH61elljofnFozKcX5u1Xfgt7ORzxd2RoNxZtHrfpVIjpeSVMNkMqL 6QedwBYXM5HPkFg8uONEzhD39EAGd7Ww1OxgQtzS+HnPc1jlXNKOCwktmIxvhCTC8FXu paWrhZvpnMCBSnty2wY0l3Hgjtmw7Y3jXX+2LUg7fjrVue6ZbAWp39xxdp32VBQXXVqY e2OB81C74cPptH6b1yf7kT4p4uR1bmqk3XSDLqC/tSu9voH+f1TpF+a0H0amVKe+IXI/ kDKDGrEIOSnDs+kEyAOUogJpywZfvovY/FA1SUCVoelPlN+DxkTWvgT7VjuQHtFBpSC0 jZKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689975063; x=1690579863; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=k27HggGtyh/7SQlalRdyM0KZT2OoJTKokQsrvPp4dcE=; b=DOO5ukWW2VWyqKbhjE/3JdgnvcbC3gt9MnVoJEfPuJE3fEYvKYVubO/yfgv3IMxEB3 mib+wBPAwY4NSDvTGGhXXJo+U1RxleA5Zx0BrDyxNSxsbv+l2pBj8TsqsA+Cr4PP00cz +g63U243Y9aXaE+NxuK25lxv15rioCPm1Xm8BwWKiulWYxn56yysU/6Ylvcxga3oOHRH WcbMADPeAuun6NqdVJZAgPxlUn8SUEUEs1f2WSnSB9LZObWNAMjQyU6fz03ks/myF1LM 2hN6OCK145SqMcaO4lT9nHeHRQvmWDTCzJhpZurfwlQEeMe/IrTuqx6DuaZ3OjDWP70s Et2Q== X-Gm-Message-State: ABy/qLatIxypzjxHSQ1zwwxhHktEmLqei1FQnOGtm2J6deIXYCqnHRYl EIYw6Db1DzOBRSUixf5hUqxul9amCcdDTiQXNxw= X-Received: by 2002:a05:620a:4501:b0:767:d6f9:452b with SMTP id t1-20020a05620a450100b00767d6f9452bmr1523390qkp.12.1689975062775; Fri, 21 Jul 2023 14:31:02 -0700 (PDT) Received: from ltnt-nyc-580testdevin.livetimenet.com (pool-71-105-132-214.nycmny.fios.verizon.net. [71.105.132.214]) by smtp.gmail.com with ESMTPSA id z21-20020ae9c115000000b0076738337cd1sm1406333qki.1.2023.07.21.14.31.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Jul 2023 14:31:02 -0700 (PDT) From: Devin Heitmueller X-Google-Original-From: Devin Heitmueller To: ffmpeg-devel@ffmpeg.org Date: Fri, 21 Jul 2023 17:30:57 -0400 Message-Id: <1689975057-22226-4-git-send-email-dheitmueller@ltnglobal.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1689975057-22226-1-git-send-email-dheitmueller@ltnglobal.com> References: <1689975057-22226-1-git-send-email-dheitmueller@ltnglobal.com> Subject: [FFmpeg-devel] [RFC][PATCH 3/3] decklink: Add support for output of HDR metadata X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Devin Heitmueller MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: fEmrWxfILCep Add HDR support to the decklink output for cards that support such functionality. This includes setting the EOTF, the colorspace, the mastering info, and the content light level info. Both the Payload Identification HANC data as well as the SMPTE ST 2108-1 VANC data are being set. Tested with in-house content as well as samples from 4kmedia.org. Testing was done with the Decklink 8K Pro and the Duo2 with 12.5.1 firmware, as well as with the Duo2 with 10.11.2 (before it supported HDR) to ensure there are no regressions. Signed-off-by: Devin Heitmueller --- libavdevice/decklink_common.cpp | 12 +++ libavdevice/decklink_common.h | 2 + libavdevice/decklink_enc.cpp | 204 ++++++++++++++++++++++++++++++++++++++-- 3 files changed, 211 insertions(+), 7 deletions(-) diff --git a/libavdevice/decklink_common.cpp b/libavdevice/decklink_common.cpp index 47de7ef..c1bcb82 100644 --- a/libavdevice/decklink_common.cpp +++ b/libavdevice/decklink_common.cpp @@ -251,6 +251,18 @@ int ff_decklink_set_configs(AVFormatContext *avctx, } } + DECKLINK_BOOL hdr_supported; + if (ctx->attr->GetFlag(BMDDeckLinkSupportsHDRMetadata, &hdr_supported) == S_OK) { + if (hdr_supported) + ctx->supports_hdr = 1; + } + + DECKLINK_BOOL colorspace_supported; + if (ctx->attr->GetFlag(BMDDeckLinkSupportsColorspaceMetadata, &colorspace_supported) == S_OK) { + if (colorspace_supported) + ctx->supports_colorspace = 1; + } + return 0; } diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h index 34ab1b9..d50007f 100644 --- a/libavdevice/decklink_common.h +++ b/libavdevice/decklink_common.h @@ -109,6 +109,8 @@ struct decklink_ctx { int bmd_height; int bmd_field_dominance; int supports_vanc; + int supports_hdr; + int supports_colorspace; /* Capture buffer queue */ DecklinkPacketQueue queue; diff --git a/libavdevice/decklink_enc.cpp b/libavdevice/decklink_enc.cpp index ffd0ad9..92d6bbe 100644 --- a/libavdevice/decklink_enc.cpp +++ b/libavdevice/decklink_enc.cpp @@ -35,6 +35,7 @@ extern "C" { #include "libavcodec/bytestream.h" #include "libavutil/internal.h" #include "libavutil/imgutils.h" +#include "libavutil/mastering_display_metadata.h" #include "avdevice.h" } @@ -47,13 +48,13 @@ extern "C" { #endif /* DeckLink callback class declaration */ -class decklink_frame : public IDeckLinkVideoFrame +class decklink_frame : public IDeckLinkVideoFrame, public IDeckLinkVideoFrameMetadataExtensions { public: decklink_frame(struct decklink_ctx *ctx, AVFrame *avframe, AVCodecID codec_id, int height, int width) : _ctx(ctx), _avframe(avframe), _avpacket(NULL), _codec_id(codec_id), _ancillary(NULL), _height(height), _width(width), _refs(1) { } decklink_frame(struct decklink_ctx *ctx, AVPacket *avpacket, AVCodecID codec_id, int height, int width) : - _ctx(ctx), _avframe(NULL), _avpacket(avpacket), _codec_id(codec_id), _ancillary(NULL), _height(height), _width(width), _refs(1) { } + _ctx(ctx), _avframe(NULL), _avpacket(avpacket), _codec_id(codec_id), _ancillary(NULL), _height(height), _width(width), _colorspace(AVCOL_SPC_BT709), _eotf(AVCOL_TRC_BT709), hdr(NULL), lighting(NULL), _refs(1) { } virtual long STDMETHODCALLTYPE GetWidth (void) { return _width; } virtual long STDMETHODCALLTYPE GetHeight (void) { return _height; } virtual long STDMETHODCALLTYPE GetRowBytes (void) @@ -72,10 +73,14 @@ public: } virtual BMDFrameFlags STDMETHODCALLTYPE GetFlags (void) { - if (_codec_id == AV_CODEC_ID_WRAPPED_AVFRAME) - return _avframe->linesize[0] < 0 ? bmdFrameFlagFlipVertical : bmdFrameFlagDefault; - else - return bmdFrameFlagDefault; + if (_codec_id == AV_CODEC_ID_WRAPPED_AVFRAME) { + return _avframe->linesize[0] < 0 ? bmdFrameFlagFlipVertical : bmdFrameFlagDefault; + } else { + if (_ctx->supports_hdr && (hdr || lighting)) + return bmdFrameFlagDefault | bmdFrameContainsHDRMetadata; + else + return bmdFrameFlagDefault; + } } virtual HRESULT STDMETHODCALLTYPE GetBytes (void **buffer) @@ -110,7 +115,176 @@ public: _ancillary->AddRef(); return S_OK; } - virtual HRESULT STDMETHODCALLTYPE QueryInterface(REFIID iid, LPVOID *ppv) { return E_NOINTERFACE; } + + virtual HRESULT STDMETHODCALLTYPE SetMetadata(enum AVColorSpace colorspace, enum AVColorTransferCharacteristic eotf) + { + _colorspace = colorspace; + _eotf = eotf; + return S_OK; + } + + // IDeckLinkVideoFrameMetadataExtensions interface + virtual HRESULT GetInt(BMDDeckLinkFrameMetadataID metadataID, int64_t* value) + { + HRESULT result = S_OK; + + switch (metadataID) { + case bmdDeckLinkFrameMetadataHDRElectroOpticalTransferFunc: + /* See CTA-861-G Sec 6.9 Dynamic Range and Mastering */ + + switch(_eotf) { + case AVCOL_TRC_SMPTEST2084: + /* PQ */ + *value = 2; + break; + case AVCOL_TRC_ARIB_STD_B67: + /* Also known as "HLG" */ + *value = 3; + break; + case AVCOL_TRC_SMPTE170M: + case AVCOL_TRC_SMPTE240M: + case AVCOL_TRC_BT709: + default: + /* SDR */ + *value = 0; + break; + } + break; + + case bmdDeckLinkFrameMetadataColorspace: + if (!_ctx->supports_colorspace) { + result = E_NOTIMPL; + break; + } + switch(_colorspace) { + case AVCOL_SPC_BT470BG: + case AVCOL_SPC_SMPTE170M: + case AVCOL_SPC_SMPTE240M: + *value = bmdColorspaceRec601; + break; + case AVCOL_SPC_BT2020_CL: + case AVCOL_SPC_BT2020_NCL: + *value = bmdColorspaceRec2020; + break; + case AVCOL_SPC_BT709: + default: + *value = bmdColorspaceRec709; + break; + } + break; + default: + result = E_INVALIDARG; + } + + return result; + } + virtual HRESULT GetFloat(BMDDeckLinkFrameMetadataID metadataID, double* value) + { + *value = 0; + + switch (metadataID) { + case bmdDeckLinkFrameMetadataHDRDisplayPrimariesRedX: + if (hdr && hdr->has_primaries) + *value = av_q2d(hdr->display_primaries[0][0]); + break; + case bmdDeckLinkFrameMetadataHDRDisplayPrimariesRedY: + if (hdr && hdr->has_primaries) + *value = av_q2d(hdr->display_primaries[0][1]); + break; + case bmdDeckLinkFrameMetadataHDRDisplayPrimariesGreenX: + if (hdr && hdr->has_primaries) + *value = av_q2d(hdr->display_primaries[1][0]); + break; + case bmdDeckLinkFrameMetadataHDRDisplayPrimariesGreenY: + if (hdr && hdr->has_primaries) + *value = av_q2d(hdr->display_primaries[1][1]); + break; + case bmdDeckLinkFrameMetadataHDRDisplayPrimariesBlueX: + if (hdr && hdr->has_primaries) + *value = av_q2d(hdr->display_primaries[2][0]); + break; + case bmdDeckLinkFrameMetadataHDRDisplayPrimariesBlueY: + if (hdr && hdr->has_primaries) + *value = av_q2d(hdr->display_primaries[2][1]); + break; + case bmdDeckLinkFrameMetadataHDRWhitePointX: + if (hdr && hdr->has_primaries) + *value = av_q2d(hdr->white_point[0]); + break; + case bmdDeckLinkFrameMetadataHDRWhitePointY: + if (hdr && hdr->has_primaries) + *value = av_q2d(hdr->white_point[1]); + break; + case bmdDeckLinkFrameMetadataHDRMaxDisplayMasteringLuminance: + if (hdr && hdr->has_luminance) + *value = av_q2d(hdr->max_luminance); + break; + case bmdDeckLinkFrameMetadataHDRMinDisplayMasteringLuminance: + if (hdr && hdr->has_luminance) + *value = av_q2d(hdr->min_luminance); + break; + case bmdDeckLinkFrameMetadataHDRMaximumContentLightLevel: + if (lighting) + *value = (float) lighting->MaxCLL; + else + *value = 0; + break; + case bmdDeckLinkFrameMetadataHDRMaximumFrameAverageLightLevel: + if (lighting) + *value = (float) lighting->MaxFALL; + else + *value = 0; + break; + default: + return E_INVALIDARG; + } + + return S_OK; + } + + virtual HRESULT GetFlag(BMDDeckLinkFrameMetadataID metadataID, bool* value) + { + *value = false; + return E_INVALIDARG; + } + virtual HRESULT GetString(BMDDeckLinkFrameMetadataID metadataID, const char** value) + { + *value = nullptr; + return E_INVALIDARG; + } + virtual HRESULT GetBytes(BMDDeckLinkFrameMetadataID metadataID, void* buffer, uint32_t* bufferSize) + { + *bufferSize = 0; + return E_INVALIDARG; + } + + virtual HRESULT STDMETHODCALLTYPE QueryInterface(REFIID iid, LPVOID *ppv) + { + CFUUIDBytes iunknown; + HRESULT result = S_OK; + + if (!ppv) + return E_INVALIDARG; + + *ppv = NULL; + + iunknown = CFUUIDGetUUIDBytes(IUnknownUUID); + if (memcmp(&iid, &iunknown, sizeof(REFIID)) == 0) { + *ppv = this; + AddRef(); + } else if (memcmp(&iid, &IID_IDeckLinkVideoFrame, sizeof(REFIID)) == 0) { + *ppv = static_cast(this); + AddRef(); + } else if (memcmp(&iid, &IID_IDeckLinkVideoFrameMetadataExtensions, sizeof(REFIID)) == 0) { + *ppv = static_cast(this); + AddRef(); + } else { + result = E_NOINTERFACE; + } + + return result; + } + virtual ULONG STDMETHODCALLTYPE AddRef(void) { return ++_refs; } virtual ULONG STDMETHODCALLTYPE Release(void) { @@ -132,6 +306,10 @@ public: IDeckLinkVideoFrameAncillary *_ancillary; int _height; int _width; + enum AVColorSpace _colorspace; + enum AVColorTransferCharacteristic _eotf; + const AVMasteringDisplayMetadata *hdr; + const AVContentLightMetadata *lighting; private: std::atomic _refs; @@ -726,6 +904,18 @@ static int decklink_write_video_packet(AVFormatContext *avctx, AVPacket *pkt) return AVERROR(EIO); } + /* Set frame metadata properties */ + size_t size; + const AVMasteringDisplayMetadata *hdr = (const AVMasteringDisplayMetadata *) av_packet_get_side_data(pkt, AV_PKT_DATA_MASTERING_DISPLAY_METADATA, &size); + if (hdr && size > 0) + frame->hdr = hdr; + + const AVContentLightMetadata *lighting = (const AVContentLightMetadata *) av_packet_get_side_data(pkt, AV_PKT_DATA_CONTENT_LIGHT_LEVEL, &size); + if (hdr && size > 0) + frame->lighting = lighting; + + frame->SetMetadata(st->codecpar->color_space, st->codecpar->color_trc); + /* Always keep at most one second of frames buffered. */ pthread_mutex_lock(&ctx->mutex); while (ctx->frames_buffer_available_spots == 0) {