From patchwork Thu May 27 01:35:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mohammad Izadi X-Patchwork-Id: 27954 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp20369iof; Wed, 26 May 2021 18:35:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzq/s5jmaLAXGDhVFuRVrWpBHBm32uO6YUHvSjMZfkxxrSfa8eJM5OUEP2XMRaOsSMa8yEN X-Received: by 2002:aa7:d853:: with SMTP id f19mr1241082eds.371.1622079327014; Wed, 26 May 2021 18:35:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622079327; cv=none; d=google.com; s=arc-20160816; b=wjw2gnee+siE6l7MR2egVh1RQs5poPSu+0feD6HdaS+7IE2XWITMxUW3TlXlftLZAG eRxJ/KCID+Ig+qwx3YuL3xhka4VB6Pc5xlfpVI9ErUgQqTkhbjACFSob+7mBmEcQpL1b JdG/+YUtunJVLeAs/BxIXqe9m8QR3o0yjPrl/htz+8LoKfjKRkFkQGNTT0BlKpMi1VSL 7uyYvQEWyNwLisVYWt2CkUYPXxv3nfJTGb64WHiwQ9f+O1wh86mHLKNCGlffGqAA+A3M CXV9A2Lct2o/zUhMnInI6aIS8FbxqYQYVTbpZVTEEgem038DAJ8JeXmbUfeo6C+PSufi 3oLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:from:references:mime-version :message-id:in-reply-to:date:dkim-signature:delivered-to; bh=p9ZGl04F48ja1aQWQcAoVCwNmhqJpPtFoH3aS513JMI=; b=0hFOASGwbtSnJToJBJcyi0vTY2KkC4AbYKrFJTHjqRvAdcxY4bgxz5e7jRsT3FqMwe vbE0MRkMmJhNsG0rU30sc+yM8fHesFNbf1MQjuc8/+9lP9J5rmH1W0Ucdhx1MgpwAHGh gc6ycbb972Hq7p9i2DysCR1R8sc8BdRbIx16wtnN9EKZCTP1s3AGEl5BXyDaXEWBDMhi uEJlu3V7v/ellnFT7vl03BGeL2z/z9u2LK6ZOTFh2OuXYwcRbwZIEp/1aOHJA1ksQBOn jffFqfGru3DlYr+dl8qU04mqQMfnUS4u42WdoKe5dIzaJCpPEJ2f8ffwfSUP6QewLJ61 chlw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20161025 header.b=aMXtyink; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b1si634412edm.343.2021.05.26.18.35.26; Wed, 26 May 2021 18:35:27 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20161025 header.b=aMXtyink; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CE44E687F8C; Thu, 27 May 2021 04:35:21 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A68F368052A for ; Thu, 27 May 2021 04:35:15 +0300 (EEST) Received: by mail-yb1-f202.google.com with SMTP id h67-20020a25d0460000b0290517e5f14ba4so3860401ybg.18 for ; Wed, 26 May 2021 18:35:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=bpA5v4BvLZdYfM992f9giOg3D9Fkh0HsR8AvHGRh+kE=; b=aMXtyinks+QhTpnslR6EtEsECqhKsIRJNcukP6AteiRJmwT9Xn/cAGEIQEqhugOL5q SrcYhb/mdTM/u2uiXnhRc4g08C7jhn+kEQh2YcQ62Nfk5hpCsS2F6OqTgoGKSgE1WkyI 1dRhhUYAq5XcC3N8MQaFwBGyZvfmwTYflk7AQ+nMMVlitvvUU6RqyGo7vAI/DB4Fy3Kl Tw2agqMgdKJ+xBqAOFK/oHwQUVllDzsdolNA8sZLRz3JJrIZoGy55tYKm6JAS1DGDIcU qk2HY4QruQtexBaonYTlK+N46h1C53LtEDz0+6OGqXrsjJeckzESMPUJqAIHlLMCD5l4 2iEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=bpA5v4BvLZdYfM992f9giOg3D9Fkh0HsR8AvHGRh+kE=; b=JV3f6bOABXyqS5ZX91TZ7FazvOzymxpsv1uQ1Ttv8Dajw4mWVUmIt9f4brHB4SP66Q fHisXikCHGFNxdTS4zuw3r19pGLQi5QazTLTH+LP5KfxaxBs9VoOYzwSQIrM+Lcp3bPW F9VxkPuqxhmIoD06MPbg76rsiuoe3c8Jk3egW072jKZ2BwnZFFTh1nIpdjePCQp24tQL YUaMWPxd25frknTB32Dcslvq+CBjpXNY8xEmg0rpIHIh4l5EQJak1B3dl2Gy5M06CPwF ZDucAFaGujDza4RZ6UA0PD9O+1AvM8GIyT4NfeN4EFzkhJjyF7WIf11/rzEDi+Lxn6cS BI4w== X-Gm-Message-State: AOAM531Y7B2oVYFin2vvIAZkpkfdJVFvf48htzKkGNjDjTzQFxlGx9Ef XoFGtSWo1Y4QFBD0OiNC5mDTkXbHjzu+Sbt3fajneb2+kxyOUAecQaKR5jw6F5Fb1OO5xS77l3a AVZ0w9Ww1/UQGBA8LhazUVy0vVXcMgqPWzTJKeJ0XA8oTiPtR5qELzEfnmWAH X-Received: from izadi.mtv.corp.google.com ([2620:0:1000:4001:87cb:2b87:3886:6]) (user=izadi job=sendgmr) by 2002:a25:bbd0:: with SMTP id c16mr1362373ybk.138.1622079313708; Wed, 26 May 2021 18:35:13 -0700 (PDT) Date: Wed, 26 May 2021 18:35:09 -0700 In-Reply-To: Message-Id: <20210527013509.3500013-1-izadi@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.31.1.818.g46aad6cb9e-goog From: Mohammad Izadi To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH] avcodec: Pass the HDR10+ metadata to the packet side data in VP9 encoder X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Mohammad Izadi Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: y/6c3ifNCpD0 HDR10+ metadata is stored in the bit stream for HEVC. The story is different for VP9 and cannot store the metadata in the bit stream. HDR10+ should be passed to packet side data an stored in the container (mkv) for VP9. This CL is taking HDR10+ from AVFrame side data in libvpxenc and is passing it to the AVPacket side data. --- doc/APIchanges | 2 + libavcodec/avpacket.c | 1 + libavcodec/decode.c | 1 + libavcodec/libvpxenc.c | 90 +++++++++++++++++++++++++++++++++++++++++- libavcodec/packet.h | 8 ++++ libavcodec/version.h | 2 +- 6 files changed, 101 insertions(+), 3 deletions(-) diff --git a/doc/APIchanges b/doc/APIchanges index c46f4d5304..60995579e5 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -13,6 +13,8 @@ libavutil: 2021-04-27 API changes, most recent first: +2021-05-25 - 8c88a66d3c - lavc 59.2.100 - packet.h + Add AV_PKT_DATA_DYNAMIC_HDR10_PLUS 2021-04-27 - cb3ac722f4 - lavc 59.0.100 - avcodec.h Constified AVCodecParserContext.parser. diff --git a/libavcodec/avpacket.c b/libavcodec/avpacket.c index 7383d12d3e..800bee3489 100644 --- a/libavcodec/avpacket.c +++ b/libavcodec/avpacket.c @@ -289,6 +289,7 @@ const char *av_packet_side_data_name(enum AVPacketSideDataType type) case AV_PKT_DATA_ICC_PROFILE: return "ICC Profile"; case AV_PKT_DATA_DOVI_CONF: return "DOVI configuration record"; case AV_PKT_DATA_S12M_TIMECODE: return "SMPTE ST 12-1:2014 timecode"; + case AV_PKT_DATA_DYNAMIC_HDR10_PLUS: return "HDR10+ Dynamic Metadata (SMPTE 2094-40)"; } return NULL; } diff --git a/libavcodec/decode.c b/libavcodec/decode.c index 75bc7ad98e..40f688e40c 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -1488,6 +1488,7 @@ int ff_decode_frame_props(AVCodecContext *avctx, AVFrame *frame) { AV_PKT_DATA_A53_CC, AV_FRAME_DATA_A53_CC }, { AV_PKT_DATA_ICC_PROFILE, AV_FRAME_DATA_ICC_PROFILE }, { AV_PKT_DATA_S12M_TIMECODE, AV_FRAME_DATA_S12M_TIMECODE }, + { AV_PKT_DATA_DYNAMIC_HDR10_PLUS, AV_FRAME_DATA_DYNAMIC_HDR_PLUS }, }; if (IS_EMPTY(pkt) && av_fifo_size(avctx->internal->pkt_props) >= sizeof(*pkt)) diff --git a/libavcodec/libvpxenc.c b/libavcodec/libvpxenc.c index 66bad444d0..3fa0dbd8e2 100644 --- a/libavcodec/libvpxenc.c +++ b/libavcodec/libvpxenc.c @@ -64,6 +64,11 @@ struct FrameListData { struct FrameListData *next; }; +typedef struct FrameHDR10Plus { + int64_t pts; + AVBufferRef *hdr10_plus; +} FrameHDR10Plus; + typedef struct VPxEncoderContext { AVClass *class; struct vpx_codec_ctx encoder; @@ -121,6 +126,8 @@ typedef struct VPxEncoderContext { int tune_content; int corpus_complexity; int tpl_model; + int discard_hdr10_plus; + AVFifoBuffer *hdr10_plus_fifo; /** * If the driver does not support ROI then warn the first time we * encounter a frame with ROI side data. @@ -316,6 +323,50 @@ static av_cold void free_frame_list(struct FrameListData *list) } } +static void add_hdr10_plus(AVFifoBuffer *fifo, struct FrameHDR10Plus *data) +{ + av_fifo_grow(fifo, sizeof(FrameHDR10Plus)); + av_fifo_generic_write(fifo, data, sizeof(FrameHDR10Plus), NULL); +} + +static av_cold void free_hdr10_plus(struct FrameHDR10Plus *p) +{ + if (!p) + return; + av_buffer_unref(&p->hdr10_plus); + av_free(p); +} + +static av_cold void free_hdr10_plus_fifo(AVFifoBuffer **fifo) +{ + FrameHDR10Plus *frame_hdr10_plus = NULL; + while (av_fifo_generic_read(*fifo, frame_hdr10_plus, sizeof(*frame_hdr10_plus), NULL) > 0) + free_hdr10_plus(frame_hdr10_plus); + av_fifo_freep(fifo); +} + +static int copy_hdr10_plus_to_pkt(AVFifoBuffer *fifo, AVPacket *pkt) +{ + FrameHDR10Plus *frame_hdr10_plus; + uint8_t *data; + if (av_fifo_size(fifo) < 1) + return 0; + + av_fifo_generic_read(fifo, frame_hdr10_plus, sizeof(*frame_hdr10_plus), NULL); + if (!frame_hdr10_plus || !pkt || !(frame_hdr10_plus->hdr10_plus) || frame_hdr10_plus->pts != pkt->pts) + return 0; + + data = av_packet_new_side_data(pkt, AV_PKT_DATA_DYNAMIC_HDR10_PLUS, frame_hdr10_plus->hdr10_plus->size); + if (!data) { + free_hdr10_plus(frame_hdr10_plus); + return AVERROR(ENOMEM); + } + memcpy(data, frame_hdr10_plus->hdr10_plus->data, frame_hdr10_plus->hdr10_plus->size); + free_hdr10_plus(frame_hdr10_plus); + + return 0; +} + static av_cold int codecctl_int(AVCodecContext *avctx, enum vp8e_enc_control_id id, int val) { @@ -384,6 +435,7 @@ static av_cold int vpx_free(AVCodecContext *avctx) av_freep(&ctx->twopass_stats.buf); av_freep(&avctx->stats_out); free_frame_list(ctx->coded_frame_list); + free_hdr10_plus_fifo(&ctx->hdr10_plus_fifo); return 0; } @@ -835,6 +887,7 @@ static av_cold int vpx_init(AVCodecContext *avctx, #endif AVDictionaryEntry* en = NULL; + ctx->discard_hdr10_plus = 1; av_log(avctx, AV_LOG_INFO, "%s\n", vpx_codec_version_str()); av_log(avctx, AV_LOG_VERBOSE, "%s\n", vpx_codec_build_config()); @@ -851,11 +904,19 @@ static av_cold int vpx_init(AVCodecContext *avctx, if (avctx->codec_id == AV_CODEC_ID_VP9) { if (set_pix_fmt(avctx, codec_caps, &enccfg, &flags, &img_fmt)) return AVERROR(EINVAL); + // Keep HDR10+ if it has bit depth higher than 8 and + // it has PQ trc (SMPTE2084). + if (enccfg.g_bit_depth > 8 && avctx->color_trc == AVCOL_TRC_SMPTE2084) { + ctx->discard_hdr10_plus = 0; + ctx->hdr10_plus_fifo = av_fifo_alloc(sizeof(FrameHDR10Plus)); + if (!ctx->hdr10_plus_fifo) + return AVERROR(ENOMEM); + } } #endif - if(!avctx->bit_rate) - if(avctx->rc_max_rate || avctx->rc_buffer_size || avctx->rc_initial_buffer_occupancy) { + if (!avctx->bit_rate) + if (avctx->rc_max_rate || avctx->rc_buffer_size || avctx->rc_initial_buffer_occupancy) { av_log( avctx, AV_LOG_ERROR, "Rate control parameters set without a bitrate\n"); return AVERROR(EINVAL); } @@ -1211,6 +1272,15 @@ static int storeframe(AVCodecContext *avctx, struct FrameListData *cx_frame, AV_WB64(side_data, 1); memcpy(side_data + 8, cx_frame->buf_alpha, cx_frame->sz_alpha); } + if (cx_frame->frame_number != -1) { + VPxContext *ctx = avctx->priv_data; + if (!ctx->discard_hdr10_plus) { + int err = copy_hdr10_plus_to_pkt(ctx->hdr10_plus_fifo, pkt); + if (err < 0) + return err; + } + } + return pkt->size; } @@ -1542,6 +1612,7 @@ static int vpx_encode(AVCodecContext *avctx, AVPacket *pkt, const struct vpx_codec_enc_cfg *enccfg = ctx->encoder.config.enc; vpx_svc_layer_id_t layer_id; int layer_id_valid = 0; + AVFrameSideData *hdr10_plus_metadata; if (frame) { const AVFrameSideData *sd = av_frame_get_side_data(frame, AV_FRAME_DATA_REGIONS_OF_INTEREST); @@ -1618,6 +1689,21 @@ static int vpx_encode(AVCodecContext *avctx, AVPacket *pkt, vp9_encode_set_roi(avctx, frame->width, frame->height, sd); } } + + if (!ctx->discard_hdr10_plus) { + // Add HDR10+ metadata to queue. + hdr10_plus_metadata = av_frame_get_side_data(frame, AV_FRAME_DATA_DYNAMIC_HDR_PLUS); + if (hdr10_plus_metadata) { + struct FrameHDR10Plus *data = av_malloc(sizeof(*data)); + if (!data) + return AVERROR(ENOMEM); + data->pts = frame->pts; + data->hdr10_plus = av_buffer_ref(hdr10_plus_metadata->buf); + if (!data->hdr10_plus) + return AVERROR(ENOMEM); + add_hdr10_plus(ctx->hdr10_plus_fifo, data); + } + } } // this is for encoding with preset temporal layering patterns defined in diff --git a/libavcodec/packet.h b/libavcodec/packet.h index fad8341c12..2fd4e88d0f 100644 --- a/libavcodec/packet.h +++ b/libavcodec/packet.h @@ -290,6 +290,14 @@ enum AVPacketSideDataType { */ AV_PKT_DATA_S12M_TIMECODE, + /** + * HDR10+ dynamic metadata associated with a video frame. The metadata is in + * the form of the AVDynamicHDRPlus struct and contains + * information for color volume transform - application 4 of + * SPMTE 2094-40:2016 standard. + */ + AV_PKT_DATA_DYNAMIC_HDR10_PLUS, + /** * The number of side data types. * This is not part of the public API/ABI in the sense that it may diff --git a/libavcodec/version.h b/libavcodec/version.h index 48165b9ac4..1288cecebe 100644 --- a/libavcodec/version.h +++ b/libavcodec/version.h @@ -28,7 +28,7 @@ #include "libavutil/version.h" #define LIBAVCODEC_VERSION_MAJOR 59 -#define LIBAVCODEC_VERSION_MINOR 1 +#define LIBAVCODEC_VERSION_MINOR 2 #define LIBAVCODEC_VERSION_MICRO 100 #define LIBAVCODEC_VERSION_INT AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \