From patchwork Fri Apr 23 15:58:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mohammad Izadi X-Patchwork-Id: 27220 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:5014:0:0:0:0:0 with SMTP id e20csp1660361iob; Fri, 23 Apr 2021 08:58:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw8EfnupdP7u0yj0s0+j9zSe/INWO8SskY/3Mv3byuIzOS9hpozD88BjiYmg91deeCcgPHE X-Received: by 2002:a05:6402:3514:: with SMTP id b20mr5261665edd.348.1619193530270; Fri, 23 Apr 2021 08:58:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1619193530; cv=none; d=google.com; s=arc-20160816; b=hCafsys/hN+6XkTZk9SG/N77M84M2bSkbm8EZBMYwFwFD+SfBe2bGPTG86bZwRuXbh BLQvLjeb7POYLwlxlQvtnib0jXafZIe6tLQ/Utw72QRLlbKqml8im53cTN/mZKpJTS6n WJxvtpbah0A4q0LbF/XrD+5ETLrCPmKx2WVstNsDNYe+KA3wj/6R8rHfzM1Q8XhmcWzK PCRAJmLMOVdHJlBJi3TYVyKfuCAH8vEa6u9c0hF8gv4+78q5yDxc+qjwhA7lkA/KHGup E3ElRnsj+g5Exi2Z+vGE+7E87Khp0SiS4RJ4L6K3SYiQF5mmAPhwlwdZuEVpLkaxKgHP 2RsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:from:mime-version:message-id:date :dkim-signature:delivered-to; bh=mYqgHZ6QpMtg3cvsqeM65AcIJT4o/mVGR6vrXjuJWvA=; b=PsdnV9vWb70LX9Zf58Vq6DuPH7IE90XEsqSPSOBulHAFrNcbHw8DbKiLgEA9wOQ/PA Bo6e1BYmIgvg3+C1PrMIAWGqA9oxaSaqIllgj0oMth7b5rRpriRIOXhAtQRUH7Y1kfln oc1OzbaHq/KFtEgCPrYJotNGdwRYJ1/kMM6qtzykArvT6/q6j10av2FkB5xqvQYzksEl JqFgZlL4Y8elAT1/KrHDYtHgq/WA7ohVbXIU2d60YKNa+RN9AeGRAPic0x97OkCdl/VT hklBxEmjS8nOxAg7HWCjR62aa23DsDlFShH/us+IsfADXuCgjUrHvmrskPSKHimLR96i E58A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20161025 header.b=tE8tyzTJ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e12si5688922ejz.266.2021.04.23.08.58.49; Fri, 23 Apr 2021 08:58:50 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20161025 header.b=tE8tyzTJ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E6E106897BB; Fri, 23 Apr 2021 18:58:45 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f201.google.com (mail-qt1-f201.google.com [209.85.160.201]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7CDC96881C2 for ; Fri, 23 Apr 2021 18:58:38 +0300 (EEST) Received: by mail-qt1-f201.google.com with SMTP id 7-20020ac857070000b02901ba5c36e1b2so8743031qtw.21 for ; Fri, 23 Apr 2021 08:58:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=UCHmf/Z9HFBkuSUUbaQRHtJ1eab3ULtDqvlmI5Rnb7o=; b=tE8tyzTJKirHOkufQNQRIP98rovb0Zk4a5XOaSyMNRz29sXxECy58rubzRtUN60OOq tsSmaHucZlI44uWDs3rlM5w7V6MdiJdTZJ7mS2uKjKk5O1eFgmhUndbyV+7WvBSpN82s earQ334j8VxGcH6CZ/PQvyTprwktuekloQlTnOeuyNsHZEUNWGVlguUEDePbIEqb5334 wtibWiWSvAjTxyU/IfrNk0/nq0KsxKavKMnNOb7mh9Nl7N3Qfq3hqeadVK8wddP/x2Ff 6pll1LQFi5+NUh2v/thO7gWz3ddun86+M8RP8e3dEpdmW/dsLuWUZSYB18WkxkOn06X4 4CDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=UCHmf/Z9HFBkuSUUbaQRHtJ1eab3ULtDqvlmI5Rnb7o=; b=L7Eju1q/OSjdRhgSM6Lw1d2WwJDjDqYB5Rx1ytHmJI7eee7HyrS4e2pJhn2iO48VbV PuRJrmNm0DFfYpslCr9GZFGGcR8FIfGn0vE32TkvMQepNKiySMom85FBSyD1rrn/ckJk s34XDEC1WZ4QE6qa2Nd31T52DcOEPRaJPTi/dEAfseIUTlA6cKbW2dQD1gOe9A6JuJ7e +WpFj7MzEA/2sIFNpK4J5nD7UvUGtNVoZBY67bBuJcvboqvgCdi9vBpZl+/S8oPqpYYH 5vD3/DFJUQf3p6C2AaMj6xLyCmxzbuxYrWVGikEC87bZoBdMZ5eeGN6sdBIDthGyikgW AEFw== X-Gm-Message-State: AOAM532zcRXkuCnuUc5kJhnlUWx30TDE+EqCGEFUPse6V0eOVjOfObZs 7qBvWgfmYuu8wUwB4gaFc2qtZC7ewhwqCN1R7LBz7x8/iSLtkyNIZYZp0FDDG2KngMTgmekgWA7 tVo8+yTukXyvhOhbIUDqSaZVWI0OCydmyU8t+wY36ytTgUNBHgEbRfaLz1PfX X-Received: from izadi.mtv.corp.google.com ([2620:0:1000:4001:8364:49ae:94d4:bea1]) (user=izadi job=sendgmr) by 2002:a05:6214:1270:: with SMTP id r16mr5082497qvv.10.1619193516877; Fri, 23 Apr 2021 08:58:36 -0700 (PDT) Date: Fri, 23 Apr 2021 08:58:28 -0700 Message-Id: <20210423155828.3433455-1-izadi@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.31.1.498.g6c1eba8ee3d-goog From: Mohammad Izadi To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH] Pass the HDR10+ metadata to the packet side data in VP9 encoder X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Mohammad Izadi Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: e+NEv8rN/kGs HDR10+ metadata is stored in the bit stream for HEVC. The story is different for VP9 and cannot store the metadata in the bit stream. HDR10+ should be passed to packet side data an stored in the container (mkv) for VP9. This CL is taking HDR10+ from AVFrame side data in libvpxenc and is passing it to the AVPacket side data. --- libavcodec/avpacket.c | 1 + libavcodec/decode.c | 1 + libavcodec/libvpxenc.c | 71 ++++++++++++++++++++++++++++++++++++++++++ libavcodec/packet.h | 10 +++++- 4 files changed, 82 insertions(+), 1 deletion(-) diff --git a/libavcodec/avpacket.c b/libavcodec/avpacket.c index b5bac5c5f2..7a3b0a73e3 100644 --- a/libavcodec/avpacket.c +++ b/libavcodec/avpacket.c @@ -416,6 +416,7 @@ const char *av_packet_side_data_name(enum AVPacketSideDataType type) case AV_PKT_DATA_ICC_PROFILE: return "ICC Profile"; case AV_PKT_DATA_DOVI_CONF: return "DOVI configuration record"; case AV_PKT_DATA_S12M_TIMECODE: return "SMPTE ST 12-1:2014 timecode"; + case AV_PKT_DATA_DYNAMIC_HDR10_PLUS: return "HDR10+ Dynamic Metadata (SMPTE 2094-40)"; } return NULL; } diff --git a/libavcodec/decode.c b/libavcodec/decode.c index 0956a6ac6f..bf5fbcca97 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -1736,6 +1736,7 @@ int ff_decode_frame_props(AVCodecContext *avctx, AVFrame *frame) { AV_PKT_DATA_A53_CC, AV_FRAME_DATA_A53_CC }, { AV_PKT_DATA_ICC_PROFILE, AV_FRAME_DATA_ICC_PROFILE }, { AV_PKT_DATA_S12M_TIMECODE, AV_FRAME_DATA_S12M_TIMECODE }, + { AV_PKT_DATA_DYNAMIC_HDR10_PLUS, AV_FRAME_DATA_DYNAMIC_HDR_PLUS }, }; if (IS_EMPTY(pkt) && av_fifo_size(avctx->internal->pkt_props) >= sizeof(*pkt)) diff --git a/libavcodec/libvpxenc.c b/libavcodec/libvpxenc.c index 3f36943c12..373c6e8418 100644 --- a/libavcodec/libvpxenc.c +++ b/libavcodec/libvpxenc.c @@ -63,6 +63,12 @@ struct FrameListData { struct FrameListData *next; }; +typedef struct FrameHDR10PlusList { + int64_t pts; + AVBufferRef *hdr10_plus; + struct FrameHDR10PlusList *next; +} FrameHDR10PlusList; + typedef struct VPxEncoderContext { AVClass *class; struct vpx_codec_ctx encoder; @@ -120,6 +126,7 @@ typedef struct VPxEncoderContext { int tune_content; int corpus_complexity; int tpl_model; + struct FrameHDR10PlusList *hdr10_plus_list; /** * If the driver does not support ROI then warn the first time we * encounter a frame with ROI side data. @@ -315,6 +322,53 @@ static av_cold void free_frame_list(struct FrameListData *list) } } + +static void add_hdr10_plus(void *list, struct FrameHDR10PlusList *data) +{ + struct FrameHDR10PlusList **p = list; + while (*p) + p = &(*p)->next; + *p = data; + data->next = NULL; +} + +static av_cold void free_hdr10_plus(struct FrameHDR10PlusList *p) +{ + av_buffer_unref(&p->hdr10_plus); + av_free(p); +} + +static av_cold void free_hdr10_plus_list(struct FrameHDR10PlusList *list) +{ + struct FrameHDR10PlusList *p = list; + while (p) { + list = list->next; + free_hdr10_plus(p); + p = list; + } +} + +static int copy_hdr10_plus_to_pkt(void *list, AVPacket *pkt) +{ + struct FrameHDR10PlusList **p = list; + struct FrameHDR10PlusList *head = *p; + + if(head && pkt && head->hdr10_plus && head->pts == pkt->pts) { + uint8_t *data; + *p = (*p)->next; + data = av_packet_new_side_data(pkt, AV_PKT_DATA_DYNAMIC_HDR10_PLUS, head->hdr10_plus->size); + + if (!data) { + free_hdr10_plus(head); + return AVERROR(ENOMEM); + } + memcpy(data, head->hdr10_plus->data, head->hdr10_plus->size); + free_hdr10_plus(head); + + } + return 0; +} + static av_cold int codecctl_int(AVCodecContext *avctx, enum vp8e_enc_control_id id, int val) { @@ -383,6 +437,7 @@ static av_cold int vpx_free(AVCodecContext *avctx) av_freep(&ctx->twopass_stats.buf); av_freep(&avctx->stats_out); free_frame_list(ctx->coded_frame_list); + free_hdr10_plus_list(ctx->hdr10_plus_list); return 0; } @@ -1245,6 +1300,12 @@ FF_ENABLE_DEPRECATION_WARNINGS AV_WB64(side_data, 1); memcpy(side_data + 8, cx_frame->buf_alpha, cx_frame->sz_alpha); } + if(cx_frame->frame_number != -1) { // Not invisible frame + VPxContext *ctx = avctx->priv_data; + int err = copy_hdr10_plus_to_pkt(&ctx->hdr10_plus_list, pkt); + if (err < 0) + return err; + } } else { return ret; } @@ -1579,6 +1640,7 @@ static int vpx_encode(AVCodecContext *avctx, AVPacket *pkt, const struct vpx_codec_enc_cfg *enccfg = ctx->encoder.config.enc; vpx_svc_layer_id_t layer_id; int layer_id_valid = 0; + AVFrameSideData *hdr10_plus_metadata; if (frame) { const AVFrameSideData *sd = av_frame_get_side_data(frame, AV_FRAME_DATA_REGIONS_OF_INTEREST); @@ -1655,6 +1717,15 @@ static int vpx_encode(AVCodecContext *avctx, AVPacket *pkt, vp9_encode_set_roi(avctx, frame->width, frame->height, sd); } } + + // Add HDR10+ metadata to queue. + hdr10_plus_metadata = av_frame_get_side_data(frame, AV_FRAME_DATA_DYNAMIC_HDR_PLUS); + if(hdr10_plus_metadata){ + struct FrameHDR10PlusList *data = av_malloc(sizeof(struct FrameHDR10PlusList)); + data->pts = frame->pts; + data->hdr10_plus = av_buffer_ref(hdr10_plus_metadata->buf); + add_hdr10_plus(&ctx->hdr10_plus_list, data); + } } // this is for encoding with preset temporal layering patterns defined in diff --git a/libavcodec/packet.h b/libavcodec/packet.h index ca18ae631f..89e683b357 100644 --- a/libavcodec/packet.h +++ b/libavcodec/packet.h @@ -298,7 +298,15 @@ enum AVPacketSideDataType { * If its value becomes huge, some code using it * needs to be updated as it assumes it to be smaller than other limits. */ - AV_PKT_DATA_NB + AV_PKT_DATA_NB, + + /** + * HDR10+ dynamic metadata associated with a video frame. The metadata is in + * the form of the AVDynamicHDRPlus struct and contains + * information for color volume transform - application 4 of + * SPMTE 2094-40:2016 standard. + */ + AV_PKT_DATA_DYNAMIC_HDR10_PLUS }; #define AV_PKT_DATA_QUALITY_FACTOR AV_PKT_DATA_QUALITY_STATS //DEPRECATED