From patchwork Mon Mar 6 16:58:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 40603 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:d046:b0:cd:afd7:272c with SMTP id hv6csp3051781pzb; Mon, 6 Mar 2023 08:59:17 -0800 (PST) X-Google-Smtp-Source: AK7set8r8f83XXMcwPJ3njLYpbwmPNM3j5gsiv4pCcB/qdWMX1kzZaU2Q/fjrt1nJQUhGf6glIIH X-Received: by 2002:a17:907:7748:b0:8e8:6b69:2093 with SMTP id kx8-20020a170907774800b008e86b692093mr14160847ejc.25.1678121957285; Mon, 06 Mar 2023 08:59:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678121957; cv=none; d=google.com; s=arc-20160816; b=PAvPTCar5laTsJVIueX13k1WXuR7GamqZlcKJU4XTQtal0bTagADFGQRsbMllrhAQF 5zbVwuLOeFvsWhxy4xA4+0UlMZuHM26XeQWPO6iWEhsPpDuw0ojMfbGM9KkRAlMw4vP1 QhB+LhBL0s6oFhmvsFaT8D4mdwuEtrlYsFQYthFEb7nAiXJTTU420UpNkXZkPfsXss9U fcHawAvpxnyHHerknPx8oFdAlznnJUhSqQ96j8tFxPl904vvBk05D/EhoN15DivVRneZ 8etPdxkZsiIw5PqbULXHdp1nQw+WVf66zL1ip2lbRbNt1gKsDqkMht1BU/8PI2/DmaLh mpJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=fpjU6qhw2M5WWf/5nZHS7B5eroDL9YFZguMQLE8hIsY=; b=IJelpcnaVdWoT5fncK0pEG+pzOgsJIIYVrz1zwvNQJh6Kas1SLJS/TUIZANy/GuH+I eOrNjiFAQpWKu/DEYaBXvyo1TsX01Jp2ZhdvpHKkUHdMysanaq2hPZ6Ve76W7Ue/idex 2xjdXbAZNro/8NVFHwQ1W7V6Z1Tepck0EQa/Y3E0NQeIuBgIBf+uIWR5vhTGWVJml6GJ BdD/TV74fF4hrhda+U13uM9DBqIOymTCln6aMwa20PKLfOG/rwswO/pZRyD3xYLrj7t4 82IKUynAJu6zrwlTdlv+jy/AthtlxSBD/3k8GDCqJlVcqgfPvnwfr1HOSBIVEbPf2sse O2eQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=cDFSDipC; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a16-20020a170906275000b008c3aa86be5csi351222ejd.585.2023.03.06.08.59.16; Mon, 06 Mar 2023 08:59:17 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=cDFSDipC; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CEE2568BA97; Mon, 6 Mar 2023 18:59:12 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9F37F688388 for ; Mon, 6 Mar 2023 18:59:06 +0200 (EET) Received: by mail-ot1-f48.google.com with SMTP id g73-20020a9d12cf000000b006943a7df072so5666516otg.11 for ; Mon, 06 Mar 2023 08:59:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678121944; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=58eQF3tMoLIYv0tCY8Q7feORlmVlqHRiX0X6pq1whnM=; b=cDFSDipCK1NW/OTHgRPoLancG5Bdmqa1x0vJkaUFskF6D7t6BFTSzELUNF5Qx+vxTU Q1unEQjL2+RDn+1SVeweWii/0K+/EZnn/zELxFaEOsm1LTJ6FhHKP4wtJ9bLNA7ILB3q j6vKaUNUzuAibisCunNl0rpLGeRyBO1boj5Ruuwge3xWTE0sn1rxv6eUoK1ymsN4hW2I TH+SdFqZqyZkrR5GSHRv1uG1iQYtDfsuIon0lK/0to+bMEzieBGWaepUxISWz/PSqLlF e1Raod/QOO/PyhdpBfLoUZgS9YpBYkySqqZJ1LJZ7pTQCFzgfFt7jp5v1YKYJ6dffPxg RvKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678121944; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=58eQF3tMoLIYv0tCY8Q7feORlmVlqHRiX0X6pq1whnM=; b=ip+aW1Zyt+HsxhgpmPqXMXR9Q8cfRSq7f+/qm2Ap7ZVYu79yd+GZlm3bMvmoJn4fI+ 4j+StUcypgTD9SiCGrHmM+OMqqHLby2E56BM6mxjdqc6xAX7KrdikG80BJnvrNUfjwX1 022kVz/o9JDC2qF2rVrZTN3B1nVhEvToRnpnMhX+w5CggwY1QeKnAlW61yU9qmh4PKPL hCpYRGzR1GNBLwKb06HpB6SUT9b7c5U4Uj25Xway59I8eVx8yhk+cPmPjCEAa54hxIvR tfgS8gP5knwlArHG1eEbGJvh5L5dYMdPqJflBEPn/JNX2s0ZL6ZMhxETZO5pq4wMzQDV 3uqA== X-Gm-Message-State: AO0yUKVgjd9beSEDVk7zqlj0OHiSIHb1UhkNKvC3+UjTd8me3wZ5ceGj SEiWmnR3gQ5uso3DEyZK0y/YpclFdAo= X-Received: by 2002:a05:6830:3101:b0:690:dcaa:727b with SMTP id b1-20020a056830310100b00690dcaa727bmr6291481ots.27.1678121944587; Mon, 06 Mar 2023 08:59:04 -0800 (PST) Received: from localhost.localdomain (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id m2-20020a9d6ac2000000b00693d998f713sm4306572otq.35.2023.03.06.08.59.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 08:59:04 -0800 (PST) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Mon, 6 Mar 2023 13:58:56 -0300 Message-Id: <20230306165856.3398-1-jamrial@gmail.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avcodec/av1dec: parse and export Metadata OBUs X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: wtwh201+t6tC This includes Mastering Display, Content light level, and some ITU-T T35 metadata like closed captions and HDR10+. Signed-off-by: James Almer --- libavcodec/av1dec.c | 163 ++++++++++++++++++++++++++++++++++++++++++++ libavcodec/av1dec.h | 8 +++ 2 files changed, 171 insertions(+) diff --git a/libavcodec/av1dec.c b/libavcodec/av1dec.c index d83c902f1f..dda9a22cf2 100644 --- a/libavcodec/av1dec.c +++ b/libavcodec/av1dec.c @@ -21,13 +21,16 @@ #include "config_components.h" #include "libavutil/film_grain_params.h" +#include "libavutil/mastering_display_metadata.h" #include "libavutil/pixdesc.h" #include "libavutil/opt.h" #include "avcodec.h" #include "av1dec.h" +#include "atsc_a53.h" #include "bytestream.h" #include "codec_internal.h" #include "decode.h" +#include "dynamic_hdr10_plus.h" #include "hwconfig.h" #include "profiles.h" #include "thread.h" @@ -645,6 +648,7 @@ fail: static av_cold int av1_decode_free(AVCodecContext *avctx) { AV1DecContext *s = avctx->priv_data; + AV1RawMetadataITUTT35 itut_t35; for (int i = 0; i < FF_ARRAY_ELEMS(s->ref); i++) { av1_frame_unref(avctx, &s->ref[i]); @@ -655,8 +659,14 @@ static av_cold int av1_decode_free(AVCodecContext *avctx) av_buffer_unref(&s->seq_ref); av_buffer_unref(&s->header_ref); + av_buffer_unref(&s->cll_ref); + av_buffer_unref(&s->mdcv_ref); av_freep(&s->tile_group_info); + while (s->itut_t35_fifo && av_fifo_read(s->itut_t35_fifo, &itut_t35, 1) >= 0) + av_buffer_unref(&itut_t35.payload_ref); + av_fifo_freep2(&s->itut_t35_fifo); + ff_cbs_fragment_free(&s->current_obu); ff_cbs_close(&s->cbc); @@ -771,6 +781,11 @@ static av_cold int av1_decode_init(AVCodecContext *avctx) if (ret < 0) return ret; + s->itut_t35_fifo = av_fifo_alloc2(1, sizeof(AV1RawMetadataITUTT35), + AV_FIFO_FLAG_AUTO_GROW); + if (!s->itut_t35_fifo) + return AVERROR(ENOMEM); + av_opt_set_int(s->cbc->priv_data, "operating_point", s->operating_point, 0); if (avctx->extradata && avctx->extradata_size) { @@ -852,6 +867,106 @@ fail: return ret; } +static int export_itut_t35(AVCodecContext *avctx, AVFrame *frame, + AV1RawMetadataITUTT35 *itut_t35) +{ + GetByteContext gb; + int ret, provider_code; + + bytestream2_init(&gb, itut_t35->payload, itut_t35->payload_size); + + provider_code = bytestream2_get_be16(&gb); + switch (provider_code) { + case 0x31: { // atsc_provider_code + uint32_t user_identifier = bytestream2_get_be32(&gb); + switch (user_identifier) { + case MKBETAG('G', 'A', '9', '4'): { // closed captions + AVBufferRef *buf = NULL; + + ret = ff_parse_a53_cc(&buf, gb.buffer, bytestream2_get_bytes_left(&gb)); + if (ret < 0) + return ret; + if (!ret) + break; + + if (!av_frame_new_side_data_from_buf(frame, AV_FRAME_DATA_A53_CC, buf)) + av_buffer_unref(&buf); + + avctx->properties |= FF_CODEC_PROPERTY_CLOSED_CAPTIONS; + break; + } + default: // ignore unsupported identifiers + break; + } + } + case 0x3C: { // smpte_provider_code + AVDynamicHDRPlus *hdrplus; + int provider_oriented_code = bytestream2_get_be16(&gb); + int application_identifier = bytestream2_get_byte(&gb); + + if (provider_oriented_code != 1 || application_identifier != 4) + break; + + hdrplus = av_dynamic_hdr_plus_create_side_data(frame); + if (!hdrplus) + return AVERROR(ENOMEM); + + ret = ff_parse_itu_t_t35_to_dynamic_hdr10_plus(hdrplus, gb.buffer, + bytestream2_get_bytes_left(&gb)); + if (ret < 0) + return ret; + break; + } + default: // ignore unsupported provider codes + break; + } + + return 0; +} + +static int export_metadata(AVCodecContext *avctx, AVFrame *frame) +{ + AV1DecContext *s = avctx->priv_data; + AV1RawMetadataITUTT35 itut_t35; + int ret = 0; + + if (s->mdcv) { + AVMasteringDisplayMetadata *mastering = av_mastering_display_metadata_create_side_data(frame); + if (!mastering) + return AVERROR(ENOMEM); + + for (int i = 0; i < 3; i++) { + mastering->display_primaries[i][0] = av_make_q(s->mdcv->primary_chromaticity_x[i], 1 << 16); + mastering->display_primaries[i][1] = av_make_q(s->mdcv->primary_chromaticity_y[i], 1 << 16); + } + mastering->white_point[0] = av_make_q(s->mdcv->white_point_chromaticity_x, 1 << 16); + mastering->white_point[1] = av_make_q(s->mdcv->white_point_chromaticity_y, 1 << 16); + + mastering->max_luminance = av_make_q(s->mdcv->luminance_max, 1 << 8); + mastering->min_luminance = av_make_q(s->mdcv->luminance_min, 1 << 14); + + mastering->has_primaries = 1; + mastering->has_luminance = 1; + } + + if (s->cll) { + AVContentLightMetadata *light = av_content_light_metadata_create_side_data(frame); + if (!light) + return AVERROR(ENOMEM); + + light->MaxCLL = s->cll->max_cll; + light->MaxFALL = s->cll->max_fall; + } + + while (av_fifo_read(s->itut_t35_fifo, &itut_t35, 1) >= 0) { + if (ret >= 0) + ret = export_itut_t35(avctx, frame, &itut_t35); + av_buffer_unref(&itut_t35.payload_ref); + } + + return ret; +} + static int export_film_grain(AVCodecContext *avctx, AVFrame *frame) { AV1DecContext *s = avctx->priv_data; @@ -928,6 +1043,10 @@ static int set_output_frame(AVCodecContext *avctx, AVFrame *frame, if (ret < 0) return ret; + ret = export_metadata(avctx, frame); + if (ret < 0) + return ret; + if (avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN) { ret = export_film_grain(avctx, frame); if (ret < 0) { @@ -1174,6 +1293,45 @@ static int av1_decode_frame(AVCodecContext *avctx, AVFrame *frame, case AV1_OBU_TEMPORAL_DELIMITER: case AV1_OBU_PADDING: case AV1_OBU_METADATA: + switch (obu->obu.metadata.metadata_type) { + case AV1_METADATA_TYPE_HDR_CLL: + av_buffer_unref(&s->cll_ref); + s->cll_ref = av_buffer_ref(unit->content_ref); + if (!s->cll_ref) { + s->cll = NULL; + ret = AVERROR(ENOMEM); + goto end; + } + s->cll = &obu->obu.metadata.metadata.hdr_cll; + break; + case AV1_METADATA_TYPE_HDR_MDCV: + av_buffer_unref(&s->mdcv_ref); + s->mdcv_ref = av_buffer_ref(unit->content_ref); + if (!s->mdcv_ref) { + s->mdcv = NULL; + ret = AVERROR(ENOMEM); + goto end; + } + s->mdcv = &obu->obu.metadata.metadata.hdr_mdcv; + break; + case AV1_METADATA_TYPE_ITUT_T35: { + AV1RawMetadataITUTT35 itut_t35; + memcpy(&itut_t35, &obu->obu.metadata.metadata.itut_t35, sizeof(itut_t35)); + itut_t35.payload_ref = av_buffer_ref(obu->obu.metadata.metadata.itut_t35.payload_ref); + if (!itut_t35.payload_ref) { + ret = AVERROR(ENOMEM); + goto end; + } + ret = av_fifo_write(s->itut_t35_fifo, &itut_t35, 1); + if (ret < 0) { + av_buffer_unref(&itut_t35.payload_ref); + goto end; + } + break; + } + default: + break; + } break; default: av_log(avctx, AV_LOG_DEBUG, @@ -1218,6 +1376,7 @@ end: static void av1_decode_flush(AVCodecContext *avctx) { AV1DecContext *s = avctx->priv_data; + AV1RawMetadataITUTT35 itut_t35; for (int i = 0; i < FF_ARRAY_ELEMS(s->ref); i++) av1_frame_unref(avctx, &s->ref[i]); @@ -1226,6 +1385,10 @@ static void av1_decode_flush(AVCodecContext *avctx) s->operating_point_idc = 0; s->raw_frame_header = NULL; s->raw_seq = NULL; + s->cll = NULL; + s->mdcv = NULL; + while (s->itut_t35_fifo && av_fifo_read(s->itut_t35_fifo, &itut_t35, 1) >= 0) + av_buffer_unref(&itut_t35.payload_ref); ff_cbs_flush(s->cbc); } diff --git a/libavcodec/av1dec.h b/libavcodec/av1dec.h index 82c7084e99..cef899f81f 100644 --- a/libavcodec/av1dec.h +++ b/libavcodec/av1dec.h @@ -23,6 +23,7 @@ #include +#include "libavutil/fifo.h" #include "libavutil/buffer.h" #include "libavutil/frame.h" #include "libavutil/pixfmt.h" @@ -73,6 +74,13 @@ typedef struct AV1DecContext { AVBufferRef *header_ref; AV1RawFrameHeader *raw_frame_header; TileGroupInfo *tile_group_info; + + AVBufferRef *cll_ref; + AV1RawMetadataHDRCLL *cll; + AVBufferRef *mdcv_ref; + AV1RawMetadataHDRMDCV *mdcv; + AVFifo *itut_t35_fifo; + uint16_t tile_num; uint16_t tg_start; uint16_t tg_end;