From patchwork Sat Mar 2 08:51:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diego Felix de Souza via ffmpeg-devel X-Patchwork-Id: 46694 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:a919:b0:19e:cdac:8cce with SMTP id cd25csp1750590pzb; Sat, 2 Mar 2024 00:51:52 -0800 (PST) X-Forwarded-Encrypted: i=2; AJvYcCVO9j3GccS2j2AmMW538kpDn4c9F/zuJ37WhHYDProeSHL8UL9F/bUB5BGxblqWCO86d9xwMDA1+HaPy3gF0kQqAj3j1K72QdS6KQ== X-Google-Smtp-Source: AGHT+IGa2TX69KznlGzUoLIalLm+k3/mZbBp9DwL4tp/FwKETwLN3Tq27FJ4zmTaoKr0lWVwT4pA X-Received: by 2002:a50:ed02:0:b0:566:c2f8:a47d with SMTP id j2-20020a50ed02000000b00566c2f8a47dmr2694641eds.1.1709369512486; Sat, 02 Mar 2024 00:51:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1709369512; cv=none; d=google.com; s=arc-20160816; b=s6q/dI/ZF4TR94vVe55d7vRjm8E5egiAGmDnLlg9n/NXoGvRHhOJN/Z1UCQrBKr8nG t1HAWh9USLmS0ChAoqdromIpLjfqX54lUuI2QOF4ZK+ZiO4unP01sVZTrH9fhPT8lySa y5Hbbjm8SqznoHvuDlX89YMMsGZKYSUzH3F/dfXrTkLV2LZShjsDFHS5YiAqn2Bmy/QW 6o0f2jPorawWnbwIPt1QqDgDlEMHfRW0mjstPYtjrXCca+8toMfhQkTvW//xTINj0TFi aRcpjKniiWB3lYnbz/0gnYMQiEmUy1QdMQrWCYvTIS0xBfH5b2rXnA4tkjgbPXLEYn6j EQeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:message-id:delivered-to; bh=nuBxoihw2PbUhr9LIzO8DWKVpw3zvgDNO/SPR9/HgSI=; fh=frUT2hx60kzuIVmkNUv7bM3Y/kw3lxlKoLH+4fdDqZk=; b=Cy6YllYt+URg7FajBdVyPaXEhqP8FUYq9BH+TiM9YDaewQQBOQs2QL3jYSp5AO9Xga 68co9s5Msqh4shmvFv7r5z1mHiPiDmAn3Ka229kz1wCwGATUSoTLls2okC4chfDfvWAb f8B9WI4GDFmpwTj20e5b2Vla3e+91yv2AcsTCgxq9XI5kX+l+q7cWnbPgIgHkuCGHfS9 /2NN3ffFZIDBUPa4Lhfm7egEFiiwT3H2VwXtIj7lQdIsV564utcmQZf310hR4ALqmkFE jfWjFBHvkyNULM58y0CZsA4XLlgNxTBEQxOzpnNUuPLGpKu2MqktpqJkteAoiL5w82og Ff7A==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c35-20020a509fa6000000b00566e2ee9621si988605edf.261.2024.03.02.00.51.51; Sat, 02 Mar 2024 00:51:52 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 431C368D2F1; Sat, 2 Mar 2024 10:51:48 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-57-49.mail.qq.com (out162-62-57-49.mail.qq.com [162.62.57.49]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C0C9F68D2A3 for ; Sat, 2 Mar 2024 10:51:40 +0200 (EET) Received: from localhost.localdomain ([163.125.127.26]) by newxmesmtplogicsvrszb6-0.qq.com (NewEsmtp) with SMTP id CE28AC25; Sat, 02 Mar 2024 16:51:34 +0800 X-QQ-mid: xmsmtpt1709369494t7cexd1px Message-ID: X-QQ-XMAILINFO: NT7dO8ok7FP2vosaY1usgY/gpnk1q8F063P+Qhyn9OKGbLp5U3B/MUGAyCmUln NyAwDevNbhJ/MqBomc012pDp84WR/DlwPo5q3gB1C2akAE/tQiGr0dr/QvhR6CWOr8MDrVO9aOGK aqz+Yi+T6Glu4br2m8IIfUqKnPaI3G0sh3XWcJd4tnB/5cWF6MSvTW0FWvDUDh11ne4JItMpgVq7 ml/9b/fQxtbvUxaiV2UO8QXOexC+y58w00SWV0SUzmsMmzOmiQWET/ap90T963hLFQB+9dE8Z0ms 9cVOTDNclt624/9oDqSbSvJk9H208u9whxlXZUlOfNaemsvUAPrujU+2VKHDPABl0HnpWyRnUDfB ATdBa8l85DYwdsgt1SBOxK2l9t6yy7713nReS5XOJCziY6s92mLkCo5PDP+5Jys3PkgzMkkl77Pp H5m7KgjWiVWk7NUHzFBc8xQryzbr6I1zw+Z7FHUUCkkia2zPUXndtb61reXA5jHKPoaGUn0qzF3n hRowt4XpmD/XR4lBRnv7ahGkeeS3TKLB+zhbNlgwLIOMyMLc7OEPHc4g7mdsR4nGneZFv27IhFTp TAaii+2EW7+FofItfJ//Pnri0hV2L9e0SeIiQ55MHO0C1scluBzPolWGZyxeHoAMiQRX5Qcf0i8e uhqWrR6b0ZCRa+JGZaSe+tp2idEbk5D4Mp6aUlVgqKTg9vDuVcw3XNbYbrbFl4RsGx8GkTLt/If1 UzaGbEXVa3hoyQqYQF8tq6CDqXz63BbGrjXkZMxsEGk0w9aLdAOU+70jRZzQj8rJbjKbbxBb7sRv VEUcnKV6t57raiJeqnFdtwNqyPYgOYxJtG9CTIZHQPvWXwdyArk4Hj1nkGdq1Lw61lgrW3uaVeIR R/ZNafg/pHo+CeN9wSbnaiGsTdVKFErEzEkz1xby6LBEWzOhj/KqBme7y1pCuhYrWh7Rm6h85ltE WPUgkSI+l6oTUyvqGE3y2kWnqrzayw+CF0PnevaC3lZL2lFNtcrSq09gOBg/SJSr1UojVyS/5OOz kDDGdBClC7lQK+Yq0a X-QQ-XMRINFO: NyFYKkN4Ny6FSmKK/uo/jdU= To: ffmpeg-devel@ffmpeg.org Date: Sat, 2 Mar 2024 16:51:25 +0800 X-OQ-MSGID: <20240302085125.23650-1-411294962@qq.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <29ed7620-f60f-4f31-bc22-3f0903456c07@gmail.com> References: <29ed7620-f60f-4f31-bc22-3f0903456c07@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 1/3] avformat/flvdec: support enhanced flv PacketTypeMetadata X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: zhupengfei via ffmpeg-devel From: Diego Felix de Souza via ffmpeg-devel Reply-To: FFmpeg development discussions and patches Cc: 411294962@qq.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: C0N5qIJiIAh2 From: Zhu Pengfei <411294962@qq.com> Signed-off-by: Zhu Pengfei <411294962@qq.com> --- libavformat/flvdec.c | 178 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 177 insertions(+), 1 deletion(-) diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index e25b5bd163..d0806c5338 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -34,6 +34,7 @@ #include "libavutil/intfloat.h" #include "libavutil/intreadwrite.h" #include "libavutil/mathematics.h" +#include "libavutil/mastering_display_metadata.h" #include "avformat.h" #include "demux.h" #include "internal.h" @@ -45,6 +46,28 @@ #define MAX_DEPTH 16 ///< arbitrary limit to prevent unbounded recursion +typedef struct FLVMasteringMeta { + double r_x; + double r_y; + double g_x; + double g_y; + double b_x; + double b_y; + double white_x; + double white_y; + double max_luminance; + double min_luminance; +} FLVMasteringMeta; + +typedef struct FLVMetaVideoColor { + uint64_t matrix_coefficients; + uint64_t transfer_characteristics; + uint64_t primaries; + uint64_t max_cll; + uint64_t max_fall; + FLVMasteringMeta mastering_meta; +} FLVMetaVideoColor; + typedef struct FLVContext { const AVClass *class; ///< Class for private options. int trust_metadata; ///< configure streams according onMetaData @@ -80,6 +103,8 @@ typedef struct FLVContext { int64_t time_offset; int64_t time_pos; + FLVMetaVideoColor *metaVideoColor; + int meta_color_info_flag; } FLVContext; /* AMF date type */ @@ -524,6 +549,7 @@ static int amf_parse_object(AVFormatContext *s, AVStream *astream, FLVContext *flv = s->priv_data; AVIOContext *ioc; AMFDataType amf_type; + FLVMetaVideoColor *meta_video_color = flv->metaVideoColor; char str_val[1024]; double num_val; amf_date date; @@ -672,6 +698,43 @@ static int amf_parse_object(AVFormatContext *s, AVStream *astream, } } + if (meta_video_color) { + if (amf_type == AMF_DATA_TYPE_NUMBER || + amf_type == AMF_DATA_TYPE_BOOL) { + if (!strcmp(key, "colorPrimaries")) { + meta_video_color->primaries = num_val; + } else if (!strcmp(key, "transferCharacteristics")) { + meta_video_color->transfer_characteristics = num_val; + } else if (!strcmp(key, "matrixCoefficients")) { + meta_video_color->matrix_coefficients = num_val; + } else if (!strcmp(key, "maxFall")) { + meta_video_color->max_fall = num_val; + } else if (!strcmp(key, "maxCLL")) { + meta_video_color->max_cll = num_val; + } else if (!strcmp(key, "redX")) { + meta_video_color->mastering_meta.r_x = num_val; + } else if (!strcmp(key, "redY")) { + meta_video_color->mastering_meta.r_y = num_val; + } else if (!strcmp(key, "greenX")) { + meta_video_color->mastering_meta.g_x = num_val; + } else if (!strcmp(key, "greenY")) { + meta_video_color->mastering_meta.g_y = num_val; + } else if (!strcmp(key, "blueX")) { + meta_video_color->mastering_meta.b_x = num_val; + } else if (!strcmp(key, "blueY")) { + meta_video_color->mastering_meta.b_y = num_val; + } else if (!strcmp(key, "whitePointX")) { + meta_video_color->mastering_meta.white_x = num_val; + } else if (!strcmp(key, "whitePointY")) { + meta_video_color->mastering_meta.white_y = num_val; + } else if (!strcmp(key, "maxLuminance")) { + meta_video_color->mastering_meta.max_luminance = num_val; + } else if (!strcmp(key, "minLuminance")) { + meta_video_color->mastering_meta.min_luminance = num_val; + } + } + } + if (amf_type == AMF_DATA_TYPE_OBJECT && s->nb_streams == 1 && ((!apar && !strcmp(key, "audiocodecid")) || (!vpar && !strcmp(key, "videocodecid")))) @@ -824,6 +887,7 @@ static int flv_read_close(AVFormatContext *s) av_freep(&flv->new_extradata[i]); av_freep(&flv->keyframe_times); av_freep(&flv->keyframe_filepositions); + av_freep(&flv->metaVideoColor); return 0; } @@ -1028,6 +1092,104 @@ static int resync(AVFormatContext *s) return AVERROR_EOF; } +static int flv_parse_video_color_info(AVFormatContext *s, AVStream *st, int64_t next_pos) +{ + FLVContext *flv = s->priv_data; + AMFDataType type; + AVIOContext *ioc; + char buffer[32]; + ioc = s->pb; + + // first object needs to be "colorInfo" string + type = avio_r8(ioc); + if (type != AMF_DATA_TYPE_STRING || + amf_get_string(ioc, buffer, sizeof(buffer)) < 0) + return TYPE_UNKNOWN; + + if (strcmp(buffer, "colorInfo")) { + av_log(s, AV_LOG_DEBUG, "Unknown type %s\n", buffer); + return TYPE_UNKNOWN; + } + + flv->metaVideoColor = av_mallocz(sizeof(FLVMetaVideoColor)); + if (!flv->metaVideoColor) { + return AVERROR(ENOMEM); + } + flv->meta_color_info_flag = 1; + amf_parse_object(s, NULL, NULL, buffer, next_pos, 0); // parse metadata + return 0; +} + +static int flv_update_video_color_info(AVFormatContext *s, AVStream *st) +{ + FLVContext *flv = s->priv_data; + const FLVMetaVideoColor* meta_video_color = flv->metaVideoColor; + const FLVMasteringMeta *mastering_meta = &meta_video_color->mastering_meta; + + int has_mastering_primaries, has_mastering_luminance; + // Mastering primaries are CIE 1931 coords, and must be > 0. + has_mastering_primaries = + mastering_meta->r_x > 0 && mastering_meta->r_y > 0 && + mastering_meta->g_x > 0 && mastering_meta->g_y > 0 && + mastering_meta->b_x > 0 && mastering_meta->b_y > 0 && + mastering_meta->white_x > 0 && mastering_meta->white_y > 0; + has_mastering_luminance = mastering_meta->max_luminance > 0 && mastering_meta->min_luminance > 0; + + if (meta_video_color->matrix_coefficients != AVCOL_SPC_RESERVED) + st->codecpar->color_space = meta_video_color->matrix_coefficients; + if (meta_video_color->primaries != AVCOL_PRI_RESERVED && + meta_video_color->primaries != AVCOL_PRI_RESERVED0) + st->codecpar->color_primaries = meta_video_color->primaries; + if (meta_video_color->transfer_characteristics != AVCOL_TRC_RESERVED && + meta_video_color->transfer_characteristics != AVCOL_TRC_RESERVED0) + st->codecpar->color_trc = meta_video_color->transfer_characteristics; + + if (meta_video_color->max_cll && meta_video_color->max_fall) { + size_t size = 0; + AVContentLightMetadata *metadata = av_content_light_metadata_alloc(&size); + if (!metadata) + return AVERROR(ENOMEM); + if (!av_packet_side_data_add(&st->codecpar->coded_side_data, &st->codecpar->nb_coded_side_data, + AV_PKT_DATA_CONTENT_LIGHT_LEVEL, metadata, size, 0)) { + av_freep(&metadata); + return AVERROR(ENOMEM); + } + metadata->MaxCLL = meta_video_color->max_cll; + metadata->MaxFALL = meta_video_color->max_fall; + } + + if (has_mastering_primaries || has_mastering_luminance) { + AVMasteringDisplayMetadata *metadata; + AVPacketSideData *sd = av_packet_side_data_new(&st->codecpar->coded_side_data, + &st->codecpar->nb_coded_side_data, + AV_PKT_DATA_MASTERING_DISPLAY_METADATA, + sizeof(AVMasteringDisplayMetadata), 0); + if (!sd) + return AVERROR(ENOMEM); + metadata = (AVMasteringDisplayMetadata*)sd->data; + memset(metadata, 0, sizeof(AVMasteringDisplayMetadata)); + // hdrCll + if (has_mastering_luminance) { + metadata->max_luminance = av_d2q(mastering_meta->max_luminance, INT_MAX); + metadata->min_luminance = av_d2q(mastering_meta->min_luminance, INT_MAX); + metadata->has_luminance = 1; + } + // hdrMdcv + if (has_mastering_primaries) { + metadata->display_primaries[0][0] = av_d2q(mastering_meta->r_x, INT_MAX); + metadata->display_primaries[0][1] = av_d2q(mastering_meta->r_y, INT_MAX); + metadata->display_primaries[1][0] = av_d2q(mastering_meta->g_x, INT_MAX); + metadata->display_primaries[1][1] = av_d2q(mastering_meta->g_y, INT_MAX); + metadata->display_primaries[2][0] = av_d2q(mastering_meta->b_x, INT_MAX); + metadata->display_primaries[2][1] = av_d2q(mastering_meta->b_y, INT_MAX); + metadata->white_point[0] = av_d2q(mastering_meta->white_x, INT_MAX); + metadata->white_point[1] = av_d2q(mastering_meta->white_y, INT_MAX); + metadata->has_primaries = 1; + } + } + return 0; +} + static int flv_read_packet(AVFormatContext *s, AVPacket *pkt) { FLVContext *flv = s->priv_data; @@ -1100,8 +1262,17 @@ retry: video_codec_id = avio_rb32(s->pb); size -= 4; } - if ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) + + if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { + type = flags & 0x0F; + if (type == PacketTypeMetadata) { + int ret = flv_parse_video_color_info(s, st, next); + av_log(s, AV_LOG_DEBUG, "enhanced flv parse metadata ret %d and skip\n", ret); + } + goto skip; + } else if ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { goto skip; + } } else if (type == FLV_TAG_TYPE_META) { stream_type=FLV_STREAM_TYPE_SUBTITLE; if (size > 13 + 1 + 4) { // Header-type metadata stuff @@ -1287,6 +1458,11 @@ retry_duration: goto leave; } + if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && flv->meta_color_info_flag) { + flv_update_video_color_info(s, st); // update av packet side data + flv->meta_color_info_flag = 0; + } + if (st->codecpar->codec_id == AV_CODEC_ID_H264 || st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || (st->codecpar->codec_id == AV_CODEC_ID_HEVC && type == PacketTypeCodedFrames)) { // sign extension