From patchwork Thu Jan 11 16:15:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diego Felix de Souza via ffmpeg-devel X-Patchwork-Id: 45568 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:bf2f:b0:199:de12:6fa6 with SMTP id gc47csp1798625pzb; Thu, 11 Jan 2024 08:16:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IHqLWUY6G/lx9ljp2JUgdXSgXQO5+gCmSBuxFlnm8cBGAe1I7i4ore0igiYqst1NW5CvxoZ X-Received: by 2002:a17:907:3f25:b0:a28:dcfb:1e68 with SMTP id hq37-20020a1709073f2500b00a28dcfb1e68mr1316239ejc.10.1704989788556; Thu, 11 Jan 2024 08:16:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704989788; cv=none; d=google.com; s=arc-20160816; b=qYIfYR67XpAUeOIfoOsDTgdw1PYnoo0IX+0pZj+CNZ5ujqVi0maDnLXcy24pC047F1 G+kE0CHL8itJsOD1GQedbcb9/nP7bHW5YKEbAp0D4abmn9Ie1P2SL2W9cAk4lI9/M7OR pBqoM95mRXqeYQlNLNU/sIb6sVORr0u/et1K07vkK755vW0DExUzkTCy5I98Z2Ddl+6b wLV5WB7/cC+OZOm5Mj7WZ6GEpQ8fQVxaU77gSXbQrbs9J7ZkhfF7QgKn/k5U0MlbWf3Y JEiAxYHAccsgoIfo8hyZ80BhfJgVR7wBEP9ddvbcTWgcot0I72REx11EPS0l7G0jKi/S bxPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:message-id:delivered-to; bh=ihIbRSWXfir5AhSrd1tV4AZ+j0X7d3Ns0iWBKt+vfvU=; fh=frUT2hx60kzuIVmkNUv7bM3Y/kw3lxlKoLH+4fdDqZk=; b=NwRiEic2uxgd6O6/MQO/8dYz/Nu7ZORYXzcOFS4eDAcPJtTY6VAfeCeOXrACa4Zmkl 80ubAy+LnpO9ysbJQE6i5FekU5qEvRin9ouWeFaFQaNOwIqjGEyDzkP0H/308eFABpEN 5ppXBv+RZluTZ6n5DzXXkAExnOYgxpFZfNALyoSDLy/Y/eV0pK+i9e4RFeJHDd1R+IKX Eo6ziUj8J8g/vb+gmlrhOqCjnNo38Grk2jBZs5K0wc5GnXDA8g7uo5mAr2SxR2Jw5FCb DrREPp6VvcDU5zKJ3NyRfZKeCNIEjr6BeEokA6kxXeghdn6uDvwojSWLHMPmpRFN/otx gCgw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b2-20020a170906038200b00a2c96f159aasi69000eja.385.2024.01.11.08.16.27; Thu, 11 Jan 2024 08:16:28 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 450E368CA1F; Thu, 11 Jan 2024 18:16:24 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-251-84.mail.qq.com (unknown [203.205.251.84]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id DC7AB68CFAA for ; Thu, 11 Jan 2024 18:16:15 +0200 (EET) Received: from localhost.localdomain ([103.102.203.208]) by newxmesmtplogicsvrszb9-0.qq.com (NewEsmtp) with SMTP id 40A1B273; Fri, 12 Jan 2024 00:16:10 +0800 X-QQ-mid: xmsmtpt1704989770tfnlnh7ni Message-ID: X-QQ-XMAILINFO: NafziRg7Bx69rvi+/XXYj25jw1cTiwaS8x+VTi2WT9XIeFK296Ry3lmxnDrrMF ME2fsIcbnTgf15Xlki1IhDKNOKqQJx0fjLqO4putMVqOoML15ToXykG/PqSHcqhiJXjDElBC24m7 TymDZtsyLdTJKbyEtHITI/MJq0/eVqay43Tc/ipvuJnu3abPcgBtC5c5CruydL/UH9KGK5nx2Y5S R90UqE6lyLnFYKNjeIl3ZDdp2odipvUn4ME679IPpiP23PUnEufpdDyCK9S4BR8W47yhAVLeGjYn pHRIfce4P5fACcBwZk5y+5TMPYzJjxv5AUX4K/CRyt7S5KbxTrft18Xx+FihJJDoE7AKwY6A2tcZ 2JGjrvKxbooeiTSY2Xv9FTk8ik6otOOHgf9T7ehg92YIh12abcFipCmkrXQlG1Lg3Qxws4Jp+Xe7 oIo+gjgTWyOmuCFoc8jq0LVjBQ+kHSvYkFcuKQpF4co3FMkzeXGnIKKkUfes1ELULuhFZFSSF44a bhHOfaE7Ibz99qwTEPRz8cU+rnz46l90TN4FVjYWtV4+cvB/4fdK9N9El1vsRclrxBoeqweTWtdB 6Tf4Hil130keFCFrIDX9WiOcov+mU++fbiyEQkebjOFl1H8FjnPSZM8Y7VqiBjOX1AgUrL8Uyxpt Pogfb2Eb73meyAGlTPpGIpMnE7/7IsRpSPMTJBkV7N5i1sllTTFFvFvTxMCP8LOmUer3vpwN//kg ki0i5wcRoRzYGIOcO/kl55JzRJFVaZrTe0pwWcx3aDYmfdeH3sTEW4JWaOYyo+13BdTE7HkSJT85 g7ym8G4y7tU2D5Ez4kd9PMuIa846qeaBeJ1SkF3VuGjlpb32KHJE/gfmAHCArlOCHzSPzlKzAHmt GmzOmp+Cuy45gKKI7p8jMoEE5UXP0C4hjPdFZOwiBrhKco/PMg1vDdvcE/DXwwyINL44IP4FUB X-QQ-XMRINFO: MPJ6Tf5t3I/ycC2BItcBVIA= To: ffmpeg-devel@ffmpeg.org Date: Fri, 12 Jan 2024 00:15:57 +0800 X-OQ-MSGID: <20240111161557.22416-1-411294962@qq.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <29ed7620-f60f-4f31-bc22-3f0903456c07@gmail.com> References: <29ed7620-f60f-4f31-bc22-3f0903456c07@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 1/2] avformat/flvdec: support enhanced flv PacketTypeMetadata X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: zhupengfei via ffmpeg-devel From: Diego Felix de Souza via ffmpeg-devel Reply-To: FFmpeg development discussions and patches Cc: 411294962@qq.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 8e3DNoHs5dxX From: Zhu Pengfei <411294962@qq.com> Signed-off-by: Zhu Pengfei <411294962@qq.com> --- libavformat/flvdec.c | 178 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 177 insertions(+), 1 deletion(-) diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index e25b5bd163..65815e51f9 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -34,6 +34,7 @@ #include "libavutil/intfloat.h" #include "libavutil/intreadwrite.h" #include "libavutil/mathematics.h" +#include "libavutil/mastering_display_metadata.h" #include "avformat.h" #include "demux.h" #include "internal.h" @@ -45,6 +46,28 @@ #define MAX_DEPTH 16 ///< arbitrary limit to prevent unbounded recursion +typedef struct FLVMasteringMeta { + double r_x; + double r_y; + double g_x; + double g_y; + double b_x; + double b_y; + double white_x; + double white_y; + double max_luminance; + double min_luminance; +} FLVMasteringMeta; + +typedef struct FLVMetaVideoColor { + uint64_t matrix_coefficients; + uint64_t transfer_characteristics; + uint64_t primaries; + uint64_t max_cll; + uint64_t max_fall; + FLVMasteringMeta mastering_meta; +} FLVMetaVideoColor; + typedef struct FLVContext { const AVClass *class; ///< Class for private options. int trust_metadata; ///< configure streams according onMetaData @@ -80,6 +103,8 @@ typedef struct FLVContext { int64_t time_offset; int64_t time_pos; + FLVMetaVideoColor *metaVideoColor; + int meta_color_info_flag; } FLVContext; /* AMF date type */ @@ -524,6 +549,7 @@ static int amf_parse_object(AVFormatContext *s, AVStream *astream, FLVContext *flv = s->priv_data; AVIOContext *ioc; AMFDataType amf_type; + FLVMetaVideoColor *meta_video_color = flv->metaVideoColor; char str_val[1024]; double num_val; amf_date date; @@ -672,6 +698,43 @@ static int amf_parse_object(AVFormatContext *s, AVStream *astream, } } + if (meta_video_color) { + if (amf_type == AMF_DATA_TYPE_NUMBER || + amf_type == AMF_DATA_TYPE_BOOL) { + if (!strcmp(key, "colorPrimaries")) { + meta_video_color->primaries = num_val; + } else if (!strcmp(key, "transferCharacteristics")) { + meta_video_color->transfer_characteristics = num_val; + } else if (!strcmp(key, "matrixCoefficients")) { + meta_video_color->matrix_coefficients = num_val; + } else if (!strcmp(key, "maxFall")) { + meta_video_color->max_fall = num_val; + } else if (!strcmp(key, "maxCLL")) { + meta_video_color->max_cll = num_val; + } else if (!strcmp(key, "redX")) { + meta_video_color->mastering_meta.r_x = num_val; + } else if (!strcmp(key, "redY")) { + meta_video_color->mastering_meta.r_y = num_val; + } else if (!strcmp(key, "greenX")) { + meta_video_color->mastering_meta.g_x = num_val; + } else if (!strcmp(key, "greenY")) { + meta_video_color->mastering_meta.g_y = num_val; + } else if (!strcmp(key, "blueX")) { + meta_video_color->mastering_meta.b_x = num_val; + } else if (!strcmp(key, "blueY")) { + meta_video_color->mastering_meta.b_y = num_val; + } else if (!strcmp(key, "whitePointX")) { + meta_video_color->mastering_meta.white_x = num_val; + } else if (!strcmp(key, "whitePointY")) { + meta_video_color->mastering_meta.white_y = num_val; + } else if (!strcmp(key, "maxLuminance")) { + meta_video_color->mastering_meta.max_luminance = num_val; + } else if (!strcmp(key, "minLuminance")) { + meta_video_color->mastering_meta.min_luminance = num_val; + } + } + } + if (amf_type == AMF_DATA_TYPE_OBJECT && s->nb_streams == 1 && ((!apar && !strcmp(key, "audiocodecid")) || (!vpar && !strcmp(key, "videocodecid")))) @@ -824,6 +887,7 @@ static int flv_read_close(AVFormatContext *s) av_freep(&flv->new_extradata[i]); av_freep(&flv->keyframe_times); av_freep(&flv->keyframe_filepositions); + av_freep(&flv->metaVideoColor); return 0; } @@ -1028,6 +1092,104 @@ static int resync(AVFormatContext *s) return AVERROR_EOF; } +static int flv_parse_video_color_info(AVFormatContext *s, AVStream *st, int64_t next_pos) +{ + FLVContext *flv = s->priv_data; + AMFDataType type; + AVIOContext *ioc; + char buffer[32]; + ioc = s->pb; + + // first object needs to be "colorInfo" string + type = avio_r8(ioc); + if (type != AMF_DATA_TYPE_STRING || + amf_get_string(ioc, buffer, sizeof(buffer)) < 0) + return TYPE_UNKNOWN; + + if (strcmp(buffer, "colorInfo")) { + av_log(s, AV_LOG_DEBUG, "Unknown type %s\n", buffer); + return TYPE_UNKNOWN; + } + + flv->metaVideoColor = av_mallocz(sizeof(FLVMetaVideoColor)); + if (!flv->metaVideoColor) { + return AVERROR(ENOMEM); + } + flv->meta_color_info_flag = 1; + amf_parse_object(s, NULL, NULL, buffer, next_pos, 0); // parse metadata + return 0; +} + +static int flv_update_video_color_info(AVFormatContext *s, AVStream *st) +{ + FLVContext *flv = s->priv_data; + const FLVMetaVideoColor* meta_video_color = flv->metaVideoColor; + const FLVMasteringMeta *mastering_meta = &meta_video_color->mastering_meta; + + int has_mastering_primaries, has_mastering_luminance; + // Mastering primaries are CIE 1931 coords, and must be > 0. + has_mastering_primaries = + mastering_meta->r_x > 0 && mastering_meta->r_y > 0 && + mastering_meta->g_x > 0 && mastering_meta->g_y > 0 && + mastering_meta->b_x > 0 && mastering_meta->b_y > 0 && + mastering_meta->white_x > 0 && mastering_meta->white_y > 0; + has_mastering_luminance = mastering_meta->max_luminance >= 0 && mastering_meta->min_luminance >= 0; + + if (meta_video_color->matrix_coefficients != AVCOL_SPC_RESERVED) + st->codecpar->color_space = meta_video_color->matrix_coefficients; + if (meta_video_color->primaries != AVCOL_PRI_RESERVED && + meta_video_color->primaries != AVCOL_PRI_RESERVED0) + st->codecpar->color_primaries = meta_video_color->primaries; + if (meta_video_color->transfer_characteristics != AVCOL_TRC_RESERVED && + meta_video_color->transfer_characteristics != AVCOL_TRC_RESERVED0) + st->codecpar->color_trc = meta_video_color->transfer_characteristics; + + if (meta_video_color->max_cll && meta_video_color->max_fall) { + size_t size = 0; + AVContentLightMetadata *metadata = av_content_light_metadata_alloc(&size); + if (!metadata) + return AVERROR(ENOMEM); + if (!av_packet_side_data_add(&st->codecpar->coded_side_data, &st->codecpar->nb_coded_side_data, + AV_PKT_DATA_CONTENT_LIGHT_LEVEL, metadata, size, 0)) { + av_freep(&metadata); + return AVERROR(ENOMEM); + } + metadata->MaxCLL = meta_video_color->max_cll; + metadata->MaxFALL = meta_video_color->max_fall; + } + + if (has_mastering_primaries || has_mastering_luminance) { + AVMasteringDisplayMetadata *metadata; + AVPacketSideData *sd = av_packet_side_data_new(&st->codecpar->coded_side_data, + &st->codecpar->nb_coded_side_data, + AV_PKT_DATA_MASTERING_DISPLAY_METADATA, + sizeof(AVMasteringDisplayMetadata), 0); + if (!sd) + return AVERROR(ENOMEM); + metadata = (AVMasteringDisplayMetadata*)sd->data; + memset(metadata, 0, sizeof(AVMasteringDisplayMetadata)); + // hdrCll + if (has_mastering_luminance) { + metadata->max_luminance = av_d2q(mastering_meta->max_luminance, INT_MAX); + metadata->min_luminance = av_d2q(mastering_meta->min_luminance, INT_MAX); + metadata->has_luminance = 1; + } + // hdrMdcv + if (has_mastering_primaries) { + metadata->display_primaries[0][0] = av_d2q(mastering_meta->r_x, INT_MAX); + metadata->display_primaries[0][1] = av_d2q(mastering_meta->r_y, INT_MAX); + metadata->display_primaries[1][0] = av_d2q(mastering_meta->g_x, INT_MAX); + metadata->display_primaries[1][1] = av_d2q(mastering_meta->g_y, INT_MAX); + metadata->display_primaries[2][0] = av_d2q(mastering_meta->b_x, INT_MAX); + metadata->display_primaries[2][1] = av_d2q(mastering_meta->b_y, INT_MAX); + metadata->white_point[0] = av_d2q(mastering_meta->white_x, INT_MAX); + metadata->white_point[1] = av_d2q(mastering_meta->white_y, INT_MAX); + metadata->has_primaries = 1; + } + } + return 0; +} + static int flv_read_packet(AVFormatContext *s, AVPacket *pkt) { FLVContext *flv = s->priv_data; @@ -1100,8 +1262,17 @@ retry: video_codec_id = avio_rb32(s->pb); size -= 4; } - if ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) + + if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { + type = flags & 0x0F; + if (type == PacketTypeMetadata) { + int ret = flv_parse_video_color_info(s, st, next); + av_log(s, AV_LOG_DEBUG, "enhanced flv parse metadata ret %d and skip\n", ret); + } + goto skip; + } else if ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { goto skip; + } } else if (type == FLV_TAG_TYPE_META) { stream_type=FLV_STREAM_TYPE_SUBTITLE; if (size > 13 + 1 + 4) { // Header-type metadata stuff @@ -1287,6 +1458,11 @@ retry_duration: goto leave; } + if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && flv->meta_color_info_flag) { + flv_update_video_color_info(s, st); // update av packet side data + flv->meta_color_info_flag = 0; + } + if (st->codecpar->codec_id == AV_CODEC_ID_H264 || st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || (st->codecpar->codec_id == AV_CODEC_ID_HEVC && type == PacketTypeCodedFrames)) { // sign extension