From patchwork Wed Nov 15 14:40:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Diego Felix de Souza via ffmpeg-devel X-Patchwork-Id: 44673 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:92a5:b0:181:818d:5e7f with SMTP id q37csp2725776pzg; Wed, 15 Nov 2023 06:40:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IH4p2If0Hkh1RSuk+jujwouHu/13VOQvrCXY7T8DMsd+Mjd8qXhWgM06/FAwCKTGCFQX/Ps X-Received: by 2002:a05:6402:1a36:b0:540:caed:3619 with SMTP id be22-20020a0564021a3600b00540caed3619mr9362459edb.24.1700059229305; Wed, 15 Nov 2023 06:40:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700059229; cv=none; d=google.com; s=arc-20160816; b=AM6Lha6EVR8J0+P032NrJN0OQn1Ao7jgAe1Xh8KRzb8Mj08S5+XKvcq3I8aJha5+6J S5/810jd4JxrrUUcRx4fIjE4mQ5+ALaAG6aUZnuiQT0gBBm+/EnQ3gknL57sIItpKUFX OT2ejmHHPjy+L19/zPnURfTcgAx/eqn8zMTV6Okv/kZwXWDBx/Od3okqsPGYY6gjwxUF xrs8zWnT4YTl3Te4oj+kN66um4+13Y7gvgpK6X8yXoujfxLho61dPxTtF8TIva3weBMW W9yJMDt1o37GU59c2fAHUQj/1W404MDB2hHG2UC6BCefnVj6ieOV9NFROVNkFSE928rY 84mA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:date:to:message-id :delivered-to; bh=NT1KwzLUfc8Qd0lKDtzKrzedVGTSODyuK6Yhrx63WEU=; fh=frUT2hx60kzuIVmkNUv7bM3Y/kw3lxlKoLH+4fdDqZk=; b=hhj7kWlRayFA+hPOfR6A3qDPmeGlYIRVf+ev1gToi6hVlxXVZsNXC8r+qx7aPgwQH9 eJ0YMjqlUnakC4LGg1NB+gY57jA8Kvq9q3MDrRz7/kIbX8KynEgygIhTC8HvSzUXrSng 6a03g5aCr5iRDSfS04T5Y0LHPCxAG31es+mwT4ztdbChxFT5Xkx3X2A+A1JwtI3Lt75a /kzOzXN7alI1pfcKp0XV/tfsnzL05Cr7T79AzoQVRGGfBmOdVJIym4pT7kuiWBaY1Ryv 0lU15IYwKOujec99KAtYDPl4wlZpqj3ly4cv6t1OfsOqwgsye+u+7OvGcyiMpCTo2H5d vqVQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z40-20020a509e2b000000b0053e2fdbb7dasi5707089ede.72.2023.11.15.06.40.28; Wed, 15 Nov 2023 06:40:29 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D520D68CCCD; Wed, 15 Nov 2023 16:40:24 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-235.mail.qq.com (out203-205-221-235.mail.qq.com [203.205.221.235]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C7CBE68CC6B for ; Wed, 15 Nov 2023 16:40:16 +0200 (EET) Received: from localhost.localdomain ([27.38.127.86]) by newxmesmtplogicsvrszc2-0.qq.com (NewEsmtp) with SMTP id A0B932E1; Wed, 15 Nov 2023 22:40:11 +0800 X-QQ-mid: xmsmtpt1700059211tsdi8nm4m Message-ID: X-QQ-XMAILINFO: NvKyM24IHTKSaYjPUcn4li2mx9K9VDpwROOh5VtQWObS/dgocOIcDgqvs//Mud nmGsiXTHI36BLiXGOp3EbQP2avHO2+HNzse/cEPJpe0afrOfHFuQlcX10pglOO5sm5/ATF+iAwMQ f1nPHZThwfo2J6uG4GhKH41E/rp8QiLbW+XaEz9xEwrc3W9UlQ2k8UsJuqXzLDWRLFdFVE8j0aqi hJ1b1gq1GVHDgsqcjRYk6mNQFMejGYNEZtvQmcSO83kTWzvIAKgg8DTbk4geg6bUkM4/faT5IZVj tZByGn2Pg4cTyvpgwaYnG18UhkPW1NB0fVKV87lRhJFdmCKWw0m9tBXGZBH4WxGOugshCVaa4eBY JiZX3OWjOqn6zfx08HeWRKCUmU4ZtNFASZKL8EifmhaQ/BQf7CnLEKw7mraY2oPbyf8Qp76mVS01 tiyXGEFFiQAsFFDH8otdBwwz14ONQ2TCkrFsoMGnXu2kSblkHeK8Tm7RikchQwIzLfv5sZvZ3cb3 i7V+5tn4xl3lK4Qln9Xdau6oZpQ5EQ0bLnXQ3S/qgZhbT4jmZG0oZ4nS94GEdSTGSP8N0DS+looI XFF37RdLNndn1wrSYArbCGw2qm1AjV69PJp+eXgzB2QL2k7hoG3G5X1esbsdMKpYffhecdLqqH/i boWShjTosw7BwEQTm+RmO014TV5FktudGRSwBDuD48p1/AvBl9FHwvrDGDFAJC1ixI7DjUZVCP/g 27UKpYlgV32rkWuyymm4YAgBOHipIrFzOAJDbmWmcYEGXqs6Ah0ef/py6yF2bnC2l80HN5Yp30Xk sE4HsjmlR1rGMDYeqCHcmy4MJZ2RRW2/DjGm2MoiQKP5BanJSvnv7I+lHKjqmy390hyjGNNKps5+ lJRmsbavi+Hm3uefip+tKSVJwQbe6zwhNnuS2UNzlhhMbtHZCYJKqiCqMsVzOXqsqQlHE7pie/X7 McuQ5msWBR+l9XZGlkf90ZV6/yaeLRIRPswgIQMFwCx2iG7y2J6CJqTyAY6JQoA6W2u3oS/NBfkd mTiP5TDQ== X-QQ-XMRINFO: NyFYKkN4Ny6FSmKK/uo/jdU= To: ffmpeg-devel@ffmpeg.org Date: Wed, 15 Nov 2023 22:40:08 +0800 X-OQ-MSGID: <20231115144010.70606-1-411294962@qq.com> X-Mailer: git-send-email 2.42.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/2] avformat/flvenc: support enhanced flv PacketTypeMetadata X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: zhupengfei via ffmpeg-devel From: Diego Felix de Souza via ffmpeg-devel Reply-To: FFmpeg development discussions and patches Cc: 411294962@qq.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Ep0q8ID/h16y From: Zhu Pengfei <411294962@qq.com> Signed-off-by: Zhu Pengfei <411294962@qq.com> --- libavformat/flvenc.c | 158 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 158 insertions(+) diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c index f6d10f331c..615a072928 100644 --- a/libavformat/flvenc.c +++ b/libavformat/flvenc.c @@ -24,6 +24,7 @@ #include "libavutil/intfloat.h" #include "libavutil/avassert.h" #include "libavutil/mathematics.h" +#include "libavutil/mastering_display_metadata.h" #include "libavcodec/codec_desc.h" #include "libavcodec/mpeg4audio.h" #include "avio.h" @@ -124,6 +125,7 @@ typedef struct FLVContext { int flags; int64_t last_ts[FLV_STREAM_TYPE_NB]; + int write_metadata_pkt; } FLVContext; static int get_audio_flags(AVFormatContext *s, AVCodecParameters *par) @@ -478,6 +480,161 @@ static void write_metadata(AVFormatContext *s, unsigned int ts) avio_wb32(pb, flv->metadata_totalsize + 11); } +static void flv_write_metadata_packet_if_needed(AVFormatContext *s, AVCodecParameters *par, unsigned int ts) +{ + AVIOContext *pb = s->pb; + FLVContext *flv = s->priv_data; + AVContentLightMetadata *lightMetadata = NULL; + AVMasteringDisplayMetadata *displayMetadata = NULL; + const int flags_size = 5; + int64_t metadata_size_pos = 0; + int64_t total_size = 0; + const AVPacketSideData *side_data = NULL; + + if (flv->write_metadata_pkt) { + return; + } + + side_data = av_packet_side_data_get(par->coded_side_data, par->nb_coded_side_data, + AV_PKT_DATA_CONTENT_LIGHT_LEVEL); + if (side_data) + lightMetadata = (AVContentLightMetadata *)side_data->data; + + side_data = av_packet_side_data_get(par->coded_side_data, par->nb_coded_side_data, + AV_PKT_DATA_MASTERING_DISPLAY_METADATA); + if (side_data) + displayMetadata = (AVMasteringDisplayMetadata *)side_data->data; + + if (!lightMetadata && !displayMetadata) { + return; + } + + /* + * Reference Enhancing FLV + * https://github.com/veovera/enhanced-rtmp/blob/main/enhanced-rtmp.pdf + * */ + avio_w8(pb, FLV_TAG_TYPE_VIDEO); //写入Video /Audio tag type + metadata_size_pos = avio_tell(pb); + avio_wb24(pb, 0 + flags_size); + put_timestamp(pb, ts); //ts = pkt->dts, gen + avio_wb24(pb, flv->reserved); + + if (par->codec_id == AV_CODEC_ID_HEVC) { + avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeMetadata| FLV_FRAME_VIDEO_INFO_CMD); // ExVideoTagHeader mode with PacketTypeMetadata + avio_write(pb, "hvc1", 4); + } else if (par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9) { + avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeMetadata| FLV_FRAME_VIDEO_INFO_CMD); + avio_write(pb, par->codec_id == AV_CODEC_ID_AV1 ? "av01" : "vp09", 4); + } else { + return; + } + + avio_w8(pb, AMF_DATA_TYPE_STRING); + put_amf_string(pb, "colorInfo"); + + avio_w8(pb, AMF_DATA_TYPE_OBJECT); + + put_amf_string(pb, "colorConfig"); // colorConfig + + /* mixed array (hash) with size and string/type/data tuples */ + avio_w8(pb, AMF_DATA_TYPE_MIXEDARRAY); + + avio_wb32(pb, 0); // write array count + + if (par->color_trc != AVCOL_TRC_UNSPECIFIED && + par->color_trc < AVCOL_TRC_NB) { + put_amf_string(pb, "transferCharacteristics"); // color_trc + put_amf_double(pb, par->color_trc); + } + + if (par->color_space != AVCOL_SPC_UNSPECIFIED && + par->color_space < AVCOL_SPC_NB) { + put_amf_string(pb, "matrixCoefficients"); // colorspace + put_amf_double(pb, par->color_space); + } + + if (par->color_primaries != AVCOL_PRI_UNSPECIFIED && + par->color_primaries < AVCOL_PRI_NB) { + put_amf_string(pb, "colorPrimaries"); // color_primaries + put_amf_double(pb, par->color_primaries); + } + + put_amf_string(pb, ""); + avio_w8(pb, AMF_END_OF_OBJECT); // array end of object + + if (lightMetadata) { + put_amf_string(pb, "hdrCll"); + + /* mixed array (hash) with size and string/type/data tuples */ + avio_w8(pb, AMF_DATA_TYPE_MIXEDARRAY); + + avio_wb32(pb, 0); // write array count + + put_amf_string(pb, "maxFall"); + put_amf_double(pb, lightMetadata->MaxFALL); + + put_amf_string(pb, "maxCLL"); + put_amf_double(pb, lightMetadata->MaxCLL); + + // array end of object + put_amf_string(pb, ""); + avio_w8(pb, AMF_END_OF_OBJECT); + } + + if (displayMetadata && (displayMetadata->has_primaries || displayMetadata->has_luminance)) { + put_amf_string(pb, "hdrMdcv"); + + /* mixed array (hash) with size and string/type/data tuples */ + avio_w8(pb, AMF_DATA_TYPE_MIXEDARRAY); + avio_wb32(pb, 0); // write array count + + if (displayMetadata->has_primaries) { + put_amf_string(pb, "redX"); + put_amf_double(pb, av_q2d(displayMetadata->display_primaries[0][0])); + + put_amf_string(pb, "redY"); + put_amf_double(pb, av_q2d(displayMetadata->display_primaries[0][1])); + + put_amf_string(pb, "greenX"); + put_amf_double(pb, av_q2d(displayMetadata->display_primaries[1][0])); + + put_amf_string(pb, "greenY"); + put_amf_double(pb, av_q2d(displayMetadata->display_primaries[1][1])); + + put_amf_string(pb, "blueX"); + put_amf_double(pb, av_q2d(displayMetadata->display_primaries[2][0])); + + put_amf_string(pb, "blueY"); + put_amf_double(pb, av_q2d(displayMetadata->display_primaries[2][1])); + + put_amf_string(pb, "whitePointX"); + put_amf_double(pb, av_q2d(displayMetadata->white_point[0])); + + put_amf_string(pb, "whitePointY"); + put_amf_double(pb, av_q2d(displayMetadata->white_point[1])); + } + if (displayMetadata->has_luminance) { + put_amf_string(pb, "maxLuminance"); + put_amf_double(pb, av_q2d(displayMetadata->max_luminance)); + + put_amf_string(pb, "minLuminance"); + put_amf_double(pb, av_q2d(displayMetadata->min_luminance)); + } + put_amf_string(pb, ""); + avio_w8(pb, AMF_END_OF_OBJECT); + } + avio_w8(pb, AMF_DATA_TYPE_OBJECT_END); + + total_size = avio_tell(pb) - metadata_size_pos - 10; + avio_seek(pb, metadata_size_pos, SEEK_SET); + avio_wb24(pb, total_size); + avio_tell(pb); + avio_skip(pb, total_size + 10 - 3); + avio_tell(pb); + avio_wb32(pb, total_size + 11); // previous tag size + flv->write_metadata_pkt = 1; +} + static int unsupported_codec(AVFormatContext *s, const char* type, int codec_id) { @@ -878,6 +1035,7 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) memcpy(par->extradata, side, side_size); flv_write_codec_header(s, par, pkt->dts); } + flv_write_metadata_packet_if_needed(s, par, pkt->dts); } if (flv->delay == AV_NOPTS_VALUE) From patchwork Wed Nov 15 14:40:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diego Felix de Souza via ffmpeg-devel X-Patchwork-Id: 44674 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:92a5:b0:181:818d:5e7f with SMTP id q37csp2725873pzg; Wed, 15 Nov 2023 06:40:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IFzkLbSguDPWCROYr2UnjveBWr28LzHm+2KDFymPbTefzkPR9sXrwx2QPNOb9H3vzhyqyNZ X-Received: by 2002:a05:6512:3e0f:b0:509:ffe5:e3e6 with SMTP id i15-20020a0565123e0f00b00509ffe5e3e6mr4434227lfv.0.1700059239366; Wed, 15 Nov 2023 06:40:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700059239; cv=none; d=google.com; s=arc-20160816; b=OABMYI6b+tUEbttOctNQYGC8IJGk/X1YAZ1m3IcGJrbg9vFmEBM7+theayg26ctnsv GRF/SUFtST8oRf/sEUl+7DEpQCWekdAA3VJjurCdEiQvfpsq60MUA3rBZR4q43iqL/hX ieM9kH86D8S7e+HjxRtlHCx007k7W9GqZ8Ag93Yaj0eglrSAOi3lDx4WWphEsrPMpYtD /wBA93MLXk2/iPShbVPNNjvWDszVBzz8V0k7z0lCFGaOSTd8Ks13S0hd0pphRs/VzVUf VRm7TFqIo+FFI0ZGNbaFXIyDH1NIw0/eUNTpb+3/0s6NmahjXc27U6HBiAx0LCcoHI1i htDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:date:to:message-id :delivered-to; bh=/iKKL99u0bq9lPq0k1LGpt4jFW59vh+vcqtPwg1pCYY=; fh=frUT2hx60kzuIVmkNUv7bM3Y/kw3lxlKoLH+4fdDqZk=; b=VG1Zks9alJtM8lI7uK8x2BqokeodIomxcIKrroezxghMd/6ZIL06dxUPK8iUhwhtFA MWE+me1VIJdHLq90VBpGyR8q1pEfH0lHi1Pug/d6M4+/BQNruMaa8c29ftlsalsSV8tH gq4/z6W7PNEaXECy5A3y8Ka0EuUcdCguP3/Z/ftdp3VNNYpJbtY4EAm9uC8aMHuhbJ8g OWPMQcDXCfYzC99ysYs2b2/baoOJ0uKUrxbVoZQMeWSGZmodapUIBwjPQhCO8DzxoS/B M9NoOq0jFyaZqDShRsAuWvCr46lwE7Kqd+EOlU7K8QkkOtZlCeEmueCCAGjFO4WbV3qx jvAw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b26-20020a170906195a00b009dbf9e6d190si5287758eje.948.2023.11.15.06.40.38; Wed, 15 Nov 2023 06:40:39 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E1A8168CCE7; Wed, 15 Nov 2023 16:40:31 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-192.mail.qq.com (out203-205-221-192.mail.qq.com [203.205.221.192]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C920B68CCB6 for ; Wed, 15 Nov 2023 16:40:24 +0200 (EET) Received: from localhost.localdomain ([27.38.127.86]) by newxmesmtplogicsvrszb9-0.qq.com (NewEsmtp) with SMTP id A13B9A88; Wed, 15 Nov 2023 22:40:19 +0800 X-QQ-mid: xmsmtpt1700059219tmvg52jy6 Message-ID: X-QQ-XMAILINFO: MyIXMys/8kCtwIJlyWGglhjVQ8KPmbBXJFAJTWKN80tOYJoZUfCiW9uzhG8lp/ Fxt4F+WEKsVTKgf31OG07WQ5rpgG8LNp/3gr2t/fvOHr2hn+2dnVx2y7e8IrUWsQ2PB8O0Pxf1GT ++NN69kyHxrVq50nBe6xfeueVAufMkMyMVRYUj8i+jU6sHK0OYPjnYepAovL/90gnqTPhimT8o1G 2Hp8lJTgMVsJJRkiG9TQxDxyx8SCXiIbMr3KOqBA8J09FlDfot0okkQ4F8Vm+Fcdiq3B1cpja7Xn nDhTEr0OtvOkBxrgaq5JutURwRDj612NUvAOWivbN4nWDTvpOMG3fFJiT/yki7f8/RTeJIQ1me+t OhTWyEuzs2UQI6PKuy+wjTuBxB1iWWt28zEhbu58sh1gDuv3cZEOTf4Qx/9TH9ZURrCCfC/kCGew UeljQ6XC0Vx3lIS0hQ3mI9aOyKHkD+ynJ+YVG6Jy91GDxWXOu/GwW8AZy3hmMFeUi0ipRCZK2iCZ SOAXykGTP+7/WqBj61trXPw3bb2hFkAYrJ4T/3SLYOCBR16lNt+skDz2za3GiEAVlo/6i/1e5wgA lu2L82CHUDfLAJ09AYoH9XgA1qNxcWXcunT0txYhRN2FTRExBdwCDuCEcyw+5AzsVjOzqjduHrKh ufna4hZIX+H4lWqzfDSaMfHZEtFitTElDwk+o+GiPsP7wxhHy4UvgYfOs/1tkM6o4i6FbftZpmDl jCwLAcnLpGpOVe1/ktsN7GbJ10zjK8GDCWEUPvOOkul02G52TeIRqKgC1tcOBbcRGVo8BkafPN16 qojAcRkT6Te+w+MihGJQi5LUanKWz89frZ0ltsBo4oQ+e4nyH5g9v+2WS/jcv9df7ZRAsOVG+6QV WCrjIcKNQmF3A8/6jB9NHxt/isuVyHouJmNhyDONXTQwTvx4lBLhW47ByqrxNn0smWKCyC1rB203 L38yNeLO4yPNZE7y25VuDZCS5XfII9uCMQ4MSEJTZcxrLLSb7gFEE96I5QMmprfQywn1Qv5KPWhR ebBvUjjA== X-QQ-XMRINFO: M/715EihBoGSf6IYSX1iLFg= To: ffmpeg-devel@ffmpeg.org Date: Wed, 15 Nov 2023 22:40:18 +0800 X-OQ-MSGID: <20231115144018.70653-1-411294962@qq.com> X-Mailer: git-send-email 2.42.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] avformat/flvdec: support enhanced flv PacketTypeMetadata X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: zhupengfei via ffmpeg-devel From: Diego Felix de Souza via ffmpeg-devel Reply-To: FFmpeg development discussions and patches Cc: 411294962@qq.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: JFOrZeSq4x+C From: Zhu Pengfei <411294962@qq.com> Signed-off-by: Zhu Pengfei <411294962@qq.com> --- libavformat/flvdec.c | 171 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 170 insertions(+), 1 deletion(-) diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index e25b5bd163..46bb0825ca 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -34,6 +34,7 @@ #include "libavutil/intfloat.h" #include "libavutil/intreadwrite.h" #include "libavutil/mathematics.h" +#include "libavutil/mastering_display_metadata.h" #include "avformat.h" #include "demux.h" #include "internal.h" @@ -45,6 +46,28 @@ #define MAX_DEPTH 16 ///< arbitrary limit to prevent unbounded recursion +typedef struct FLVMasteringMeta { + double r_x; + double r_y; + double g_x; + double g_y; + double b_x; + double b_y; + double white_x; + double white_y; + double max_luminance; + double min_luminance; +} FLVMasteringMeta; + +typedef struct FLVMetaVideoColor { + uint64_t matrix_coefficients; + uint64_t transfer_characteristics; + uint64_t primaries; + uint64_t max_cll; + uint64_t max_fall; + FLVMasteringMeta mastering_meta; +} FLVMetaVideoColor; + typedef struct FLVContext { const AVClass *class; ///< Class for private options. int trust_metadata; ///< configure streams according onMetaData @@ -80,6 +103,8 @@ typedef struct FLVContext { int64_t time_offset; int64_t time_pos; + FLVMetaVideoColor *metaVideoColor; + int meta_color_info_flag; } FLVContext; /* AMF date type */ @@ -524,6 +549,7 @@ static int amf_parse_object(AVFormatContext *s, AVStream *astream, FLVContext *flv = s->priv_data; AVIOContext *ioc; AMFDataType amf_type; + FLVMetaVideoColor *meta_video_color = flv->metaVideoColor; char str_val[1024]; double num_val; amf_date date; @@ -655,6 +681,36 @@ static int amf_parse_object(AVFormatContext *s, AVStream *astream, } else if (!strcmp(key, "height") && vpar) { vpar->height = num_val; } + } else if (!strcmp(key, "colorPrimaries") && meta_video_color) { + meta_video_color->primaries = num_val; + } else if (!strcmp(key, "transferCharacteristics") && meta_video_color) { + meta_video_color->transfer_characteristics = num_val; + } else if (!strcmp(key, "matrixCoefficients") && meta_video_color) { + meta_video_color->matrix_coefficients = num_val; + } else if (!strcmp(key, "maxFall") && meta_video_color) { + meta_video_color->max_fall = num_val; + } else if (!strcmp(key, "maxCLL") && meta_video_color) { + meta_video_color->max_cll = num_val; + } else if (!strcmp(key, "redX") && meta_video_color) { + meta_video_color->mastering_meta.r_x = num_val; + } else if (!strcmp(key, "redY") && meta_video_color) { + meta_video_color->mastering_meta.r_y = num_val; + } else if (!strcmp(key, "greenX") && meta_video_color) { + meta_video_color->mastering_meta.g_x = num_val; + } else if (!strcmp(key, "greenY") && meta_video_color) { + meta_video_color->mastering_meta.g_y = num_val; + } else if (!strcmp(key, "blueX") && meta_video_color) { + meta_video_color->mastering_meta.b_x = num_val; + } else if (!strcmp(key, "blueY") && meta_video_color) { + meta_video_color->mastering_meta.b_y = num_val; + } else if (!strcmp(key, "whitePointX") && meta_video_color) { + meta_video_color->mastering_meta.white_x = num_val; + } else if (!strcmp(key, "whitePointY") && meta_video_color) { + meta_video_color->mastering_meta.white_y = num_val; + } else if (!strcmp(key, "maxLuminance") && meta_video_color) { + meta_video_color->mastering_meta.max_luminance = num_val; + } else if (!strcmp(key, "minLuminance") && meta_video_color) { + meta_video_color->mastering_meta.min_luminance = num_val; } } if (amf_type == AMF_DATA_TYPE_STRING) { @@ -824,6 +880,7 @@ static int flv_read_close(AVFormatContext *s) av_freep(&flv->new_extradata[i]); av_freep(&flv->keyframe_times); av_freep(&flv->keyframe_filepositions); + av_freep(&flv->metaVideoColor); return 0; } @@ -1028,6 +1085,104 @@ static int resync(AVFormatContext *s) return AVERROR_EOF; } +static int flv_parse_video_color_info(AVFormatContext *s, AVStream *st, int64_t next_pos) +{ + FLVContext *flv = s->priv_data; + AMFDataType type; + AVIOContext *ioc; + char buffer[32]; + ioc = s->pb; + + // first object needs to be "colorInfo" string + type = avio_r8(ioc); + if (type != AMF_DATA_TYPE_STRING || + amf_get_string(ioc, buffer, sizeof(buffer)) < 0) + return TYPE_UNKNOWN; + + if (strcmp(buffer, "colorInfo")) { + av_log(s, AV_LOG_DEBUG, "Unknown type %s\n", buffer); + return TYPE_UNKNOWN; + } + + flv->metaVideoColor = av_mallocz(sizeof(FLVMetaVideoColor)); + if (!flv->metaVideoColor) { + return AVERROR(ENOMEM); + } + flv->meta_color_info_flag = 1; + amf_parse_object(s, NULL, NULL, buffer, next_pos, 0); // parse metadata + return 0; +} + +static int flv_update_video_color_info(AVFormatContext *s, AVStream *st) +{ + FLVContext *flv = s->priv_data; + const FLVMetaVideoColor* meta_video_color = flv->metaVideoColor; + const FLVMasteringMeta *mastering_meta = &meta_video_color->mastering_meta; + + int has_mastering_primaries, has_mastering_luminance; + // Mastering primaries are CIE 1931 coords, and must be > 0. + has_mastering_primaries = + mastering_meta->r_x > 0 && mastering_meta->r_y > 0 && + mastering_meta->g_x > 0 && mastering_meta->g_y > 0 && + mastering_meta->b_x > 0 && mastering_meta->b_y > 0 && + mastering_meta->white_x > 0 && mastering_meta->white_y > 0; + has_mastering_luminance = mastering_meta->max_luminance >= 0 && mastering_meta->min_luminance >= 0; + + if (meta_video_color->matrix_coefficients != AVCOL_SPC_RESERVED) + st->codecpar->color_space = meta_video_color->matrix_coefficients; + if (meta_video_color->primaries != AVCOL_PRI_RESERVED && + meta_video_color->primaries != AVCOL_PRI_RESERVED0) + st->codecpar->color_primaries = meta_video_color->primaries; + if (meta_video_color->transfer_characteristics != AVCOL_TRC_RESERVED && + meta_video_color->transfer_characteristics != AVCOL_TRC_RESERVED0) + st->codecpar->color_trc = meta_video_color->transfer_characteristics; + + if (meta_video_color->max_cll && meta_video_color->max_fall) { + size_t size = 0; + AVContentLightMetadata *metadata = av_content_light_metadata_alloc(&size); + if (!metadata) + return AVERROR(ENOMEM); + if (!av_packet_side_data_add(&st->codecpar->coded_side_data, &st->codecpar->nb_coded_side_data, + AV_PKT_DATA_CONTENT_LIGHT_LEVEL, metadata, size, 0)) { + av_freep(&metadata); + return AVERROR(ENOMEM); + } + metadata->MaxCLL = meta_video_color->max_cll; + metadata->MaxFALL = meta_video_color->max_fall; + } + + if (has_mastering_primaries || has_mastering_luminance) { + AVMasteringDisplayMetadata *metadata; + AVPacketSideData *sd = av_packet_side_data_new(&st->codecpar->coded_side_data, + &st->codecpar->nb_coded_side_data, + AV_PKT_DATA_MASTERING_DISPLAY_METADATA, + sizeof(AVMasteringDisplayMetadata), 0); + if (!sd) + return AVERROR(ENOMEM); + metadata = (AVMasteringDisplayMetadata*)sd->data; + memset(metadata, 0, sizeof(AVMasteringDisplayMetadata)); + // hdrCll + if (has_mastering_luminance) { + metadata->max_luminance = av_d2q(mastering_meta->max_luminance, INT_MAX); + metadata->min_luminance = av_d2q(mastering_meta->min_luminance, INT_MAX); + metadata->has_luminance = 1; + } + // hdrMdcv + if (has_mastering_primaries) { + metadata->display_primaries[0][0] = av_d2q(mastering_meta->r_x, INT_MAX); + metadata->display_primaries[0][1] = av_d2q(mastering_meta->r_y, INT_MAX); + metadata->display_primaries[1][0] = av_d2q(mastering_meta->g_x, INT_MAX); + metadata->display_primaries[1][1] = av_d2q(mastering_meta->g_y, INT_MAX); + metadata->display_primaries[2][0] = av_d2q(mastering_meta->b_x, INT_MAX); + metadata->display_primaries[2][1] = av_d2q(mastering_meta->b_y, INT_MAX); + metadata->white_point[0] = av_d2q(mastering_meta->white_x, INT_MAX); + metadata->white_point[1] = av_d2q(mastering_meta->white_y, INT_MAX); + metadata->has_primaries = 1; + } + } + return 0; +} + static int flv_read_packet(AVFormatContext *s, AVPacket *pkt) { FLVContext *flv = s->priv_data; @@ -1100,8 +1255,17 @@ retry: video_codec_id = avio_rb32(s->pb); size -= 4; } - if ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) + + if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { + type = flags & 0x0F; + if (type == PacketTypeMetadata) { + int ret = flv_parse_video_color_info(s, st, next); + av_log(s, AV_LOG_INFO, "enhanced flv parse metadata ret %d and skip\n", ret); + } + goto skip; + } else if ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { goto skip; + } } else if (type == FLV_TAG_TYPE_META) { stream_type=FLV_STREAM_TYPE_SUBTITLE; if (size > 13 + 1 + 4) { // Header-type metadata stuff @@ -1287,6 +1451,11 @@ retry_duration: goto leave; } + if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && flv->meta_color_info_flag) { + flv_update_video_color_info(s, st); // update av packet side data + flv->meta_color_info_flag = 0; + } + if (st->codecpar->codec_id == AV_CODEC_ID_H264 || st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || (st->codecpar->codec_id == AV_CODEC_ID_HEVC && type == PacketTypeCodedFrames)) { // sign extension