From patchwork Mon Apr 1 16:16:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Dennis_S=C3=A4dtler?= X-Patchwork-Id: 47693 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:9f96:b0:1a3:b6bb:3029 with SMTP id mm22csp838920pzb; Mon, 1 Apr 2024 09:17:05 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXaGMQPnuGYu7pOFh6XYk8rFHxvC/EtE8bJGDHlkyy1Qf8qd6sCXzmy4BdWvguK+vSsyTrImCK1YyZ2JAs91AG/0kLd70eHjLZSfg== X-Google-Smtp-Source: AGHT+IF4X44048lRZqY6kQ8CgbUKW3jEYO6xLs5NkV36849QgLGNN7ayVUuaa4NR8a0ldmgu/DR4 X-Received: by 2002:ac2:4108:0:b0:513:572f:88f1 with SMTP id b8-20020ac24108000000b00513572f88f1mr7891256lfi.27.1711988224943; Mon, 01 Apr 2024 09:17:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1711988224; cv=none; d=google.com; s=arc-20160816; b=ig23WE6QUptMwonydIAbj/N8bD190/pEfVGNd1xYjgoCgz67xJUnrXTr49haO5IB+n hwe4ywsu941jBir6nIXITfzrhCODQzWJgqGQw5WZ72n0SG76S/Z00kUG5otRs7PtM1MF Cd5OyBBRke52BmY+4PGyBSK0e/Gs/1lqWvxyn8Yq3gs5IK9zK5ysFNI8VTprQ09WpqKI GtUeQBCWVkwMnsOfEr6HG5OnKCkqvoFk5+yshS4rEvkrdgdp6EWEKVjHVOklyOck867e C32OBno8k/0nDYI3CSpGGCVQnTS1FCpebz/DoanRMQB/OGCvh1nUJMKsopM3krHY5taA h+DQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to :delivered-to; bh=4f4rDyULCHVVynlaVYhwIaLYqb02aYyqbpWGWQPa6JQ=; fh=z6qEiAy/zTNWGFjO2bmtKlCIU6WL+yiW+LZenWsDoNM=; b=Wn17oOk6yLE2EkVhm5Jjc/iv33hmZp8ReOnIz5yAZy1QEx/RcWFDHvOEqo5cQVXeIJ eFw0YOv5IfP3DKRFWz5FHNUssMsSH00JDeRHrbibovVFQV7tRmHNBSJKp02BO8x9tQrY 2IvPWlH1n1u9kj5Q5fUhH1Xo6GRVLmlG9qrlEmKjrT2x5mmR1NJ3Eq/BBd0wMws5qyNL 4haUdD1pSIljamu4csAtYK+pI0ItHfV8ptK+hW2BUadcJSOnu9E0D8YJH62NPHNlW5Mx UiW0vW/DAt7RKkpnZkbgsxPK1Mg+H1newAT0BJaJbRUM76JWlPGkhBq5dQrougMKp0Tc MXVw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j10-20020a170906474a00b00a4753c03615si4873967ejs.32.2024.04.01.09.17.04; Mon, 01 Apr 2024 09:17:04 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 425A568CEBF; Mon, 1 Apr 2024 19:17:01 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9745B68C308 for ; Mon, 1 Apr 2024 19:16:54 +0300 (EEST) Received: by mail-lj1-f170.google.com with SMTP id 38308e7fff4ca-2d485886545so69508911fa.2 for ; Mon, 01 Apr 2024 09:16:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711988213; x=1712593013; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ep7M6tcr2uAdmNeXreyStuwauaesKcE0kdpHXBSuCCw=; b=xOk6BfaIMAH2+SHA3fLn98BSL2cg1iNNY/lllzW3jcjQwtt1lPjtZ6JxkUCFQE8FDQ dLsIAOx7f2AnXD4o0/8CYZiQcdOjJL8gij4xT6ptjl9AXlZPBf+wWhuQYCHd8ft+ZwbO RndmTITxKG/VoyGoW4tv2L+ewJ50t4zUXcvJp/d5BM2wZKayjEZx2L8Z1oC1QAxjYoDB FrzcZZNw+Mdj1eOMDs5fe0UImiTPmAaYBaLWjNRSGA9Tko4RARYKR9biHFa4rE1dtz2t eGDEjwfjILJfF9Z/Q0vQYHCLz5ryxNwhOS+yizae1fqSjq25BGxBOvtx/Q1BU0eMTsiu uXDg== X-Gm-Message-State: AOJu0YwNQSryp+ro4Q2rPJ4wHfjAfyFgv3WUkZTajOD13T2HvJ+EGamJ xO7A753pruTA0n3soWFC3xfiR1Qd49pQxcjdM7Dh5/VTK7MK2XFMKRlstRBLD4lKLsfsocQyPaE 4 X-Received: by 2002:a2e:a4d8:0:b0:2d4:7455:89f6 with SMTP id p24-20020a2ea4d8000000b002d4745589f6mr6813090ljm.40.1711988213298; Mon, 01 Apr 2024 09:16:53 -0700 (PDT) Received: from Denniss-MBP.fritz.box (ip-088-152-103-184.um26.pools.vodafone-ip.de. [88.152.103.184]) by smtp.gmail.com with ESMTPSA id kf16-20020a17090776d000b00a46bf6d890bsm5491341ejc.91.2024.04.01.09.16.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 01 Apr 2024 09:16:52 -0700 (PDT) To: ffmpeg-devel@ffmpeg.org Date: Mon, 1 Apr 2024 18:16:49 +0200 Message-Id: <20240401161650.34063-1-dennis@obsproject.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/2] avformat/flvenc: Implement support for multi-track video X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: =?utf-8?q?Dennis_S=C3=A4dtler_via_ffmpeg-devel?= From: =?utf-8?q?Dennis_S=C3=A4dtler?= Reply-To: FFmpeg development discussions and patches Cc: =?utf-8?q?Dennis_S=C3=A4dtler?= Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 5UH3M0zlPpwK Based on enhanced-rtmp v2 spec published by Veovera: https://veovera.github.io/enhanced-rtmp/docs/enhanced/enhanced-rtmp-v2 This implementation maintains some backwards compatibility by only writing the track information for track indices > 0. This means that older FFmpeg versions - and possibly other software - can still read the first video track properly and skip over unsupported packets. Signed-off-by: Dennis Sädtler --- libavformat/flv.h | 7 ++ libavformat/flvenc.c | 160 ++++++++++++++++++++++++++++++------------- 2 files changed, 119 insertions(+), 48 deletions(-) diff --git a/libavformat/flv.h b/libavformat/flv.h index f710963b92..653c2bc82c 100644 --- a/libavformat/flv.h +++ b/libavformat/flv.h @@ -125,6 +125,13 @@ enum { PacketTypeCodedFramesX = 3, PacketTypeMetadata = 4, PacketTypeMPEG2TSSequenceStart = 5, + PacketTypeMultitrack = 6, +}; + +enum { + MultitrackTypeOneTrack = 0x00, + MultitrackTypeManyTracks = 0x10, + MultitrackTypeManyTracksManyCodecs = 0x20, }; enum { diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c index aba1d7d80b..82f275777f 100644 --- a/libavformat/flvenc.c +++ b/libavformat/flvenc.c @@ -125,8 +125,9 @@ typedef struct FLVContext { AVCodecParameters *data_par; int flags; - int64_t last_ts[FLV_STREAM_TYPE_NB]; - int metadata_pkt_written; + int64_t *last_ts; + int *metadata_pkt_written; + int *video_track_idx_map; } FLVContext; static int get_audio_flags(AVFormatContext *s, AVCodecParameters *par) @@ -484,7 +485,7 @@ static void write_metadata(AVFormatContext *s, unsigned int ts) avio_wb32(pb, flv->metadata_totalsize + 11); } -static void flv_write_metadata_packet(AVFormatContext *s, AVCodecParameters *par, unsigned int ts) +static void flv_write_metadata_packet(AVFormatContext *s, AVCodecParameters *par, unsigned int ts, int stream_idx) { AVIOContext *pb = s->pb; FLVContext *flv = s->priv_data; @@ -494,7 +495,9 @@ static void flv_write_metadata_packet(AVFormatContext *s, AVCodecParameters *par int64_t total_size = 0; const AVPacketSideData *side_data = NULL; - if (flv->metadata_pkt_written) return; + if (flv->metadata_pkt_written[stream_idx]) + return; + if (par->codec_id == AV_CODEC_ID_HEVC || par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9) { int flags_size = 5; @@ -616,7 +619,7 @@ static void flv_write_metadata_packet(AVFormatContext *s, AVCodecParameters *par avio_wb24(pb, total_size); avio_skip(pb, total_size + 10 - 3); avio_wb32(pb, total_size + 11); // previous tag size - flv->metadata_pkt_written = 1; + flv->metadata_pkt_written[stream_idx] = 1; } } @@ -631,7 +634,7 @@ static int unsupported_codec(AVFormatContext *s, return AVERROR(ENOSYS); } -static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts) { +static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts, int stream_index) { int64_t data_size; AVIOContext *pb = s->pb; FLVContext *flv = s->priv_data; @@ -681,12 +684,32 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i } avio_write(pb, par->extradata, par->extradata_size); } else { - if (par->codec_id == AV_CODEC_ID_HEVC) { - avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeSequenceStart | FLV_FRAME_KEY); // ExVideoTagHeader mode with PacketTypeSequenceStart - avio_write(pb, "hvc1", 4); - } else if (par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9) { - avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeSequenceStart | FLV_FRAME_KEY); - avio_write(pb, par->codec_id == AV_CODEC_ID_AV1 ? "av01" : "vp09", 4); + int track_idx = flv->video_track_idx_map[stream_index]; + // If video stream has track_idx > 0 we need to send H.264 as extended video packet + int extended_flv = (par->codec_id == AV_CODEC_ID_H264 && track_idx) + || par->codec_id == AV_CODEC_ID_HEVC + || par->codec_id == AV_CODEC_ID_AV1 + || par->codec_id == AV_CODEC_ID_VP9; + + if (extended_flv) { + if (track_idx) { + avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeMultitrack | FLV_FRAME_KEY); + avio_w8(pb, MultitrackTypeOneTrack | PacketTypeSequenceStart); + } else { + avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeSequenceStart | FLV_FRAME_KEY); + } + + if (par->codec_id == AV_CODEC_ID_H264) + avio_write(pb, "avc1", 4); + else if (par->codec_id == AV_CODEC_ID_HEVC) + avio_write(pb, "hvc1", 4); + else if (par->codec_id == AV_CODEC_ID_AV1) + avio_write(pb, "av01", 4); + else if (par->codec_id == AV_CODEC_ID_VP9) + avio_write(pb, "vp09", 4); + + if (track_idx) + avio_w8(pb, track_idx); } else { avio_w8(pb, par->codec_tag | FLV_FRAME_KEY); // flags avio_w8(pb, 0); // AVC sequence header @@ -769,13 +792,12 @@ static int shift_data(AVFormatContext *s) static int flv_init(struct AVFormatContext *s) { int i; + int video_ctr = 0; FLVContext *flv = s->priv_data; - - if (s->nb_streams > FLV_STREAM_TYPE_NB) { - av_log(s, AV_LOG_ERROR, "invalid number of streams %d\n", - s->nb_streams); - return AVERROR(EINVAL); - } + + flv->last_ts = av_mallocz(sizeof(*flv->last_ts) * s->nb_streams); + flv->metadata_pkt_written = av_mallocz(sizeof(*flv->metadata_pkt_written) * s->nb_streams); + flv->video_track_idx_map = av_mallocz(sizeof(*flv->video_track_idx_map) * s->nb_streams); for (i = 0; i < s->nb_streams; i++) { AVCodecParameters *par = s->streams[i]->codecpar; @@ -786,12 +808,17 @@ static int flv_init(struct AVFormatContext *s) s->streams[i]->avg_frame_rate.num) { flv->framerate = av_q2d(s->streams[i]->avg_frame_rate); } - if (flv->video_par) { + flv->video_track_idx_map[i] = video_ctr++; + if (flv->video_par && flv->flags & FLV_ADD_KEYFRAME_INDEX) { av_log(s, AV_LOG_ERROR, - "at most one video stream is supported in flv\n"); + "at most one video stream is supported in flv with keyframe index\n"); return AVERROR(EINVAL); + } else if (flv->video_par) { + av_log(s, AV_LOG_WARNING, + "more than one video stream is not supported by most flv demuxers.\n"); } - flv->video_par = par; + if (!flv->video_par) + flv->video_par = par; if (!ff_codec_get_tag(flv_video_codec_ids, par->codec_id)) return unsupported_codec(s, "Video", par->codec_id); @@ -881,7 +908,7 @@ static int flv_write_header(AVFormatContext *s) } for (i = 0; i < s->nb_streams; i++) { - flv_write_codec_header(s, s->streams[i]->codecpar, 0); + flv_write_codec_header(s, s->streams[i]->codecpar, 0, i); } flv->datastart_offset = avio_tell(pb); @@ -989,6 +1016,7 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) uint8_t frametype = pkt->flags & AV_PKT_FLAG_KEY ? FLV_FRAME_KEY : FLV_FRAME_INTER; int flags = -1, flags_size, ret = 0; int64_t cur_offset = avio_tell(pb); + int track_idx = flv->video_track_idx_map[pkt->stream_index]; if (par->codec_type == AVMEDIA_TYPE_AUDIO && !pkt->size) { av_log(s, AV_LOG_WARNING, "Empty audio Packet\n"); @@ -1005,7 +1033,12 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) else flags_size = 1; - if (par->codec_id == AV_CODEC_ID_HEVC && pkt->pts != pkt->dts) + if (par->codec_type == AVMEDIA_TYPE_VIDEO && track_idx) + flags_size += 2; // additional header bytes for multi-track video + + if ((par->codec_id == AV_CODEC_ID_HEVC || + (par->codec_id == AV_CODEC_ID_H264 && track_idx)) + && pkt->pts != pkt->dts) flags_size += 3; if (par->codec_id == AV_CODEC_ID_AAC || par->codec_id == AV_CODEC_ID_H264 @@ -1018,9 +1051,9 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) if (ret < 0) return ret; memcpy(par->extradata, side, side_size); - flv_write_codec_header(s, par, pkt->dts); + flv_write_codec_header(s, par, pkt->dts, pkt->stream_index); } - flv_write_metadata_packet(s, par, pkt->dts); + flv_write_metadata_packet(s, par, pkt->dts, pkt->stream_index); } if (flv->delay == AV_NOPTS_VALUE) @@ -1142,32 +1175,59 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) avio_seek(pb, data_size + 10 - 3, SEEK_CUR); avio_wb32(pb, data_size + 11); } else { - av_assert1(flags>=0); - if (par->codec_id == AV_CODEC_ID_HEVC) { - int pkttype = (pkt->pts != pkt->dts) ? PacketTypeCodedFrames : PacketTypeCodedFramesX; - avio_w8(pb, FLV_IS_EX_HEADER | pkttype | frametype); // ExVideoTagHeader mode with PacketTypeCodedFrames(X) - avio_write(pb, "hvc1", 4); - if (pkttype == PacketTypeCodedFrames) + int extended_flv = (par->codec_id == AV_CODEC_ID_H264 && track_idx) + || par->codec_id == AV_CODEC_ID_HEVC + || par->codec_id == AV_CODEC_ID_AV1 + || par->codec_id == AV_CODEC_ID_VP9; + + if (extended_flv) { + int h2645 = par->codec_id == AV_CODEC_ID_H264 || + par->codec_id == AV_CODEC_ID_HEVC; + int pkttype = PacketTypeCodedFrames; + // Optimisation for HEVC/H264: Do not send composition time if DTS == PTS + if (h2645 && pkt->pts == pkt->dts) + pkttype = PacketTypeCodedFramesX; + + if (track_idx) { + avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeMultitrack | frametype); + avio_w8(pb, MultitrackTypeOneTrack | pkttype); + } else { + avio_w8(pb, FLV_IS_EX_HEADER | pkttype | frametype); + } + + if (par->codec_id == AV_CODEC_ID_H264) + avio_write(pb, "avc1", 4); + else if (par->codec_id == AV_CODEC_ID_HEVC) + avio_write(pb, "hvc1", 4); + else if (par->codec_id == AV_CODEC_ID_AV1) + avio_write(pb, "av01", 4); + else if (par->codec_id == AV_CODEC_ID_VP9) + avio_write(pb, "vp09", 4); + + if (track_idx) + avio_w8(pb, track_idx); + if (h2645 && pkttype == PacketTypeCodedFrames) avio_wb24(pb, pkt->pts - pkt->dts); - } else if (par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9) { - avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeCodedFrames | frametype); - avio_write(pb, par->codec_id == AV_CODEC_ID_AV1 ? "av01" : "vp09", 4); } else { + av_assert1(flags>=0); avio_w8(pb, flags); - } - if (par->codec_id == AV_CODEC_ID_VP6) - avio_w8(pb,0); - if (par->codec_id == AV_CODEC_ID_VP6F || par->codec_id == AV_CODEC_ID_VP6A) { - if (par->extradata_size) - avio_w8(pb, par->extradata[0]); - else - avio_w8(pb, ((FFALIGN(par->width, 16) - par->width) << 4) | - (FFALIGN(par->height, 16) - par->height)); - } else if (par->codec_id == AV_CODEC_ID_AAC) - avio_w8(pb, 1); // AAC raw - else if (par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4) { - avio_w8(pb, 1); // AVC NALU - avio_wb24(pb, pkt->pts - pkt->dts); + + if (par->codec_id == AV_CODEC_ID_VP6) { + avio_w8(pb,0); + } else if (par->codec_id == AV_CODEC_ID_VP6F || + par->codec_id == AV_CODEC_ID_VP6A) { + if (par->extradata_size) + avio_w8(pb, par->extradata[0]); + else + avio_w8(pb, ((FFALIGN(par->width, 16) - par->width) << 4) | + (FFALIGN(par->height, 16) - par->height)); + } else if (par->codec_id == AV_CODEC_ID_AAC) { + avio_w8(pb, 1); // AAC raw + } else if (par->codec_id == AV_CODEC_ID_H264 || + par->codec_id == AV_CODEC_ID_MPEG4) { + avio_w8(pb, 1); // AVC NALU + avio_wb24(pb, pkt->pts - pkt->dts); + } } avio_write(pb, data ? data : pkt->data, size); @@ -1234,6 +1294,10 @@ static void flv_deinit(AVFormatContext *s) } flv->filepositions = flv->head_filepositions = NULL; flv->filepositions_count = 0; + + av_freep(&flv->last_ts); + av_freep(&flv->metadata_pkt_written); + av_freep(&flv->video_track_idx_map); } static const AVOption options[] = { From patchwork Mon Apr 1 16:16:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Dennis_S=C3=A4dtler?= X-Patchwork-Id: 47694 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:9f96:b0:1a3:b6bb:3029 with SMTP id mm22csp839029pzb; Mon, 1 Apr 2024 09:17:14 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCV8QfTs6d6iExsAOYyDCGGvW+ZDdlnYuxhGmr9MA3zXklcsIxNwe8ki0CCqwTBXfgsL4ASjRUAzqbWcbUUxhF57MDcVfYLSbRG8qg== X-Google-Smtp-Source: AGHT+IGte2pcy6XIH+htQw+X5Bay811/RCoVqcQOmAin3B3S8XqBbhBOCG6wGRfbzHxvSqht15Mz X-Received: by 2002:a17:907:da5:b0:a4e:3956:6ad8 with SMTP id go37-20020a1709070da500b00a4e39566ad8mr7045628ejc.44.1711988234467; Mon, 01 Apr 2024 09:17:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1711988234; cv=none; d=google.com; s=arc-20160816; b=p4/+rf/RHB4qVGEgxYYv/SFt45AjOeNbrTpUy8U+4aALtEagafq3hO9dj8m9GzcTL0 bJ8SqAmWsGOJsmiZs6zeAK8loCCXNZI2eyG//aani+Rmo1aZjhwA41qeQV8GX11miutR IHNmIr4HOx9AgPxgjzMhRyNd3ahEj3jyZTHxJYiZYaKCATiK0kYqvBIPbbHtRqfRMNbT m4Nnzt3JGWdYhU8hBrzZE6jP2bIaKYVo46ZnTEMbahfKmPzg0KwC8EAoDXL1T2wzpTZc Yfhm4O7DFSDXKlS1ow+xRAiyy8RXcCFKVW6w8Mgfpw48OlyznJOn0GJF37iBemBeublS GlBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:delivered-to; bh=Ssa1aghLI/Ev2sMwZEjE+zHw3zDQd7kHOx4ZXyrkoaM=; fh=z6qEiAy/zTNWGFjO2bmtKlCIU6WL+yiW+LZenWsDoNM=; b=TvH97i2aikZfFPlLwF0ikXeaE+QtC/EraSP2lHN2k3NWbKR253kUd/UEvsHed+xXLt hxIkDreFXCMKQj6rw79YAV2ZauXd5K/5Mkxyyjd9+PDeLWXCd7ktP3mdeD4ppg1hGf7E nS9N037IP00C1dzkUbys+vlk7Yj/ksSNgTtT6LSIoYfFMbdyzqgs8712TLtFnJ1nCqZQ aRSVDdlLwvFpks+vAaKlQeFX4Lupw+pYtWAZdAJKabfOP1aJqWjKNTGh43HM6KUlw8JB UhkJ7xUsxlCvQbJSkNAAZzj0AMJlKtCK987QGSHzCCuln7cxrEmy7AgSaTbIsT0hPijV F2lw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id k22-20020a1709065fd600b00a4dffd58150si4650499ejv.914.2024.04.01.09.17.14; Mon, 01 Apr 2024 09:17:14 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4CBEB68CEE6; Mon, 1 Apr 2024 19:17:03 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2FA7668CEBC for ; Mon, 1 Apr 2024 19:16:56 +0300 (EEST) Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a4702457ccbso519960466b.3 for ; Mon, 01 Apr 2024 09:16:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711988215; x=1712593015; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kT5tWvGg0NbvxJ+geO3otb3e1vCe32iv5/uAhiQLddg=; b=ErC2D+sgeRB/t0DGJSvxB5AQheI/dQPSS+dRSAW5d/xHGzJIOKj+9N3238QmG20TXV iBM90kWkduZi99HZvEUDSC21KwR0HVaw0rAntv4FmoFyLmLnNH6t5gnjCzWXHkWUHWLe /YDF2dOKGorD59tgnmVBqMe/FsoFN/SnwuTgpM/zSQc3BPDnbmooGeSnuNgOUvR/J6CJ +RdkpRzaPjtZg5zIbQtxvJxrBtiXYzH5OpZaSwCwtq5bR18BrhgK+B0B9k4ru0T73gJE cQMYFVKvexd9dS9p3QXEy/pvKNcxUpm+iITn/NbcjKoLTLgyO/qEOhUj0m3wmpPojLIn QmSA== X-Gm-Message-State: AOJu0YyO9oZagE/AWd1W075bLXd+zOjfcQFNwLBmLs5ilAmf1K5ir4iK wX2A3bZYtqR+raDzS05Krm2wJVmzcfHZgud4q1rO9oL/zeI2KjMVs8i3sBPALgp0wgPwz5zTD4l T X-Received: by 2002:a17:906:fac4:b0:a4d:f5d4:fb02 with SMTP id lu4-20020a170906fac400b00a4df5d4fb02mr6671620ejb.51.1711988215060; Mon, 01 Apr 2024 09:16:55 -0700 (PDT) Received: from Denniss-MBP.fritz.box (ip-088-152-103-184.um26.pools.vodafone-ip.de. [88.152.103.184]) by smtp.gmail.com with ESMTPSA id kf16-20020a17090776d000b00a46bf6d890bsm5491341ejc.91.2024.04.01.09.16.54 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 01 Apr 2024 09:16:54 -0700 (PDT) To: ffmpeg-devel@ffmpeg.org Date: Mon, 1 Apr 2024 18:16:50 +0200 Message-Id: <20240401161650.34063-2-dennis@obsproject.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20240401161650.34063-1-dennis@obsproject.com> References: <20240401161650.34063-1-dennis@obsproject.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] avformat/flvdec: Add support for demuxing multi-track FLV X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: =?utf-8?q?Dennis_S=C3=A4dtler_via_ffmpeg-devel?= From: =?utf-8?q?Dennis_S=C3=A4dtler?= Reply-To: FFmpeg development discussions and patches Cc: =?utf-8?q?Dennis_S=C3=A4dtler?= Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: la993p237oOd Based on enhanced-rtmp v2 spec published by Veovera: https://veovera.github.io/enhanced-rtmp/docs/enhanced/enhanced-rtmp-v2 Signed-off-by: Dennis Sädtler --- libavformat/flvdec.c | 117 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 96 insertions(+), 21 deletions(-) diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index 22a9b9e4a7..39d01d3b1f 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -105,6 +105,10 @@ typedef struct FLVContext { FLVMetaVideoColor *metaVideoColor; int meta_color_info_flag; + + uint8_t **mt_extradata; + int *mt_extradata_sz; + int mt_extradata_cnt; } FLVContext; /* AMF date type */ @@ -187,13 +191,18 @@ static void add_keyframes_index(AVFormatContext *s) } } -static AVStream *create_stream(AVFormatContext *s, int codec_type) +static AVStream *create_stream(AVFormatContext *s, int codec_type, int track_idx) { FLVContext *flv = s->priv_data; AVStream *st = avformat_new_stream(s, NULL); if (!st) return NULL; st->codecpar->codec_type = codec_type; + st->id = track_idx; + avpriv_set_pts_info(st, 32, 1, 1000); /* 32 bit pts in ms */ + if (track_idx) + return st; + if (s->nb_streams>=3 ||( s->nb_streams==2 && s->streams[0]->codecpar->codec_type != AVMEDIA_TYPE_SUBTITLE && s->streams[1]->codecpar->codec_type != AVMEDIA_TYPE_SUBTITLE @@ -210,8 +219,6 @@ static AVStream *create_stream(AVFormatContext *s, int codec_type) st->avg_frame_rate = flv->framerate; } - - avpriv_set_pts_info(st, 32, 1, 1000); /* 32 bit pts in ms */ flv->last_keyframe_stream_index = s->nb_streams - 1; add_keyframes_index(s); return st; @@ -351,6 +358,7 @@ static int flv_same_video_codec(AVCodecParameters *vpar, uint32_t flv_codecid) case FLV_CODECID_VP6A: return vpar->codec_id == AV_CODEC_ID_VP6A; case FLV_CODECID_H264: + case MKBETAG('a', 'v', 'c', '1'): return vpar->codec_id == AV_CODEC_ID_H264; default: return vpar->codec_tag == flv_codecid; @@ -407,6 +415,7 @@ static int flv_set_video_codec(AVFormatContext *s, AVStream *vstream, ret = 1; // 1 byte body size adjustment for flv_read_packet() break; case FLV_CODECID_H264: + case MKBETAG('a', 'v', 'c', '1'): par->codec_id = AV_CODEC_ID_H264; vstreami->need_parsing = AVSTREAM_PARSE_HEADERS; break; @@ -676,7 +685,7 @@ static int amf_parse_object(AVFormatContext *s, AVStream *astream, } else if (!strcmp(key, "height") && vpar) { vpar->height = num_val; } else if (!strcmp(key, "datastream")) { - AVStream *st = create_stream(s, AVMEDIA_TYPE_SUBTITLE); + AVStream *st = create_stream(s, AVMEDIA_TYPE_SUBTITLE, 0); if (!st) return AVERROR(ENOMEM); st->codecpar->codec_id = AV_CODEC_ID_TEXT; @@ -885,6 +894,9 @@ static int flv_read_close(AVFormatContext *s) FLVContext *flv = s->priv_data; for (i=0; inew_extradata[i]); + for (i=0; i < flv->mt_extradata_cnt; i++) + av_freep(&flv->mt_extradata[i]); + av_freep(&flv->mt_extradata_sz); av_freep(&flv->keyframe_times); av_freep(&flv->keyframe_filepositions); av_freep(&flv->metaVideoColor); @@ -904,18 +916,47 @@ static int flv_get_extradata(AVFormatContext *s, AVStream *st, int size) } static int flv_queue_extradata(FLVContext *flv, AVIOContext *pb, int stream, - int size) + int size, int multitrack) { if (!size) return 0; - av_free(flv->new_extradata[stream]); - flv->new_extradata[stream] = av_mallocz(size + - AV_INPUT_BUFFER_PADDING_SIZE); - if (!flv->new_extradata[stream]) - return AVERROR(ENOMEM); - flv->new_extradata_size[stream] = size; - avio_read(pb, flv->new_extradata[stream], size); + if (!multitrack) { + av_free(flv->new_extradata[stream]); + flv->new_extradata[stream] = av_mallocz(size + + AV_INPUT_BUFFER_PADDING_SIZE); + if (!flv->new_extradata[stream]) + return AVERROR(ENOMEM); + flv->new_extradata_size[stream] = size; + avio_read(pb, flv->new_extradata[stream], size); + } else { + int new_count = stream + 1; + + if (flv->mt_extradata_cnt < new_count) { + flv->mt_extradata = av_realloc(flv->mt_extradata, + sizeof(*flv->mt_extradata) * + new_count); + flv->mt_extradata_sz = av_realloc(flv->mt_extradata_sz, + sizeof(*flv->mt_extradata_sz) * + new_count); + if (!flv->mt_extradata || !flv->mt_extradata_sz) + return AVERROR(ENOMEM); + // Set newly allocated pointers/sizes to 0 + for (int i = flv->mt_extradata_cnt; i < new_count; i++) { + flv->mt_extradata[i] = NULL; + flv->mt_extradata_sz[i] = 0; + } + flv->mt_extradata_cnt = new_count; + } + + av_free(flv->mt_extradata[stream]); + flv->mt_extradata[stream] = av_mallocz(size + AV_INPUT_BUFFER_PADDING_SIZE); + if (!flv->mt_extradata[stream]) + return AVERROR(ENOMEM); + flv->mt_extradata_sz[stream] = size; + avio_read(pb, flv->mt_extradata[stream], size); + } + return 0; } @@ -1031,7 +1072,7 @@ static int flv_data_packet(AVFormatContext *s, AVPacket *pkt, } if (i == s->nb_streams) { - st = create_stream(s, AVMEDIA_TYPE_SUBTITLE); + st = create_stream(s, AVMEDIA_TYPE_SUBTITLE, 0); if (!st) return AVERROR(ENOMEM); st->codecpar->codec_id = AV_CODEC_ID_TEXT; @@ -1203,6 +1244,9 @@ static int flv_read_packet(AVFormatContext *s, AVPacket *pkt) int last = -1; int orig_size; int enhanced_flv = 0; + int multitrack = 0; + int pkt_type = 0; + uint8_t track_idx = 0; uint32_t video_codec_id = 0; retry: @@ -1256,14 +1300,33 @@ retry: * https://github.com/veovera/enhanced-rtmp/blob/main/enhanced-rtmp-v1.pdf * */ enhanced_flv = (flags >> 7) & 1; + pkt_type = enhanced_flv ? video_codec_id : 0; size--; + + if (pkt_type == PacketTypeMultitrack) { + uint8_t types = avio_r8(s->pb); + int multitrack_type = types >> 4; + pkt_type = types & 0xF; + + if (multitrack_type != MultitrackTypeOneTrack) { + av_log(s, AV_LOG_ERROR, "Multitrack types other than MultitrackTypeOneTrack are unsupported!\n"); + return AVERROR_PATCHWELCOME; + } + + multitrack = 1; + size--; + } + if (enhanced_flv) { video_codec_id = avio_rb32(s->pb); size -= 4; } + if (multitrack) { + track_idx = avio_r8(s->pb); + size--; + } - if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { - int pkt_type = flags & 0x0F; + if (enhanced_flv && (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { if (pkt_type == PacketTypeMetadata) { int ret = flv_parse_video_color_info(s, st, next); av_log(s, AV_LOG_DEBUG, "enhanced flv parse metadata ret %d and skip\n", ret); @@ -1327,7 +1390,8 @@ skip: break; } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO && - (s->video_codec_id || flv_same_video_codec(st->codecpar, video_codec_id))) + (s->video_codec_id || flv_same_video_codec(st->codecpar, video_codec_id)) && + st->id == track_idx) break; } else if (stream_type == FLV_STREAM_TYPE_SUBTITLE) { if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) @@ -1339,7 +1403,7 @@ skip: } if (i == s->nb_streams) { static const enum AVMediaType stream_types[] = {AVMEDIA_TYPE_VIDEO, AVMEDIA_TYPE_AUDIO, AVMEDIA_TYPE_SUBTITLE, AVMEDIA_TYPE_DATA}; - st = create_stream(s, stream_types[stream_type]); + st = create_stream(s, stream_types[stream_type], track_idx); if (!st) return AVERROR(ENOMEM); } @@ -1446,7 +1510,7 @@ retry_duration: st->codecpar->codec_id == AV_CODEC_ID_VP9) { int type = 0; if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO) { - type = flags & 0x0F; + type = pkt_type; } else { type = avio_r8(s->pb); size--; @@ -1462,7 +1526,8 @@ retry_duration: flv->meta_color_info_flag = 0; } - if (st->codecpar->codec_id == AV_CODEC_ID_H264 || st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || + if (st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || + (st->codecpar->codec_id == AV_CODEC_ID_H264 && (!enhanced_flv || type == PacketTypeCodedFrames)) || (st->codecpar->codec_id == AV_CODEC_ID_HEVC && type == PacketTypeCodedFrames)) { // sign extension int32_t cts = (avio_rb24(s->pb) + 0xff800000) ^ 0xff800000; @@ -1485,7 +1550,7 @@ retry_duration: AVDictionaryEntry *t; if (st->codecpar->extradata) { - if ((ret = flv_queue_extradata(flv, s->pb, stream_type, size)) < 0) + if ((ret = flv_queue_extradata(flv, s->pb, multitrack ? track_idx : stream_type, size, multitrack)) < 0) return ret; ret = FFERROR_REDO; goto leave; @@ -1516,7 +1581,7 @@ retry_duration: pkt->pts = pts == AV_NOPTS_VALUE ? dts : pts; pkt->stream_index = st->index; pkt->pos = pos; - if (flv->new_extradata[stream_type]) { + if (!multitrack && flv->new_extradata[stream_type]) { int ret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, flv->new_extradata[stream_type], flv->new_extradata_size[stream_type]); @@ -1524,6 +1589,16 @@ retry_duration: flv->new_extradata[stream_type] = NULL; flv->new_extradata_size[stream_type] = 0; } + } else if (multitrack + && flv->mt_extradata_cnt > track_idx + && flv->mt_extradata[track_idx]) { + int ret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, + flv->mt_extradata[track_idx], + flv->mt_extradata_sz[track_idx]); + if (ret >= 0) { + flv->mt_extradata[track_idx] = NULL; + flv->mt_extradata_sz[track_idx] = 0; + } } if (stream_type == FLV_STREAM_TYPE_AUDIO && (sample_rate != flv->last_sample_rate ||