From patchwork Mon Apr 1 16:16:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Dennis_S=C3=A4dtler?= X-Patchwork-Id: 47694 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:9f96:b0:1a3:b6bb:3029 with SMTP id mm22csp839029pzb; Mon, 1 Apr 2024 09:17:14 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCV8QfTs6d6iExsAOYyDCGGvW+ZDdlnYuxhGmr9MA3zXklcsIxNwe8ki0CCqwTBXfgsL4ASjRUAzqbWcbUUxhF57MDcVfYLSbRG8qg== X-Google-Smtp-Source: AGHT+IGte2pcy6XIH+htQw+X5Bay811/RCoVqcQOmAin3B3S8XqBbhBOCG6wGRfbzHxvSqht15Mz X-Received: by 2002:a17:907:da5:b0:a4e:3956:6ad8 with SMTP id go37-20020a1709070da500b00a4e39566ad8mr7045628ejc.44.1711988234467; Mon, 01 Apr 2024 09:17:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1711988234; cv=none; d=google.com; s=arc-20160816; b=p4/+rf/RHB4qVGEgxYYv/SFt45AjOeNbrTpUy8U+4aALtEagafq3hO9dj8m9GzcTL0 bJ8SqAmWsGOJsmiZs6zeAK8loCCXNZI2eyG//aani+Rmo1aZjhwA41qeQV8GX11miutR IHNmIr4HOx9AgPxgjzMhRyNd3ahEj3jyZTHxJYiZYaKCATiK0kYqvBIPbbHtRqfRMNbT m4Nnzt3JGWdYhU8hBrzZE6jP2bIaKYVo46ZnTEMbahfKmPzg0KwC8EAoDXL1T2wzpTZc Yfhm4O7DFSDXKlS1ow+xRAiyy8RXcCFKVW6w8Mgfpw48OlyznJOn0GJF37iBemBeublS GlBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:delivered-to; bh=Ssa1aghLI/Ev2sMwZEjE+zHw3zDQd7kHOx4ZXyrkoaM=; fh=z6qEiAy/zTNWGFjO2bmtKlCIU6WL+yiW+LZenWsDoNM=; b=TvH97i2aikZfFPlLwF0ikXeaE+QtC/EraSP2lHN2k3NWbKR253kUd/UEvsHed+xXLt hxIkDreFXCMKQj6rw79YAV2ZauXd5K/5Mkxyyjd9+PDeLWXCd7ktP3mdeD4ppg1hGf7E nS9N037IP00C1dzkUbys+vlk7Yj/ksSNgTtT6LSIoYfFMbdyzqgs8712TLtFnJ1nCqZQ aRSVDdlLwvFpks+vAaKlQeFX4Lupw+pYtWAZdAJKabfOP1aJqWjKNTGh43HM6KUlw8JB UhkJ7xUsxlCvQbJSkNAAZzj0AMJlKtCK987QGSHzCCuln7cxrEmy7AgSaTbIsT0hPijV F2lw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id k22-20020a1709065fd600b00a4dffd58150si4650499ejv.914.2024.04.01.09.17.14; Mon, 01 Apr 2024 09:17:14 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4CBEB68CEE6; Mon, 1 Apr 2024 19:17:03 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2FA7668CEBC for ; Mon, 1 Apr 2024 19:16:56 +0300 (EEST) Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a4702457ccbso519960466b.3 for ; Mon, 01 Apr 2024 09:16:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711988215; x=1712593015; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kT5tWvGg0NbvxJ+geO3otb3e1vCe32iv5/uAhiQLddg=; b=ErC2D+sgeRB/t0DGJSvxB5AQheI/dQPSS+dRSAW5d/xHGzJIOKj+9N3238QmG20TXV iBM90kWkduZi99HZvEUDSC21KwR0HVaw0rAntv4FmoFyLmLnNH6t5gnjCzWXHkWUHWLe /YDF2dOKGorD59tgnmVBqMe/FsoFN/SnwuTgpM/zSQc3BPDnbmooGeSnuNgOUvR/J6CJ +RdkpRzaPjtZg5zIbQtxvJxrBtiXYzH5OpZaSwCwtq5bR18BrhgK+B0B9k4ru0T73gJE cQMYFVKvexd9dS9p3QXEy/pvKNcxUpm+iITn/NbcjKoLTLgyO/qEOhUj0m3wmpPojLIn QmSA== X-Gm-Message-State: AOJu0YyO9oZagE/AWd1W075bLXd+zOjfcQFNwLBmLs5ilAmf1K5ir4iK wX2A3bZYtqR+raDzS05Krm2wJVmzcfHZgud4q1rO9oL/zeI2KjMVs8i3sBPALgp0wgPwz5zTD4l T X-Received: by 2002:a17:906:fac4:b0:a4d:f5d4:fb02 with SMTP id lu4-20020a170906fac400b00a4df5d4fb02mr6671620ejb.51.1711988215060; Mon, 01 Apr 2024 09:16:55 -0700 (PDT) Received: from Denniss-MBP.fritz.box (ip-088-152-103-184.um26.pools.vodafone-ip.de. [88.152.103.184]) by smtp.gmail.com with ESMTPSA id kf16-20020a17090776d000b00a46bf6d890bsm5491341ejc.91.2024.04.01.09.16.54 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 01 Apr 2024 09:16:54 -0700 (PDT) To: ffmpeg-devel@ffmpeg.org Date: Mon, 1 Apr 2024 18:16:50 +0200 Message-Id: <20240401161650.34063-2-dennis@obsproject.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20240401161650.34063-1-dennis@obsproject.com> References: <20240401161650.34063-1-dennis@obsproject.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] avformat/flvdec: Add support for demuxing multi-track FLV X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: =?utf-8?q?Dennis_S=C3=A4dtler_via_ffmpeg-devel?= From: =?utf-8?q?Dennis_S=C3=A4dtler?= Reply-To: FFmpeg development discussions and patches Cc: =?utf-8?q?Dennis_S=C3=A4dtler?= Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: la993p237oOd Based on enhanced-rtmp v2 spec published by Veovera: https://veovera.github.io/enhanced-rtmp/docs/enhanced/enhanced-rtmp-v2 Signed-off-by: Dennis Sädtler --- libavformat/flvdec.c | 117 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 96 insertions(+), 21 deletions(-) diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index 22a9b9e4a7..39d01d3b1f 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -105,6 +105,10 @@ typedef struct FLVContext { FLVMetaVideoColor *metaVideoColor; int meta_color_info_flag; + + uint8_t **mt_extradata; + int *mt_extradata_sz; + int mt_extradata_cnt; } FLVContext; /* AMF date type */ @@ -187,13 +191,18 @@ static void add_keyframes_index(AVFormatContext *s) } } -static AVStream *create_stream(AVFormatContext *s, int codec_type) +static AVStream *create_stream(AVFormatContext *s, int codec_type, int track_idx) { FLVContext *flv = s->priv_data; AVStream *st = avformat_new_stream(s, NULL); if (!st) return NULL; st->codecpar->codec_type = codec_type; + st->id = track_idx; + avpriv_set_pts_info(st, 32, 1, 1000); /* 32 bit pts in ms */ + if (track_idx) + return st; + if (s->nb_streams>=3 ||( s->nb_streams==2 && s->streams[0]->codecpar->codec_type != AVMEDIA_TYPE_SUBTITLE && s->streams[1]->codecpar->codec_type != AVMEDIA_TYPE_SUBTITLE @@ -210,8 +219,6 @@ static AVStream *create_stream(AVFormatContext *s, int codec_type) st->avg_frame_rate = flv->framerate; } - - avpriv_set_pts_info(st, 32, 1, 1000); /* 32 bit pts in ms */ flv->last_keyframe_stream_index = s->nb_streams - 1; add_keyframes_index(s); return st; @@ -351,6 +358,7 @@ static int flv_same_video_codec(AVCodecParameters *vpar, uint32_t flv_codecid) case FLV_CODECID_VP6A: return vpar->codec_id == AV_CODEC_ID_VP6A; case FLV_CODECID_H264: + case MKBETAG('a', 'v', 'c', '1'): return vpar->codec_id == AV_CODEC_ID_H264; default: return vpar->codec_tag == flv_codecid; @@ -407,6 +415,7 @@ static int flv_set_video_codec(AVFormatContext *s, AVStream *vstream, ret = 1; // 1 byte body size adjustment for flv_read_packet() break; case FLV_CODECID_H264: + case MKBETAG('a', 'v', 'c', '1'): par->codec_id = AV_CODEC_ID_H264; vstreami->need_parsing = AVSTREAM_PARSE_HEADERS; break; @@ -676,7 +685,7 @@ static int amf_parse_object(AVFormatContext *s, AVStream *astream, } else if (!strcmp(key, "height") && vpar) { vpar->height = num_val; } else if (!strcmp(key, "datastream")) { - AVStream *st = create_stream(s, AVMEDIA_TYPE_SUBTITLE); + AVStream *st = create_stream(s, AVMEDIA_TYPE_SUBTITLE, 0); if (!st) return AVERROR(ENOMEM); st->codecpar->codec_id = AV_CODEC_ID_TEXT; @@ -885,6 +894,9 @@ static int flv_read_close(AVFormatContext *s) FLVContext *flv = s->priv_data; for (i=0; inew_extradata[i]); + for (i=0; i < flv->mt_extradata_cnt; i++) + av_freep(&flv->mt_extradata[i]); + av_freep(&flv->mt_extradata_sz); av_freep(&flv->keyframe_times); av_freep(&flv->keyframe_filepositions); av_freep(&flv->metaVideoColor); @@ -904,18 +916,47 @@ static int flv_get_extradata(AVFormatContext *s, AVStream *st, int size) } static int flv_queue_extradata(FLVContext *flv, AVIOContext *pb, int stream, - int size) + int size, int multitrack) { if (!size) return 0; - av_free(flv->new_extradata[stream]); - flv->new_extradata[stream] = av_mallocz(size + - AV_INPUT_BUFFER_PADDING_SIZE); - if (!flv->new_extradata[stream]) - return AVERROR(ENOMEM); - flv->new_extradata_size[stream] = size; - avio_read(pb, flv->new_extradata[stream], size); + if (!multitrack) { + av_free(flv->new_extradata[stream]); + flv->new_extradata[stream] = av_mallocz(size + + AV_INPUT_BUFFER_PADDING_SIZE); + if (!flv->new_extradata[stream]) + return AVERROR(ENOMEM); + flv->new_extradata_size[stream] = size; + avio_read(pb, flv->new_extradata[stream], size); + } else { + int new_count = stream + 1; + + if (flv->mt_extradata_cnt < new_count) { + flv->mt_extradata = av_realloc(flv->mt_extradata, + sizeof(*flv->mt_extradata) * + new_count); + flv->mt_extradata_sz = av_realloc(flv->mt_extradata_sz, + sizeof(*flv->mt_extradata_sz) * + new_count); + if (!flv->mt_extradata || !flv->mt_extradata_sz) + return AVERROR(ENOMEM); + // Set newly allocated pointers/sizes to 0 + for (int i = flv->mt_extradata_cnt; i < new_count; i++) { + flv->mt_extradata[i] = NULL; + flv->mt_extradata_sz[i] = 0; + } + flv->mt_extradata_cnt = new_count; + } + + av_free(flv->mt_extradata[stream]); + flv->mt_extradata[stream] = av_mallocz(size + AV_INPUT_BUFFER_PADDING_SIZE); + if (!flv->mt_extradata[stream]) + return AVERROR(ENOMEM); + flv->mt_extradata_sz[stream] = size; + avio_read(pb, flv->mt_extradata[stream], size); + } + return 0; } @@ -1031,7 +1072,7 @@ static int flv_data_packet(AVFormatContext *s, AVPacket *pkt, } if (i == s->nb_streams) { - st = create_stream(s, AVMEDIA_TYPE_SUBTITLE); + st = create_stream(s, AVMEDIA_TYPE_SUBTITLE, 0); if (!st) return AVERROR(ENOMEM); st->codecpar->codec_id = AV_CODEC_ID_TEXT; @@ -1203,6 +1244,9 @@ static int flv_read_packet(AVFormatContext *s, AVPacket *pkt) int last = -1; int orig_size; int enhanced_flv = 0; + int multitrack = 0; + int pkt_type = 0; + uint8_t track_idx = 0; uint32_t video_codec_id = 0; retry: @@ -1256,14 +1300,33 @@ retry: * https://github.com/veovera/enhanced-rtmp/blob/main/enhanced-rtmp-v1.pdf * */ enhanced_flv = (flags >> 7) & 1; + pkt_type = enhanced_flv ? video_codec_id : 0; size--; + + if (pkt_type == PacketTypeMultitrack) { + uint8_t types = avio_r8(s->pb); + int multitrack_type = types >> 4; + pkt_type = types & 0xF; + + if (multitrack_type != MultitrackTypeOneTrack) { + av_log(s, AV_LOG_ERROR, "Multitrack types other than MultitrackTypeOneTrack are unsupported!\n"); + return AVERROR_PATCHWELCOME; + } + + multitrack = 1; + size--; + } + if (enhanced_flv) { video_codec_id = avio_rb32(s->pb); size -= 4; } + if (multitrack) { + track_idx = avio_r8(s->pb); + size--; + } - if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { - int pkt_type = flags & 0x0F; + if (enhanced_flv && (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { if (pkt_type == PacketTypeMetadata) { int ret = flv_parse_video_color_info(s, st, next); av_log(s, AV_LOG_DEBUG, "enhanced flv parse metadata ret %d and skip\n", ret); @@ -1327,7 +1390,8 @@ skip: break; } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO && - (s->video_codec_id || flv_same_video_codec(st->codecpar, video_codec_id))) + (s->video_codec_id || flv_same_video_codec(st->codecpar, video_codec_id)) && + st->id == track_idx) break; } else if (stream_type == FLV_STREAM_TYPE_SUBTITLE) { if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) @@ -1339,7 +1403,7 @@ skip: } if (i == s->nb_streams) { static const enum AVMediaType stream_types[] = {AVMEDIA_TYPE_VIDEO, AVMEDIA_TYPE_AUDIO, AVMEDIA_TYPE_SUBTITLE, AVMEDIA_TYPE_DATA}; - st = create_stream(s, stream_types[stream_type]); + st = create_stream(s, stream_types[stream_type], track_idx); if (!st) return AVERROR(ENOMEM); } @@ -1446,7 +1510,7 @@ retry_duration: st->codecpar->codec_id == AV_CODEC_ID_VP9) { int type = 0; if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO) { - type = flags & 0x0F; + type = pkt_type; } else { type = avio_r8(s->pb); size--; @@ -1462,7 +1526,8 @@ retry_duration: flv->meta_color_info_flag = 0; } - if (st->codecpar->codec_id == AV_CODEC_ID_H264 || st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || + if (st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || + (st->codecpar->codec_id == AV_CODEC_ID_H264 && (!enhanced_flv || type == PacketTypeCodedFrames)) || (st->codecpar->codec_id == AV_CODEC_ID_HEVC && type == PacketTypeCodedFrames)) { // sign extension int32_t cts = (avio_rb24(s->pb) + 0xff800000) ^ 0xff800000; @@ -1485,7 +1550,7 @@ retry_duration: AVDictionaryEntry *t; if (st->codecpar->extradata) { - if ((ret = flv_queue_extradata(flv, s->pb, stream_type, size)) < 0) + if ((ret = flv_queue_extradata(flv, s->pb, multitrack ? track_idx : stream_type, size, multitrack)) < 0) return ret; ret = FFERROR_REDO; goto leave; @@ -1516,7 +1581,7 @@ retry_duration: pkt->pts = pts == AV_NOPTS_VALUE ? dts : pts; pkt->stream_index = st->index; pkt->pos = pos; - if (flv->new_extradata[stream_type]) { + if (!multitrack && flv->new_extradata[stream_type]) { int ret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, flv->new_extradata[stream_type], flv->new_extradata_size[stream_type]); @@ -1524,6 +1589,16 @@ retry_duration: flv->new_extradata[stream_type] = NULL; flv->new_extradata_size[stream_type] = 0; } + } else if (multitrack + && flv->mt_extradata_cnt > track_idx + && flv->mt_extradata[track_idx]) { + int ret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, + flv->mt_extradata[track_idx], + flv->mt_extradata_sz[track_idx]); + if (ret >= 0) { + flv->mt_extradata[track_idx] = NULL; + flv->mt_extradata_sz[track_idx] = 0; + } } if (stream_type == FLV_STREAM_TYPE_AUDIO && (sample_rate != flv->last_sample_rate ||