From patchwork Tue May 21 09:02:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49086 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5096766pzb; Tue, 21 May 2024 02:04:02 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCV1KIfaofZG38JELaxlS5RoJqJ14ArcZtIxipMnJchUtGrvE8nJcPoA0JV1ao4IFBcwuaaW/JUHb6bd9aqt153vNRtB9m7eqaznsw== X-Google-Smtp-Source: AGHT+IEAGwn1Um24fi8mkYJUppiUlp17nC83mFqMh48q4kEnIkfkptPdON5hEix5+9sJYQGyRWY7 X-Received: by 2002:a17:907:1b14:b0:a59:d0fc:7ac5 with SMTP id a640c23a62f3a-a5a2d57d672mr2614503466b.32.1716282241864; Tue, 21 May 2024 02:04:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282241; cv=none; d=google.com; s=arc-20160816; b=fF4rxV+T2zK/B8PGc/7EEUadwvV5w2e9XK2Qy/uc+X32b0qiDjdeVyx27/Vad9I6bF WTO74u6mBf3ANwmSDAWG3YhjdFFnKiiPsdvA3g0by5i/y6gtxakVDycDlOieow3gxy71 Tir8arWxus31hQGeHmCGFq0fhxX7OvfZTm9FcyfQrIlylLoAunL8B4Sl69HHdpFcr2yj ItrZWV9e4DW3u88UUKhJgTKQaO9gWF39q2O0aH169SCwJEdcpuhdJhNc5ypriRGCpDL2 NMT//ECPUbGnRv9jNNLbabEBG6Ft0UXE+tyXlzfnFi4Acjf87L8JHMdEWOBWALbhkpG4 sSQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=H+s2dUYrzpYQbgqMj02DbPj5AVcHGZG4uqPyZXRSrr8=; fh=z6qEiAy/zTNWGFjO2bmtKlCIU6WL+yiW+LZenWsDoNM=; b=rPZ3Cjz2MyqTLjWghn83vtdq2ZU73m4GjojhPyyzdnjVWocRq1CDQNw4CLISJLTyFg oZZP/1OTu4HwME5+YTCKloNFBi6Qn/20y6b6CRmeZIb02TxkGCetF+GlXkOqQXSGepuQ 0Lzv5Oft7W3BKV/DczbBgS4wSM1clvzFevaDHyC9M3rDVVZ7mHhVcpk65/hrgatGufdZ oY9M8LmuigARyPD2BxBd18SN5CQufeSkzJyQgn7QpN5f9iNgHdXy3LxrAKq+mjKsqdAn g5b6CFlqjlfkRe9O2BSQtpDqqnBTFlASCgSOedO43j7j0ryLhBJ1aoxWUa6NAF1LBgYN yIwA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=lUTukQKT; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a640c23a62f3a-a5a17ba57d7si1326039166b.656.2024.05.21.02.04.00; Tue, 21 May 2024 02:04:01 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=lUTukQKT; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EED3C68D353; Tue, 21 May 2024 12:03:44 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id DCA3168AD99 for ; Tue, 21 May 2024 12:03:37 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id 2E6FE27FFD906; Tue, 21 May 2024 11:03:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282217; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=daFX0fvzxqrp+dqdwXTpbUJoa4Un4S5ZAyBHIDyruvQ=; b=lUTukQKT8sIeLq+pAZ0Vwofum1I/mHjtkIV2VF9wmSqv7lNuESn+WEq4weroKQ7u6KEJre 4STavGU+dDGCINLgleYqoxZdBR81nC2lrR5QwunuyiYo/XzQucRAFUU9Mpi9aiKiN7plEH ff4cx5JirCvXATcP4KfJ5Noi7e+Xhz7MVTSjoOLKrpWuXWvKSQyAUfuF7KRJ+ov4yLEmHc OjVROPdet8B3W+CfuSv+L2aNLWQq4w4Jt3EL78iig8oJ8D/aUmvrZIsfilh8Q7H9mtK33U WvOTlYTj8bzSo6WTmWbP/KB1trRR2L4LqRhd0bM1pZ4DembWwFVDog1m4HmjhA== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:10 +0200 Message-ID: <20240521090316.782-2-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 01/13] avformat/flvenc: Implement support for multi-track video X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: =?utf-8?q?Dennis_S=C3=A4dtler?= Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: F0+NpheNdaf7 From: Dennis Sädtler via ffmpeg-devel Based on enhanced-rtmp v2 spec published by Veovera: https://veovera.github.io/enhanced-rtmp/docs/enhanced/enhanced-rtmp-v2 This implementation maintains some backwards compatibility by only writing the track information for track indices > 0. This means that older FFmpeg versions - and possibly other software - can still read the first video track properly and skip over unsupported packets. Signed-off-by: Dennis Sädtler --- libavformat/flv.h | 7 ++ libavformat/flvenc.c | 158 ++++++++++++++++++++++++++++++------------- 2 files changed, 118 insertions(+), 47 deletions(-) diff --git a/libavformat/flv.h b/libavformat/flv.h index f710963b92..653c2bc82c 100644 --- a/libavformat/flv.h +++ b/libavformat/flv.h @@ -125,6 +125,13 @@ enum { PacketTypeCodedFramesX = 3, PacketTypeMetadata = 4, PacketTypeMPEG2TSSequenceStart = 5, + PacketTypeMultitrack = 6, +}; + +enum { + MultitrackTypeOneTrack = 0x00, + MultitrackTypeManyTracks = 0x10, + MultitrackTypeManyTracksManyCodecs = 0x20, }; enum { diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c index aba1d7d80b..7ddc59e369 100644 --- a/libavformat/flvenc.c +++ b/libavformat/flvenc.c @@ -125,8 +125,9 @@ typedef struct FLVContext { AVCodecParameters *data_par; int flags; - int64_t last_ts[FLV_STREAM_TYPE_NB]; - int metadata_pkt_written; + int64_t *last_ts; + int *metadata_pkt_written; + int *video_track_idx_map; } FLVContext; static int get_audio_flags(AVFormatContext *s, AVCodecParameters *par) @@ -484,7 +485,7 @@ static void write_metadata(AVFormatContext *s, unsigned int ts) avio_wb32(pb, flv->metadata_totalsize + 11); } -static void flv_write_metadata_packet(AVFormatContext *s, AVCodecParameters *par, unsigned int ts) +static void flv_write_metadata_packet(AVFormatContext *s, AVCodecParameters *par, unsigned int ts, int stream_idx) { AVIOContext *pb = s->pb; FLVContext *flv = s->priv_data; @@ -494,7 +495,9 @@ static void flv_write_metadata_packet(AVFormatContext *s, AVCodecParameters *par int64_t total_size = 0; const AVPacketSideData *side_data = NULL; - if (flv->metadata_pkt_written) return; + if (flv->metadata_pkt_written[stream_idx]) + return; + if (par->codec_id == AV_CODEC_ID_HEVC || par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9) { int flags_size = 5; @@ -616,7 +619,7 @@ static void flv_write_metadata_packet(AVFormatContext *s, AVCodecParameters *par avio_wb24(pb, total_size); avio_skip(pb, total_size + 10 - 3); avio_wb32(pb, total_size + 11); // previous tag size - flv->metadata_pkt_written = 1; + flv->metadata_pkt_written[stream_idx] = 1; } } @@ -631,7 +634,7 @@ static int unsupported_codec(AVFormatContext *s, return AVERROR(ENOSYS); } -static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts) { +static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts, int stream_index) { int64_t data_size; AVIOContext *pb = s->pb; FLVContext *flv = s->priv_data; @@ -681,12 +684,32 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i } avio_write(pb, par->extradata, par->extradata_size); } else { - if (par->codec_id == AV_CODEC_ID_HEVC) { - avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeSequenceStart | FLV_FRAME_KEY); // ExVideoTagHeader mode with PacketTypeSequenceStart - avio_write(pb, "hvc1", 4); - } else if (par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9) { - avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeSequenceStart | FLV_FRAME_KEY); - avio_write(pb, par->codec_id == AV_CODEC_ID_AV1 ? "av01" : "vp09", 4); + int track_idx = flv->video_track_idx_map[stream_index]; + // If video stream has track_idx > 0 we need to send H.264 as extended video packet + int extended_flv = (par->codec_id == AV_CODEC_ID_H264 && track_idx) + || par->codec_id == AV_CODEC_ID_HEVC + || par->codec_id == AV_CODEC_ID_AV1 + || par->codec_id == AV_CODEC_ID_VP9; + + if (extended_flv) { + if (track_idx) { + avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeMultitrack | FLV_FRAME_KEY); + avio_w8(pb, MultitrackTypeOneTrack | PacketTypeSequenceStart); + } else { + avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeSequenceStart | FLV_FRAME_KEY); + } + + if (par->codec_id == AV_CODEC_ID_H264) + avio_write(pb, "avc1", 4); + else if (par->codec_id == AV_CODEC_ID_HEVC) + avio_write(pb, "hvc1", 4); + else if (par->codec_id == AV_CODEC_ID_AV1) + avio_write(pb, "av01", 4); + else if (par->codec_id == AV_CODEC_ID_VP9) + avio_write(pb, "vp09", 4); + + if (track_idx) + avio_w8(pb, track_idx); } else { avio_w8(pb, par->codec_tag | FLV_FRAME_KEY); // flags avio_w8(pb, 0); // AVC sequence header @@ -769,13 +792,12 @@ static int shift_data(AVFormatContext *s) static int flv_init(struct AVFormatContext *s) { int i; + int video_ctr = 0; FLVContext *flv = s->priv_data; - if (s->nb_streams > FLV_STREAM_TYPE_NB) { - av_log(s, AV_LOG_ERROR, "invalid number of streams %d\n", - s->nb_streams); - return AVERROR(EINVAL); - } + flv->last_ts = av_mallocz(sizeof(*flv->last_ts) * s->nb_streams); + flv->metadata_pkt_written = av_mallocz(sizeof(*flv->metadata_pkt_written) * s->nb_streams); + flv->video_track_idx_map = av_mallocz(sizeof(*flv->video_track_idx_map) * s->nb_streams); for (i = 0; i < s->nb_streams; i++) { AVCodecParameters *par = s->streams[i]->codecpar; @@ -786,12 +808,17 @@ static int flv_init(struct AVFormatContext *s) s->streams[i]->avg_frame_rate.num) { flv->framerate = av_q2d(s->streams[i]->avg_frame_rate); } - if (flv->video_par) { + flv->video_track_idx_map[i] = video_ctr++; + if (flv->video_par && flv->flags & FLV_ADD_KEYFRAME_INDEX) { av_log(s, AV_LOG_ERROR, - "at most one video stream is supported in flv\n"); + "at most one video stream is supported in flv with keyframe index\n"); return AVERROR(EINVAL); + } else if (flv->video_par) { + av_log(s, AV_LOG_WARNING, + "more than one video stream is not supported by most flv demuxers.\n"); } - flv->video_par = par; + if (!flv->video_par) + flv->video_par = par; if (!ff_codec_get_tag(flv_video_codec_ids, par->codec_id)) return unsupported_codec(s, "Video", par->codec_id); @@ -881,7 +908,7 @@ static int flv_write_header(AVFormatContext *s) } for (i = 0; i < s->nb_streams; i++) { - flv_write_codec_header(s, s->streams[i]->codecpar, 0); + flv_write_codec_header(s, s->streams[i]->codecpar, 0, i); } flv->datastart_offset = avio_tell(pb); @@ -989,6 +1016,7 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) uint8_t frametype = pkt->flags & AV_PKT_FLAG_KEY ? FLV_FRAME_KEY : FLV_FRAME_INTER; int flags = -1, flags_size, ret = 0; int64_t cur_offset = avio_tell(pb); + int track_idx = flv->video_track_idx_map[pkt->stream_index]; if (par->codec_type == AVMEDIA_TYPE_AUDIO && !pkt->size) { av_log(s, AV_LOG_WARNING, "Empty audio Packet\n"); @@ -1005,7 +1033,12 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) else flags_size = 1; - if (par->codec_id == AV_CODEC_ID_HEVC && pkt->pts != pkt->dts) + if (par->codec_type == AVMEDIA_TYPE_VIDEO && track_idx) + flags_size += 2; // additional header bytes for multi-track video + + if ((par->codec_id == AV_CODEC_ID_HEVC || + (par->codec_id == AV_CODEC_ID_H264 && track_idx)) + && pkt->pts != pkt->dts) flags_size += 3; if (par->codec_id == AV_CODEC_ID_AAC || par->codec_id == AV_CODEC_ID_H264 @@ -1018,9 +1051,9 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) if (ret < 0) return ret; memcpy(par->extradata, side, side_size); - flv_write_codec_header(s, par, pkt->dts); + flv_write_codec_header(s, par, pkt->dts, pkt->stream_index); } - flv_write_metadata_packet(s, par, pkt->dts); + flv_write_metadata_packet(s, par, pkt->dts, pkt->stream_index); } if (flv->delay == AV_NOPTS_VALUE) @@ -1142,32 +1175,59 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) avio_seek(pb, data_size + 10 - 3, SEEK_CUR); avio_wb32(pb, data_size + 11); } else { - av_assert1(flags>=0); - if (par->codec_id == AV_CODEC_ID_HEVC) { - int pkttype = (pkt->pts != pkt->dts) ? PacketTypeCodedFrames : PacketTypeCodedFramesX; - avio_w8(pb, FLV_IS_EX_HEADER | pkttype | frametype); // ExVideoTagHeader mode with PacketTypeCodedFrames(X) - avio_write(pb, "hvc1", 4); - if (pkttype == PacketTypeCodedFrames) + int extended_flv = (par->codec_id == AV_CODEC_ID_H264 && track_idx) + || par->codec_id == AV_CODEC_ID_HEVC + || par->codec_id == AV_CODEC_ID_AV1 + || par->codec_id == AV_CODEC_ID_VP9; + + if (extended_flv) { + int h2645 = par->codec_id == AV_CODEC_ID_H264 || + par->codec_id == AV_CODEC_ID_HEVC; + int pkttype = PacketTypeCodedFrames; + // Optimisation for HEVC/H264: Do not send composition time if DTS == PTS + if (h2645 && pkt->pts == pkt->dts) + pkttype = PacketTypeCodedFramesX; + + if (track_idx) { + avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeMultitrack | frametype); + avio_w8(pb, MultitrackTypeOneTrack | pkttype); + } else { + avio_w8(pb, FLV_IS_EX_HEADER | pkttype | frametype); + } + + if (par->codec_id == AV_CODEC_ID_H264) + avio_write(pb, "avc1", 4); + else if (par->codec_id == AV_CODEC_ID_HEVC) + avio_write(pb, "hvc1", 4); + else if (par->codec_id == AV_CODEC_ID_AV1) + avio_write(pb, "av01", 4); + else if (par->codec_id == AV_CODEC_ID_VP9) + avio_write(pb, "vp09", 4); + + if (track_idx) + avio_w8(pb, track_idx); + if (h2645 && pkttype == PacketTypeCodedFrames) avio_wb24(pb, pkt->pts - pkt->dts); - } else if (par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9) { - avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeCodedFrames | frametype); - avio_write(pb, par->codec_id == AV_CODEC_ID_AV1 ? "av01" : "vp09", 4); } else { + av_assert1(flags>=0); avio_w8(pb, flags); - } - if (par->codec_id == AV_CODEC_ID_VP6) - avio_w8(pb,0); - if (par->codec_id == AV_CODEC_ID_VP6F || par->codec_id == AV_CODEC_ID_VP6A) { - if (par->extradata_size) - avio_w8(pb, par->extradata[0]); - else - avio_w8(pb, ((FFALIGN(par->width, 16) - par->width) << 4) | - (FFALIGN(par->height, 16) - par->height)); - } else if (par->codec_id == AV_CODEC_ID_AAC) - avio_w8(pb, 1); // AAC raw - else if (par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4) { - avio_w8(pb, 1); // AVC NALU - avio_wb24(pb, pkt->pts - pkt->dts); + + if (par->codec_id == AV_CODEC_ID_VP6) { + avio_w8(pb,0); + } else if (par->codec_id == AV_CODEC_ID_VP6F || + par->codec_id == AV_CODEC_ID_VP6A) { + if (par->extradata_size) + avio_w8(pb, par->extradata[0]); + else + avio_w8(pb, ((FFALIGN(par->width, 16) - par->width) << 4) | + (FFALIGN(par->height, 16) - par->height)); + } else if (par->codec_id == AV_CODEC_ID_AAC) { + avio_w8(pb, 1); // AAC raw + } else if (par->codec_id == AV_CODEC_ID_H264 || + par->codec_id == AV_CODEC_ID_MPEG4) { + avio_w8(pb, 1); // AVC NALU + avio_wb24(pb, pkt->pts - pkt->dts); + } } avio_write(pb, data ? data : pkt->data, size); @@ -1234,6 +1294,10 @@ static void flv_deinit(AVFormatContext *s) } flv->filepositions = flv->head_filepositions = NULL; flv->filepositions_count = 0; + + av_freep(&flv->last_ts); + av_freep(&flv->metadata_pkt_written); + av_freep(&flv->video_track_idx_map); } static const AVOption options[] = { From patchwork Tue May 21 09:02:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49087 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5096853pzb; Tue, 21 May 2024 02:04:13 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVS7qZesB49bv0fdQSQerE3QCrulAXL14WqaVjytClXNEvPMLNUXu0gRCSBTYKvU7iKR8gnSlpme+wQM+Fg4J9uXnx3fYyGs4t6JQ== X-Google-Smtp-Source: AGHT+IGwjvLVoYJ2cuQOgPJQQLXrLggxxqB7WNnA2S0KUgeW/F4bBm56K9cR7fp/vbt0I+gYJMih X-Received: by 2002:a17:906:d41:b0:a5d:e05:c5eb with SMTP id a640c23a62f3a-a5d0e05c639mr689067766b.65.1716282253441; Tue, 21 May 2024 02:04:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282253; cv=none; d=google.com; s=arc-20160816; b=boFQ1S1SBDuv7g4J2pAm3CysIDX5u00+SY2KdLvt12BB7nR/RYI8hGtsEr4Tka+n/8 wygAVMrUQca8urNO32UGRO7XauIZmzQFHAoF+2NE4lnn5qLJQekGbuuFyuit8/Iaehwh bPQlzr3ibBQGXzJa538c3pgDwowR2t46x/RxNNUCnXJvxlBHQH1Pdsx3HR33I0xpKf4P vl9fwrwDFQglLpnMhtAtjf3e3TG1z20CPulkkg7AjhwbAq3Ch5HqOMsKXiwDEYhBBTKr f4cRMpM7U41usL1oOB5UBXQox9iwaZzErGrr6uTFlQIu1SttkSjsxKtqDBOyv/5EX9ou RjLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=uT58JSxaHp9tSdQaQe1KYGg23Jk81dqHj4J8O5i8Ys0=; fh=z6qEiAy/zTNWGFjO2bmtKlCIU6WL+yiW+LZenWsDoNM=; b=epTkFX2xJFQKdhZk0gJjBIGyCtUVdAqDlX8fqRGRTLfH5WSmyfXDWuiysanOV07rdE Uvf0SXnqLtZxfQ/IejObpHdDOgyo27z+pgNitOLGPcX7u3OM7oWM20MdWILS7pE62I/C o6PeHome7x3fzhxKJ8KHQOVH7Jx47B1IH4JxkhLjweqL2G5rdq9k3Q7MR2aJGx9/UM5Z ErSWxvlzmSAL6cO4z2vo81b+1C1dmbZeUvobvgCJOHNHAi1GGtTVglukPKdDRR2wUfR6 sHllIlsRTWu9t1gK4FMcenkMC8atV9dh8WKkVpIqIUct/80w8wH3NJTwNdMUOIX0NWKl FS+g==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=bgBdh4Fl; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a640c23a62f3a-a61c5f55457si163795066b.1033.2024.05.21.02.04.12; Tue, 21 May 2024 02:04:13 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=bgBdh4Fl; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CC5A868CDAC; Tue, 21 May 2024 12:03:45 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 18FFD68D2D4 for ; Tue, 21 May 2024 12:03:38 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id 99D2E27FFD908; Tue, 21 May 2024 11:03:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282217; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JL0NFd9iqU2qyZIhx2VRuQKz1d1IG+pdubnNgF9EcrQ=; b=bgBdh4FlREh35wtQsLasBCVYVwvf8ea10wv4RnG8GQhoDfNDbTg36BNau7QpqmI6g7Z75H zczTxMR+iGDQ+dEVGe1ylUTmYlyUYs9Jjmalfivph6NBxZdhyB5lKnqquQv9SZD8TGKJfP I+mY3NnlUw34VRTHmGDImQ36r9rYjFXvcvohQ/CEWcY+uGczHbLi62sDhgLFRcAOr0afkX b3Sq4QS5jKYhGYQ4KvpFBU+nqjQuInFJjLCA50qy42JgwK2mr8MxjcGQlb+8edrimqVCkT 35XpJmFoc0+Uk+bFHmIE9rrn8fMzeTbqNoR6zr9soXaq/V5TLdSKzX8JAnR3Ag== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:11 +0200 Message-ID: <20240521090316.782-3-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 02/13] avformat/flvdec: Add support for demuxing multi-track FLV X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: =?utf-8?q?Dennis_S=C3=A4dtler?= Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: M5t3hnjmZEN9 From: Dennis Sädtler via ffmpeg-devel Based on enhanced-rtmp v2 spec published by Veovera: https://veovera.github.io/enhanced-rtmp/docs/enhanced/enhanced-rtmp-v2 Signed-off-by: Dennis Sädtler --- libavformat/flvdec.c | 117 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 96 insertions(+), 21 deletions(-) diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index 22a9b9e4a7..2445d1fd5e 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -105,6 +105,10 @@ typedef struct FLVContext { FLVMetaVideoColor *metaVideoColor; int meta_color_info_flag; + + uint8_t **mt_extradata; + int *mt_extradata_sz; + int mt_extradata_cnt; } FLVContext; /* AMF date type */ @@ -187,13 +191,18 @@ static void add_keyframes_index(AVFormatContext *s) } } -static AVStream *create_stream(AVFormatContext *s, int codec_type) +static AVStream *create_stream(AVFormatContext *s, int codec_type, int track_idx) { FLVContext *flv = s->priv_data; AVStream *st = avformat_new_stream(s, NULL); if (!st) return NULL; st->codecpar->codec_type = codec_type; + st->id = track_idx; + avpriv_set_pts_info(st, 32, 1, 1000); /* 32 bit pts in ms */ + if (track_idx) + return st; + if (s->nb_streams>=3 ||( s->nb_streams==2 && s->streams[0]->codecpar->codec_type != AVMEDIA_TYPE_SUBTITLE && s->streams[1]->codecpar->codec_type != AVMEDIA_TYPE_SUBTITLE @@ -210,8 +219,6 @@ static AVStream *create_stream(AVFormatContext *s, int codec_type) st->avg_frame_rate = flv->framerate; } - - avpriv_set_pts_info(st, 32, 1, 1000); /* 32 bit pts in ms */ flv->last_keyframe_stream_index = s->nb_streams - 1; add_keyframes_index(s); return st; @@ -351,6 +358,7 @@ static int flv_same_video_codec(AVCodecParameters *vpar, uint32_t flv_codecid) case FLV_CODECID_VP6A: return vpar->codec_id == AV_CODEC_ID_VP6A; case FLV_CODECID_H264: + case MKBETAG('a', 'v', 'c', '1'): return vpar->codec_id == AV_CODEC_ID_H264; default: return vpar->codec_tag == flv_codecid; @@ -407,6 +415,7 @@ static int flv_set_video_codec(AVFormatContext *s, AVStream *vstream, ret = 1; // 1 byte body size adjustment for flv_read_packet() break; case FLV_CODECID_H264: + case MKBETAG('a', 'v', 'c', '1'): par->codec_id = AV_CODEC_ID_H264; vstreami->need_parsing = AVSTREAM_PARSE_HEADERS; break; @@ -676,7 +685,7 @@ static int amf_parse_object(AVFormatContext *s, AVStream *astream, } else if (!strcmp(key, "height") && vpar) { vpar->height = num_val; } else if (!strcmp(key, "datastream")) { - AVStream *st = create_stream(s, AVMEDIA_TYPE_SUBTITLE); + AVStream *st = create_stream(s, AVMEDIA_TYPE_SUBTITLE, 0); if (!st) return AVERROR(ENOMEM); st->codecpar->codec_id = AV_CODEC_ID_TEXT; @@ -885,6 +894,9 @@ static int flv_read_close(AVFormatContext *s) FLVContext *flv = s->priv_data; for (i=0; inew_extradata[i]); + for (i=0; i < flv->mt_extradata_cnt; i++) + av_freep(&flv->mt_extradata[i]); + av_freep(&flv->mt_extradata_sz); av_freep(&flv->keyframe_times); av_freep(&flv->keyframe_filepositions); av_freep(&flv->metaVideoColor); @@ -904,18 +916,47 @@ static int flv_get_extradata(AVFormatContext *s, AVStream *st, int size) } static int flv_queue_extradata(FLVContext *flv, AVIOContext *pb, int stream, - int size) + int size, int multitrack) { if (!size) return 0; - av_free(flv->new_extradata[stream]); - flv->new_extradata[stream] = av_mallocz(size + - AV_INPUT_BUFFER_PADDING_SIZE); - if (!flv->new_extradata[stream]) - return AVERROR(ENOMEM); - flv->new_extradata_size[stream] = size; - avio_read(pb, flv->new_extradata[stream], size); + if (!multitrack) { + av_free(flv->new_extradata[stream]); + flv->new_extradata[stream] = av_mallocz(size + + AV_INPUT_BUFFER_PADDING_SIZE); + if (!flv->new_extradata[stream]) + return AVERROR(ENOMEM); + flv->new_extradata_size[stream] = size; + avio_read(pb, flv->new_extradata[stream], size); + } else { + int new_count = stream + 1; + + if (flv->mt_extradata_cnt < new_count) { + flv->mt_extradata = av_realloc(flv->mt_extradata, + sizeof(*flv->mt_extradata) * + new_count); + flv->mt_extradata_sz = av_realloc(flv->mt_extradata_sz, + sizeof(*flv->mt_extradata_sz) * + new_count); + if (!flv->mt_extradata || !flv->mt_extradata_sz) + return AVERROR(ENOMEM); + // Set newly allocated pointers/sizes to 0 + for (int i = flv->mt_extradata_cnt; i < new_count; i++) { + flv->mt_extradata[i] = NULL; + flv->mt_extradata_sz[i] = 0; + } + flv->mt_extradata_cnt = new_count; + } + + av_free(flv->mt_extradata[stream]); + flv->mt_extradata[stream] = av_mallocz(size + AV_INPUT_BUFFER_PADDING_SIZE); + if (!flv->mt_extradata[stream]) + return AVERROR(ENOMEM); + flv->mt_extradata_sz[stream] = size; + avio_read(pb, flv->mt_extradata[stream], size); + } + return 0; } @@ -1031,7 +1072,7 @@ static int flv_data_packet(AVFormatContext *s, AVPacket *pkt, } if (i == s->nb_streams) { - st = create_stream(s, AVMEDIA_TYPE_SUBTITLE); + st = create_stream(s, AVMEDIA_TYPE_SUBTITLE, 0); if (!st) return AVERROR(ENOMEM); st->codecpar->codec_id = AV_CODEC_ID_TEXT; @@ -1203,6 +1244,9 @@ static int flv_read_packet(AVFormatContext *s, AVPacket *pkt) int last = -1; int orig_size; int enhanced_flv = 0; + int multitrack = 0; + int pkt_type = 0; + uint8_t track_idx = 0; uint32_t video_codec_id = 0; retry: @@ -1256,14 +1300,33 @@ retry: * https://github.com/veovera/enhanced-rtmp/blob/main/enhanced-rtmp-v1.pdf * */ enhanced_flv = (flags >> 7) & 1; + pkt_type = enhanced_flv ? video_codec_id : 0; size--; + + if (pkt_type == PacketTypeMultitrack) { + uint8_t types = avio_r8(s->pb); + int multitrack_type = types >> 4; + pkt_type = types & 0xF; + + if (multitrack_type != MultitrackTypeOneTrack) { + av_log(s, AV_LOG_ERROR, "Multitrack types other than MultitrackTypeOneTrack are unsupported!\n"); + return AVERROR_PATCHWELCOME; + } + + multitrack = 1; + size--; + } + if (enhanced_flv) { video_codec_id = avio_rb32(s->pb); size -= 4; } + if (multitrack) { + track_idx = avio_r8(s->pb); + size--; + } - if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { - int pkt_type = flags & 0x0F; + if (enhanced_flv && (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { if (pkt_type == PacketTypeMetadata) { int ret = flv_parse_video_color_info(s, st, next); av_log(s, AV_LOG_DEBUG, "enhanced flv parse metadata ret %d and skip\n", ret); @@ -1327,7 +1390,8 @@ skip: break; } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO && - (s->video_codec_id || flv_same_video_codec(st->codecpar, video_codec_id))) + (s->video_codec_id || flv_same_video_codec(st->codecpar, video_codec_id)) && + st->id == track_idx) break; } else if (stream_type == FLV_STREAM_TYPE_SUBTITLE) { if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) @@ -1339,7 +1403,7 @@ skip: } if (i == s->nb_streams) { static const enum AVMediaType stream_types[] = {AVMEDIA_TYPE_VIDEO, AVMEDIA_TYPE_AUDIO, AVMEDIA_TYPE_SUBTITLE, AVMEDIA_TYPE_DATA}; - st = create_stream(s, stream_types[stream_type]); + st = create_stream(s, stream_types[stream_type], track_idx); if (!st) return AVERROR(ENOMEM); } @@ -1446,7 +1510,7 @@ retry_duration: st->codecpar->codec_id == AV_CODEC_ID_VP9) { int type = 0; if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO) { - type = flags & 0x0F; + type = pkt_type; } else { type = avio_r8(s->pb); size--; @@ -1462,7 +1526,8 @@ retry_duration: flv->meta_color_info_flag = 0; } - if (st->codecpar->codec_id == AV_CODEC_ID_H264 || st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || + if (st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || + (st->codecpar->codec_id == AV_CODEC_ID_H264 && (!enhanced_flv || type == PacketTypeCodedFrames)) || (st->codecpar->codec_id == AV_CODEC_ID_HEVC && type == PacketTypeCodedFrames)) { // sign extension int32_t cts = (avio_rb24(s->pb) + 0xff800000) ^ 0xff800000; @@ -1485,7 +1550,7 @@ retry_duration: AVDictionaryEntry *t; if (st->codecpar->extradata) { - if ((ret = flv_queue_extradata(flv, s->pb, stream_type, size)) < 0) + if ((ret = flv_queue_extradata(flv, s->pb, multitrack ? track_idx : stream_type, size, multitrack)) < 0) return ret; ret = FFERROR_REDO; goto leave; @@ -1516,7 +1581,7 @@ retry_duration: pkt->pts = pts == AV_NOPTS_VALUE ? dts : pts; pkt->stream_index = st->index; pkt->pos = pos; - if (flv->new_extradata[stream_type]) { + if (!multitrack && flv->new_extradata[stream_type]) { int ret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, flv->new_extradata[stream_type], flv->new_extradata_size[stream_type]); @@ -1524,6 +1589,16 @@ retry_duration: flv->new_extradata[stream_type] = NULL; flv->new_extradata_size[stream_type] = 0; } + } else if (multitrack + && flv->mt_extradata_cnt > track_idx + && flv->mt_extradata[track_idx]) { + int ret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, + flv->mt_extradata[track_idx], + flv->mt_extradata_sz[track_idx]); + if (ret >= 0) { + flv->mt_extradata[track_idx] = NULL; + flv->mt_extradata_sz[track_idx] = 0; + } } if (stream_type == FLV_STREAM_TYPE_AUDIO && (sample_rate != flv->last_sample_rate || From patchwork Tue May 21 09:02:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49088 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5096953pzb; Tue, 21 May 2024 02:04:25 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCU/8Fp7gSrbRqyOR1HSxjlcCSUmJBwTiKUfyHhNzw0vfiknQSLCXjw1S7f/93cBpWQsgmqnF/LMdzVUj67JLiFjv31gD0b93Q+jzA== X-Google-Smtp-Source: AGHT+IGdZwBdCKF+K4hUJHx+jqUIxlg11a+DHEZL3e4cXNiNzh77FVLY3UPSg2Rt6jeEwSV+ar9M X-Received: by 2002:a19:5e55:0:b0:522:2990:7134 with SMTP id 2adb3069b0e04-52407ce92edmr2284728e87.33.1716282265290; Tue, 21 May 2024 02:04:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282265; cv=none; d=google.com; s=arc-20160816; b=SOreZfaxfHrH9gzYmlQb8S7GhER9kpFnIRUmFPo0E2yAMEQ0bcNB3R+ZhCDm8unT82 rn6ax00K8OpbEgkZqeTNq5P4Piwypx1G57E3AMRS37ZrGpCYKxtUbWU1CTXRDmnLqBwS 5aKavlV0mPSqOf1DKbLPs/+3Tg0QpTtJAi9pWRFOvttaBK7LdhdgxnbN1OCLSykpAyxI zWj4WxUIcTmJ5spmN3SEdAc3fw6QEUYyM8pKd5A2F9GDCVFl5Ui5jGHoAm8adha//eB+ zV0bV5RwZQIwopsUpYHfR9DQamtJp4/Z4tpJQsgHulqPSP7eFe0t7O+S5frO6zF0IOwd NpHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=l3Odh+JNDmsQYmHeSkVFsIRlQ6l6l6M82ZsDh0edm1U=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=X8hR1NV+rGvnls6qN7c6xSbmNPkGJyo8/wsQtzPGlQyAD1c1fR6KxH/wm619a3yHfM EIk6y5rH+jEQq/2fcgsAn3Mnv9ALE/aE4UkCgfbisG/i1yFxQ1Hm7Qs5r4+b89Cp1O3d iZScGqygs9b0CnfYGeQtuBx+rn1dLwHTo90alOAqR9vSb4bou8YR9fbTi7LXfhjGZQOY XkkPbK59TgaPLFshtxcYuVqwxfmoMr9vKInURg1GCMQHsNh8zD/b1IkjWU9fAWvkn5pb Jtx5po7qzCmv3NfGSIgXCrdn3HEjTKzpQuq9nndtKa6cS0ypsalOPYeRIWA8vsqNNm8Y dUCA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=qJ4hXYdu; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-521f38d416fsi8921193e87.246.2024.05.21.02.04.24; Tue, 21 May 2024 02:04:25 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=qJ4hXYdu; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9C62F68D373; Tue, 21 May 2024 12:03:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4CBB068D2D4 for ; Tue, 21 May 2024 12:03:38 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id CBC9627FFD841; Tue, 21 May 2024 11:03:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282217; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QkVKMHFA1KKxEuPDOv/lSbYjcdELBSSDQQNtIZmMVFQ=; b=qJ4hXYduvLMob0p4SFL9nF06XeFW8gTrVsJjLZ5E1twroYY5q6VPt3ndur85keMVU+6BHT NB4lFtkTcnPrI/9xZ7mK7NNChdBEKO63bpCGV77xX9YHybk/QuPHg563dVJDjxJ9tdeIrf NQmW/s27l95NHZK2FWXlKrzfvOoG6B07Q6reTUyKIgUoeYuEM2GfS/bcz6rvPVHSAZ3nu9 hdZtmJlhxRB/kkCmz4gxljSDt6joeEXcXzLGdK8Jsc4BmzuuMtVT4zMlWtXAJskr2UIQSu hLUEGDST71uJNCbeYMOUl74h9dDNGNJHXcxcOeqmveUo7V7HSfkSIkxFE1Y5Fw== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:12 +0200 Message-ID: <20240521090316.782-4-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 03/13] avformat/flvenc: add enhanced audio codecs X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Oy4ozGoVvEda --- libavformat/flv.h | 6 ++ libavformat/flvenc.c | 143 +++++++++++++++++++++++++++++++------------ 2 files changed, 111 insertions(+), 38 deletions(-) diff --git a/libavformat/flv.h b/libavformat/flv.h index 653c2bc82c..d030d576f3 100644 --- a/libavformat/flv.h +++ b/libavformat/flv.h @@ -103,6 +103,7 @@ enum { FLV_CODECID_NELLYMOSER = 6 << FLV_AUDIO_CODECID_OFFSET, FLV_CODECID_PCM_ALAW = 7 << FLV_AUDIO_CODECID_OFFSET, FLV_CODECID_PCM_MULAW = 8 << FLV_AUDIO_CODECID_OFFSET, + FLV_CODECID_EX_HEADER = 9 << FLV_AUDIO_CODECID_OFFSET, FLV_CODECID_AAC = 10<< FLV_AUDIO_CODECID_OFFSET, FLV_CODECID_SPEEX = 11<< FLV_AUDIO_CODECID_OFFSET, }; @@ -128,6 +129,11 @@ enum { PacketTypeMultitrack = 6, }; +enum { + AudioPacketTypeSequenceStart = 0, + AudioPacketTypeCodedFrames = 1, +}; + enum { MultitrackTypeOneTrack = 0x00, MultitrackTypeManyTracks = 0x10, diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c index 7ddc59e369..4e79b06499 100644 --- a/libavformat/flvenc.c +++ b/libavformat/flvenc.c @@ -68,6 +68,10 @@ static const AVCodecTag flv_audio_codec_ids[] = { { AV_CODEC_ID_PCM_MULAW, FLV_CODECID_PCM_MULAW >> FLV_AUDIO_CODECID_OFFSET }, { AV_CODEC_ID_PCM_ALAW, FLV_CODECID_PCM_ALAW >> FLV_AUDIO_CODECID_OFFSET }, { AV_CODEC_ID_SPEEX, FLV_CODECID_SPEEX >> FLV_AUDIO_CODECID_OFFSET }, + { AV_CODEC_ID_OPUS, MKBETAG('O', 'p', 'u', 's') }, + { AV_CODEC_ID_FLAC, MKBETAG('f', 'L', 'a', 'C') }, + { AV_CODEC_ID_AC3, MKBETAG('a', 'c', '-', '3') }, + { AV_CODEC_ID_EAC3, MKBETAG('e', 'c', '-', '3') }, { AV_CODEC_ID_NONE, 0 } }; @@ -138,6 +142,9 @@ static int get_audio_flags(AVFormatContext *s, AVCodecParameters *par) if (par->codec_id == AV_CODEC_ID_AAC) // specs force these parameters return FLV_CODECID_AAC | FLV_SAMPLERATE_44100HZ | FLV_SAMPLESSIZE_16BIT | FLV_STEREO; + if (par->codec_id == AV_CODEC_ID_OPUS || par->codec_id == AV_CODEC_ID_FLAC + || par->codec_id == AV_CODEC_ID_AC3 || par->codec_id == AV_CODEC_ID_EAC3) + return FLV_CODECID_EX_HEADER; // only needed for codec support check else if (par->codec_id == AV_CODEC_ID_SPEEX) { if (par->sample_rate != 16000) { av_log(s, AV_LOG_ERROR, @@ -634,6 +641,42 @@ static int unsupported_codec(AVFormatContext *s, return AVERROR(ENOSYS); } +static void flv_write_aac_header(AVFormatContext* s, AVCodecParameters* par) +{ + AVIOContext *pb = s->pb; + FLVContext *flv = s->priv_data; + + if (!par->extradata_size && (flv->flags & FLV_AAC_SEQ_HEADER_DETECT)) { + PutBitContext pbc; + int samplerate_index; + int channels = par->ch_layout.nb_channels + - (par->ch_layout.nb_channels == 8 ? 1 : 0); + uint8_t data[2]; + + for (samplerate_index = 0; samplerate_index < 16; + samplerate_index++) + if (par->sample_rate + == ff_mpeg4audio_sample_rates[samplerate_index]) + break; + + init_put_bits(&pbc, data, sizeof(data)); + put_bits(&pbc, 5, par->profile + 1); //profile + put_bits(&pbc, 4, samplerate_index); //sample rate index + put_bits(&pbc, 4, channels); + put_bits(&pbc, 1, 0); //frame length - 1024 samples + put_bits(&pbc, 1, 0); //does not depend on core coder + put_bits(&pbc, 1, 0); //is not extension + flush_put_bits(&pbc); + + avio_w8(pb, data[0]); + avio_w8(pb, data[1]); + + av_log(s, AV_LOG_WARNING, "AAC sequence header: %02x %02x.\n", + data[0], data[1]); + } + avio_write(pb, par->extradata, par->extradata_size); +} + static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts, int stream_index) { int64_t data_size; AVIOContext *pb = s->pb; @@ -641,7 +684,9 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i if (par->codec_id == AV_CODEC_ID_AAC || par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4 || par->codec_id == AV_CODEC_ID_HEVC - || par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9) { + || par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9 + || par->codec_id == AV_CODEC_ID_OPUS || par->codec_id == AV_CODEC_ID_FLAC + || par->codec_id == AV_CODEC_ID_AC3 || par->codec_id == AV_CODEC_ID_EAC3) { int64_t pos; avio_w8(pb, par->codec_type == AVMEDIA_TYPE_VIDEO ? @@ -650,39 +695,38 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i put_timestamp(pb, ts); avio_wb24(pb, 0); // streamid pos = avio_tell(pb); - if (par->codec_id == AV_CODEC_ID_AAC) { - avio_w8(pb, get_audio_flags(s, par)); - avio_w8(pb, 0); // AAC sequence header - - if (!par->extradata_size && (flv->flags & FLV_AAC_SEQ_HEADER_DETECT)) { - PutBitContext pbc; - int samplerate_index; - int channels = par->ch_layout.nb_channels - - (par->ch_layout.nb_channels == 8 ? 1 : 0); - uint8_t data[2]; - - for (samplerate_index = 0; samplerate_index < 16; - samplerate_index++) - if (par->sample_rate - == ff_mpeg4audio_sample_rates[samplerate_index]) - break; - - init_put_bits(&pbc, data, sizeof(data)); - put_bits(&pbc, 5, par->profile + 1); //profile - put_bits(&pbc, 4, samplerate_index); //sample rate index - put_bits(&pbc, 4, channels); - put_bits(&pbc, 1, 0); //frame length - 1024 samples - put_bits(&pbc, 1, 0); //does not depend on core coder - put_bits(&pbc, 1, 0); //is not extension - flush_put_bits(&pbc); - - avio_w8(pb, data[0]); - avio_w8(pb, data[1]); - - av_log(s, AV_LOG_WARNING, "AAC sequence header: %02x %02x.\n", - data[0], data[1]); + if (par->codec_type == AVMEDIA_TYPE_AUDIO) { + int extended_flv = par->codec_id == AV_CODEC_ID_OPUS + || par->codec_id == AV_CODEC_ID_FLAC + || par->codec_id == AV_CODEC_ID_AC3 + || par->codec_id == AV_CODEC_ID_EAC3; + + if (extended_flv) { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeSequenceStart); + + if (par->codec_id == AV_CODEC_ID_AAC) { + avio_write(pb, "mp4a", 4); + flv_write_aac_header(s, par); + } else if (par->codec_id == AV_CODEC_ID_OPUS) { + avio_write(pb, "Opus", 4); + av_assert0(par->extradata_size); + avio_write(pb, par->extradata, par->extradata_size); + } else if (par->codec_id == AV_CODEC_ID_FLAC) { + avio_write(pb, "fLaC", 4); + av_assert0(par->extradata_size); + avio_write(pb, par->extradata, par->extradata_size); + } else if (par->codec_id == AV_CODEC_ID_MP3) + avio_write(pb, ".mp3", 4); + else if (par->codec_id == AV_CODEC_ID_AC3) + avio_write(pb, "ac-3", 4); + else if (par->codec_id == AV_CODEC_ID_EAC3) + avio_write(pb, "ec-3", 4); + } else if (par->codec_id == AV_CODEC_ID_AAC) { + avio_w8(pb, get_audio_flags(s, par)); + avio_w8(pb, 0); // AAC sequence header + + flv_write_aac_header(s, par); } - avio_write(pb, par->extradata, par->extradata_size); } else { int track_idx = flv->video_track_idx_map[stream_index]; // If video stream has track_idx > 0 we need to send H.264 as extended video packet @@ -1018,13 +1062,20 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) int64_t cur_offset = avio_tell(pb); int track_idx = flv->video_track_idx_map[pkt->stream_index]; + int extended_audio = par->codec_id == AV_CODEC_ID_OPUS + || par->codec_id == AV_CODEC_ID_FLAC + || par->codec_id == AV_CODEC_ID_AC3 + || par->codec_id == AV_CODEC_ID_EAC3; + if (par->codec_type == AVMEDIA_TYPE_AUDIO && !pkt->size) { av_log(s, AV_LOG_WARNING, "Empty audio Packet\n"); return AVERROR(EINVAL); } - if (par->codec_id == AV_CODEC_ID_VP6F || par->codec_id == AV_CODEC_ID_VP6A || - par->codec_id == AV_CODEC_ID_VP6 || par->codec_id == AV_CODEC_ID_AAC) + if (extended_audio) + flags_size = 5; + else if (par->codec_id == AV_CODEC_ID_VP6F || par->codec_id == AV_CODEC_ID_VP6A || + par->codec_id == AV_CODEC_ID_VP6 || par->codec_id == AV_CODEC_ID_AAC) flags_size = 2; else if (par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4 || par->codec_id == AV_CODEC_ID_HEVC || par->codec_id == AV_CODEC_ID_AV1 || @@ -1043,7 +1094,8 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) if (par->codec_id == AV_CODEC_ID_AAC || par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4 || par->codec_id == AV_CODEC_ID_HEVC - || par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9) { + || par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9 + || par->codec_id == AV_CODEC_ID_OPUS || par->codec_id == AV_CODEC_ID_FLAC) { size_t side_size; uint8_t *side = av_packet_get_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, &side_size); if (side && side_size > 0 && (side_size != par->extradata_size || memcmp(side, par->extradata, side_size))) { @@ -1175,12 +1227,12 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) avio_seek(pb, data_size + 10 - 3, SEEK_CUR); avio_wb32(pb, data_size + 11); } else { - int extended_flv = (par->codec_id == AV_CODEC_ID_H264 && track_idx) + int extended_video = (par->codec_id == AV_CODEC_ID_H264 && track_idx) || par->codec_id == AV_CODEC_ID_HEVC || par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9; - if (extended_flv) { + if (extended_video) { int h2645 = par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_HEVC; int pkttype = PacketTypeCodedFrames; @@ -1208,6 +1260,21 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) avio_w8(pb, track_idx); if (h2645 && pkttype == PacketTypeCodedFrames) avio_wb24(pb, pkt->pts - pkt->dts); + } else if (extended_audio) { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeCodedFrames); + + if (par->codec_id == AV_CODEC_ID_AAC) + avio_write(pb, "mp4a", 4); + else if (par->codec_id == AV_CODEC_ID_OPUS) + avio_write(pb, "Opus", 4); + else if (par->codec_id == AV_CODEC_ID_FLAC) + avio_write(pb, "fLaC", 4); + else if (par->codec_id == AV_CODEC_ID_MP3) + avio_write(pb, ".mp3", 4); + else if (par->codec_id == AV_CODEC_ID_AC3) + avio_write(pb, "ac-3", 4); + else if (par->codec_id == AV_CODEC_ID_EAC3) + avio_write(pb, "ec-3", 4); } else { av_assert1(flags>=0); avio_w8(pb, flags); From patchwork Tue May 21 09:02:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49089 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097025pzb; Tue, 21 May 2024 02:04:35 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVLaZNsCc+XdarQ4kUFgW5OEis884niQyeoOF/8nq5TjnTf8HZjLAzE15zMLdPr84zeakErAHW1nw/zxd7Kaen2Nkne330Y6PoMDg== X-Google-Smtp-Source: AGHT+IHu256f/lhL5Xa2qSP649zsMw5uIhp3qzL+uwwxJuheU0YCDUyBP3uTdJhn2X4srnabjiGU X-Received: by 2002:a05:6512:110c:b0:523:41ec:125a with SMTP id 2adb3069b0e04-52341ec136cmr20338342e87.46.1716282275365; Tue, 21 May 2024 02:04:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282275; cv=none; d=google.com; s=arc-20160816; b=ed8KynUP2W5Tm63fOCRiOEmwJv45J0uOPfgUpvPbAGtWR6sRa5eowKlsSiwwe999Xd IjW3bO+Ey/d8UuBs8K1HANyZDHA/1YgeWeUd0fP9fsnqFVLWO5amQpQTX1WyMBfKbpMn eRz3NnVQuSrL9vG/e50ul287B/moxH6nVwSSb8yRY2aqmQXrHzMW6bHMe+jM7mdA7rlQ kBKsJbd1lFOd1WTb3fPhslR4OWo9Y9wqQ6k990k2aFVFj8sCe0ouST99n1Tdx2gbqwuu bOFfYR+irRNx6XXj8JxZ2GHEjxJM+f3SEYF/yg8+7Q5glGj7TRrb+0J5rw0luMxdr9Qx xANg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=gp1I8Mfzt6myyzMmurRfkq9JwgZoZO7SMG3Dr59Wl/Y=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=uFJL96fvbyrxzJN5yJ2n+90g1wYM8FgvXk1/oVGbXRAWUeaq9YYaPO8ZNtGblyQUY2 iqswkwr0zeHaW7dC39+6BeYS4OZIoahruetDFNwIUUPaJZDJbe9bxEAm9NBpPWDCbd4X fcS0U32OTcHwHmU87nAWPhKIHYRye46MN6QaDQr2fyAH9X8azBKQm+nF5hWtSLYeQCbx +Aj9tWwdoWbNdvhXRa7p+fWV1O276t/3TwFkaNUuzpxHAj+UFuYy5UKf2TytYf5/KASp +O4yxwdWgj5ZND0Fv2pN87saJxAIriyI6Y/SixsNTE436zCjIGQXLWZbtUp5cijBWiwm aE2A==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=UfWpYbE3; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5733c32c3f8si13391664a12.427.2024.05.21.02.04.34; Tue, 21 May 2024 02:04:35 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=UfWpYbE3; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9CC3968D3B3; Tue, 21 May 2024 12:03:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8114E68D2FE for ; Tue, 21 May 2024 12:03:38 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id 0AFA727FFD85E; Tue, 21 May 2024 11:03:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282218; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rnU1IqMThkpyF5ZexU7Ap8xowabnLxMuL5BA79sio0c=; b=UfWpYbE32MNopdxsLSjwDYjsGSnS/h5MmJmFgLjEpLEgDLrHQ3KG6lq1jsAMAKj6bgNjOk rXEjKID54wQMTkYIuatlVkcBEYYQhqfCriYsBAIxayjmjypC+KIh5GU7a9Ci9ecrCcJHm6 sXNxH4wmjNBSjllBGiFCjBjqOhpf/qkkvwzJ39KSZ8ZCkTzpCfZEhFbdgK5ZqqlQj7mCNq gccBwK4AUw5cUOenx7Vg+EiiJCkfiraVoNk+1ajqbR8RG9vHAUNc/LcN1svQsN06lh3MEW KsRdETC9Y+ct9Py4SVQMpCm0zOm2hza1Y6TrYKeZB+oaqGTXUygKGTWsXyAvdg== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:13 +0200 Message-ID: <20240521090316.782-5-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 04/13] avformat/flvenc: remove !size check for audio packets X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 0yMzdYVjlL5C --- libavformat/flvenc.c | 7 ------- 1 file changed, 7 deletions(-) diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c index 4e79b06499..13e06aedbb 100644 --- a/libavformat/flvenc.c +++ b/libavformat/flvenc.c @@ -1067,11 +1067,6 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) || par->codec_id == AV_CODEC_ID_AC3 || par->codec_id == AV_CODEC_ID_EAC3; - if (par->codec_type == AVMEDIA_TYPE_AUDIO && !pkt->size) { - av_log(s, AV_LOG_WARNING, "Empty audio Packet\n"); - return AVERROR(EINVAL); - } - if (extended_audio) flags_size = 5; else if (par->codec_id == AV_CODEC_ID_VP6F || par->codec_id == AV_CODEC_ID_VP6A || @@ -1146,8 +1141,6 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) case AVMEDIA_TYPE_AUDIO: flags = get_audio_flags(s, par); - av_assert0(size); - avio_w8(pb, FLV_TAG_TYPE_AUDIO); break; case AVMEDIA_TYPE_SUBTITLE: From patchwork Tue May 21 09:02:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49093 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097362pzb; Tue, 21 May 2024 02:05:19 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCW1Ds5bZTTKxVOv+Rt59bmEM5MZKT6eFsj9QRFVfwykq0q9e/PNBg2409rsrwnewuErMlvfMC2hccRrlDMvu/bVmP9dwGPxUZo9Ug== X-Google-Smtp-Source: AGHT+IHqdXjVN+RDR+9HLA2818qbrnWc/CdEHWwj8PYg1A0XqghGs5GEWTm7uNjiR3QpEP4E4pHF X-Received: by 2002:ac2:46ca:0:b0:518:95b6:176f with SMTP id 2adb3069b0e04-5220fe7a008mr18345142e87.50.1716282319687; Tue, 21 May 2024 02:05:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282319; cv=none; d=google.com; s=arc-20160816; b=EkCYwC7pLC4YGngAWZAAE5LW7Qc9MaV+PMWbwlVq/ZW8hfsMNSAHUMLF2iC2QZM69x 0bxHuQC40ujqrBRdOrlYD5tHJpgUEbINNfMYAR5Hpqc1oGO9sf4iRGos32ca0lAvr/TO Zm+OmWvUB63eUW6K2KRji7SvOGR8Oy/3UmB5K/xPcgWBA9owFatCwUr6gjGqjJIBX5Zh S+OkOQOAvcTy2AmajM31ETEmxhjdMN0jp5sYxxn1bnws2lrylzAL1XUEYeqHdcbXqUqx 3aV2xUV+0MXsBojG7vIOOKV+NJnyCE6WLwwPESdTNDrVT1wGJcV0ASPQ8oCJ/iS03pZy AgtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=GFYv6waLrlvxQLdHvGTdYn67PgR0rUzZntqxVv5nupY=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=Bk7pjRbU2SQ3bPu+ZjTtksv73h23hq9+4EwLsa0oGeeCoSZVZt0PdVPQ8WJV5Y2ULm syYJKGbxxb/ioRDlQ9P+NmIdoQJky8FdJNJDmEgbK4VrWyC6UltQ4rW1CgfLCL2bIK7U OvByTKqbZtmqJZiU+1492cvWP3XFYrPOwOhoA+LE6kfhUkrBJfaTL57/deBeYlb/NlUC LkeGdwuhzDe5IcP2UzmkxuRLVp6STkIV45v1ItCZzeoBdsIIE9o9roUI5/K+Rp7c3XAh mhVWAjZ8r1+aV3jf18KLB1P7f5AFNVwSSYPp8zeEUUq6j0JwRbJg6UtfZyU7tqjyXboh idgA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=Sw75+Co2; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-521f39d363dsi8446506e87.467.2024.05.21.02.05.19; Tue, 21 May 2024 02:05:19 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=Sw75+Co2; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5FF5868D396; Tue, 21 May 2024 12:03:54 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CB29068D33A for ; Tue, 21 May 2024 12:03:42 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id 3D64627FFD878; Tue, 21 May 2024 11:03:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282218; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/0c8sMaimKisyS6/by8+9OBxBCMItf4pglhmgeXekrk=; b=Sw75+Co2oADdphtvb9UYzTIgF3N4vt5LDUm/z9IMOECQ81zXjdqQZZ3vcl/9ZjtZD/ozkT sUTqhm4+EPGFdanuHk9kQhCMN6rJVdeQnCDJTcHzOBFHHtpCevtUAPPn5iD0oLhv516vvP njIxOI2z5TWQx3q2CleJOkrYpSMl480yVmYXye0qEU/D0JrUhdE6gm75yGno9bxS00Sul/ jQebGKPVpBEeHqnP53j4vlYpVdgcMJe0oTKJXX7gM1DZSzMoOxSz5ZUwTImcOERE0dtKYW JaMARxxGUii//ntHF4zhLe7hvxT1Atd2k1qa5yeQddA4h9dwHZ9h9QE6EY9orA== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:14 +0200 Message-ID: <20240521090316.782-6-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 05/13] avformat/flvdec: add enhanced audio codecs X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: GLKroROrVymY --- libavformat/flv.h | 8 +++ libavformat/flvdec.c | 119 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 116 insertions(+), 11 deletions(-) diff --git a/libavformat/flv.h b/libavformat/flv.h index d030d576f3..8926db6388 100644 --- a/libavformat/flv.h +++ b/libavformat/flv.h @@ -132,6 +132,14 @@ enum { enum { AudioPacketTypeSequenceStart = 0, AudioPacketTypeCodedFrames = 1, + AudioPacketTypeMultichannelConfig = 4, + AudioPacketTypeMultitrack = 5, +}; + +enum { + AudioChannelOrderUnspecified = 0, + AudioChannelOrderNative = 1, + AudioChannelOrderCustom = 2, }; enum { diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index 2445d1fd5e..ee2a1931b2 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -224,12 +224,33 @@ static AVStream *create_stream(AVFormatContext *s, int codec_type, int track_idx return st; } -static int flv_same_audio_codec(AVCodecParameters *apar, int flags) +static int flv_same_audio_codec(AVCodecParameters *apar, int flags, uint32_t codec_fourcc) { int bits_per_coded_sample = (flags & FLV_AUDIO_SAMPLESIZE_MASK) ? 16 : 8; int flv_codecid = flags & FLV_AUDIO_CODECID_MASK; int codec_id; + switch (codec_fourcc) { + case MKBETAG('m', 'p', '4', 'a'): + return apar->codec_id == AV_CODEC_ID_AAC; + case MKBETAG('O', 'p', 'u', 's'): + return apar->codec_id == AV_CODEC_ID_OPUS; + case MKBETAG('.', 'm', 'p', '3'): + return apar->codec_id == AV_CODEC_ID_MP3; + case MKBETAG('f', 'L', 'a', 'C'): + return apar->codec_id == AV_CODEC_ID_FLAC; + case MKBETAG('a', 'c', '-', '3'): + return apar->codec_id == AV_CODEC_ID_AC3; + case MKBETAG('e', 'c', '-', '3'): + return apar->codec_id == AV_CODEC_ID_EAC3; + case 0: + // Not enhanced flv, continue as normal. + break; + default: + // Unknown FOURCC + return 0; + } + if (!apar->codec_id && !apar->codec_tag) return 1; @@ -328,6 +349,24 @@ static void flv_set_audio_codec(AVFormatContext *s, AVStream *astream, apar->sample_rate = 8000; apar->codec_id = AV_CODEC_ID_PCM_ALAW; break; + case MKBETAG('m', 'p', '4', 'a'): + apar->codec_id = AV_CODEC_ID_AAC; + return; + case MKBETAG('O', 'p', 'u', 's'): + apar->codec_id = AV_CODEC_ID_OPUS; + return; + case MKBETAG('.', 'm', 'p', '3'): + apar->codec_id = AV_CODEC_ID_MP3; + return; + case MKBETAG('f', 'L', 'a', 'C'): + apar->codec_id = AV_CODEC_ID_FLAC; + return; + case MKBETAG('a', 'c', '-', '3'): + apar->codec_id = AV_CODEC_ID_AC3; + return; + case MKBETAG('e', 'c', '-', '3'): + apar->codec_id = AV_CODEC_ID_EAC3; + return; default: avpriv_request_sample(s, "Audio codec (%x)", flv_codecid >> FLV_AUDIO_CODECID_OFFSET); @@ -1247,7 +1286,7 @@ static int flv_read_packet(AVFormatContext *s, AVPacket *pkt) int multitrack = 0; int pkt_type = 0; uint8_t track_idx = 0; - uint32_t video_codec_id = 0; + uint32_t codec_id = 0; retry: /* pkt size is repeated at end. skip it */ @@ -1291,16 +1330,31 @@ retry: stream_type = FLV_STREAM_TYPE_AUDIO; flags = avio_r8(s->pb); size--; + + if ((flags & FLV_AUDIO_CODECID_MASK) == FLV_CODECID_EX_HEADER) { + enhanced_flv = 1; + pkt_type = flags & ~FLV_AUDIO_CODECID_MASK; + + channels = 0; + + if (pkt_type == AudioPacketTypeMultitrack) { + av_log(s, AV_LOG_ERROR, "Multitrack audio is unsupported!\n"); + return AVERROR_PATCHWELCOME; + } + + codec_id = avio_rb32(s->pb); + size -= 4; + } } else if (type == FLV_TAG_TYPE_VIDEO) { stream_type = FLV_STREAM_TYPE_VIDEO; flags = avio_r8(s->pb); - video_codec_id = flags & FLV_VIDEO_CODECID_MASK; + codec_id = flags & FLV_VIDEO_CODECID_MASK; /* * Reference Enhancing FLV 2023-03-v1.0.0-B.8 * https://github.com/veovera/enhanced-rtmp/blob/main/enhanced-rtmp-v1.pdf * */ enhanced_flv = (flags >> 7) & 1; - pkt_type = enhanced_flv ? video_codec_id : 0; + pkt_type = enhanced_flv ? codec_id : 0; size--; if (pkt_type == PacketTypeMultitrack) { @@ -1318,7 +1372,7 @@ retry: } if (enhanced_flv) { - video_codec_id = avio_rb32(s->pb); + codec_id = avio_rb32(s->pb); size -= 4; } if (multitrack) { @@ -1386,11 +1440,11 @@ skip: st = s->streams[i]; if (stream_type == FLV_STREAM_TYPE_AUDIO) { if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO && - (s->audio_codec_id || flv_same_audio_codec(st->codecpar, flags))) + (s->audio_codec_id || flv_same_audio_codec(st->codecpar, flags, codec_id))) break; } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO && - (s->video_codec_id || flv_same_video_codec(st->codecpar, video_codec_id)) && + (s->video_codec_id || flv_same_video_codec(st->codecpar, codec_id)) && st->id == track_idx) break; } else if (stream_type == FLV_STREAM_TYPE_SUBTITLE) { @@ -1459,7 +1513,7 @@ retry_duration: flv->searched_for_end = 1; } - if (stream_type == FLV_STREAM_TYPE_AUDIO) { + if (stream_type == FLV_STREAM_TYPE_AUDIO && !enhanced_flv) { int bits_per_coded_sample; channels = (flags & FLV_AUDIO_CHANNEL_MASK) == FLV_STEREO ? 2 : 1; sample_rate = 44100 << ((flags & FLV_AUDIO_SAMPLERATE_MASK) >> @@ -1491,8 +1545,48 @@ retry_duration: sample_rate = par->sample_rate; avcodec_parameters_free(&par); } + } else if (stream_type == FLV_STREAM_TYPE_AUDIO) { + if (!st->codecpar->codec_id) { + flv_set_audio_codec(s, st, st->codecpar, + codec_id ? codec_id : (flags & FLV_AUDIO_CODECID_MASK)); + flv->last_sample_rate = 0; + flv->last_channels = 0; + } + + // These are not signalled in the flags anymore + channels = 0; + sample_rate = 0; + + if (pkt_type == AudioPacketTypeMultichannelConfig) { + int channel_order = avio_r8(s->pb); + channels = avio_r8(s->pb); + size -= 2; + + if (channel_order == AudioChannelOrderCustom) { + for (i = 0; i < channels; i++) { + avio_r8(s->pb); // audio channel mapping + size--; + } + } else if (channel_order == AudioChannelOrderNative) { + avio_rb32(s->pb); // audio channel flags + size -= 4; + } + + if (!av_channel_layout_check(&st->codecpar->ch_layout)) { + ///TODO: This can be much more sophisticated with above info. + av_channel_layout_default(&st->codecpar->ch_layout, channels); + flv->last_channels = channels; + } + + if (channels != flv->last_channels) { + flv->last_channels = channels; + ff_add_param_change(pkt, channels, 0, 0, 0, 0); + } + + goto leave; + } } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { - int ret = flv_set_video_codec(s, st, video_codec_id, 1); + int ret = flv_set_video_codec(s, st, codec_id, 1); if (ret < 0) return ret; size -= ret; @@ -1503,13 +1597,15 @@ retry_duration: } if (st->codecpar->codec_id == AV_CODEC_ID_AAC || + st->codecpar->codec_id == AV_CODEC_ID_OPUS || + st->codecpar->codec_id == AV_CODEC_ID_FLAC || st->codecpar->codec_id == AV_CODEC_ID_H264 || st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || st->codecpar->codec_id == AV_CODEC_ID_HEVC || st->codecpar->codec_id == AV_CODEC_ID_AV1 || st->codecpar->codec_id == AV_CODEC_ID_VP9) { int type = 0; - if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO) { + if (enhanced_flv) { type = pkt_type; } else { type = avio_r8(s->pb); @@ -1545,6 +1641,7 @@ retry_duration: size -= 3; } if (type == 0 && (!st->codecpar->extradata || st->codecpar->codec_id == AV_CODEC_ID_AAC || + st->codecpar->codec_id == AV_CODEC_ID_OPUS || st->codecpar->codec_id == AV_CODEC_ID_FLAC || st->codecpar->codec_id == AV_CODEC_ID_H264 || st->codecpar->codec_id == AV_CODEC_ID_HEVC || st->codecpar->codec_id == AV_CODEC_ID_AV1 || st->codecpar->codec_id == AV_CODEC_ID_VP9)) { AVDictionaryEntry *t; @@ -1600,7 +1697,7 @@ retry_duration: flv->mt_extradata_sz[track_idx] = 0; } } - if (stream_type == FLV_STREAM_TYPE_AUDIO && + if (stream_type == FLV_STREAM_TYPE_AUDIO && !enhanced_flv && (sample_rate != flv->last_sample_rate || channels != flv->last_channels)) { flv->last_sample_rate = sample_rate; From patchwork Tue May 21 09:02:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49094 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097496pzb; Tue, 21 May 2024 02:05:36 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCW7NN/RVONvtg3ut5m+YAzSh5/9q0roDzHzw3TBjyB3UVJi2NurxdV0gycMrrbGCL2BpgsWimOXjspstAM3a8uvsbwWJb4pW+tw4Q== X-Google-Smtp-Source: AGHT+IGgyBP0r8QiYOt0B1yNMwHHx0gmW60N3G/ItEcP0cMo81Nt/kIFrIzyOFv1r23uJBx95PSl X-Received: by 2002:a05:6402:b89:b0:56f:e715:d1c7 with SMTP id 4fb4d7f45d1cf-5734d59d8b3mr19926496a12.1.1716282335901; Tue, 21 May 2024 02:05:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282335; cv=none; d=google.com; s=arc-20160816; b=KlD9wgpyGgKgUxwR8R4p9i6I636I2PqXAMyG9+RFdZzdQcjBwwp79lcFRIPBqxb0WX ebHV6FsPGUG4HrT4Xw6+36uabqSEiyDw5yoKmF9ejJ+Jj2kY3+LNTSELwhsbRFSpnzX2 wBOFdZORkZHhet4cni2ts74ywcOplsvh9FQ+ZGyLqZ8IP3tQGUGm0jBc318mWtYqaSCs RsglCQfltIx8dHADM3Y4maVHxLRyAKPh8JgSDAqP7sdn0Fxvj7vF7NgptFKg0oqVNxkS z3qYMdTi9+L+57aKs8WCNbKsAwiZY5ZeAu6T1zS29XpUoT75TLy2Qfs28Vn1T7I6LuFn Ykgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=+PaBZ4wvlbuNjtR/Yjv2SSFeF319TybKKKinpaa1yAY=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=hxg69g0Vy+THZS5z1iO2ZLwCkMecnn7uKJZ4hgYjr8XKaxqBT3TXsJtE4GH5m8yE0z sK/DT2nJOas3qPb7sx80BK+xF2fuWzWDFNOkogdnf3La/g8CM6mGtaE41tnx2Y4yBtsw 1EvzH8HEgk9GNqW3eqpro1uwy/qXW/e5342gTM2oRSJMQY3Jn+SOSmtOuGXxlx+99CjW caGJ8mnzr9SFLP7OcDv97GH8jleaqJlQOTAcb9/axJ7qIbeeNVsC04T6uFFp9mBUJuIk K2CTQiglQw8jshrxFk0/Dhh+G8IoujKm1Xy1QprIIFrxOGtT+tCRgY3xJ4h/veQbQKY7 lbSw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=onGA3KZH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-574ebc7960esi8359959a12.432.2024.05.21.02.05.29; Tue, 21 May 2024 02:05:35 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=onGA3KZH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 83F0A68D392; Tue, 21 May 2024 12:03:55 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D14DC68D37F for ; Tue, 21 May 2024 12:03:42 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id 705B227FFD7D1; Tue, 21 May 2024 11:03:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282218; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aUfbhxM6AUy77FCGK4lkFlb3PoLjOwJGGbfenUx2heg=; b=onGA3KZHLN7NlZ/Devp6BUVtCDA9VsOA++Bry/CmfWdXrP63XYQIyfkNGmARyL7DA90Zn0 19pOfM6D+GTTpCKjRYqUTvzIHPo7p0ekDy2ozyamqVgOiMhhHC9G/xme1dVqV/WG9K4uDy dG/cYUm9DNWeV9VayqYDk38t78WpCpmdnf89WvUjleVHh8LljQu89RgeazGNEwnYUcaJKO FeZk08e+AOjJ6KkRucyHi+cGUWNO8sf27ZGdYj8By8nytFvOeChdisWWhF0ID3sx0IvQ5T SizRJO4+RZx2sGBzSxV3VDrQY26Zonhsg7/A0/N7Xce6C3PH6OFnUEEG3zMVbQ== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:15 +0200 Message-ID: <20240521090316.782-7-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 06/13] avformat/flvenc: refactor fourcc writing X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 7ucg6VgXD9As --- libavformat/flvenc.c | 96 ++++++++++++++++++++++---------------------- 1 file changed, 47 insertions(+), 49 deletions(-) diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c index 13e06aedbb..94d633fbca 100644 --- a/libavformat/flvenc.c +++ b/libavformat/flvenc.c @@ -492,6 +492,45 @@ static void write_metadata(AVFormatContext *s, unsigned int ts) avio_wb32(pb, flv->metadata_totalsize + 11); } +static void write_codec_fourcc(AVIOContext *pb, enum AVCodecID codec_id) +{ + switch (codec_id) { + case AV_CODEC_ID_AAC: + avio_write(pb, "mp4a", 4); + return; + case AV_CODEC_ID_OPUS: + avio_write(pb, "Opus", 4); + return; + case AV_CODEC_ID_FLAC: + avio_write(pb, "fLaC", 4); + return; + case AV_CODEC_ID_MP3: + avio_write(pb, ".mp3", 4); + return; + case AV_CODEC_ID_AC3: + avio_write(pb, "ac-3", 4); + return; + case AV_CODEC_ID_EAC3: + avio_write(pb, "ec-3", 4); + return; + case AV_CODEC_ID_H264: + avio_write(pb, "avc1", 4); + return; + case AV_CODEC_ID_HEVC: + avio_write(pb, "hvc1", 4); + return; + case AV_CODEC_ID_AV1: + avio_write(pb, "av01", 4); + return; + case AV_CODEC_ID_VP9: + avio_write(pb, "vp09", 4); + return; + default: + av_log(NULL, AV_LOG_ERROR, "Invalid codec FourCC write requested.\n"); + av_assert0(0); + } +} + static void flv_write_metadata_packet(AVFormatContext *s, AVCodecParameters *par, unsigned int ts, int stream_idx) { AVIOContext *pb = s->pb; @@ -528,13 +567,8 @@ static void flv_write_metadata_packet(AVFormatContext *s, AVCodecParameters *par put_timestamp(pb, ts); //ts = pkt->dts, gen avio_wb24(pb, flv->reserved); - if (par->codec_id == AV_CODEC_ID_HEVC) { - avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeMetadata| FLV_FRAME_VIDEO_INFO_CMD); // ExVideoTagHeader mode with PacketTypeMetadata - avio_write(pb, "hvc1", 4); - } else if (par->codec_id == AV_CODEC_ID_AV1 || par->codec_id == AV_CODEC_ID_VP9) { - avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeMetadata| FLV_FRAME_VIDEO_INFO_CMD); - avio_write(pb, par->codec_id == AV_CODEC_ID_AV1 ? "av01" : "vp09", 4); - } + avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeMetadata | FLV_FRAME_VIDEO_INFO_CMD); // ExVideoTagHeader mode with PacketTypeMetadata + write_codec_fourcc(pb, par->codec_id); avio_w8(pb, AMF_DATA_TYPE_STRING); put_amf_string(pb, "colorInfo"); @@ -703,24 +737,14 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i if (extended_flv) { avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeSequenceStart); + write_codec_fourcc(pb, par->codec_id); if (par->codec_id == AV_CODEC_ID_AAC) { - avio_write(pb, "mp4a", 4); flv_write_aac_header(s, par); - } else if (par->codec_id == AV_CODEC_ID_OPUS) { - avio_write(pb, "Opus", 4); + } else if (par->codec_id == AV_CODEC_ID_OPUS || par->codec_id == AV_CODEC_ID_FLAC) { av_assert0(par->extradata_size); avio_write(pb, par->extradata, par->extradata_size); - } else if (par->codec_id == AV_CODEC_ID_FLAC) { - avio_write(pb, "fLaC", 4); - av_assert0(par->extradata_size); - avio_write(pb, par->extradata, par->extradata_size); - } else if (par->codec_id == AV_CODEC_ID_MP3) - avio_write(pb, ".mp3", 4); - else if (par->codec_id == AV_CODEC_ID_AC3) - avio_write(pb, "ac-3", 4); - else if (par->codec_id == AV_CODEC_ID_EAC3) - avio_write(pb, "ec-3", 4); + } } else if (par->codec_id == AV_CODEC_ID_AAC) { avio_w8(pb, get_audio_flags(s, par)); avio_w8(pb, 0); // AAC sequence header @@ -743,14 +767,7 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeSequenceStart | FLV_FRAME_KEY); } - if (par->codec_id == AV_CODEC_ID_H264) - avio_write(pb, "avc1", 4); - else if (par->codec_id == AV_CODEC_ID_HEVC) - avio_write(pb, "hvc1", 4); - else if (par->codec_id == AV_CODEC_ID_AV1) - avio_write(pb, "av01", 4); - else if (par->codec_id == AV_CODEC_ID_VP9) - avio_write(pb, "vp09", 4); + write_codec_fourcc(pb, par->codec_id); if (track_idx) avio_w8(pb, track_idx); @@ -1240,14 +1257,7 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) avio_w8(pb, FLV_IS_EX_HEADER | pkttype | frametype); } - if (par->codec_id == AV_CODEC_ID_H264) - avio_write(pb, "avc1", 4); - else if (par->codec_id == AV_CODEC_ID_HEVC) - avio_write(pb, "hvc1", 4); - else if (par->codec_id == AV_CODEC_ID_AV1) - avio_write(pb, "av01", 4); - else if (par->codec_id == AV_CODEC_ID_VP9) - avio_write(pb, "vp09", 4); + write_codec_fourcc(pb, par->codec_id); if (track_idx) avio_w8(pb, track_idx); @@ -1255,19 +1265,7 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) avio_wb24(pb, pkt->pts - pkt->dts); } else if (extended_audio) { avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeCodedFrames); - - if (par->codec_id == AV_CODEC_ID_AAC) - avio_write(pb, "mp4a", 4); - else if (par->codec_id == AV_CODEC_ID_OPUS) - avio_write(pb, "Opus", 4); - else if (par->codec_id == AV_CODEC_ID_FLAC) - avio_write(pb, "fLaC", 4); - else if (par->codec_id == AV_CODEC_ID_MP3) - avio_write(pb, ".mp3", 4); - else if (par->codec_id == AV_CODEC_ID_AC3) - avio_write(pb, "ac-3", 4); - else if (par->codec_id == AV_CODEC_ID_EAC3) - avio_write(pb, "ec-3", 4); + write_codec_fourcc(pb, par->codec_id); } else { av_assert1(flags>=0); avio_w8(pb, flags); From patchwork Tue May 21 09:02:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49085 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097551pzb; Tue, 21 May 2024 02:05:42 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUWYpnWeXBlwd5kc93UkryKsAf0Pw5XE4P6eltLWfYoMOq6MfZI/3wk5Y9WjExBhJSsfG2A5mwIgQ0yTRP0Hic35hoIWqyqRb3oMg== X-Google-Smtp-Source: AGHT+IGxAGA8Gg440s3O34+ew9z/jG7xRgEpzrW6TQlSbnRM10SGyplkwP31LwlwbF5zyfEFKb0h X-Received: by 2002:a50:a40c:0:b0:56d:c928:ad76 with SMTP id 4fb4d7f45d1cf-5734d67e8d0mr21019727a12.26.1716282341971; Tue, 21 May 2024 02:05:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282341; cv=none; d=google.com; s=arc-20160816; b=PC3nG84uz1z86rJA1FgQhiSydZhH61dkcJJN9+6Klh0xTTlS/BSl8uLeq8oynsLM37 hg/fYYHYCMTYxqCryw1/i6nsCm1cPhZF91z60oOI+tm5RmcSJklkO+MO2htZHYw0tlqn icupTh3hdTy48XFV9ThPzjR9d9tBBjRCaNMkjIZz0oHQQDVo7KGSV/YDrCA0fu9G/0Rn JBvbcUbfcmfvwPRq/Btv2LvTjAe7Sr4FxP9tAciMhnuKDP71o2ln/bT+ieb1H/5liJ9S 8BPdMqPZ5h9W+cj41CJtipN1z95Vab6lRAx4kkJpLueP+HD2ERd4cEEb4//R1UJQsQ21 YCcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=1KHRd8u8QQZhg6bLQnQnsM9mzlac56If9TRfTqhU7pk=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=ubMA0sR69g46UfV8UHWNGf2PtDmjbyEQzxrGIxw7OXXMi5+Ak0H6p3PBVsyx1ge23v /P7s98piBRNLtpq9wDf8Mtah9iTxRPBxLJD8LX9kxp7nwb8aU7C8Wh8WowwlRQdsBjJB kICjRuaZRoR+go3B+20uhFhr2LdYjXpDbs1bp16INkqZHzXhc9h8pxK7fJ8FpLoWtyYU rRBMIhpN14YtrbcRtysk3FcuAg2dKwONri6sHYNlOe6QQOkbUCzxbfiuqOIVwbtjyLKY F092AJfaKnAA3P4KZPrmGHk1P6Wk5NT3LEvEw70FsslOy5m60GnlYUzR4OT/LvzfBD1v KSIg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=QV1NkIuw; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5733c3787c5si13518597a12.656.2024.05.21.02.05.41; Tue, 21 May 2024 02:05:41 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=QV1NkIuw; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EE41C68D415; Tue, 21 May 2024 12:03:56 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4017968D37F for ; Tue, 21 May 2024 12:03:43 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id A334427FFD7D2; Tue, 21 May 2024 11:03:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282218; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1MAb2I2Ks68z+wPkLaZvy4LLtmERKeBhHCzRfj7ZM0c=; b=QV1NkIuwDVboRf5dBn5IWkhlTKfCOw5OoO6QTsfuyFVcQRsW6Zlis5w46Oh1kyuLmDIhsY hxhUp3hH5LRsapnQNJus0XJ32/5jAT/gsk+dYdQhSXGoTtNVAEdBF49CI+rxwUkIT5bAK6 2MkqCAixCS3sIwNcZTjQii/accyFj0o/IgvcZbEgOuOkn0eMx6LeFYnBCLh2AuArVYGbmg 4K0hmoHfh7UmAJw2X4GT5Hj+/U5WIdGRHfxrR1kaJ5WZDYOYN7H93dsWWpa+XWu4zNJRUI FvDkr8Qsjs37zxkGOjDkRIUkxNAk1h/0VYS/gtVsVeMJHzBzvUYNnrP2GGsS3w== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:16 +0200 Message-ID: <20240521090316.782-8-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 07/13] avformat/flvenc: write enhanced rtmp multichannel info for audio with more than two channels X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ifMTWmpz4W6Z --- libavformat/flvenc.c | 93 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 85 insertions(+), 8 deletions(-) diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c index 94d633fbca..056940afc5 100644 --- a/libavformat/flvenc.c +++ b/libavformat/flvenc.c @@ -711,10 +711,82 @@ static void flv_write_aac_header(AVFormatContext* s, AVCodecParameters* par) avio_write(pb, par->extradata, par->extradata_size); } +static void flv_write_multichannel_body(AVFormatContext* s, AVCodecParameters* par) +{ + AVIOContext *pb = s->pb; + + switch (par->ch_layout.order) { + case AV_CHANNEL_ORDER_NATIVE: + avio_w8(pb, AudioChannelOrderNative); + break; + case AV_CHANNEL_ORDER_CUSTOM: + avio_w8(pb, AudioChannelOrderCustom); + break; + default: + avio_w8(pb, AudioChannelOrderUnspecified); + break; + } + + avio_w8(pb, par->ch_layout.nb_channels); + + if (par->ch_layout.order == AV_CHANNEL_ORDER_NATIVE) { + // The first 18 entries are identical between FFmpeg and flv + uint32_t mask = par->ch_layout.u.mask & 0x03FFFF; + // The remaining 6 flv entries are in the right order, but start at AV_CHAN_LOW_FREQUENCY_2 + mask |= (par->ch_layout.u.mask >> (AV_CHAN_LOW_FREQUENCY_2 - 18)) & 0xFC0000; + + avio_wb32(pb, mask); + } else if (par->ch_layout.order == AV_CHANNEL_ORDER_CUSTOM) { + for (int i = 0; i < par->ch_layout.nb_channels; i++) { + enum AVChannel id = par->ch_layout.u.map[i].id; + if (id >= AV_CHAN_FRONT_LEFT && id <= AV_CHAN_TOP_BACK_RIGHT) { + avio_w8(pb, id - AV_CHAN_FRONT_LEFT + 0); + } else if (id >= AV_CHAN_LOW_FREQUENCY_2 && id <= AV_CHAN_BOTTOM_FRONT_RIGHT) { + avio_w8(pb, id - AV_CHAN_LOW_FREQUENCY_2 + 18); + } else if (id == AV_CHAN_UNUSED) { + avio_w8(pb, 0xFE); + } else { + avio_w8(pb, 0xFF); // unknown + } + } + } +} + +static int flv_get_multichannel_body_size(AVCodecParameters* par) +{ + int res = 2; + + if (par->ch_layout.order == AV_CHANNEL_ORDER_NATIVE) + res += 4; + else if (par->ch_layout.order == AV_CHANNEL_ORDER_CUSTOM) + res += par->ch_layout.nb_channels; + + return res; +} + +static void flv_write_multichannel_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts) +{ + AVIOContext *pb = s->pb; + int data_size = flv_get_multichannel_body_size(par); + + avio_w8(pb, FLV_TAG_TYPE_AUDIO); + avio_wb24(pb, data_size + 5); // size + put_timestamp(pb, ts); + avio_wb24(pb, 0); // streamid + + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultichannelConfig); + write_codec_fourcc(pb, par->codec_id); + + flv_write_multichannel_body(s, par); + + avio_wb32(pb, data_size + 5 + 11); // previous tag size +} + static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts, int stream_index) { int64_t data_size; AVIOContext *pb = s->pb; FLVContext *flv = s->priv_data; + int extended_flv = 0; if (par->codec_id == AV_CODEC_ID_AAC || par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4 || par->codec_id == AV_CODEC_ID_HEVC @@ -730,10 +802,10 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i avio_wb24(pb, 0); // streamid pos = avio_tell(pb); if (par->codec_type == AVMEDIA_TYPE_AUDIO) { - int extended_flv = par->codec_id == AV_CODEC_ID_OPUS - || par->codec_id == AV_CODEC_ID_FLAC - || par->codec_id == AV_CODEC_ID_AC3 - || par->codec_id == AV_CODEC_ID_EAC3; + extended_flv = par->codec_id == AV_CODEC_ID_OPUS + || par->codec_id == AV_CODEC_ID_FLAC + || par->codec_id == AV_CODEC_ID_AC3 + || par->codec_id == AV_CODEC_ID_EAC3; if (extended_flv) { avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeSequenceStart); @@ -754,10 +826,10 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i } else { int track_idx = flv->video_track_idx_map[stream_index]; // If video stream has track_idx > 0 we need to send H.264 as extended video packet - int extended_flv = (par->codec_id == AV_CODEC_ID_H264 && track_idx) - || par->codec_id == AV_CODEC_ID_HEVC - || par->codec_id == AV_CODEC_ID_AV1 - || par->codec_id == AV_CODEC_ID_VP9; + extended_flv = (par->codec_id == AV_CODEC_ID_H264 && track_idx) + || par->codec_id == AV_CODEC_ID_HEVC + || par->codec_id == AV_CODEC_ID_AV1 + || par->codec_id == AV_CODEC_ID_VP9; if (extended_flv) { if (track_idx) { @@ -792,6 +864,11 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i avio_skip(pb, data_size + 10 - 3); avio_wb32(pb, data_size + 11); // previous tag size } + + if (par->codec_type == AVMEDIA_TYPE_AUDIO && (extended_flv || + (av_channel_layout_compare(&par->ch_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO) == 1 && + av_channel_layout_compare(&par->ch_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_MONO) == 1))) + flv_write_multichannel_header(s, par, ts); } static int flv_append_keyframe_info(AVFormatContext *s, FLVContext *flv, double ts, int64_t pos) From patchwork Tue May 21 09:02:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49095 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097639pzb; Tue, 21 May 2024 02:05:53 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWgQa8aODLIBxekryTboNqnmdTB7Eb9wgvc0sdNLHYzNYgT7eBskuxeUIwzg29BK+dIGbZvzP8w+hkwp4DosrT6yxu2FrIrjQzWEQ== X-Google-Smtp-Source: AGHT+IHLvasxJAqWDeJRIVRApwj7ry/JZyQRfeg6CzFBWizVujvzVy6X2FkpoW5zz8WLp49O++tr X-Received: by 2002:a2e:9e14:0:b0:2e1:c97b:6f1f with SMTP id 38308e7fff4ca-2e5203a795emr202055491fa.51.1716282353170; Tue, 21 May 2024 02:05:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282353; cv=none; d=google.com; s=arc-20160816; b=ko9cI0Q3F37748l3MbPRTXH2lCoJzUcB0hk3jd4lhdEI5e0H8trENWH7x57T2Lsuot eMWS3aaxMVu7QXSq03nG3rPnG4X0AZ9lHTXsSao53Wc5/ggPKkdiytu+iSjBPC8hWszp iSxmkOEpP6iTcC8WLyA0WgzITI//G5g1cc8taxE+Oreu/GkgoU+hpctQa5ronRCSMaRS ePWwenK4lhMHNSJ/MxHXaxPBCxavmGlsF/KKX3e1Kuw9VfMpj5/mWqfQhE0CJsbJgrko oobVRo7ZtKK1G2O3rrwgjbHwJxPIAq59Wlya7T8o6pkUtOB7kvcyBMeoxqwbfTlMDSM9 oSJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=N7XW3DzJEH1NveE7KDz+f1m+YWuP+EPM4PzgtmunYkQ=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=vhTlJtYNLdK5GS6+91xpTOZMQke0Jo1cBWiHVc47VTzpw9IkROz5E2EqBgH83QDxeL xyeNYFbkgtCpNuxRp0CDTbknxHlNN2FK7tvxWDU/K4bFdWpSqBJ1r3HKuzSzpeVZOA+u 7epQXJdTvR1ysJi98z9QmRNPm4h7fA0spbWzd76IFNrQheO/5kgpojondD08RkZBpPuE GUeIF2ZpNOgJ6DaalGB9wLnYgmW00+F3qhzUGG2Q7J9vzQ9bH97I/EKzmt6FH7qCFJRA FUCYbGo3ff6bAfLUHzH1QjGgM1uFiOgmKKtofWivt6bW5ttt4zELCmdbE/KyyTJIHpOT 9lOQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=TMBPAs9B; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 38308e7fff4ca-2e4d0ce30c6si77288201fa.157.2024.05.21.02.05.52; Tue, 21 May 2024 02:05:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=TMBPAs9B; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 280E268D294; Tue, 21 May 2024 12:03:58 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 472FD68D392 for ; Tue, 21 May 2024 12:03:43 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id D754427FFD298; Tue, 21 May 2024 11:03:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282218; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U3P6IKRdJP8K+iN/gEUIVGT/h7SBYq7YFLCQrmFiMhg=; b=TMBPAs9BbnenoyUFYgnvzl7+fHm/dQ+hd6cY9PGnbp3ghwRNFRAnCdFwYIQTJNVhrhAhQ/ aN8QojK18fUwiG7j5D0zZhqo3ohu2Iz4R17o0F9RMB8Te0cWLXB+d0P0GrqbRCZugo7eHC icifcTodkyyWawyix70H7x4hU8+4y1hP5zvKbA/jGD0LyAxv4chzs3X7ck/2hIMQcsWUpM 10jIxh69NBJXDQTLFTWFjYDh4NwHWc+eAZzdJOBGG/spCNg0Jd60W3bQoYNUFviokGlBG9 a/kh+mUckb6Sm6u4t3db1HSv7I2JzDUpuvBB0JiOFvpU64bfttJXl4ONg1Lw4A== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:17 +0200 Message-ID: <20240521090316.782-9-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 08/13] avformat/flvdec: parse enhanced rtmp multichannel info X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: BmSrXJx2Tywe --- libavformat/flvdec.c | 41 +++++++++++++++++++++++++---------------- 1 file changed, 25 insertions(+), 16 deletions(-) diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index ee2a1931b2..890958351a 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -1335,8 +1335,6 @@ retry: enhanced_flv = 1; pkt_type = flags & ~FLV_AUDIO_CODECID_MASK; - channels = 0; - if (pkt_type == AudioPacketTypeMultitrack) { av_log(s, AV_LOG_ERROR, "Multitrack audio is unsupported!\n"); return AVERROR_PATCHWELCOME; @@ -1546,12 +1544,9 @@ retry_duration: avcodec_parameters_free(&par); } } else if (stream_type == FLV_STREAM_TYPE_AUDIO) { - if (!st->codecpar->codec_id) { + if (!st->codecpar->codec_id) flv_set_audio_codec(s, st, st->codecpar, codec_id ? codec_id : (flags & FLV_AUDIO_CODECID_MASK)); - flv->last_sample_rate = 0; - flv->last_channels = 0; - } // These are not signalled in the flags anymore channels = 0; @@ -1562,26 +1557,40 @@ retry_duration: channels = avio_r8(s->pb); size -= 2; + av_channel_layout_uninit(&st->codecpar->ch_layout); + if (channel_order == AudioChannelOrderCustom) { + ret = av_channel_layout_custom_init(&st->codecpar->ch_layout, channels); + if (ret < 0) + return ret; + for (i = 0; i < channels; i++) { - avio_r8(s->pb); // audio channel mapping + uint8_t id = avio_r8(s->pb); size--; + + if (id < 18) + st->codecpar->ch_layout.u.map[i].id = id; + else if (id >= 18 && id <= 23) + st->codecpar->ch_layout.u.map[i].id = id - 18 + AV_CHAN_LOW_FREQUENCY_2; + else if (id == 0xFE) + st->codecpar->ch_layout.u.map[i].id = AV_CHAN_UNUSED; + else + st->codecpar->ch_layout.u.map[i].id = AV_CHAN_UNKNOWN; } } else if (channel_order == AudioChannelOrderNative) { - avio_rb32(s->pb); // audio channel flags + uint64_t mask = avio_rb32(s->pb); size -= 4; - } - if (!av_channel_layout_check(&st->codecpar->ch_layout)) { - ///TODO: This can be much more sophisticated with above info. + // The first 18 entries in the mask match ours, but the remaining 6 entries start at AV_CHAN_LOW_FREQUENCY_2 + mask = (mask & 0x3FFFF) | ((mask & 0xFC0000) << (AV_CHAN_LOW_FREQUENCY_2 - 18)); + ret = av_channel_layout_from_mask(&st->codecpar->ch_layout, mask); + if (ret < 0) + return ret; + } else { av_channel_layout_default(&st->codecpar->ch_layout, channels); - flv->last_channels = channels; } - if (channels != flv->last_channels) { - flv->last_channels = channels; - ff_add_param_change(pkt, channels, 0, 0, 0, 0); - } + av_log(s, AV_LOG_DEBUG, "Set channel data from MultiChannel info.\n"); goto leave; } From patchwork Tue May 21 09:02:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49096 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097713pzb; Tue, 21 May 2024 02:06:03 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWtyOIFvnn2xBCcy0X0VQ5ITp95N5eNnLG5cFzotJEtI/1wz4naDO+O08vgfdCXeMwMqHzCScVWHrWvaK36vZgETjyrG+B21CHYJw== X-Google-Smtp-Source: AGHT+IFfFHmZlwcOh4Cchwx8y3weMDv9cIj4Z7pSb6/+Kza0AYIpO3wUlUk/eFwyJidbidI4rHl7 X-Received: by 2002:ac2:52b0:0:b0:51f:9549:9c0d with SMTP id 2adb3069b0e04-52210278795mr27170598e87.48.1716282363133; Tue, 21 May 2024 02:06:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282363; cv=none; d=google.com; s=arc-20160816; b=QgPxfO29e2dMuRAnfD5CkL7FaCYrkft3uLRJ1C69UYt1cL9TGCW9lTbrhFAPi7iu7/ x3R5OVUCE2wXfNsxwksWIuYElcUbWa+srJohZrhBo0y1UWD75ewFmqk8dCZZJSPiO6Ap +d4YAtCy22ROZopfG23y6lTjSoxMTZrtU5EgRS3dni4H6QiLWRSrlqPekKpAEO7PcsLB K1+SHNSPFJGWPY2v3Ra7g9Mi+2KRMAHuPoiRSi5yakI17MHox9ilBkaClX+40Wk06y+B r36SITLzbEWNfRSrDWdtoem7fcmybgWmMmuJyTcmvnu+dcCSU3qHeeu0Ummk/qBfWb8c 178w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=B/ynEYiWdRzM4ZVa1EM8OE9Oquz2H1bUC6xGVzfMr9s=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=G0SHlXs1yxXjaOWf2Ja0/3W0dzSEybv92vTj4FGTnYTzRmd1/hBctnkJwdtJPFV14A bjVbH3rfx5tTvw4IRH8zY5Jelm3OXmWh7bmLAIbBlfYXitxsVwwlek16XYGqI1iJ3M4T id4d6FqVk+G7f6t/G8IB55KL8UaKvUfr3ADdXpq5BQCPWz9f1ANFGxdWWvlee6hLFDI4 8FjcC8Ka9yodq7siZfDXZI0RR2+/xwqJx09mUX8/4e1Jt8VMe6Cim7wbOMdhfsgru/MB RtLpfir6bsAQBwyh0zvlQTLzWLt/Oa1yjsa9OWiRnj8SM78ykTJlVy/tCsPLllDpOgfx 3hTg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=kbefL6JI; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a640c23a62f3a-a5d1079016bsi447271966b.591.2024.05.21.02.06.02; Tue, 21 May 2024 02:06:03 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=kbefL6JI; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 29F1A68D428; Tue, 21 May 2024 12:03:59 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6BC0E68D3A1 for ; Tue, 21 May 2024 12:03:43 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id 151D827FFD299; Tue, 21 May 2024 11:03:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282219; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IA/sOPRBNpG3crCFlmE4JrSaO9nKC0pACnxE9/XsPo4=; b=kbefL6JIcaslWJQK4c3Q4fwTTBFYCceEtJHqaAipQ1Hsgqzu1NzbrOYmltroi5f4muQDbh N4eiLWNDvicagCMXRs2OGQugM61rPVm9MpODZDI8c75Gh09yvie7LruLU/E5Y332hfAAVo MhcIdHyTXjJy2tKsN6yE9B7EYZjqI4MSGsHA74D42L9JC1ldpuJ2zxyIIFtJGAYHwEfmNr mPJNGcMJ50TbePknKXBbQrkrwnnOjIN7PwUfcmnG4pJtdZ5k5vagY3mRoeWdCctynR6U4w QkfrPfmAcBlv4TZJr1Cq8rSgljsvpSGlFXB8mvu4Zu0bxPzmlC2aXi1jt+h4DA== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:18 +0200 Message-ID: <20240521090316.782-10-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 09/13] avformat/flvenc: add support for writing multi track audio X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: jw2bCbnr173L --- libavformat/flvenc.c | 90 +++++++++++++++++++++++++++++++++----------- 1 file changed, 67 insertions(+), 23 deletions(-) diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c index 056940afc5..ab729702a5 100644 --- a/libavformat/flvenc.c +++ b/libavformat/flvenc.c @@ -131,7 +131,7 @@ typedef struct FLVContext { int flags; int64_t *last_ts; int *metadata_pkt_written; - int *video_track_idx_map; + int *track_idx_map; } FLVContext; static int get_audio_flags(AVFormatContext *s, AVCodecParameters *par) @@ -764,19 +764,33 @@ static int flv_get_multichannel_body_size(AVCodecParameters* par) return res; } -static void flv_write_multichannel_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts) +static void flv_write_multichannel_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts, int stream_index) { AVIOContext *pb = s->pb; + FLVContext *flv = s->priv_data; + + int track_idx = flv->track_idx_map[stream_index]; int data_size = flv_get_multichannel_body_size(par); + if (track_idx) + data_size += 2; avio_w8(pb, FLV_TAG_TYPE_AUDIO); avio_wb24(pb, data_size + 5); // size put_timestamp(pb, ts); avio_wb24(pb, 0); // streamid - avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultichannelConfig); + if (track_idx) { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultitrack); + avio_w8(pb, MultitrackTypeOneTrack | AudioPacketTypeMultichannelConfig); + } else { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultichannelConfig); + } + write_codec_fourcc(pb, par->codec_id); + if (track_idx) + avio_w8(pb, track_idx); + flv_write_multichannel_body(s, par); avio_wb32(pb, data_size + 5 + 11); // previous tag size @@ -786,6 +800,7 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i int64_t data_size; AVIOContext *pb = s->pb; FLVContext *flv = s->priv_data; + int track_idx = flv->track_idx_map[stream_index]; int extended_flv = 0; if (par->codec_id == AV_CODEC_ID_AAC || par->codec_id == AV_CODEC_ID_H264 @@ -802,15 +817,26 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i avio_wb24(pb, 0); // streamid pos = avio_tell(pb); if (par->codec_type == AVMEDIA_TYPE_AUDIO) { - extended_flv = par->codec_id == AV_CODEC_ID_OPUS - || par->codec_id == AV_CODEC_ID_FLAC - || par->codec_id == AV_CODEC_ID_AC3 - || par->codec_id == AV_CODEC_ID_EAC3; + extended_flv = (par->codec_id == AV_CODEC_ID_AAC && track_idx) + || (par->codec_id == AV_CODEC_ID_MP3 && track_idx) + || par->codec_id == AV_CODEC_ID_OPUS + || par->codec_id == AV_CODEC_ID_FLAC + || par->codec_id == AV_CODEC_ID_AC3 + || par->codec_id == AV_CODEC_ID_EAC3; if (extended_flv) { - avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeSequenceStart); + if (track_idx) { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultitrack); + avio_w8(pb, MultitrackTypeOneTrack | AudioPacketTypeSequenceStart); + } else { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeSequenceStart); + } + write_codec_fourcc(pb, par->codec_id); + if (track_idx) + avio_w8(pb, track_idx); + if (par->codec_id == AV_CODEC_ID_AAC) { flv_write_aac_header(s, par); } else if (par->codec_id == AV_CODEC_ID_OPUS || par->codec_id == AV_CODEC_ID_FLAC) { @@ -824,7 +850,6 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i flv_write_aac_header(s, par); } } else { - int track_idx = flv->video_track_idx_map[stream_index]; // If video stream has track_idx > 0 we need to send H.264 as extended video packet extended_flv = (par->codec_id == AV_CODEC_ID_H264 && track_idx) || par->codec_id == AV_CODEC_ID_HEVC @@ -868,7 +893,7 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i if (par->codec_type == AVMEDIA_TYPE_AUDIO && (extended_flv || (av_channel_layout_compare(&par->ch_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO) == 1 && av_channel_layout_compare(&par->ch_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_MONO) == 1))) - flv_write_multichannel_header(s, par, ts); + flv_write_multichannel_header(s, par, ts, stream_index); } static int flv_append_keyframe_info(AVFormatContext *s, FLVContext *flv, double ts, int64_t pos) @@ -930,12 +955,12 @@ static int shift_data(AVFormatContext *s) static int flv_init(struct AVFormatContext *s) { int i; - int video_ctr = 0; + int video_ctr = 0, audio_ctr = 0; FLVContext *flv = s->priv_data; flv->last_ts = av_mallocz(sizeof(*flv->last_ts) * s->nb_streams); flv->metadata_pkt_written = av_mallocz(sizeof(*flv->metadata_pkt_written) * s->nb_streams); - flv->video_track_idx_map = av_mallocz(sizeof(*flv->video_track_idx_map) * s->nb_streams); + flv->track_idx_map = av_mallocz(sizeof(*flv->track_idx_map) * s->nb_streams); for (i = 0; i < s->nb_streams; i++) { AVCodecParameters *par = s->streams[i]->codecpar; @@ -946,7 +971,7 @@ static int flv_init(struct AVFormatContext *s) s->streams[i]->avg_frame_rate.num) { flv->framerate = av_q2d(s->streams[i]->avg_frame_rate); } - flv->video_track_idx_map[i] = video_ctr++; + flv->track_idx_map[i] = video_ctr++; if (flv->video_par && flv->flags & FLV_ADD_KEYFRAME_INDEX) { av_log(s, AV_LOG_ERROR, "at most one video stream is supported in flv with keyframe index\n"); @@ -977,12 +1002,22 @@ static int flv_init(struct AVFormatContext *s) } break; case AVMEDIA_TYPE_AUDIO: - if (flv->audio_par) { - av_log(s, AV_LOG_ERROR, - "at most one audio stream is supported in flv\n"); + if (audio_ctr && + par->codec_id != AV_CODEC_ID_AAC && + par->codec_id != AV_CODEC_ID_MP3 && + par->codec_id != AV_CODEC_ID_OPUS && + par->codec_id != AV_CODEC_ID_FLAC && + par->codec_id != AV_CODEC_ID_AC3 && + par->codec_id != AV_CODEC_ID_EAC3) { + av_log(s, AV_LOG_ERROR, "Unsupported multi-track codec.\n"); return AVERROR(EINVAL); } - flv->audio_par = par; + flv->track_idx_map[i] = audio_ctr++; + if (flv->audio_par) + av_log(s, AV_LOG_WARNING, + "more than one audio stream is not supported by most flv demuxers.\n"); + else + flv->audio_par = par; if (get_audio_flags(s, par) < 0) return unsupported_codec(s, "Audio", par->codec_id); if (par->codec_id == AV_CODEC_ID_PCM_S16BE) @@ -1154,9 +1189,11 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) uint8_t frametype = pkt->flags & AV_PKT_FLAG_KEY ? FLV_FRAME_KEY : FLV_FRAME_INTER; int flags = -1, flags_size, ret = 0; int64_t cur_offset = avio_tell(pb); - int track_idx = flv->video_track_idx_map[pkt->stream_index]; + int track_idx = flv->track_idx_map[pkt->stream_index]; - int extended_audio = par->codec_id == AV_CODEC_ID_OPUS + int extended_audio = (par->codec_id == AV_CODEC_ID_AAC && track_idx) + || (par->codec_id == AV_CODEC_ID_MP3 && track_idx) + || par->codec_id == AV_CODEC_ID_OPUS || par->codec_id == AV_CODEC_ID_FLAC || par->codec_id == AV_CODEC_ID_AC3 || par->codec_id == AV_CODEC_ID_EAC3; @@ -1173,8 +1210,8 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) else flags_size = 1; - if (par->codec_type == AVMEDIA_TYPE_VIDEO && track_idx) - flags_size += 2; // additional header bytes for multi-track video + if ((par->codec_type == AVMEDIA_TYPE_VIDEO || par->codec_type == AVMEDIA_TYPE_AUDIO) && track_idx) + flags_size += 2; // additional header bytes for multi-track flv if ((par->codec_id == AV_CODEC_ID_HEVC || (par->codec_id == AV_CODEC_ID_H264 && track_idx)) @@ -1341,8 +1378,15 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) if (h2645 && pkttype == PacketTypeCodedFrames) avio_wb24(pb, pkt->pts - pkt->dts); } else if (extended_audio) { - avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeCodedFrames); + if (track_idx) { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultitrack); + avio_w8(pb, MultitrackTypeOneTrack | AudioPacketTypeCodedFrames); + } else { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeCodedFrames); + } write_codec_fourcc(pb, par->codec_id); + if (track_idx) + avio_w8(pb, track_idx); } else { av_assert1(flags>=0); avio_w8(pb, flags); @@ -1432,7 +1476,7 @@ static void flv_deinit(AVFormatContext *s) av_freep(&flv->last_ts); av_freep(&flv->metadata_pkt_written); - av_freep(&flv->video_track_idx_map); + av_freep(&flv->track_idx_map); } static const AVOption options[] = { From patchwork Tue May 21 09:02:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49097 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097794pzb; Tue, 21 May 2024 02:06:13 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUYkcShje/b4D6h3/XcjmZpsIBzGsv3jwYT5i66px05bOorD7aWU+QDBhR7v/Nk3Atso9Hcb+AxFPVw+FvcEJD0qV3jfXnvTPHGbw== X-Google-Smtp-Source: AGHT+IFP8Vx7eBSvL/fBcOz8MqTZhqakZtukyO3PbVvoob1Kg4yY4DEA+LuPnyV/8TpWNwhWAyEe X-Received: by 2002:a50:d61c:0:b0:56e:2f39:c5d2 with SMTP id 4fb4d7f45d1cf-5767de62f93mr4801866a12.7.1716282373199; Tue, 21 May 2024 02:06:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282373; cv=none; d=google.com; s=arc-20160816; b=n2Z2hX6wurNBEzsaqJtRpg0hDsbYh+oHU3ZfBugoS/tV2GEudROC7FlTJzI5Jnb2XM YNAN3q9prtU+WMTrFIhdGIajKQbMS2DDWVDic/SnWKtb8ROLcAYRSbqbODMEs8w0LtaB +J9b9DkvGldVDCdKyfI7Czla4b3mJl8onCszg3xAVWyFLFimhJcbQMGeyzPgcYg9mNQm qcGBAIeGPkSJSyQxfJjkXjReZF1JNLTRYlW79l2DxEceeu0eb+iMuwMYuuutAkotX/VQ Ye/KGET4rnSi+isMH+liWLdrmOFlPhGnSoUlrI6EpXWyuOLCK9vwkTP1jkSWp5UEInKP 3znw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=MdXTY4e+RgSq7MuJVGREEIRf3fzecGsoTn3vCj5T+Cg=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=lGYx04P91cPzDSCQ7K8B2irJEPBQkoe6M0EUeq78kOaiRtCstlFTp6lizZ0T9RNfzk sgi1aBZHXLXCfzvxTOMxspZQsRbUlhJQ4L+T1AwGEnjNZE/YqFvwt/lRnGoiW3rN09AG vPusMuSPIhfLOq1YwlL1VjmQKn7Su+GVuUmOzcrH7jrab59Vn/Ar857cw97v8E4H1d7K OgdwRRuw7uVgqFN75wqHcSmnbcy7twtag9hNrlNiilgoIuebQJwNtGzop8bO2R239Kgi XJs/xEO3PCYWcE1316NbVPhhtznIEvqD3is2UKb2oEVj1bu8cTyTk0IGAbVE0yqa657N YzXw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=hTCelBSo; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5733bec099fsi13583449a12.144.2024.05.21.02.06.12; Tue, 21 May 2024 02:06:13 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=hTCelBSo; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4729968D321; Tue, 21 May 2024 12:04:00 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6E94C68D3B3 for ; Tue, 21 May 2024 12:03:43 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id 46BE227FFD2B6; Tue, 21 May 2024 11:03:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282219; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=crAwDE76BC/pGFeZJpR6iKXLZNwtRn1/591jEwjsmmA=; b=hTCelBSoVX7GWWS6qXZlM/P+Dp5D+y21G+7vwqInxAXo2Y6Kx1ZjL7Qrq4NLPSJryputX3 YQZ3OHt99Sd5+iKyXGRYXnKFqPTSCI2LFKOgB47vjGHuKu2lTyx6QSV7zqvKc8/1VifGe/ oaSOOv9K8yFwSOvhmVTGOD7blBGcxw9U0Wuc9SfzxTqxQZvGPeLEcBQxcznBhqX0mE13Eu MU0FqqK1U7sGz0wu4HhStUGrOOhSxuccYeYa64aR9oi59lJ5EpB5tnqVYqn/6MMPcEwpqn vC/IBsPIAAUTK5/4TcHll8Gc61AbZRzauOHDWokQr1z9/M0WhdSZclwdoAYsgQ== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:19 +0200 Message-ID: <20240521090316.782-11-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 10/13] avformat/flvdec: add support for reading multi track audio X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: p1WaODV3gmbK --- libavformat/flvdec.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index 890958351a..5877828c52 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -1336,12 +1336,26 @@ retry: pkt_type = flags & ~FLV_AUDIO_CODECID_MASK; if (pkt_type == AudioPacketTypeMultitrack) { - av_log(s, AV_LOG_ERROR, "Multitrack audio is unsupported!\n"); - return AVERROR_PATCHWELCOME; + uint8_t types = avio_r8(s->pb); + int multitrack_type = types >> 4; + pkt_type = types & 0xF; + + if (multitrack_type != MultitrackTypeOneTrack) { + av_log(s, AV_LOG_ERROR, "Audio multitrack types other than MultitrackTypeOneTrack are unsupported!\n"); + return AVERROR_PATCHWELCOME; + } + + multitrack = 1; + size--; } codec_id = avio_rb32(s->pb); size -= 4; + + if (multitrack) { + track_idx = avio_r8(s->pb); + size--; + } } } else if (type == FLV_TAG_TYPE_VIDEO) { stream_type = FLV_STREAM_TYPE_VIDEO; @@ -1438,7 +1452,8 @@ skip: st = s->streams[i]; if (stream_type == FLV_STREAM_TYPE_AUDIO) { if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO && - (s->audio_codec_id || flv_same_audio_codec(st->codecpar, flags, codec_id))) + (s->audio_codec_id || flv_same_audio_codec(st->codecpar, flags, codec_id)) && + st->id == track_idx) break; } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO && From patchwork Tue May 21 09:02:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49090 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097092pzb; Tue, 21 May 2024 02:04:46 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXf0cA4qhupQSymlkSLXAN0VQjF2EXW8xkAVTpVflHmJDNiLpKnoyd4w7VLhf7pfiSTidUuDV4U3sbeqzDVjMcA9GsDUmOdr0Rywg== X-Google-Smtp-Source: AGHT+IHyJmyilPolQBAdR/Umz8RcXTtdywCRfRTtSmmfIDi2MdKSqECG/sTJuRA1wC8OcCzAnv1W X-Received: by 2002:a50:cdd6:0:b0:574:ecfd:8306 with SMTP id 4fb4d7f45d1cf-574ecfd8f4amr12449958a12.25.1716282286556; Tue, 21 May 2024 02:04:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282286; cv=none; d=google.com; s=arc-20160816; b=kcQMUMmt1z+iLYFMw+OwXR12u//wFFTNVQxf/DHM58uHSE+dgI/3eU7XGemOxy5ohE cQERkVhNvkOljiH0ieGRhwJzH8E9CWxTNEEf77gL3XipPmIagA6VSrQCMt7wH1w2jo6m Xs8dSxbHcw9g7Z0PFsP1pWjO2y66IlTOu8ZpsrTKsIJl6hiZrV4WNAfKEREUglO9Jxuk IQjuVSrhDlIXokfTMG5ashpB3MQ9ooZXNsrNOkbOFbbDClaES7FgSXXk6Gt6axn9rX8J UViO8uyESlQw1Z72quGwh2DRq9PvYKqoZG/WpFlp0LgV3I7lEZnVJoXe6FBCwqgyQvtk IfBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=OE8Llew74JkiPLAQ3XYG3CY3bxlHizRhobF1PlNSTzE=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=umhr+2+ibLKMXlxU5KkB1IaRwOsJicD9jmqZAAXCDMmm6V6qLwv2TCohw70aULMZkX LuNMcsHqAcmbHL0ZsUoFqFam4fopmmP9HefMizftJMj+EnoYtsN98TBF/on9pbKR0jLJ eFC94Z+ZtNpPqfU2W2YRzXiX5sM4OI1sLJuVhKYf6jJP09U/onHOM4yJVOHfUaiuvdLI NumTBc9LoqhUdeM352tBRilq1uBTc2HJPaQ1198KGg5a+KFp1fz0nVHTpizJxP0nkX0b wGrbBHapzQjonw/24X9jUv2ZDlFqou+Jdb6SdGTL64fFyqNdrIRpu5bgd53J0Fd5yT53 geiw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=RhWc9vCw; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-574f5cf4c50si7338218a12.325.2024.05.21.02.04.46; Tue, 21 May 2024 02:04:46 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=RhWc9vCw; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 09FF168D331; Tue, 21 May 2024 12:03:50 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9DFA268D319 for ; Tue, 21 May 2024 12:03:43 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id 7724427FFD2B7; Tue, 21 May 2024 11:03:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282219; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KRB4cqTFEgIpugRE+GOYEmHfiItbIEC1b2g0tYJPy80=; b=RhWc9vCwCV5GX3cOBQ7FQFT8y8s8KKmoKL3LmSABOj8U1K7FTYY2pvT9nObKctIh+4C1aN 8gR/hBstNW4pYhSQNyp7mnuRinMFTVy8DjBJwZeg5dbWYmnCZqdJSdsAarFDE9L6XlXfNF 2ga8Jp/8FDRxzmf9xcGTmUBJC36e7iUJASqvkzb9Hg6ReJlZK3LU55nlYFe8wJmnGuq17i ZkfNL7R5cUD09EPsvEJez+qaWblfs/XWUQz7p1YHuPvBhkrGYrlNjcycrQZw7QtarBHFso vRhjT+PcdXnwvkR4urEHFqWNBivmSwgCusy3PdXK4Py6mQGke7WiR4c3C+VMnA== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:20 +0200 Message-ID: <20240521090316.782-12-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 11/13] avformat/rtmpproto: add more enhanced rtmp codecs X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: tmJz8GOZ3tU6 --- libavformat/rtmpproto.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/libavformat/rtmpproto.c b/libavformat/rtmpproto.c index bc3d9df7b9..7bbb6d5313 100644 --- a/libavformat/rtmpproto.c +++ b/libavformat/rtmpproto.c @@ -356,9 +356,16 @@ static int gen_connect(URLContext *s, RTMPContext *rt) while(fourcc_data - rt->enhanced_codecs < fourcc_str_len) { unsigned char fourcc[5]; - if (!strncmp(fourcc_data, "hvc1", 4) || + if (!strncmp(fourcc_data, "avc1", 4) || + !strncmp(fourcc_data, "hvc1", 4) || !strncmp(fourcc_data, "av01", 4) || - !strncmp(fourcc_data, "vp09", 4)) { + !strncmp(fourcc_data, "vp09", 4) || + !strncmp(fourcc_data, "mp4a", 4) || + !strncmp(fourcc_data, "Opus", 4) || + !strncmp(fourcc_data, "fLaC", 4) || + !strncmp(fourcc_data, ".mp3", 4) || + !strncmp(fourcc_data, "ac-3", 4) || + !strncmp(fourcc_data, "ec-3", 4)) { av_strlcpy(fourcc, fourcc_data, sizeof(fourcc)); ff_amf_write_string(&p, fourcc); } else { From patchwork Tue May 21 09:02:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49091 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097193pzb; Tue, 21 May 2024 02:04:58 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVXZrtND7FPUK2YUnQQkdWtLUYLKn20h5an/wwvW1v4jek3NW3uM6IDXy+w5IPdZ4Kjf9JynFyF+2LmhJQZBZXaZDll1buCLa4PeA== X-Google-Smtp-Source: AGHT+IF/NV50JDLMVhGEiIBSrNwIPN9vUAMBmfHxytIFTXLtvaVuUIypcWY6AaIaoJF3M1fM/c2W X-Received: by 2002:a17:907:320b:b0:a59:b177:22d7 with SMTP id a640c23a62f3a-a5a2d53b75emr3814361066b.8.1716282298074; Tue, 21 May 2024 02:04:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282298; cv=none; d=google.com; s=arc-20160816; b=Yf5IMHKkrTWMwuXuOWpfha/gdXkazAVIRrhjTl59Jo7jy4HZy4dhMwNUCLjFPsyPcB dCQz3m1jLBdU7Igv3giuMyZAa1QJTqqAMN8oWM8EFfVHwUNQt+EUIt5z5ZnS14XmRlId TlbpwKOIkxTQmBY7bGEoErggbD+lor8ZxwJ0jaBYriDu9VXwiYmOotWnl9ckKWE6O6hl 98hxIp7xpYQxPwXG3IPxTap7lwVP69sl+sDNfPRhAirNPa78nCWDmmK1yrRTh2SUsKow /yBMrzJ1xPnXYHp0zFJcGbGWZoKblH25oOr8fArMk6CQFiSjo236+nQa/aHhdRr+jxeT pbrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=z+yY3QEYX3fMx0KHnVPOXf4wf4FHvdyaBE0aWvWjEqE=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=rB9hkNQqsWjHmrkh8AIr2OBufl/UHJ9Ksn90RIe4bXd0JlLq0O8+1FJpdU63hUz/G0 kzmr26Vto80vWgTVjHkK8aIahG+rzZ4f6kthwLnI1mRkUKJCGnkGYAJalEk33WSJD31v KUumVKOqO7OjH5TGC1wKf93o1x2RdEpmRzyCoSURee9Bvfwa4so9kJ8jdoQqHWM2dOcd fqpfjibSwJZWgvxoQf+Fc4TF6+1xvAcEgySZNcbDBtlCRRpvMVJfYq5C2KGlyHTbkbCh foPM9Nw68gKmxeBErZisX3mVcbyH1m1Q7/8klHAdrmoMP1wKsK4U5gH52XueS4D/TTxz aEQA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=WUNNDXcu; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a640c23a62f3a-a5cea7434c2si613099666b.194.2024.05.21.02.04.57; Tue, 21 May 2024 02:04:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=WUNNDXcu; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3607968D3D7; Tue, 21 May 2024 12:03:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A694A68D321 for ; Tue, 21 May 2024 12:03:43 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id A9BCD27FFD2B8; Tue, 21 May 2024 11:03:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282219; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3Ruj1l7ZbyODxvmvPk9pkUZuZDQaQKfwYvmfpmhPoR4=; b=WUNNDXcunkZAATK1oAj6n0WWAcMMjMNAT+tWXzlDohAhZc6j2Pud82af5OnosNn17fDL38 63icPDK3MCUrpqbRyQJ8OtRUTqXAwGc646pwa+fuVO/qg2+MCs58gh8WbKCS2MkLAJ8sZ3 mUPsCqBiRYH0WZOcUT0gBSQiLuQLTZmVL0y1QSMgPYg+gPghMEG1tMgiwop4VhmFXo0OiD 6Qg8+YPc4myaYV7+KwoZmx+lJoAXAF3LSSe9yi7+ddBmCYoKeV74adU2TDBlLnSQfKHuN+ QtfCEZzKvKChs+ypT8mB4+I2yc2/KZ8wQIuAFVAgw+KmmY7RHsVOLZncI5tkHQ== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:21 +0200 Message-ID: <20240521090316.782-13-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 12/13] avformat/flvdec: stop shadowing local variables X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: QMtKSPbVU8wq --- libavformat/flvdec.c | 44 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index 5877828c52..4f98ff348c 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -1394,8 +1394,8 @@ retry: if (enhanced_flv && (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { if (pkt_type == PacketTypeMetadata) { - int ret = flv_parse_video_color_info(s, st, next); - av_log(s, AV_LOG_DEBUG, "enhanced flv parse metadata ret %d and skip\n", ret); + int sret = flv_parse_video_color_info(s, st, next); + av_log(s, AV_LOG_DEBUG, "enhanced flv parse metadata ret %d and skip\n", sret); } goto skip; } else if ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_VIDEO_INFO_CMD) { @@ -1498,25 +1498,25 @@ skip: if ((s->pb->seekable & AVIO_SEEKABLE_NORMAL) && (!s->duration || s->duration == AV_NOPTS_VALUE) && !flv->searched_for_end) { - int size; + int final_size; const int64_t pos = avio_tell(s->pb); // Read the last 4 bytes of the file, this should be the size of the // previous FLV tag. Use the timestamp of its payload as duration. int64_t fsize = avio_size(s->pb); retry_duration: avio_seek(s->pb, fsize - 4, SEEK_SET); - size = avio_rb32(s->pb); - if (size > 0 && size < fsize) { - // Seek to the start of the last FLV tag at position (fsize - 4 - size) + final_size = avio_rb32(s->pb); + if (final_size > 0 && final_size < fsize) { + // Seek to the start of the last FLV tag at position (fsize - 4 - final_size) // but skip the byte indicating the type. - avio_seek(s->pb, fsize - 3 - size, SEEK_SET); - if (size == avio_rb24(s->pb) + 11) { + avio_seek(s->pb, fsize - 3 - final_size, SEEK_SET); + if (final_size == avio_rb24(s->pb) + 11) { uint32_t ts = avio_rb24(s->pb); ts |= (unsigned)avio_r8(s->pb) << 24; if (ts) s->duration = ts * (int64_t)AV_TIME_BASE / 1000; - else if (fsize >= 8 && fsize - 8 >= size) { - fsize -= size+4; + else if (fsize >= 8 && fsize - 8 >= final_size) { + fsize -= final_size+4; goto retry_duration; } } @@ -1610,10 +1610,10 @@ retry_duration: goto leave; } } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { - int ret = flv_set_video_codec(s, st, codec_id, 1); - if (ret < 0) - return ret; - size -= ret; + int sret = flv_set_video_codec(s, st, codec_id, 1); + if (sret < 0) + return sret; + size -= sret; } else if (stream_type == FLV_STREAM_TYPE_SUBTITLE) { st->codecpar->codec_id = AV_CODEC_ID_TEXT; } else if (stream_type == FLV_STREAM_TYPE_DATA) { @@ -1703,20 +1703,20 @@ retry_duration: pkt->stream_index = st->index; pkt->pos = pos; if (!multitrack && flv->new_extradata[stream_type]) { - int ret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, - flv->new_extradata[stream_type], - flv->new_extradata_size[stream_type]); - if (ret >= 0) { + int sret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, + flv->new_extradata[stream_type], + flv->new_extradata_size[stream_type]); + if (sret >= 0) { flv->new_extradata[stream_type] = NULL; flv->new_extradata_size[stream_type] = 0; } } else if (multitrack && flv->mt_extradata_cnt > track_idx && flv->mt_extradata[track_idx]) { - int ret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, - flv->mt_extradata[track_idx], - flv->mt_extradata_sz[track_idx]); - if (ret >= 0) { + int sret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, + flv->mt_extradata[track_idx], + flv->mt_extradata_sz[track_idx]); + if (sret >= 0) { flv->mt_extradata[track_idx] = NULL; flv->mt_extradata_sz[track_idx] = 0; } From patchwork Tue May 21 09:02:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49092 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097272pzb; Tue, 21 May 2024 02:05:09 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUkN6cqynzac+JnvTYrIUNsjLvjVvas7aP8gDZZ4j9uhDtMxZsZsaX23CBan6Xj1whT2EmUoXVdW6mdxy8Rp7iFU7d/b0Ik3p8BcQ== X-Google-Smtp-Source: AGHT+IELHWF5yXFbXPwa1lC0rCyjfqNJmpHSBT861XaP/dFtTpCfy/m+h/l05qAXwSicjwmL4RfJ X-Received: by 2002:a17:906:8472:b0:a61:9d64:e79c with SMTP id a640c23a62f3a-a619d64e8afmr260728066b.0.1716282309167; Tue, 21 May 2024 02:05:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282309; cv=none; d=google.com; s=arc-20160816; b=y5y0qKtTDboFdSJ9R8k2T2FgJxWGeatgA+GRlQS6KGiBBEgBZkxRKgmHUnwMml+sXz F0EzQT2qmRfRqNt3Skd2gEku8JpoAQh3BqcQlyrNDTR5lvMZGld7KS4+PXU/Kkd3WMQG 63XadBa1ExisHg2erxEHb0oHJr6jfiZB01jj30BOi6pWkb/nRzzj72QrKDqM2gt6tAQ7 xQcrObZ1moopZDtoJix5O92t49m8V9B6z9Zd1Stms4M8O7radyC47QhPKolgOD9xHK4H 1iGRaMDQD9DuoT3MJpQkDQJoLJzAwLsF3xvSEcelXvJTLPLMgOMMYegcax8WWadqbjht EOZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=trVa/GD/jXGWxpQ8ITalNoOetPnMTDlKOl6qDKCBwa4=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=k6THQF0+r5Ta3/Y2gLaExIg6jl3d5r+fveQQjlB9qEKya9T6h8jEP0vslnZ2cWTzFm 33iqSOYFXFIfkW6lMkvl01EZ+4DOyjpBfBru/9ne5fyqsG2SebryNy4ENFG7xWA8kRxP 9WLnOqRt/113EZYEIP9a/BKhB5odnPM+g5yN0BXH4pjHqC/eXJ/9KorTcAsgqhI3/jTD eKnjL6qVkLmxFP19UnSksKG6wXVB2KwcqYUAhy5nd57y51TNftfPUpY6KHnuBdRpQ91R gZUvbcmv/E3WOmuwwSxP5NcscOxvsSlXKXHbmTgvXed8ICbHTrZ0fc8D3JJvVjlvXQWy RT5Q==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=JAQeWzv8; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a640c23a62f3a-a5a17c2d11esi1411691166b.1019.2024.05.21.02.05.08; Tue, 21 May 2024 02:05:09 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=JAQeWzv8; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9516268D3C9; Tue, 21 May 2024 12:03:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D20FB68D321 for ; Tue, 21 May 2024 12:03:43 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id DB1C327FFD2B9; Tue, 21 May 2024 11:03:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282219; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c9RMJmTKpZzPJHcpqYa1ZPwzv4Ii8WxSeaxT4qg+wlc=; b=JAQeWzv80Pz6HFJ+dL8ED6x8sAjCJliwf/DZutIG2W4N0mcnJmQX3QlNT9D/LIctJ7SdxC CvCAURbUl/8lbPtXIwhOIPqsQJr/d1L3eS5V5xudy8Wl1Of2CrRBj8HeuzkvZqzm/GfSq/ CfSMmkD6KtkRMiLD4R09OnOAVZ5dUWDZVGIhJ9aczAFl2sQHk4EZpk/QfLf2rMJfxETOg2 DXr2q0sn2KkapvKWA7CcPEDxcu3bKYOhKVL5yRWqvSt7zFZPp/9JCFoanDjnUHVOA5PTsB TNEMGdHhTtSxQUpj7m8emJFgqrYTMObUG4rhzpQIZyvstd81Dz4RaNI9TVykFQ== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:22 +0200 Message-ID: <20240521090316.782-14-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 13/13] avformat/flvdec: support all multi-track modes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: gzC0LmD8th6i --- libavformat/flvdec.c | 570 +++++++++++++++++++++++-------------------- 1 file changed, 306 insertions(+), 264 deletions(-) diff --git a/libavformat/flvdec.c b/libavformat/flvdec.c index 4f98ff348c..ee22db2ce2 100644 --- a/libavformat/flvdec.c +++ b/libavformat/flvdec.c @@ -1273,6 +1273,7 @@ static int flv_read_packet(AVFormatContext *s, AVPacket *pkt) { FLVContext *flv = s->priv_data; int ret, i, size, flags; + int res = 0; enum FlvTagType type; int stream_type=-1; int64_t next, pos, meta_pos; @@ -1287,6 +1288,7 @@ static int flv_read_packet(AVFormatContext *s, AVPacket *pkt) int pkt_type = 0; uint8_t track_idx = 0; uint32_t codec_id = 0; + int multitrack_type = MultitrackTypeOneTrack; retry: /* pkt size is repeated at end. skip it */ @@ -1337,14 +1339,9 @@ retry: if (pkt_type == AudioPacketTypeMultitrack) { uint8_t types = avio_r8(s->pb); - int multitrack_type = types >> 4; + multitrack_type = types & 0xF0; pkt_type = types & 0xF; - if (multitrack_type != MultitrackTypeOneTrack) { - av_log(s, AV_LOG_ERROR, "Audio multitrack types other than MultitrackTypeOneTrack are unsupported!\n"); - return AVERROR_PATCHWELCOME; - } - multitrack = 1; size--; } @@ -1371,14 +1368,9 @@ retry: if (pkt_type == PacketTypeMultitrack) { uint8_t types = avio_r8(s->pb); - int multitrack_type = types >> 4; + multitrack_type = types & 0xF0; pkt_type = types & 0xF; - if (multitrack_type != MultitrackTypeOneTrack) { - av_log(s, AV_LOG_ERROR, "Multitrack types other than MultitrackTypeOneTrack are unsupported!\n"); - return AVERROR_PATCHWELCOME; - } - multitrack = 1; size--; } @@ -1447,293 +1439,343 @@ skip: goto leave; } - /* now find stream */ - for (i = 0; i < s->nb_streams; i++) { - st = s->streams[i]; - if (stream_type == FLV_STREAM_TYPE_AUDIO) { - if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO && - (s->audio_codec_id || flv_same_audio_codec(st->codecpar, flags, codec_id)) && - st->id == track_idx) - break; - } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { - if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO && - (s->video_codec_id || flv_same_video_codec(st->codecpar, codec_id)) && - st->id == track_idx) - break; - } else if (stream_type == FLV_STREAM_TYPE_SUBTITLE) { - if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) - break; - } else if (stream_type == FLV_STREAM_TYPE_DATA) { - if (st->codecpar->codec_type == AVMEDIA_TYPE_DATA) - break; + for (;;) { + int track_size = size; + + if (multitrack_type != MultitrackTypeOneTrack) { + track_size = avio_rb24(s->pb); + size -= 3; } - } - if (i == s->nb_streams) { - static const enum AVMediaType stream_types[] = {AVMEDIA_TYPE_VIDEO, AVMEDIA_TYPE_AUDIO, AVMEDIA_TYPE_SUBTITLE, AVMEDIA_TYPE_DATA}; - st = create_stream(s, stream_types[stream_type], track_idx); - if (!st) - return AVERROR(ENOMEM); - } - av_log(s, AV_LOG_TRACE, "%d %X %d \n", stream_type, flags, st->discard); - if (flv->time_pos <= pos) { - dts += flv->time_offset; - } + /* now find stream */ + for (i = 0; i < s->nb_streams; i++) { + st = s->streams[i]; + if (stream_type == FLV_STREAM_TYPE_AUDIO) { + if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO && + (s->audio_codec_id || flv_same_audio_codec(st->codecpar, flags, codec_id)) && + st->id == track_idx) + break; + } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { + if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO && + (s->video_codec_id || flv_same_video_codec(st->codecpar, codec_id)) && + st->id == track_idx) + break; + } else if (stream_type == FLV_STREAM_TYPE_SUBTITLE) { + if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) + break; + } else if (stream_type == FLV_STREAM_TYPE_DATA) { + if (st->codecpar->codec_type == AVMEDIA_TYPE_DATA) + break; + } + } + if (i == s->nb_streams) { + static const enum AVMediaType stream_types[] = {AVMEDIA_TYPE_VIDEO, AVMEDIA_TYPE_AUDIO, AVMEDIA_TYPE_SUBTITLE, AVMEDIA_TYPE_DATA}; + st = create_stream(s, stream_types[stream_type], track_idx); + if (!st) + return AVERROR(ENOMEM); + } + av_log(s, AV_LOG_TRACE, "%d %X %d \n", stream_type, flags, st->discard); - if ((s->pb->seekable & AVIO_SEEKABLE_NORMAL) && - ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_KEY || - stream_type == FLV_STREAM_TYPE_AUDIO)) - av_add_index_entry(st, pos, dts, size, 0, AVINDEX_KEYFRAME); + if (flv->time_pos <= pos) { + dts += flv->time_offset; + } - if ((st->discard >= AVDISCARD_NONKEY && !((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_KEY || stream_type == FLV_STREAM_TYPE_AUDIO)) || - (st->discard >= AVDISCARD_BIDIR && ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_DISP_INTER && stream_type == FLV_STREAM_TYPE_VIDEO)) || - st->discard >= AVDISCARD_ALL) { - avio_seek(s->pb, next, SEEK_SET); - ret = FFERROR_REDO; - goto leave; - } + if ((s->pb->seekable & AVIO_SEEKABLE_NORMAL) && + ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_KEY || + stream_type == FLV_STREAM_TYPE_AUDIO)) + av_add_index_entry(st, pos, dts, track_size, 0, AVINDEX_KEYFRAME); + + if ((st->discard >= AVDISCARD_NONKEY && !((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_KEY || stream_type == FLV_STREAM_TYPE_AUDIO)) || + (st->discard >= AVDISCARD_BIDIR && ((flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_DISP_INTER && stream_type == FLV_STREAM_TYPE_VIDEO)) || + st->discard >= AVDISCARD_ALL) { + avio_seek(s->pb, next, SEEK_SET); + ret = FFERROR_REDO; + goto leave; + } - // if not streamed and no duration from metadata then seek to end to find - // the duration from the timestamps - if ((s->pb->seekable & AVIO_SEEKABLE_NORMAL) && - (!s->duration || s->duration == AV_NOPTS_VALUE) && - !flv->searched_for_end) { - int final_size; - const int64_t pos = avio_tell(s->pb); - // Read the last 4 bytes of the file, this should be the size of the - // previous FLV tag. Use the timestamp of its payload as duration. - int64_t fsize = avio_size(s->pb); + // if not streamed and no duration from metadata then seek to end to find + // the duration from the timestamps + if ((s->pb->seekable & AVIO_SEEKABLE_NORMAL) && + (!s->duration || s->duration == AV_NOPTS_VALUE) && + !flv->searched_for_end) { + int final_size; + const int64_t pos = avio_tell(s->pb); + // Read the last 4 bytes of the file, this should be the size of the + // previous FLV tag. Use the timestamp of its payload as duration. + int64_t fsize = avio_size(s->pb); retry_duration: - avio_seek(s->pb, fsize - 4, SEEK_SET); - final_size = avio_rb32(s->pb); - if (final_size > 0 && final_size < fsize) { - // Seek to the start of the last FLV tag at position (fsize - 4 - final_size) - // but skip the byte indicating the type. - avio_seek(s->pb, fsize - 3 - final_size, SEEK_SET); - if (final_size == avio_rb24(s->pb) + 11) { - uint32_t ts = avio_rb24(s->pb); - ts |= (unsigned)avio_r8(s->pb) << 24; - if (ts) - s->duration = ts * (int64_t)AV_TIME_BASE / 1000; - else if (fsize >= 8 && fsize - 8 >= final_size) { - fsize -= final_size+4; - goto retry_duration; + avio_seek(s->pb, fsize - 4, SEEK_SET); + final_size = avio_rb32(s->pb); + if (final_size > 0 && final_size < fsize) { + // Seek to the start of the last FLV tag at position (fsize - 4 - final_size) + // but skip the byte indicating the type. + avio_seek(s->pb, fsize - 3 - final_size, SEEK_SET); + if (final_size == avio_rb24(s->pb) + 11) { + uint32_t ts = avio_rb24(s->pb); + ts |= (unsigned)avio_r8(s->pb) << 24; + if (ts) + s->duration = ts * (int64_t)AV_TIME_BASE / 1000; + else if (fsize >= 8 && fsize - 8 >= final_size) { + fsize -= final_size+4; + goto retry_duration; + } } } + + avio_seek(s->pb, pos, SEEK_SET); + flv->searched_for_end = 1; } - avio_seek(s->pb, pos, SEEK_SET); - flv->searched_for_end = 1; - } + if (stream_type == FLV_STREAM_TYPE_AUDIO && !enhanced_flv) { + int bits_per_coded_sample; + channels = (flags & FLV_AUDIO_CHANNEL_MASK) == FLV_STEREO ? 2 : 1; + sample_rate = 44100 << ((flags & FLV_AUDIO_SAMPLERATE_MASK) >> + FLV_AUDIO_SAMPLERATE_OFFSET) >> 3; + bits_per_coded_sample = (flags & FLV_AUDIO_SAMPLESIZE_MASK) ? 16 : 8; + if (!av_channel_layout_check(&st->codecpar->ch_layout) || + !st->codecpar->sample_rate || + !st->codecpar->bits_per_coded_sample) { + av_channel_layout_default(&st->codecpar->ch_layout, channels); + st->codecpar->sample_rate = sample_rate; + st->codecpar->bits_per_coded_sample = bits_per_coded_sample; + } + if (!st->codecpar->codec_id) { + flv_set_audio_codec(s, st, st->codecpar, + flags & FLV_AUDIO_CODECID_MASK); + flv->last_sample_rate = + sample_rate = st->codecpar->sample_rate; + flv->last_channels = + channels = st->codecpar->ch_layout.nb_channels; + } else { + AVCodecParameters *par = avcodec_parameters_alloc(); + if (!par) { + ret = AVERROR(ENOMEM); + goto leave; + } + par->sample_rate = sample_rate; + par->bits_per_coded_sample = bits_per_coded_sample; + flv_set_audio_codec(s, st, par, flags & FLV_AUDIO_CODECID_MASK); + sample_rate = par->sample_rate; + avcodec_parameters_free(&par); + } + } else if (stream_type == FLV_STREAM_TYPE_AUDIO) { + if (!st->codecpar->codec_id) + flv_set_audio_codec(s, st, st->codecpar, + codec_id ? codec_id : (flags & FLV_AUDIO_CODECID_MASK)); + + // These are not signalled in the flags anymore + channels = 0; + sample_rate = 0; + + if (pkt_type == AudioPacketTypeMultichannelConfig) { + int channel_order = avio_r8(s->pb); + channels = avio_r8(s->pb); + size -= 2; + track_size -= 2; + + av_channel_layout_uninit(&st->codecpar->ch_layout); + + if (channel_order == AudioChannelOrderCustom) { + ret = av_channel_layout_custom_init(&st->codecpar->ch_layout, channels); + if (ret < 0) + return ret; + + for (i = 0; i < channels; i++) { + uint8_t id = avio_r8(s->pb); + size--; + + if (id < 18) + st->codecpar->ch_layout.u.map[i].id = id; + else if (id >= 18 && id <= 23) + st->codecpar->ch_layout.u.map[i].id = id - 18 + AV_CHAN_LOW_FREQUENCY_2; + else if (id == 0xFE) + st->codecpar->ch_layout.u.map[i].id = AV_CHAN_UNUSED; + else + st->codecpar->ch_layout.u.map[i].id = AV_CHAN_UNKNOWN; + } + } else if (channel_order == AudioChannelOrderNative) { + uint64_t mask = avio_rb32(s->pb); + size -= 4; + track_size -= 4; + + // The first 18 entries in the mask match ours, but the remaining 6 entries start at AV_CHAN_LOW_FREQUENCY_2 + mask = (mask & 0x3FFFF) | ((mask & 0xFC0000) << (AV_CHAN_LOW_FREQUENCY_2 - 18)); + ret = av_channel_layout_from_mask(&st->codecpar->ch_layout, mask); + if (ret < 0) + return ret; + } else { + av_channel_layout_default(&st->codecpar->ch_layout, channels); + } + + av_log(s, AV_LOG_DEBUG, "Set channel data from MultiChannel info.\n"); - if (stream_type == FLV_STREAM_TYPE_AUDIO && !enhanced_flv) { - int bits_per_coded_sample; - channels = (flags & FLV_AUDIO_CHANNEL_MASK) == FLV_STEREO ? 2 : 1; - sample_rate = 44100 << ((flags & FLV_AUDIO_SAMPLERATE_MASK) >> - FLV_AUDIO_SAMPLERATE_OFFSET) >> 3; - bits_per_coded_sample = (flags & FLV_AUDIO_SAMPLESIZE_MASK) ? 16 : 8; - if (!av_channel_layout_check(&st->codecpar->ch_layout) || - !st->codecpar->sample_rate || - !st->codecpar->bits_per_coded_sample) { - av_channel_layout_default(&st->codecpar->ch_layout, channels); - st->codecpar->sample_rate = sample_rate; - st->codecpar->bits_per_coded_sample = bits_per_coded_sample; - } - if (!st->codecpar->codec_id) { - flv_set_audio_codec(s, st, st->codecpar, - flags & FLV_AUDIO_CODECID_MASK); - flv->last_sample_rate = - sample_rate = st->codecpar->sample_rate; - flv->last_channels = - channels = st->codecpar->ch_layout.nb_channels; - } else { - AVCodecParameters *par = avcodec_parameters_alloc(); - if (!par) { - ret = AVERROR(ENOMEM); goto leave; } - par->sample_rate = sample_rate; - par->bits_per_coded_sample = bits_per_coded_sample; - flv_set_audio_codec(s, st, par, flags & FLV_AUDIO_CODECID_MASK); - sample_rate = par->sample_rate; - avcodec_parameters_free(&par); + } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { + int sret = flv_set_video_codec(s, st, codec_id, 1); + if (sret < 0) + return sret; + size -= sret; + track_size -= sret; + } else if (stream_type == FLV_STREAM_TYPE_SUBTITLE) { + st->codecpar->codec_id = AV_CODEC_ID_TEXT; + } else if (stream_type == FLV_STREAM_TYPE_DATA) { + st->codecpar->codec_id = AV_CODEC_ID_NONE; // Opaque AMF data } - } else if (stream_type == FLV_STREAM_TYPE_AUDIO) { - if (!st->codecpar->codec_id) - flv_set_audio_codec(s, st, st->codecpar, - codec_id ? codec_id : (flags & FLV_AUDIO_CODECID_MASK)); - - // These are not signalled in the flags anymore - channels = 0; - sample_rate = 0; - if (pkt_type == AudioPacketTypeMultichannelConfig) { - int channel_order = avio_r8(s->pb); - channels = avio_r8(s->pb); - size -= 2; + if (st->codecpar->codec_id == AV_CODEC_ID_AAC || + st->codecpar->codec_id == AV_CODEC_ID_OPUS || + st->codecpar->codec_id == AV_CODEC_ID_FLAC || + st->codecpar->codec_id == AV_CODEC_ID_H264 || + st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || + st->codecpar->codec_id == AV_CODEC_ID_HEVC || + st->codecpar->codec_id == AV_CODEC_ID_AV1 || + st->codecpar->codec_id == AV_CODEC_ID_VP9) { + int type = 0; + if (enhanced_flv) { + type = pkt_type; + } else { + type = avio_r8(s->pb); + size--; + track_size--; + } - av_channel_layout_uninit(&st->codecpar->ch_layout); + if (size < 0 || track_size < 0) { + ret = AVERROR_INVALIDDATA; + goto leave; + } - if (channel_order == AudioChannelOrderCustom) { - ret = av_channel_layout_custom_init(&st->codecpar->ch_layout, channels); - if (ret < 0) - return ret; + if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && flv->meta_color_info_flag) { + flv_update_video_color_info(s, st); // update av packet side data + flv->meta_color_info_flag = 0; + } - for (i = 0; i < channels; i++) { - uint8_t id = avio_r8(s->pb); - size--; - - if (id < 18) - st->codecpar->ch_layout.u.map[i].id = id; - else if (id >= 18 && id <= 23) - st->codecpar->ch_layout.u.map[i].id = id - 18 + AV_CHAN_LOW_FREQUENCY_2; - else if (id == 0xFE) - st->codecpar->ch_layout.u.map[i].id = AV_CHAN_UNUSED; - else - st->codecpar->ch_layout.u.map[i].id = AV_CHAN_UNKNOWN; + if (st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || + (st->codecpar->codec_id == AV_CODEC_ID_H264 && (!enhanced_flv || type == PacketTypeCodedFrames)) || + (st->codecpar->codec_id == AV_CODEC_ID_HEVC && type == PacketTypeCodedFrames)) { + // sign extension + int32_t cts = (avio_rb24(s->pb) + 0xff800000) ^ 0xff800000; + pts = av_sat_add64(dts, cts); + if (cts < 0) { // dts might be wrong + if (!flv->wrong_dts) + av_log(s, AV_LOG_WARNING, + "Negative cts, previous timestamps might be wrong.\n"); + flv->wrong_dts = 1; + } else if (FFABS(dts - pts) > 1000*60*15) { + av_log(s, AV_LOG_WARNING, + "invalid timestamps %"PRId64" %"PRId64"\n", dts, pts); + dts = pts = AV_NOPTS_VALUE; } - } else if (channel_order == AudioChannelOrderNative) { - uint64_t mask = avio_rb32(s->pb); - size -= 4; - - // The first 18 entries in the mask match ours, but the remaining 6 entries start at AV_CHAN_LOW_FREQUENCY_2 - mask = (mask & 0x3FFFF) | ((mask & 0xFC0000) << (AV_CHAN_LOW_FREQUENCY_2 - 18)); - ret = av_channel_layout_from_mask(&st->codecpar->ch_layout, mask); - if (ret < 0) - return ret; - } else { - av_channel_layout_default(&st->codecpar->ch_layout, channels); + size -= 3; + track_size -= 3; } + if (type == 0 && (!st->codecpar->extradata || st->codecpar->codec_id == AV_CODEC_ID_AAC || + st->codecpar->codec_id == AV_CODEC_ID_OPUS || st->codecpar->codec_id == AV_CODEC_ID_FLAC || + st->codecpar->codec_id == AV_CODEC_ID_H264 || st->codecpar->codec_id == AV_CODEC_ID_HEVC || + st->codecpar->codec_id == AV_CODEC_ID_AV1 || st->codecpar->codec_id == AV_CODEC_ID_VP9)) { + AVDictionaryEntry *t; + + if (st->codecpar->extradata) { + if ((ret = flv_queue_extradata(flv, s->pb, multitrack ? track_idx : stream_type, track_size, multitrack)) < 0) + return ret; + ret = FFERROR_REDO; + goto leave; + } + if ((ret = flv_get_extradata(s, st, track_size)) < 0) + return ret; - av_log(s, AV_LOG_DEBUG, "Set channel data from MultiChannel info.\n"); - - goto leave; - } - } else if (stream_type == FLV_STREAM_TYPE_VIDEO) { - int sret = flv_set_video_codec(s, st, codec_id, 1); - if (sret < 0) - return sret; - size -= sret; - } else if (stream_type == FLV_STREAM_TYPE_SUBTITLE) { - st->codecpar->codec_id = AV_CODEC_ID_TEXT; - } else if (stream_type == FLV_STREAM_TYPE_DATA) { - st->codecpar->codec_id = AV_CODEC_ID_NONE; // Opaque AMF data - } + /* Workaround for buggy Omnia A/XE encoder */ + t = av_dict_get(s->metadata, "Encoder", NULL, 0); + if (st->codecpar->codec_id == AV_CODEC_ID_AAC && t && !strcmp(t->value, "Omnia A/XE")) + st->codecpar->extradata_size = 2; - if (st->codecpar->codec_id == AV_CODEC_ID_AAC || - st->codecpar->codec_id == AV_CODEC_ID_OPUS || - st->codecpar->codec_id == AV_CODEC_ID_FLAC || - st->codecpar->codec_id == AV_CODEC_ID_H264 || - st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || - st->codecpar->codec_id == AV_CODEC_ID_HEVC || - st->codecpar->codec_id == AV_CODEC_ID_AV1 || - st->codecpar->codec_id == AV_CODEC_ID_VP9) { - int type = 0; - if (enhanced_flv) { - type = pkt_type; - } else { - type = avio_r8(s->pb); - size--; + ret = FFERROR_REDO; + goto leave; + } } - if (size < 0) { - ret = AVERROR_INVALIDDATA; + /* skip empty data packets */ + if (!size) { + ret = FFERROR_REDO; goto leave; } - if (enhanced_flv && stream_type == FLV_STREAM_TYPE_VIDEO && flv->meta_color_info_flag) { - flv_update_video_color_info(s, st); // update av packet side data - flv->meta_color_info_flag = 0; - } - - if (st->codecpar->codec_id == AV_CODEC_ID_MPEG4 || - (st->codecpar->codec_id == AV_CODEC_ID_H264 && (!enhanced_flv || type == PacketTypeCodedFrames)) || - (st->codecpar->codec_id == AV_CODEC_ID_HEVC && type == PacketTypeCodedFrames)) { - // sign extension - int32_t cts = (avio_rb24(s->pb) + 0xff800000) ^ 0xff800000; - pts = av_sat_add64(dts, cts); - if (cts < 0) { // dts might be wrong - if (!flv->wrong_dts) - av_log(s, AV_LOG_WARNING, - "Negative cts, previous timestamps might be wrong.\n"); - flv->wrong_dts = 1; - } else if (FFABS(dts - pts) > 1000*60*15) { - av_log(s, AV_LOG_WARNING, - "invalid timestamps %"PRId64" %"PRId64"\n", dts, pts); - dts = pts = AV_NOPTS_VALUE; + /* skip empty data track */ + if (!track_size) + goto next_track; + + ret = av_get_packet(s->pb, pkt, track_size); + if (ret < 0) + return ret; + + track_size -= ret; + size -= ret; + + pkt->dts = dts; + pkt->pts = pts == AV_NOPTS_VALUE ? dts : pts; + pkt->stream_index = st->index; + pkt->pos = pos; + if (!multitrack && flv->new_extradata[stream_type]) { + ret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, + flv->new_extradata[stream_type], + flv->new_extradata_size[stream_type]); + if (ret >= 0) { + flv->new_extradata[stream_type] = NULL; + flv->new_extradata_size[stream_type] = 0; } - size -= 3; - } - if (type == 0 && (!st->codecpar->extradata || st->codecpar->codec_id == AV_CODEC_ID_AAC || - st->codecpar->codec_id == AV_CODEC_ID_OPUS || st->codecpar->codec_id == AV_CODEC_ID_FLAC || - st->codecpar->codec_id == AV_CODEC_ID_H264 || st->codecpar->codec_id == AV_CODEC_ID_HEVC || - st->codecpar->codec_id == AV_CODEC_ID_AV1 || st->codecpar->codec_id == AV_CODEC_ID_VP9)) { - AVDictionaryEntry *t; - - if (st->codecpar->extradata) { - if ((ret = flv_queue_extradata(flv, s->pb, multitrack ? track_idx : stream_type, size, multitrack)) < 0) - return ret; - ret = FFERROR_REDO; - goto leave; + } else if (multitrack + && flv->mt_extradata_cnt > track_idx + && flv->mt_extradata[track_idx]) { + ret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, + flv->mt_extradata[track_idx], + flv->mt_extradata_sz[track_idx]); + if (ret >= 0) { + flv->mt_extradata[track_idx] = NULL; + flv->mt_extradata_sz[track_idx] = 0; } - if ((ret = flv_get_extradata(s, st, size)) < 0) - return ret; - - /* Workaround for buggy Omnia A/XE encoder */ - t = av_dict_get(s->metadata, "Encoder", NULL, 0); - if (st->codecpar->codec_id == AV_CODEC_ID_AAC && t && !strcmp(t->value, "Omnia A/XE")) - st->codecpar->extradata_size = 2; + } + if (stream_type == FLV_STREAM_TYPE_AUDIO && !enhanced_flv && + (sample_rate != flv->last_sample_rate || + channels != flv->last_channels)) { + flv->last_sample_rate = sample_rate; + flv->last_channels = channels; + ff_add_param_change(pkt, channels, 0, sample_rate, 0, 0); + } - ret = FFERROR_REDO; - goto leave; + if (stream_type == FLV_STREAM_TYPE_AUDIO || + (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_KEY || + stream_type == FLV_STREAM_TYPE_SUBTITLE || + stream_type == FLV_STREAM_TYPE_DATA) + pkt->flags |= AV_PKT_FLAG_KEY; + + ret = ff_buffer_packet(s, pkt); + if (ret < 0) + return ret; + res = FFERROR_REDO; + + if (track_size) { + av_log(s, AV_LOG_WARNING, "Track size mismatch: %d!\n", track_size); + avio_skip(s->pb, track_size); + size -= track_size; } - } - /* skip empty data packets */ - if (!size) { - ret = FFERROR_REDO; - goto leave; - } + if (!size) + break; - ret = av_get_packet(s->pb, pkt, size); - if (ret < 0) - return ret; - pkt->dts = dts; - pkt->pts = pts == AV_NOPTS_VALUE ? dts : pts; - pkt->stream_index = st->index; - pkt->pos = pos; - if (!multitrack && flv->new_extradata[stream_type]) { - int sret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, - flv->new_extradata[stream_type], - flv->new_extradata_size[stream_type]); - if (sret >= 0) { - flv->new_extradata[stream_type] = NULL; - flv->new_extradata_size[stream_type] = 0; +next_track: + if (multitrack_type == MultitrackTypeOneTrack) { + av_log(s, AV_LOG_ERROR, "Attempted to read next track in single-track mode.\n"); + return AVERROR_BUG; } - } else if (multitrack - && flv->mt_extradata_cnt > track_idx - && flv->mt_extradata[track_idx]) { - int sret = av_packet_add_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, - flv->mt_extradata[track_idx], - flv->mt_extradata_sz[track_idx]); - if (sret >= 0) { - flv->mt_extradata[track_idx] = NULL; - flv->mt_extradata_sz[track_idx] = 0; + + if (multitrack_type == MultitrackTypeManyTracksManyCodecs) { + codec_id = avio_rb32(s->pb); + size -= 4; } - } - if (stream_type == FLV_STREAM_TYPE_AUDIO && !enhanced_flv && - (sample_rate != flv->last_sample_rate || - channels != flv->last_channels)) { - flv->last_sample_rate = sample_rate; - flv->last_channels = channels; - ff_add_param_change(pkt, channels, 0, sample_rate, 0, 0); - } - if (stream_type == FLV_STREAM_TYPE_AUDIO || - (flags & FLV_VIDEO_FRAMETYPE_MASK) == FLV_FRAME_KEY || - stream_type == FLV_STREAM_TYPE_SUBTITLE || - stream_type == FLV_STREAM_TYPE_DATA) - pkt->flags |= AV_PKT_FLAG_KEY; + track_idx = avio_r8(s->pb); + size--; + } leave: last = avio_rb32(s->pb); @@ -1755,7 +1797,7 @@ leave: if (ret >= 0) flv->last_ts = pkt->dts; - return ret; + return ret ? ret : res; } static int flv_read_seek(AVFormatContext *s, int stream_index,