From patchwork Tue May 21 09:02:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 49096 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:3a48:b0:1af:fc2d:ff5a with SMTP id zu8csp5097713pzb; Tue, 21 May 2024 02:06:03 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWtyOIFvnn2xBCcy0X0VQ5ITp95N5eNnLG5cFzotJEtI/1wz4naDO+O08vgfdCXeMwMqHzCScVWHrWvaK36vZgETjyrG+B21CHYJw== X-Google-Smtp-Source: AGHT+IFfFHmZlwcOh4Cchwx8y3weMDv9cIj4Z7pSb6/+Kza0AYIpO3wUlUk/eFwyJidbidI4rHl7 X-Received: by 2002:ac2:52b0:0:b0:51f:9549:9c0d with SMTP id 2adb3069b0e04-52210278795mr27170598e87.48.1716282363133; Tue, 21 May 2024 02:06:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716282363; cv=none; d=google.com; s=arc-20160816; b=QgPxfO29e2dMuRAnfD5CkL7FaCYrkft3uLRJ1C69UYt1cL9TGCW9lTbrhFAPi7iu7/ x3R5OVUCE2wXfNsxwksWIuYElcUbWa+srJohZrhBo0y1UWD75ewFmqk8dCZZJSPiO6Ap +d4YAtCy22ROZopfG23y6lTjSoxMTZrtU5EgRS3dni4H6QiLWRSrlqPekKpAEO7PcsLB K1+SHNSPFJGWPY2v3Ra7g9Mi+2KRMAHuPoiRSi5yakI17MHox9ilBkaClX+40Wk06y+B r36SITLzbEWNfRSrDWdtoem7fcmybgWmMmuJyTcmvnu+dcCSU3qHeeu0Ummk/qBfWb8c 178w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=B/ynEYiWdRzM4ZVa1EM8OE9Oquz2H1bUC6xGVzfMr9s=; fh=LnlYe9qYwgML7nWWXqAumr7YCmPjjpEPjQf6GasgJC0=; b=G0SHlXs1yxXjaOWf2Ja0/3W0dzSEybv92vTj4FGTnYTzRmd1/hBctnkJwdtJPFV14A bjVbH3rfx5tTvw4IRH8zY5Jelm3OXmWh7bmLAIbBlfYXitxsVwwlek16XYGqI1iJ3M4T id4d6FqVk+G7f6t/G8IB55KL8UaKvUfr3ADdXpq5BQCPWz9f1ANFGxdWWvlee6hLFDI4 8FjcC8Ka9yodq7siZfDXZI0RR2+/xwqJx09mUX8/4e1Jt8VMe6Cim7wbOMdhfsgru/MB RtLpfir6bsAQBwyh0zvlQTLzWLt/Oa1yjsa9OWiRnj8SM78ykTJlVy/tCsPLllDpOgfx 3hTg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=kbefL6JI; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a640c23a62f3a-a5d1079016bsi447271966b.591.2024.05.21.02.06.02; Tue, 21 May 2024 02:06:03 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=kbefL6JI; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 29F1A68D428; Tue, 21 May 2024 12:03:59 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [144.76.60.213]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6BC0E68D3A1 for ; Tue, 21 May 2024 12:03:43 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id 151D827FFD299; Tue, 21 May 2024 11:03:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1716282219; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IA/sOPRBNpG3crCFlmE4JrSaO9nKC0pACnxE9/XsPo4=; b=kbefL6JIcaslWJQK4c3Q4fwTTBFYCceEtJHqaAipQ1Hsgqzu1NzbrOYmltroi5f4muQDbh N4eiLWNDvicagCMXRs2OGQugM61rPVm9MpODZDI8c75Gh09yvie7LruLU/E5Y332hfAAVo MhcIdHyTXjJy2tKsN6yE9B7EYZjqI4MSGsHA74D42L9JC1ldpuJ2zxyIIFtJGAYHwEfmNr mPJNGcMJ50TbePknKXBbQrkrwnnOjIN7PwUfcmnG4pJtdZ5k5vagY3mRoeWdCctynR6U4w QkfrPfmAcBlv4TZJr1Cq8rSgljsvpSGlFXB8mvu4Zu0bxPzmlC2aXi1jt+h4DA== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 May 2024 11:02:18 +0200 Message-ID: <20240521090316.782-10-timo@rothenpieler.org> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240521090316.782-1-timo@rothenpieler.org> References: <20240521090316.782-1-timo@rothenpieler.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 09/13] avformat/flvenc: add support for writing multi track audio X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: jw2bCbnr173L --- libavformat/flvenc.c | 90 +++++++++++++++++++++++++++++++++----------- 1 file changed, 67 insertions(+), 23 deletions(-) diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c index 056940afc5..ab729702a5 100644 --- a/libavformat/flvenc.c +++ b/libavformat/flvenc.c @@ -131,7 +131,7 @@ typedef struct FLVContext { int flags; int64_t *last_ts; int *metadata_pkt_written; - int *video_track_idx_map; + int *track_idx_map; } FLVContext; static int get_audio_flags(AVFormatContext *s, AVCodecParameters *par) @@ -764,19 +764,33 @@ static int flv_get_multichannel_body_size(AVCodecParameters* par) return res; } -static void flv_write_multichannel_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts) +static void flv_write_multichannel_header(AVFormatContext* s, AVCodecParameters* par, int64_t ts, int stream_index) { AVIOContext *pb = s->pb; + FLVContext *flv = s->priv_data; + + int track_idx = flv->track_idx_map[stream_index]; int data_size = flv_get_multichannel_body_size(par); + if (track_idx) + data_size += 2; avio_w8(pb, FLV_TAG_TYPE_AUDIO); avio_wb24(pb, data_size + 5); // size put_timestamp(pb, ts); avio_wb24(pb, 0); // streamid - avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultichannelConfig); + if (track_idx) { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultitrack); + avio_w8(pb, MultitrackTypeOneTrack | AudioPacketTypeMultichannelConfig); + } else { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultichannelConfig); + } + write_codec_fourcc(pb, par->codec_id); + if (track_idx) + avio_w8(pb, track_idx); + flv_write_multichannel_body(s, par); avio_wb32(pb, data_size + 5 + 11); // previous tag size @@ -786,6 +800,7 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i int64_t data_size; AVIOContext *pb = s->pb; FLVContext *flv = s->priv_data; + int track_idx = flv->track_idx_map[stream_index]; int extended_flv = 0; if (par->codec_id == AV_CODEC_ID_AAC || par->codec_id == AV_CODEC_ID_H264 @@ -802,15 +817,26 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i avio_wb24(pb, 0); // streamid pos = avio_tell(pb); if (par->codec_type == AVMEDIA_TYPE_AUDIO) { - extended_flv = par->codec_id == AV_CODEC_ID_OPUS - || par->codec_id == AV_CODEC_ID_FLAC - || par->codec_id == AV_CODEC_ID_AC3 - || par->codec_id == AV_CODEC_ID_EAC3; + extended_flv = (par->codec_id == AV_CODEC_ID_AAC && track_idx) + || (par->codec_id == AV_CODEC_ID_MP3 && track_idx) + || par->codec_id == AV_CODEC_ID_OPUS + || par->codec_id == AV_CODEC_ID_FLAC + || par->codec_id == AV_CODEC_ID_AC3 + || par->codec_id == AV_CODEC_ID_EAC3; if (extended_flv) { - avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeSequenceStart); + if (track_idx) { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultitrack); + avio_w8(pb, MultitrackTypeOneTrack | AudioPacketTypeSequenceStart); + } else { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeSequenceStart); + } + write_codec_fourcc(pb, par->codec_id); + if (track_idx) + avio_w8(pb, track_idx); + if (par->codec_id == AV_CODEC_ID_AAC) { flv_write_aac_header(s, par); } else if (par->codec_id == AV_CODEC_ID_OPUS || par->codec_id == AV_CODEC_ID_FLAC) { @@ -824,7 +850,6 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i flv_write_aac_header(s, par); } } else { - int track_idx = flv->video_track_idx_map[stream_index]; // If video stream has track_idx > 0 we need to send H.264 as extended video packet extended_flv = (par->codec_id == AV_CODEC_ID_H264 && track_idx) || par->codec_id == AV_CODEC_ID_HEVC @@ -868,7 +893,7 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i if (par->codec_type == AVMEDIA_TYPE_AUDIO && (extended_flv || (av_channel_layout_compare(&par->ch_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO) == 1 && av_channel_layout_compare(&par->ch_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_MONO) == 1))) - flv_write_multichannel_header(s, par, ts); + flv_write_multichannel_header(s, par, ts, stream_index); } static int flv_append_keyframe_info(AVFormatContext *s, FLVContext *flv, double ts, int64_t pos) @@ -930,12 +955,12 @@ static int shift_data(AVFormatContext *s) static int flv_init(struct AVFormatContext *s) { int i; - int video_ctr = 0; + int video_ctr = 0, audio_ctr = 0; FLVContext *flv = s->priv_data; flv->last_ts = av_mallocz(sizeof(*flv->last_ts) * s->nb_streams); flv->metadata_pkt_written = av_mallocz(sizeof(*flv->metadata_pkt_written) * s->nb_streams); - flv->video_track_idx_map = av_mallocz(sizeof(*flv->video_track_idx_map) * s->nb_streams); + flv->track_idx_map = av_mallocz(sizeof(*flv->track_idx_map) * s->nb_streams); for (i = 0; i < s->nb_streams; i++) { AVCodecParameters *par = s->streams[i]->codecpar; @@ -946,7 +971,7 @@ static int flv_init(struct AVFormatContext *s) s->streams[i]->avg_frame_rate.num) { flv->framerate = av_q2d(s->streams[i]->avg_frame_rate); } - flv->video_track_idx_map[i] = video_ctr++; + flv->track_idx_map[i] = video_ctr++; if (flv->video_par && flv->flags & FLV_ADD_KEYFRAME_INDEX) { av_log(s, AV_LOG_ERROR, "at most one video stream is supported in flv with keyframe index\n"); @@ -977,12 +1002,22 @@ static int flv_init(struct AVFormatContext *s) } break; case AVMEDIA_TYPE_AUDIO: - if (flv->audio_par) { - av_log(s, AV_LOG_ERROR, - "at most one audio stream is supported in flv\n"); + if (audio_ctr && + par->codec_id != AV_CODEC_ID_AAC && + par->codec_id != AV_CODEC_ID_MP3 && + par->codec_id != AV_CODEC_ID_OPUS && + par->codec_id != AV_CODEC_ID_FLAC && + par->codec_id != AV_CODEC_ID_AC3 && + par->codec_id != AV_CODEC_ID_EAC3) { + av_log(s, AV_LOG_ERROR, "Unsupported multi-track codec.\n"); return AVERROR(EINVAL); } - flv->audio_par = par; + flv->track_idx_map[i] = audio_ctr++; + if (flv->audio_par) + av_log(s, AV_LOG_WARNING, + "more than one audio stream is not supported by most flv demuxers.\n"); + else + flv->audio_par = par; if (get_audio_flags(s, par) < 0) return unsupported_codec(s, "Audio", par->codec_id); if (par->codec_id == AV_CODEC_ID_PCM_S16BE) @@ -1154,9 +1189,11 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) uint8_t frametype = pkt->flags & AV_PKT_FLAG_KEY ? FLV_FRAME_KEY : FLV_FRAME_INTER; int flags = -1, flags_size, ret = 0; int64_t cur_offset = avio_tell(pb); - int track_idx = flv->video_track_idx_map[pkt->stream_index]; + int track_idx = flv->track_idx_map[pkt->stream_index]; - int extended_audio = par->codec_id == AV_CODEC_ID_OPUS + int extended_audio = (par->codec_id == AV_CODEC_ID_AAC && track_idx) + || (par->codec_id == AV_CODEC_ID_MP3 && track_idx) + || par->codec_id == AV_CODEC_ID_OPUS || par->codec_id == AV_CODEC_ID_FLAC || par->codec_id == AV_CODEC_ID_AC3 || par->codec_id == AV_CODEC_ID_EAC3; @@ -1173,8 +1210,8 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) else flags_size = 1; - if (par->codec_type == AVMEDIA_TYPE_VIDEO && track_idx) - flags_size += 2; // additional header bytes for multi-track video + if ((par->codec_type == AVMEDIA_TYPE_VIDEO || par->codec_type == AVMEDIA_TYPE_AUDIO) && track_idx) + flags_size += 2; // additional header bytes for multi-track flv if ((par->codec_id == AV_CODEC_ID_HEVC || (par->codec_id == AV_CODEC_ID_H264 && track_idx)) @@ -1341,8 +1378,15 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt) if (h2645 && pkttype == PacketTypeCodedFrames) avio_wb24(pb, pkt->pts - pkt->dts); } else if (extended_audio) { - avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeCodedFrames); + if (track_idx) { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeMultitrack); + avio_w8(pb, MultitrackTypeOneTrack | AudioPacketTypeCodedFrames); + } else { + avio_w8(pb, FLV_CODECID_EX_HEADER | AudioPacketTypeCodedFrames); + } write_codec_fourcc(pb, par->codec_id); + if (track_idx) + avio_w8(pb, track_idx); } else { av_assert1(flags>=0); avio_w8(pb, flags); @@ -1432,7 +1476,7 @@ static void flv_deinit(AVFormatContext *s) av_freep(&flv->last_ts); av_freep(&flv->metadata_pkt_written); - av_freep(&flv->video_track_idx_map); + av_freep(&flv->track_idx_map); } static const AVOption options[] = {