From patchwork Tue Jan 30 17:32:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 45913 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:8786:b0:199:de12:6fa6 with SMTP id ph6csp2315290pzb; Tue, 30 Jan 2024 09:33:03 -0800 (PST) X-Google-Smtp-Source: AGHT+IE5VUFECuwq/NxR5bN9hN1fa5wtbKoIgy+G61M5EaqllYcAzQanU9xNlyGhOByuFe2aBFJz X-Received: by 2002:a17:906:b347:b0:a34:b125:7975 with SMTP id cd7-20020a170906b34700b00a34b1257975mr1865023ejb.27.1706635983473; Tue, 30 Jan 2024 09:33:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1706635983; cv=none; d=google.com; s=arc-20160816; b=IgqtmiGMrTZwqTiiCNieXlDD4au2gxh/Uz5jxsLs/wTwuTuiRJvGr4UQvpPnf27J6W Vr7BtNDJWLHHJsmVi0asdtyAHoA/HtlzbsGZwAYSthmtAJOzGrsWrit29yCTsvFTSbGt dRLlrzYrSzGJyQcBMprZwhgKOTSFfwz/LVauMqpD1KuSjrNFPqXJM8uK73qX+pIXmX/3 sOxAy84uqT7MQZtMwPs3/g9cXOkFAThPmrVBc5mwkwE3H4l7po65azqAQzEXhPIQy5oD MYy1MgPYL6uRDP5OKOVDv4prAcIFU7r1gKVpdUu2zvygZGw1wB/af7WkwcUY8lJMXM+o dgzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=Ty3LxkOgloGf6QStq3G5rGySIGO0883LRCb823LhBEA=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=dDgXiXXP5jlv7GQSwvvElTvzkT2cqQDdaz7eBd7724H/xHyVl4acoKIrM66AGl6Lx6 wuWjlgTqOjmsR28x7LW6fOlUhJ6w9QyNIEHUjGa+E6n7voAJ794IvIVevIfXoTB2zG+q T2sjJmxagNLNdGzuPYnW7I93RfIj+y2654xhMw6dcQuFR+obqx3EIr5U1aDYJ9UXvQwJ j91jOBw52MXdbSW1IGgPRjtDqaoH+DWIBxXZ0pnyjE6HNDhSmBuP4ePa/hTSpdp/INjI uEKvDvVwZ+doBtsjs/QO/9TD2qk+M6i0tBwyGLHN35GWCa5PTudWCYC22HGR8HtIAFNd cAOQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=QAvlA7i2; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id zh17-20020a170906881100b00a34976db3e9si4625953ejb.1047.2024.01.30.09.33.02; Tue, 30 Jan 2024 09:33:03 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=QAvlA7i2; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A39C068D118; Tue, 30 Jan 2024 19:32:23 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 016C168D0CA for ; Tue, 30 Jan 2024 19:32:15 +0200 (EET) Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1d8aadc624dso19873085ad.0 for ; Tue, 30 Jan 2024 09:32:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706635933; x=1707240733; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=JL7/QrDolVSn7rxngFVHyXnfJW7oGfGONyahMUxvmx4=; b=QAvlA7i2qhk0htDFm3p4YmiEWkEdl6Bra1nEHQtuBgPBW6kFxqsNGcg4ng0EuPDAU9 LDyRhN2f8qtwOdstFxHTTicorRdIMC/UGI8dqyNHkxRMaq4qby/CEJlAP0D44xPok6RA zfV2LIQtJOp3/baTOvx6PWmi57ERxk26eDkjn7kh+Km2k5BlXZzQCBXm12HaJX6xTQI2 ur2w/fGdaSFeE743EKWspVAn8DDdbI+RnuDLrazPoBJaQGURJi7x8Pa6gqJOgvt0hUyU d6230yx7L7RmYkj9DzyFhuvOhA+OFCgewr1wpZFZ/jll2B8tx4uzOPbdiPZPA6wUhTQZ UlWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706635933; x=1707240733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JL7/QrDolVSn7rxngFVHyXnfJW7oGfGONyahMUxvmx4=; b=DInKV37qJuKrS5fIZ91UxFRhKlu8pRPovprX8GnQxRIMlsEKSns7FMdNhvr5LpTPr9 MW9NU5FQ1B27ZK9yYif09r/l9cpvVW3nW0jby3BV4UyEDuxd7Iq2diYoijEXivEibUtH rUoijOAwnzfMGdJWtTKXR2Ce5EYS34Hd+Bqfp86E5lyqfmrvs/uCNZj1R+p29Xh1FAi8 cG8VQFj+TK2jQIrnct4IGs/PCNuQYvPnKg9SabShIjxCroxnn90Kh/KXw5aAj2F72gLF G0NPHp0D+ILnYh0gQZJAS6jqVm2f+LD7b6VK/w3rtJhzErLDaYqy5GSk8bho79ZD5mGC ZiaA== X-Gm-Message-State: AOJu0YzH2KoamAucTYat9cH/TVNqg5KNnrGYEDbZOhaiYbA4TQHTOPra tlcaoaLlcm9oaSb3WeA+ULEU7s4aWYQ5v8YuKY2q/0mmA5Stavh2fM2B3eyh X-Received: by 2002:a17:903:181:b0:1d8:dcd2:663 with SMTP id z1-20020a170903018100b001d8dcd20663mr5246050plg.19.1706635933317; Tue, 30 Jan 2024 09:32:13 -0800 (PST) Received: from localhost.localdomain (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id k12-20020a170902f28c00b001d73702e0b7sm4107415plc.212.2024.01.30.09.32.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jan 2024 09:32:12 -0800 (PST) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Jan 2024 14:32:16 -0300 Message-ID: <20240130173218.63297-5-jamrial@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240130173218.63297-1-jamrial@gmail.com> References: <20240130173218.63297-1-jamrial@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 5/7] avcodec: add an Immersive Audio Model and Formats frame merge bsf X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: S8hM443mYZ6n Signed-off-by: James Almer --- libavcodec/bitstream_filters.c | 1 + libavcodec/bsf/Makefile | 1 + libavcodec/bsf/iamf_stream_merge_bsf.c | 227 +++++++++++++++++++++++++ 3 files changed, 229 insertions(+) create mode 100644 libavcodec/bsf/iamf_stream_merge_bsf.c diff --git a/libavcodec/bitstream_filters.c b/libavcodec/bitstream_filters.c index 640b821413..cc3895295e 100644 --- a/libavcodec/bitstream_filters.c +++ b/libavcodec/bitstream_filters.c @@ -42,6 +42,7 @@ extern const FFBitStreamFilter ff_h264_redundant_pps_bsf; extern const FFBitStreamFilter ff_hapqa_extract_bsf; extern const FFBitStreamFilter ff_hevc_metadata_bsf; extern const FFBitStreamFilter ff_hevc_mp4toannexb_bsf; +extern const FFBitStreamFilter ff_iamf_stream_merge_bsf; extern const FFBitStreamFilter ff_iamf_stream_split_bsf; extern const FFBitStreamFilter ff_imx_dump_header_bsf; extern const FFBitStreamFilter ff_media100_to_mjpegb_bsf; diff --git a/libavcodec/bsf/Makefile b/libavcodec/bsf/Makefile index 80dcdf94fb..34e26f1bc4 100644 --- a/libavcodec/bsf/Makefile +++ b/libavcodec/bsf/Makefile @@ -21,6 +21,7 @@ OBJS-$(CONFIG_HAPQA_EXTRACT_BSF) += bsf/hapqa_extract.o OBJS-$(CONFIG_HEVC_METADATA_BSF) += bsf/h265_metadata.o OBJS-$(CONFIG_HEVC_MP4TOANNEXB_BSF) += bsf/hevc_mp4toannexb.o OBJS-$(CONFIG_IAMF_STREAM_MERGE_BSF) += bsf/iamf_stream_merge_bsf.o +OBJS-$(CONFIG_IAMF_STREAM_SPLIT_BSF) += bsf/iamf_stream_split_bsf.o OBJS-$(CONFIG_IMX_DUMP_HEADER_BSF) += bsf/imx_dump_header.o OBJS-$(CONFIG_MEDIA100_TO_MJPEGB_BSF) += bsf/media100_to_mjpegb.o OBJS-$(CONFIG_MJPEG2JPEG_BSF) += bsf/mjpeg2jpeg.o diff --git a/libavcodec/bsf/iamf_stream_merge_bsf.c b/libavcodec/bsf/iamf_stream_merge_bsf.c new file mode 100644 index 0000000000..c62698d374 --- /dev/null +++ b/libavcodec/bsf/iamf_stream_merge_bsf.c @@ -0,0 +1,227 @@ +/* + * Copyright (c) 2023 James Almer + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include +#include + +#include "libavutil/dict.h" +#include "libavutil/fifo.h" +#include "libavutil/opt.h" +#include "libavformat/iamf.h" +#include "bsf.h" +#include "bsf_internal.h" +#include "bytestream.h" +#include "get_bits.h" +#include "put_bits.h" + +typedef struct IAMFMergeContext { + AVClass *class; + + AVFifo *fifo; + + // AVOptions + AVDictionary *index_mapping; + int stream_count; + int out_index; +} IAMFMergeContext; + +static int find_id_from_idx(AVBSFContext *ctx, int idx) +{ + IAMFMergeContext *const c = ctx->priv_data; + const AVDictionaryEntry *e = NULL; + + while (e = av_dict_iterate(c->index_mapping, e)) { + char *endptr = NULL; + int id, map_idx = strtol(e->key, &endptr, 0); + if (!endptr || *endptr) + return AVERROR_INVALIDDATA; + endptr = NULL; + id = strtol(e->value, &endptr, 0); + if (!endptr || *endptr) + return AVERROR_INVALIDDATA; + if (map_idx == idx) + return id; + } + + av_log(ctx, AV_LOG_ERROR, "Invalid stream idx %d\n", idx); + return AVERROR_INVALIDDATA; +} + +static int iamf_stream_merge_filter(AVBSFContext *ctx, AVPacket *out) +{ + IAMFMergeContext *const c = ctx->priv_data; + AVPacket *pkt; + int ret; + + while (av_fifo_can_write(c->fifo)) { + ret = ff_bsf_get_packet(ctx, &pkt); + if (ret < 0) + return ret; + av_fifo_write(c->fifo, &pkt, 1); + } + + pkt = NULL; + while (av_fifo_can_read(c->fifo)) { + PutBitContext pb; + PutByteContext p; + uint8_t *side_data, header[MAX_IAMF_OBU_HEADER_SIZE], obu[8]; + unsigned int obu_header; + unsigned int skip_samples = 0, discard_padding = 0; + size_t side_data_size; + int header_size, obu_size, old_out_size = out->size; + int id, type; + + av_packet_free(&pkt); + av_fifo_read(c->fifo, &pkt, 1); + id = find_id_from_idx(ctx, pkt->stream_index); + if (id < 0) + return AVERROR_INVALIDDATA; + + type = id <= 17 ? id + IAMF_OBU_IA_AUDIO_FRAME_ID0 : IAMF_OBU_IA_AUDIO_FRAME; + + side_data = av_packet_get_side_data(pkt, AV_PKT_DATA_SKIP_SAMPLES, + &side_data_size); + + if (side_data && side_data_size >= 10) { + skip_samples = AV_RL32(side_data); + discard_padding = AV_RL32(side_data + 4); + } + + init_put_bits(&pb, (uint8_t *)&obu_header, sizeof(obu_header)); + put_bits(&pb, 5, type); + put_bits(&pb, 1, 0); // obu_redundant_copy + put_bits(&pb, 1, skip_samples || discard_padding); + put_bits(&pb, 1, 0); // obu_extension_flag + flush_put_bits(&pb); + + init_put_bits(&pb, header, sizeof(header)); + if (skip_samples || discard_padding) { + put_leb(&pb, discard_padding); + put_leb(&pb, skip_samples); + } + if (id > 17) + put_leb(&pb, id); + flush_put_bits(&pb); + + header_size = put_bytes_count(&pb, 1); + + init_put_bits(&pb, obu, sizeof(obu)); + put_leb(&pb, header_size + pkt->size); + flush_put_bits(&pb); + + obu_size = put_bytes_count(&pb, 1); + + ret = av_grow_packet(out, 1 + obu_size + header_size + pkt->size); + if (ret < 0) + goto fail; + + bytestream2_init_writer(&p, out->data + old_out_size, 1 + obu_size + header_size + pkt->size); + bytestream2_put_byteu(&p, obu_header); + bytestream2_put_bufferu(&p, obu, obu_size); + bytestream2_put_bufferu(&p, header, header_size); + bytestream2_put_bufferu(&p, pkt->data, pkt->size); + } + + ret = av_packet_copy_props(out, pkt); + if (ret < 0) + goto fail; + out->stream_index = c->out_index; + + ret = 0; +fail: + av_packet_free(&pkt); + if (ret < 0) + av_packet_free(&out); + return ret; +} + +static int iamf_stream_merge_init(AVBSFContext *ctx) +{ + IAMFMergeContext *const c = ctx->priv_data; + + if (!c->index_mapping) { + av_log(ctx, AV_LOG_ERROR, "Empty index map\n"); + return AVERROR(EINVAL); + } + + c->fifo = av_fifo_alloc2(av_dict_count(c->index_mapping), sizeof(AVPacket*), 0); + if (!c->fifo) + return AVERROR(ENOMEM); + + return 0; +} + +static void iamf_stream_merge_flush(AVBSFContext *ctx) +{ + IAMFMergeContext *const c = ctx->priv_data; + + while (av_fifo_can_read(c->fifo)) { + AVPacket *pkt; + av_fifo_read(c->fifo, &pkt, 1); + av_packet_free(&pkt); + } + av_fifo_reset2(c->fifo); +} + +static void iamf_stream_merge_close(AVBSFContext *ctx) +{ + IAMFMergeContext *const c = ctx->priv_data; + + if (c->fifo) + iamf_stream_merge_flush(ctx); + av_fifo_freep2(&c->fifo); +} + +#define OFFSET(x) offsetof(IAMFMergeContext, x) +#define FLAGS (AV_OPT_FLAG_AUDIO_PARAM|AV_OPT_FLAG_BSF_PARAM) +static const AVOption iamf_stream_merge_options[] = { + { "index_mapping", "a :-separated list of stream_index=audio_substream_id entries " + "to set stream id in output Audio Frame OBUs", + OFFSET(index_mapping), AV_OPT_TYPE_DICT, { .str = NULL }, 0, 0, FLAGS }, + { "out_index", "Stream index to in output packets", + OFFSET(out_index), AV_OPT_TYPE_INT, { 0 }, 0, INT_MAX, FLAGS }, + { NULL } +}; + +static const AVClass iamf_stream_merge_class = { + .class_name = "iamf_stream_merge_bsf", + .item_name = av_default_item_name, + .option = iamf_stream_merge_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +static const enum AVCodecID iamf_stream_merge_codec_ids[] = { + AV_CODEC_ID_PCM_S16LE, AV_CODEC_ID_PCM_S16BE, + AV_CODEC_ID_PCM_S24LE, AV_CODEC_ID_PCM_S24BE, + AV_CODEC_ID_PCM_S32LE, AV_CODEC_ID_PCM_S32BE, + AV_CODEC_ID_OPUS, AV_CODEC_ID_AAC, + AV_CODEC_ID_FLAC, AV_CODEC_ID_NONE, +}; + +const FFBitStreamFilter ff_iamf_stream_merge_bsf = { + .p.name = "iamf_stream_merge", + .p.codec_ids = iamf_stream_merge_codec_ids, + .p.priv_class = &iamf_stream_merge_class, + .priv_data_size = sizeof(IAMFMergeContext), + .init = iamf_stream_merge_init, + .flush = iamf_stream_merge_flush, + .close = iamf_stream_merge_close, + .filter = iamf_stream_merge_filter, +};