From patchwork Tue Dec 5 22:43:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 44936 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:9153:b0:181:818d:5e7f with SMTP id x19csp658943pzc; Tue, 5 Dec 2023 14:44:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IHKgLWrBVyoE2Zbwoo98gZQ75BZzVur2ZxChVTjy0iaVxd/1h+kSijQeKEm1GmVDXTYfUp0 X-Received: by 2002:a2e:130a:0:b0:2c9:f509:d81e with SMTP id 10-20020a2e130a000000b002c9f509d81emr25733ljt.0.1701816268413; Tue, 05 Dec 2023 14:44:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701816268; cv=none; d=google.com; s=arc-20160816; b=fDm6nwmpfuxeWQW04Y7D3tYtv/GOkWIFyYjX7/hvkypSRwjRDDTRGn8TPBTY2iyWWi 2O0iHs2zxhl+zT25w0AafebvR3hPgJKw8ermpR+C35LTxuw5YOTn+ZrnbtQ0/qnAwkiE FhcBodT5uAfIUENlLSJBRh6wG3B8kZeLne2dXuma6aD1g9XtdoRkOKsu3qdGoxSjHpxV 7ZdRc5aZ/b1mLblfKIxp2Oe8h2KHKK21zT0wO6MjgsHNCe9sgYjd88qFxveQMjN59dgJ XJtvxF8MRt0uTkcuhC9btf4t67Li1JCSHNiYGKWFKzUnmWROTlhkwZPZdOBQlZLgo2Yc 6HUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=r7CM24+BepWT9uLO2E4SFh6j6DJ4jDy09hM/jKZe8FY=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=owN8Kxs37gSl3iSuSgfu6N9xG9tuC4dv6aWj69LAEqalk7REpRsU6s6OCOj/enpMat CnWLLx+D4eo/gLHfsufHjyF0mejxOarsAVqfEv62b1wxj4YXA776abaMom5cENjg6OUF bruUevaZbIcg+Lub1oWangubMpmTNE5/WJzihcI29Z5PjWqbjPS3NyeI2ViGpqOqLAeO P6PMqjsB0bs+0HPtQViNm/5+URdFQPY1f8NlJLfkw7n5uLQC/eqR+agLeuNM8xv5hEFK W+wiJw/Dnw+kHIg7+azro9+MzGg35s+2Xpw4mqO+VxDPdpotv5348JcClG17Y5h1Hzjb MsfA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=mWrUXZwB; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id di8-20020a170906730800b00a1d18360a11si564770ejc.311.2023.12.05.14.44.27; Tue, 05 Dec 2023 14:44:28 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=mWrUXZwB; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 404E468CA8F; Wed, 6 Dec 2023 00:43:56 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0C6BD68CEFC for ; Wed, 6 Dec 2023 00:43:49 +0200 (EET) Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-58d08497aa1so4392010eaf.0 for ; Tue, 05 Dec 2023 14:43:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701816227; x=1702421027; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=OekuwNgONjjVHnJE+dDtobgDVcbuHsOjZx0p67JmfQY=; b=mWrUXZwBYo2sydqOXW/fZAUnQu94BkV6es1rP4kbEE2j/od/ulaEGr2u8uNA4s3Vsk o+YfwN7kcKt1vNUicGM2xr14k8dJUQAV+zekSgtxMusybEOAFSWemId+ncvF0lhYKa6g ogeArlTXElJ73K/zt1S0x+D98HFHmoFp7VxAFOOtTz1iIMGWogKddcehcm6+D0MAYisF Z0f4wtbsz4+2WPRu8m8GDwOqIiSHR2lBKYGn7NVrH2PEKAZ4SA3+KUZZird32VuOrQKc rWqO0nflktPOMcRS+Mqc26jJGtzuthpevw2to3ol3PzhnE5vR+9QWuo2umBtuMu1Phce oefg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701816227; x=1702421027; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OekuwNgONjjVHnJE+dDtobgDVcbuHsOjZx0p67JmfQY=; b=d7inl+Zc+jXN1F8h/5Ai3+4DbLUPaPrKVJyf2eN05H6n7X1RJFcVb4TnSgP3r7/wFz FtHib4oYJx1jWe/cf2M1L9jqWAcW7IS9R9ex0MAbz9IkjFebiOKvaaZz2LqnfamtGj9z VlT4NWyqrBeyMxvIDq4MuCfpJ5/vkSER0ZKzuf+kwdc5w3gNs8aWcjuhcRSG0l6fxSBx zWl7KQv2mQbZn4t/aEzrC4uwaECDxQt+C/fMWt9pIhviZdt39LG23lmXrOh/DJcDppKG 35jqk9IDzfbt/gtaxkNiPHcnOUtPMWy/OrEV84cqa06Z8j8zgeDv6ok1FYf6Hy/rp7TB U3pw== X-Gm-Message-State: AOJu0YxThMJx+9OOuNIiP73wHBOjV2mC9ROs6fSngA4ptekZAvDuaMBO LAWB26byoh6AgARzZjyGrzRDqJX/Fw4= X-Received: by 2002:a05:6358:2919:b0:170:8ef:d223 with SMTP id y25-20020a056358291900b0017008efd223mr54513rwb.25.1701816226571; Tue, 05 Dec 2023 14:43:46 -0800 (PST) Received: from localhost.localdomain (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id hq25-20020a056a00681900b0064fd4a6b306sm2037688pfb.76.2023.12.05.14.43.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Dec 2023 14:43:45 -0800 (PST) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Tue, 5 Dec 2023 19:43:57 -0300 Message-ID: <20231205224402.14540-4-jamrial@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231205224402.14540-1-jamrial@gmail.com> References: <20231205224402.14540-1-jamrial@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/8] ffmpeg: add support for muxing AVStreamGroups X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: V1zGFVZsNpt2 Starting with IAMF support. Signed-off-by: James Almer --- fftools/ffmpeg.h | 2 + fftools/ffmpeg_mux_init.c | 335 ++++++++++++++++++++++++++++++++++++++ fftools/ffmpeg_opt.c | 2 + 3 files changed, 339 insertions(+) diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index 41935d39d5..057535adbb 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -262,6 +262,8 @@ typedef struct OptionsContext { int nb_disposition; SpecifierOpt *program; int nb_program; + SpecifierOpt *stream_groups; + int nb_stream_groups; SpecifierOpt *time_bases; int nb_time_bases; SpecifierOpt *enc_time_bases; diff --git a/fftools/ffmpeg_mux_init.c b/fftools/ffmpeg_mux_init.c index 63a25a350f..7648f2a2f1 100644 --- a/fftools/ffmpeg_mux_init.c +++ b/fftools/ffmpeg_mux_init.c @@ -39,6 +39,7 @@ #include "libavutil/dict.h" #include "libavutil/display.h" #include "libavutil/getenv_utf8.h" +#include "libavutil/iamf.h" #include "libavutil/intreadwrite.h" #include "libavutil/log.h" #include "libavutil/mem.h" @@ -1943,6 +1944,336 @@ static int setup_sync_queues(Muxer *mux, AVFormatContext *oc, int64_t buf_size_u return 0; } +static int of_parse_iamf_audio_element_layers(Muxer *mux, AVStreamGroup *stg, char **ptr) +{ + AVIAMFAudioElement *audio_element = stg->params.iamf_audio_element; + AVDictionary *dict = NULL; + const char *token; + int ret = 0; + + audio_element->demixing_info = + av_iamf_param_definition_alloc(AV_IAMF_PARAMETER_DEFINITION_DEMIXING, 1, NULL); + audio_element->recon_gain_info = + av_iamf_param_definition_alloc(AV_IAMF_PARAMETER_DEFINITION_RECON_GAIN, 1, NULL); + + if (!audio_element->demixing_info || + !audio_element->recon_gain_info) + return AVERROR(ENOMEM); + + /* process manually set layers and parameters */ + token = av_strtok(NULL, ",", ptr); + while (token) { + const AVDictionaryEntry *e; + int demixing = 0, recon_gain = 0; + int layer = 0; + + if (av_strstart(token, "layer=", &token)) + layer = 1; + else if (av_strstart(token, "demixing=", &token)) + demixing = 1; + else if (av_strstart(token, "recon_gain=", &token)) + recon_gain = 1; + + av_dict_free(&dict); + ret = av_dict_parse_string(&dict, token, "=", ":", 0); + if (ret < 0) { + av_log(mux, AV_LOG_ERROR, "Error parsing audio element specification %s\n", token); + goto fail; + } + + if (layer) { + AVIAMFLayer *audio_layer = av_iamf_audio_element_add_layer(audio_element); + if (!audio_layer) { + av_log(mux, AV_LOG_ERROR, "Error adding layer to stream group %d\n", stg->index); + ret = AVERROR(ENOMEM); + goto fail; + } + av_opt_set_dict(audio_layer, &dict); + } else if (demixing || recon_gain) { + AVIAMFParamDefinition *param = demixing ? audio_element->demixing_info + : audio_element->recon_gain_info; + void *subblock = av_iamf_param_definition_get_subblock(param, 0); + + av_opt_set_dict(param, &dict); + av_opt_set_dict(subblock, &dict); + + /* Hardcode spec parameters */ + param->param_definition_mode = 0; + param->parameter_rate = stg->streams[0]->codecpar->sample_rate; + param->duration = + param->constant_subblock_duration = stg->streams[0]->codecpar->frame_size; + } + + // make sure that no entries are left in the dict + e = NULL; + if (e = av_dict_iterate(dict, e)) { + av_log(mux, AV_LOG_FATAL, "Unknown layer key %s.\n", e->key); + ret = AVERROR(EINVAL); + goto fail; + } + token = av_strtok(NULL, ",", ptr); + } + +fail: + av_dict_free(&dict); + if (!ret && !audio_element->nb_layers) { + av_log(mux, AV_LOG_ERROR, "No layer in audio element specification\n"); + ret = AVERROR(EINVAL); + } + + return ret; +} + +static int of_parse_iamf_submixes(Muxer *mux, AVStreamGroup *stg, char **ptr) +{ + AVFormatContext *oc = mux->fc; + AVIAMFMixPresentation *mix = stg->params.iamf_mix_presentation; + AVDictionary *dict = NULL; + const char *token; + char *submix_str = NULL; + int ret = 0; + + /* process manually set submixes */ + token = av_strtok(NULL, ",", ptr); + while (token) { + AVIAMFSubmix *submix = NULL; + const char *subtoken; + char *subptr = NULL; + + if (!av_strstart(token, "submix=", &token)) { + av_log(mux, AV_LOG_ERROR, "No submix in mix presentation specification \"%s\"\n", token); + goto fail; + } + + submix_str = av_strdup(token); + if (!submix_str) + goto fail; + + submix = av_iamf_mix_presentation_add_submix(mix); + if (!submix) { + av_log(mux, AV_LOG_ERROR, "Error adding submix to stream group %d\n", stg->index); + ret = AVERROR(ENOMEM); + goto fail; + } + submix->output_mix_config = + av_iamf_param_definition_alloc(AV_IAMF_PARAMETER_DEFINITION_MIX_GAIN, 0, NULL); + if (!submix->output_mix_config) { + ret = AVERROR(ENOMEM); + goto fail; + } + + submix->output_mix_config->parameter_rate = stg->streams[0]->codecpar->sample_rate; + + subptr = NULL; + subtoken = av_strtok(submix_str, "|", &subptr); + while (subtoken) { + const AVDictionaryEntry *e; + int element = 0, layout = 0; + + if (av_strstart(subtoken, "element=", &subtoken)) + element = 1; + else if (av_strstart(subtoken, "layout=", &subtoken)) + layout = 1; + + av_dict_free(&dict); + ret = av_dict_parse_string(&dict, subtoken, "=", ":", 0); + if (ret < 0) { + av_log(mux, AV_LOG_ERROR, "Error parsing submix specification \"%s\"\n", subtoken); + goto fail; + } + + if (element) { + AVIAMFSubmixElement *submix_element; + int idx = -1; + + if (e = av_dict_get(dict, "stg", NULL, 0)) + idx = strtol(e->value, NULL, 0); + av_dict_set(&dict, "stg", NULL, 0); + if (idx < 0 || idx >= oc->nb_stream_groups) { + av_log(mux, AV_LOG_ERROR, "Invalid or missing stream group index in " + "submix element specification \"%s\"\n", subtoken); + ret = AVERROR(EINVAL); + goto fail; + } + submix_element = av_iamf_submix_add_element(submix); + if (!submix_element) { + av_log(mux, AV_LOG_ERROR, "Error adding element to submix\n"); + ret = AVERROR(ENOMEM); + goto fail; + } + + submix_element->audio_element_id = oc->stream_groups[idx]->id; + + submix_element->element_mix_config = + av_iamf_param_definition_alloc(AV_IAMF_PARAMETER_DEFINITION_MIX_GAIN, 0, NULL); + if (!submix_element->element_mix_config) + ret = AVERROR(ENOMEM); + av_opt_set_dict2(submix_element, &dict, AV_OPT_SEARCH_CHILDREN); + submix_element->element_mix_config->parameter_rate = stg->streams[0]->codecpar->sample_rate; + } else if (layout) { + AVIAMFSubmixLayout *submix_layout = av_iamf_submix_add_layout(submix); + if (!submix_layout) { + av_log(mux, AV_LOG_ERROR, "Error adding layout to submix\n"); + ret = AVERROR(ENOMEM); + goto fail; + } + av_opt_set_dict(submix_layout, &dict); + } else + av_opt_set_dict2(submix, &dict, AV_OPT_SEARCH_CHILDREN); + + if (ret < 0) { + goto fail; + } + + // make sure that no entries are left in the dict + e = NULL; + while (e = av_dict_iterate(dict, e)) { + av_log(mux, AV_LOG_FATAL, "Unknown submix key %s.\n", e->key); + ret = AVERROR(EINVAL); + goto fail; + } + subtoken = av_strtok(NULL, "|", &subptr); + } + av_freep(&submix_str); + + if (!submix->nb_elements) { + av_log(mux, AV_LOG_ERROR, "No audio elements in submix specification \"%s\"\n", token); + ret = AVERROR(EINVAL); + } + token = av_strtok(NULL, ",", ptr); + } + +fail: + av_dict_free(&dict); + av_free(submix_str); + + return ret; +} + +static int of_add_groups(Muxer *mux, const OptionsContext *o) +{ + AVFormatContext *oc = mux->fc; + int ret; + + /* process manually set groups */ + for (int i = 0; i < o->nb_stream_groups; i++) { + AVDictionary *dict = NULL, *tmp = NULL; + const AVDictionaryEntry *e; + AVStreamGroup *stg = NULL; + int type; + const char *token; + char *str, *ptr = NULL; + const AVOption opts[] = { + { "type", "Set group type", offsetof(AVStreamGroup, type), AV_OPT_TYPE_INT, + { .i64 = 0 }, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM, "type" }, + { "iamf_audio_element", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = AV_STREAM_GROUP_PARAMS_IAMF_AUDIO_ELEMENT }, .unit = "type" }, + { "iamf_mix_presentation", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = AV_STREAM_GROUP_PARAMS_IAMF_MIX_PRESENTATION }, .unit = "type" }, + { NULL }, + }; + const AVClass class = { + .class_name = "StreamGroupType", + .item_name = av_default_item_name, + .option = opts, + .version = LIBAVUTIL_VERSION_INT, + }; + const AVClass *pclass = &class; + + str = av_strdup(o->stream_groups[i].u.str); + if (!str) + goto end; + + token = av_strtok(str, ",", &ptr); + if (token) { + ret = av_dict_parse_string(&dict, token, "=", ":", AV_DICT_MULTIKEY); + if (ret < 0) { + av_log(mux, AV_LOG_ERROR, "Error parsing group specification %s\n", token); + goto end; + } + + // "type" is not a user settable option in AVStreamGroup + e = av_dict_get(dict, "type", NULL, 0); + if (!e) { + av_log(mux, AV_LOG_ERROR, "No type define for Steam Group %d\n", i); + ret = AVERROR(EINVAL); + goto end; + } + + ret = av_opt_eval_int(&pclass, opts, e->value, &type); + if (ret < 0 || type == AV_STREAM_GROUP_PARAMS_NONE) { + av_log(mux, AV_LOG_ERROR, "Invalid group type \"%s\"\n", e->value); + goto end; + } + + av_dict_copy(&tmp, dict, 0); + stg = avformat_stream_group_create(oc, type, &tmp); + if (!stg) { + ret = AVERROR(ENOMEM); + goto end; + } + av_dict_set(&tmp, "type", NULL, 0); + + e = NULL; + while (e = av_dict_get(dict, "st", e, 0)) { + unsigned int idx = strtol(e->value, NULL, 0); + if (idx >= oc->nb_streams) { + av_log(mux, AV_LOG_ERROR, "Invalid stream index %d\n", idx); + ret = AVERROR(EINVAL); + goto end; + } + avformat_stream_group_add_stream(stg, oc->streams[idx]); + } + while (e = av_dict_get(dict, "stg", e, 0)) { + unsigned int idx = strtol(e->value, NULL, 0); + if (idx >= oc->nb_stream_groups || idx == stg->index) { + av_log(mux, AV_LOG_ERROR, "Invalid stream group index %d\n", idx); + ret = AVERROR(EINVAL); + goto end; + } + for (int j = 0; j < oc->stream_groups[idx]->nb_streams; j++) + avformat_stream_group_add_stream(stg, oc->stream_groups[idx]->streams[j]); + } + + switch(type) { + case AV_STREAM_GROUP_PARAMS_IAMF_AUDIO_ELEMENT: + ret = of_parse_iamf_audio_element_layers(mux, stg, &ptr); + break; + case AV_STREAM_GROUP_PARAMS_IAMF_MIX_PRESENTATION: + ret = of_parse_iamf_submixes(mux, stg, &ptr); + break; + default: + av_log(mux, AV_LOG_FATAL, "Unknown group type %d.\n", type); + ret = AVERROR(EINVAL); + break; + } + + if (ret < 0) + goto end; + + // make sure that nothing but "st" and "stg" entries are left in the dict + e = NULL; + while (e = av_dict_iterate(tmp, e)) { + if (!strcmp(e->key, "st") || !strcmp(e->key, "stg")) + continue; + + av_log(mux, AV_LOG_FATAL, "Unknown group key %s.\n", e->key); + ret = AVERROR(EINVAL); + goto end; + } + } + +end: + av_dict_free(&dict); + av_dict_free(&tmp); + av_free(str); + if (ret < 0) + return ret; + } + + return 0; +} + static int of_add_programs(Muxer *mux, const OptionsContext *o) { AVFormatContext *oc = mux->fc; @@ -2740,6 +3071,10 @@ int of_open(const OptionsContext *o, const char *filename) if (err < 0) return err; + err = of_add_groups(mux, o); + if (err < 0) + return err; + err = of_add_programs(mux, o); if (err < 0) return err; diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index 304471dd03..1144f64f89 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -1491,6 +1491,8 @@ const OptionDef options[] = { "add metadata", "string=string" }, { "program", HAS_ARG | OPT_STRING | OPT_SPEC | OPT_OUTPUT, { .off = OFFSET(program) }, "add program with specified streams", "title=string:st=number..." }, + { "stream_group", HAS_ARG | OPT_STRING | OPT_SPEC | OPT_OUTPUT, { .off = OFFSET(stream_groups) }, + "add stream group with specified streams and group type-specific arguments", "id=number:st=number..." }, { "dframes", HAS_ARG | OPT_PERFILE | OPT_EXPERT | OPT_OUTPUT, { .func_arg = opt_data_frames }, "set the number of data frames to output", "number" },