From patchwork Sun Nov 26 01:28:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 44801 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:bca6:b0:181:818d:5e7f with SMTP id fx38csp2452169pzb; Sat, 25 Nov 2023 17:30:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IF/mA4CRUyF59BA4+QO0S7KnkkzxcHPBt0BMX/ms6iJ4wBnOsy3HxwmvtuSKUpX7YPhTyMN X-Received: by 2002:a17:906:da:b0:9e5:1db7:31b1 with SMTP id 26-20020a17090600da00b009e51db731b1mr5237791eji.2.1700962240571; Sat, 25 Nov 2023 17:30:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700962240; cv=none; d=google.com; s=arc-20160816; b=rqCqTRocQvaicf+T0ehoJFZwKZVfpk0llREysF3rl4eJJ5K4pDFVtipBrEzfUBdhSs 36b1wt2wvCfhfzDNK8e3Ox+kAfXI4k69I42alfa8odp+KhuXFQn0/wJKWCnnUvn6jPhl KpKWhkAGEwDQgF3PxcSoIwPfIyS2d+hunoXqZLXg3eJFrf24Zy2TKjGwEIHiCk5Z0iBu UtPZyICrRbFepuEQ4c8w815xkF7WTlkKUMbc1XD1kJukkOQcTbnbgkCefc2nv6urDpgP zhidI/DFvCuKSFRcbJyBYWvkeDZHj/r2oNKgmu34jmL9JKDOUl71PTn5i1eIHOjf7VBv 8A/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=KCY4TlUfPP3xohhuwIasSwC0F29kkNAWf4V3mx1z8DY=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=YXmfK8P5Wg3Zpg8pAy8I9Yt5qIx+QaMwSd1bdOJo7D8/w9/FPLprMCmJWLewcjlR54 gZt+tUnLTXXn64B3Q6BV3ZVWThlG/5Esi5qirc18bECuzvoakzeZWi0KMftaxT1IrB3v RHOhoUwY1JdbrqfgDIOhHqimFvFuYjWGIEToLDQXxAeVki+1MlvqIPP6IYNyinHrBVIW W6Ozt056f3Hjzo6SOyRBjPyKGPqFYbdGi4qwSn/a4myvJg9q8AD3FcIWZnUMmrK4am5Z f7gua788BI+GIcIYSivT6gtsRwXzwSeTSGCztwPAGAWcy6S7sa/EYbxQgiCfWrUddlz7 MiRw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=TyUZZx32; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ck23-20020a170906c45700b00a0c68b554a7si819583ejb.498.2023.11.25.17.30.40; Sat, 25 Nov 2023 17:30:40 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=TyUZZx32; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 01C6B68CF4B; Sun, 26 Nov 2023 03:29:40 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B149068CF65 for ; Sun, 26 Nov 2023 03:29:29 +0200 (EET) Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-3b83398cfc7so1959367b6e.3 for ; Sat, 25 Nov 2023 17:29:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700962167; x=1701566967; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=GIgGdnhjdF8feXtt+F2EvuYtL54sQh4ya1Crg+4P6do=; b=TyUZZx32ivVrXK8/JrL09g+ZHZUemxcJbYHdl4S7qmTF2p3Y3gNwDaAa3O0dk03igg 4bWvV7E6MzKohupdSVUxcIL7ARBi5KlDjuzstHqTPkA+zBJf9OfUE5dkzphU0aes4/jl c9IEHkWmDgXLmMmeJ9yNE+6OUj42G1yh/XIeGgdkBIdpPf1lwI4XePuHpTlX2ToqyJ5U Hm91mFqEhL8WQywAvmTDziBjrUoaFUKNFHUEhGvtVz6OfKLB/GwiDZz3CaMN3SMlRAno fuQSoXd5k/HjDHtbpVGeroK3wWBWMVTFjapAbcABHcu31YERd+KA0DrBMsirJcC2obt0 mwxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700962167; x=1701566967; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GIgGdnhjdF8feXtt+F2EvuYtL54sQh4ya1Crg+4P6do=; b=oEOsAZhRskfgd9x/sf2ZcWfjxePVDmiC6ZRuYVBKJwpkn+HrMeYWOCdIcSG/Vgm6T5 8ICXLYuPpeseezZJAoG115DP+Z0DqdPKqEmaEj9rG9oUSUeR0XQMHJuHrf9odfYIaKcz pJWVff0ttZcSnDnEaULXuw+Vy7kiFVxHRVVJCqxlCXyWE/ne9oHLA39yFEC5/eKza4iZ QpMJ9nrq27ijaVZjxJUXhrO4OexVj2uvUN6GAO9WAxbxH7kMmiWj6tsLNkzt+cmNHStO 84f8HbSICSmsN9co4PNplMsFYmvE8gIjfB1vtRd20zYZA0EFzqmSu9ClHFTXWn24p8DX ywag== X-Gm-Message-State: AOJu0YyxsGpiGgj1n/p7uafy7VrSAdKmsPOS20E5/1pEB4/+Oq92ZETW 9GzcrdI5fB7u+i4CqumR18Gu8t5P87w= X-Received: by 2002:a05:6808:1921:b0:3ae:21d1:8040 with SMTP id bf33-20020a056808192100b003ae21d18040mr11095070oib.36.1700962167396; Sat, 25 Nov 2023 17:29:27 -0800 (PST) Received: from localhost.localdomain (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id g3-20020a62e303000000b0068a13b0b300sm5049519pfh.11.2023.11.25.17.29.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Nov 2023 17:29:26 -0800 (PST) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Sat, 25 Nov 2023 22:28:55 -0300 Message-ID: <20231126012858.40388-7-jamrial@gmail.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231126012858.40388-1-jamrial@gmail.com> References: <20231126012858.40388-1-jamrial@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 6/9] ffmpeg: add support for muxing AVStreamGroups X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: IbfSy/efd/Be Starting with IAMF support. Signed-off-by: James Almer --- fftools/ffmpeg.h | 2 + fftools/ffmpeg_mux_init.c | 327 ++++++++++++++++++++++++++++++++++++++ fftools/ffmpeg_opt.c | 2 + 3 files changed, 331 insertions(+) diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index 41935d39d5..057535adbb 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -262,6 +262,8 @@ typedef struct OptionsContext { int nb_disposition; SpecifierOpt *program; int nb_program; + SpecifierOpt *stream_groups; + int nb_stream_groups; SpecifierOpt *time_bases; int nb_time_bases; SpecifierOpt *enc_time_bases; diff --git a/fftools/ffmpeg_mux_init.c b/fftools/ffmpeg_mux_init.c index 63a25a350f..a4c564e5ec 100644 --- a/fftools/ffmpeg_mux_init.c +++ b/fftools/ffmpeg_mux_init.c @@ -39,6 +39,7 @@ #include "libavutil/dict.h" #include "libavutil/display.h" #include "libavutil/getenv_utf8.h" +#include "libavutil/iamf.h" #include "libavutil/intreadwrite.h" #include "libavutil/log.h" #include "libavutil/mem.h" @@ -1943,6 +1944,328 @@ static int setup_sync_queues(Muxer *mux, AVFormatContext *oc, int64_t buf_size_u return 0; } +static int of_parse_iamf_audio_element_layers(Muxer *mux, AVStreamGroup *stg, char **ptr) +{ + AVIAMFAudioElement *audio_element = stg->params.iamf_audio_element; + AVDictionary *dict = NULL; + const char *token; + int ret = 0; + + audio_element->demixing_info = + av_iamf_param_definition_alloc(AV_IAMF_PARAMETER_DEFINITION_DEMIXING, NULL, 1, NULL, NULL); + audio_element->recon_gain_info = + av_iamf_param_definition_alloc(AV_IAMF_PARAMETER_DEFINITION_RECON_GAIN, NULL, 1, NULL, NULL); + + if (!audio_element->demixing_info || + !audio_element->recon_gain_info) + return AVERROR(ENOMEM); + + /* process manually set layers and parameters */ + token = av_strtok(NULL, ",", ptr); + while (token) { + const AVDictionaryEntry *e; + int demixing = 0, recon_gain = 0; + int layer = 0; + + if (av_strstart(token, "layer=", &token)) + layer = 1; + else if (av_strstart(token, "demixing=", &token)) + demixing = 1; + else if (av_strstart(token, "recon_gain=", &token)) + recon_gain = 1; + + av_dict_free(&dict); + ret = av_dict_parse_string(&dict, token, "=", ":", 0); + if (ret < 0) { + av_log(mux, AV_LOG_ERROR, "Error parsing audio element specification %s\n", token); + goto fail; + } + + if (layer) { + ret = av_iamf_audio_element_add_layer(audio_element, &dict); + if (ret < 0) { + av_log(mux, AV_LOG_ERROR, "Error adding layer to stream group %d\n", stg->index); + goto fail; + } + } else if (demixing || recon_gain) { + AVIAMFParamDefinition *param = demixing ? audio_element->demixing_info + : audio_element->recon_gain_info; + void *subblock = av_iamf_param_definition_get_subblock(param, 0); + + av_opt_set_dict(param, &dict); + av_opt_set_dict(subblock, &dict); + + /* Hardcode spec parameters */ + param->param_definition_mode = 0; + param->parameter_rate = stg->streams[0]->codecpar->sample_rate; + param->duration = + param->constant_subblock_duration = stg->streams[0]->codecpar->frame_size; + } + + // make sure that no entries are left in the dict + e = NULL; + if (e = av_dict_iterate(dict, e)) { + av_log(mux, AV_LOG_FATAL, "Unknown layer key %s.\n", e->key); + ret = AVERROR(EINVAL); + goto fail; + } + token = av_strtok(NULL, ",", ptr); + } + +fail: + av_dict_free(&dict); + if (!ret && !audio_element->num_layers) { + av_log(mux, AV_LOG_ERROR, "No layer in audio element specification\n"); + ret = AVERROR(EINVAL); + } + + return ret; +} + +static int of_parse_iamf_submixes(Muxer *mux, AVStreamGroup *stg, char **ptr) +{ + AVFormatContext *oc = mux->fc; + AVIAMFMixPresentation *mix = stg->params.iamf_mix_presentation; + AVDictionary *dict = NULL; + const char *token; + char *submix_str = NULL; + int ret = 0; + + /* process manually set submixes */ + token = av_strtok(NULL, ",", ptr); + while (token) { + AVIAMFSubmix *submix = NULL; + const char *subtoken; + char *subptr = NULL; + + if (!av_strstart(token, "submix=", &token)) { + av_log(mux, AV_LOG_ERROR, "No submix in mix presentation specification \"%s\"\n", token); + goto fail; + } + + submix_str = av_strdup(token); + if (!submix_str) + goto fail; + + ret = av_iamf_mix_presentation_add_submix(mix, NULL); + if (!ret) { + submix = mix->submixes[mix->num_submixes - 1]; + submix->output_mix_config = + av_iamf_param_definition_alloc(AV_IAMF_PARAMETER_DEFINITION_MIX_GAIN, NULL, 0, NULL, NULL); + if (!submix->output_mix_config) + ret = AVERROR(ENOMEM); + } + if (ret < 0) { + av_log(mux, AV_LOG_ERROR, "Error adding submix to stream group %d\n", stg->index); + goto fail; + } + + submix->output_mix_config->parameter_rate = stg->streams[0]->codecpar->sample_rate; + + subptr = NULL; + subtoken = av_strtok(submix_str, "|", &subptr); + while (subtoken) { + const AVDictionaryEntry *e; + int element = 0, layout = 0; + + if (av_strstart(subtoken, "element=", &subtoken)) + element = 1; + else if (av_strstart(subtoken, "layout=", &subtoken)) + layout = 1; + + av_dict_free(&dict); + ret = av_dict_parse_string(&dict, subtoken, "=", ":", 0); + if (ret < 0) { + av_log(mux, AV_LOG_ERROR, "Error parsing submix specification \"%s\"\n", subtoken); + goto fail; + } + + if (element) { + AVIAMFSubmixElement *submix_element; + int idx = -1; + + if (e = av_dict_get(dict, "stg", NULL, 0)) + idx = strtol(e->value, NULL, 0); + av_dict_set(&dict, "stg", NULL, 0); + if (idx < 0 || idx >= oc->nb_stream_groups) { + av_log(mux, AV_LOG_ERROR, "Invalid or missing stream group index in " + "submix element specification \"%s\"\n", subtoken); + ret = AVERROR(EINVAL); + goto fail; + } + ret = av_iamf_submix_add_element(submix, NULL); + if (ret < 0) + av_log(mux, AV_LOG_ERROR, "Error adding element to submix\n"); + + submix_element = submix->elements[submix->num_elements - 1]; + submix_element->audio_element_id = oc->stream_groups[idx]->id; + + submix_element->element_mix_config = + av_iamf_param_definition_alloc(AV_IAMF_PARAMETER_DEFINITION_MIX_GAIN, NULL, 0, NULL, NULL); + if (!submix_element->element_mix_config) + ret = AVERROR(ENOMEM); + av_opt_set_dict2(submix_element, &dict, AV_OPT_SEARCH_CHILDREN); + submix_element->element_mix_config->parameter_rate = stg->streams[0]->codecpar->sample_rate; + } else if (layout) { + ret = av_iamf_submix_add_layout(submix, &dict); + if (ret < 0) + av_log(mux, AV_LOG_ERROR, "Error adding layout to submix\n"); + } else + av_opt_set_dict2(submix, &dict, AV_OPT_SEARCH_CHILDREN); + + if (ret < 0) { + goto fail; + } + + // make sure that no entries are left in the dict + e = NULL; + while (e = av_dict_iterate(dict, e)) { + av_log(mux, AV_LOG_FATAL, "Unknown submix key %s.\n", e->key); + ret = AVERROR(EINVAL); + goto fail; + } + subtoken = av_strtok(NULL, "|", &subptr); + } + av_freep(&submix_str); + + if (!submix->num_elements) { + av_log(mux, AV_LOG_ERROR, "No audio elements in submix specification \"%s\"\n", token); + ret = AVERROR(EINVAL); + } + token = av_strtok(NULL, ",", ptr); + } + +fail: + av_dict_free(&dict); + av_free(submix_str); + + return ret; +} + +static int of_add_groups(Muxer *mux, const OptionsContext *o) +{ + AVFormatContext *oc = mux->fc; + int ret; + + /* process manually set groups */ + for (int i = 0; i < o->nb_stream_groups; i++) { + AVDictionary *dict = NULL, *tmp = NULL; + const AVDictionaryEntry *e; + AVStreamGroup *stg = NULL; + int type; + const char *token; + char *str, *ptr = NULL; + const AVOption opts[] = { + { "type", "Set group type", offsetof(AVStreamGroup, type), AV_OPT_TYPE_INT, + { .i64 = 0 }, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM, "type" }, + { "iamf_audio_element", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = AV_STREAM_GROUP_PARAMS_IAMF_AUDIO_ELEMENT }, .unit = "type" }, + { "iamf_mix_presentation", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = AV_STREAM_GROUP_PARAMS_IAMF_MIX_PRESENTATION }, .unit = "type" }, + { NULL }, + }; + const AVClass class = { + .class_name = "StreamGroupType", + .item_name = av_default_item_name, + .option = opts, + .version = LIBAVUTIL_VERSION_INT, + }; + const AVClass *pclass = &class; + + str = av_strdup(o->stream_groups[i].u.str); + if (!str) + goto end; + + token = av_strtok(str, ",", &ptr); + if (token) { + ret = av_dict_parse_string(&dict, token, "=", ":", AV_DICT_MULTIKEY); + if (ret < 0) { + av_log(mux, AV_LOG_ERROR, "Error parsing group specification %s\n", token); + goto end; + } + + // "type" is not a user settable option in AVStreamGroup + e = av_dict_get(dict, "type", NULL, 0); + if (!e) { + av_log(mux, AV_LOG_ERROR, "No type define for Steam Group %d\n", i); + ret = AVERROR(EINVAL); + goto end; + } + + ret = av_opt_eval_int(&pclass, opts, e->value, &type); + if (ret < 0 || type == AV_STREAM_GROUP_PARAMS_NONE) { + av_log(mux, AV_LOG_ERROR, "Invalid group type \"%s\"\n", e->value); + goto end; + } + + av_dict_copy(&tmp, dict, 0); + stg = avformat_stream_group_create(oc, type, &tmp); + if (!stg) { + ret = AVERROR(ENOMEM); + goto end; + } + av_dict_set(&tmp, "type", NULL, 0); + + e = NULL; + while (e = av_dict_get(dict, "st", e, 0)) { + unsigned int idx = strtol(e->value, NULL, 0); + if (idx >= oc->nb_streams) { + av_log(mux, AV_LOG_ERROR, "Invalid stream index %d\n", idx); + ret = AVERROR(EINVAL); + goto end; + } + avformat_stream_group_add_stream(stg, oc->streams[idx]); + } + while (e = av_dict_get(dict, "stg", e, 0)) { + unsigned int idx = strtol(e->value, NULL, 0); + if (idx >= oc->nb_stream_groups || idx == stg->index) { + av_log(mux, AV_LOG_ERROR, "Invalid stream group index %d\n", idx); + ret = AVERROR(EINVAL); + goto end; + } + for (int j = 0; j < oc->stream_groups[idx]->nb_streams; j++) + avformat_stream_group_add_stream(stg, oc->stream_groups[idx]->streams[j]); + } + + switch(type) { + case AV_STREAM_GROUP_PARAMS_IAMF_AUDIO_ELEMENT: + ret = of_parse_iamf_audio_element_layers(mux, stg, &ptr); + break; + case AV_STREAM_GROUP_PARAMS_IAMF_MIX_PRESENTATION: + ret = of_parse_iamf_submixes(mux, stg, &ptr); + break; + default: + av_log(mux, AV_LOG_FATAL, "Unknown group type %d.\n", type); + ret = AVERROR(EINVAL); + break; + } + + if (ret < 0) + goto end; + + // make sure that nothing but "st" and "stg" entries are left in the dict + e = NULL; + while (e = av_dict_iterate(tmp, e)) { + if (!strcmp(e->key, "st") || !strcmp(e->key, "stg")) + continue; + + av_log(mux, AV_LOG_FATAL, "Unknown group key %s.\n", e->key); + ret = AVERROR(EINVAL); + goto end; + } + } + +end: + av_dict_free(&dict); + av_dict_free(&tmp); + av_free(str); + if (ret < 0) + return ret; + } + + return 0; +} + static int of_add_programs(Muxer *mux, const OptionsContext *o) { AVFormatContext *oc = mux->fc; @@ -2740,6 +3063,10 @@ int of_open(const OptionsContext *o, const char *filename) if (err < 0) return err; + err = of_add_groups(mux, o); + if (err < 0) + return err; + err = of_add_programs(mux, o); if (err < 0) return err; diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index 304471dd03..1144f64f89 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -1491,6 +1491,8 @@ const OptionDef options[] = { "add metadata", "string=string" }, { "program", HAS_ARG | OPT_STRING | OPT_SPEC | OPT_OUTPUT, { .off = OFFSET(program) }, "add program with specified streams", "title=string:st=number..." }, + { "stream_group", HAS_ARG | OPT_STRING | OPT_SPEC | OPT_OUTPUT, { .off = OFFSET(stream_groups) }, + "add stream group with specified streams and group type-specific arguments", "id=number:st=number..." }, { "dframes", HAS_ARG | OPT_PERFILE | OPT_EXPERT | OPT_OUTPUT, { .func_arg = opt_data_frames }, "set the number of data frames to output", "number" },