From patchwork Sun Feb 11 18:56:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 46188 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:9002:b0:19e:cdac:8cce with SMTP id d2csp310725pzc; Sun, 11 Feb 2024 10:57:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IGNetXRYZr1ukLOtJVYfqB72YSminrr0rl066ET6D1Z50KRPVtiuy06sidyIcDUjo/FkL69 X-Received: by 2002:aa7:c68b:0:b0:55f:5187:b18d with SMTP id n11-20020aa7c68b000000b0055f5187b18dmr2827597edq.20.1707677846800; Sun, 11 Feb 2024 10:57:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1707677846; cv=none; d=google.com; s=arc-20160816; b=l0NeEHPPRt1r4SYC58MURyozdIwyv3h3W1hojNzdnkHJYvHOpgWukCqXcOcud+YgIP uilQZqx6UBG8Foq1A+ePpEolV3Aqk+z4e1qlQ7vomh98f6gs3nQkPeD6t7JJuqYS0m6q he22SGu+dBRpXmLo+a1RVyhGcPow74yS70z0coWV9gVdGCcGfyOzrtF5mZVztajKxpAx waZCpCqz8C/8BBXWHn97xqv9fbJzt1QWNSRUl7uxSokhHT5bQV7XJDtlXstNm4U6ZSJV zog0gxa1Pof9KcJHQDzJqSIUsH1XLwQ68mZvAk7YaiTrbNUGr/qopRrRsCxi7IiJYB5l /sgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=14o9k0mYlk++DYgPF7hdikvvRI2FJCCGVjJh/Df6qaI=; fh=1ysdB1T9aUa/LKYXJMVy6Lsp4PFpzMcmWoNt8VVRQFw=; b=zPgOujryNAgWdf8AEI1Z8yn0XiwmdFBhFMS5HGKjEUaS1WOCFH7yVgA7u34RKL7hxT Op++j7NUWdOoqodHbqKyLtOiTsP+zTKkTTsZDyI6gVh2aaDJ+H91cnuHat5vN9pjlR2X uTkgX8ljktm55OEO3VHoSb5spafX6BVMY9WGR/jzlBiMNBI3xEFYaWkt/XB+xUnb6sQz eW6xzScYKyA5RARivWM5utlTBNYU9eMOEBmXqYJbbb2+DUN5vlt1lMQBsvI3v6k6hzpl TcZWMaM2S5AdTp5dBhM/4WBA9TStyKhmcbxtkkdhHqshDYey+PjsS7F/GjtWN+iFgKCz LLvw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=ALyFVueE; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=1; AJvYcCX0OVD8Gd/PY8zn4FBWRtlbDasBGGr3qjRdQJMnl+4DNRf3r826yIlBIl0FpNp2LXLwU4G3C59XuQy53tMxSkK8JV7gt6Nleya+1FBvRo4z3c5/3FU6/WIBSaACmJJxBpBJCUmnmoIqp3Ba0bmesAA/iZ3/c3X1jP5SlIQwdgmb3MrkJ9yzXOnx7P35tPUGqlOa0TDJkdRQC9KRrqJfsBrN7Qo6q1Y0BlEyMEa9U9/5s186ntXuzJJcDcGZ7Lyq5lv+8UM1JrClvY+sPn24u6SATB04Pn5ZKhcH7cj1Tc+SH7Go6s86JXgO14h2y6W81W6RDEcNqPap1846NRlnACpy5c1/GkgMQ6slD4afQ5Jp0O7z+t36tmeQJoTzDk1GIO155lL97JHbn4FopTsgueHF09rWQL6vgX/zFtniL4oIc/0YG67x14Zdr2HrOx8mOBunMZ94LKwFDb8Hfhb9g+rl9bUuY/VNbcUX4HMviycKiOylQdAQwOE4iIqBDFTTOm1Lf0jnl+A9AHg7rNevq+ba1ocvR0TAXzimXI1J2+p3M9ZDOkFUcWkRSNLXHXkAIihuGXglZWbNc+c3w6rJ23sEd2M0mr304LgnSefkX0ngJsiPW9QXJMxsbkx2zto1dXu3+0A4UcdAFvs1HtnrwuD4AIp9Nx/iKZEZGgPsUWuXzpl8FuoegIHjnCuoWxMx8BX2vq4j2hx+lFjxX4P+GNvV0+7rA5hD4Xw4gfG4s/uv38zGZqvXIqyvPqwd6PkqKHJRoHgyWVv4inKaHi2VMeMdln5RaG3xA3ZuYbWuv+ZLXUhJ6QTeuE3fahi0hHnyk+Gki+SZewIHlc4WGUu//m/KKovlfmGJWwKeEI3oSBuin8JkKuMpn6Kikf37CA3yTpeI9kpNW/QoEn0WWp1FRT50gepiu2cgjFABw83mRb3Q5aLmn/q07mKt0sWK8KnBGlLIM4 hh1l53JU/yR+cE0mW1wzfnGalYXYf8rBFGf4FGdG69tkW3xcms4OqliiddPyJLGDhBm2jnGgWF5Fzw5TQayWzxNuTlmtZanVbJSm3BsBGBNTx2251Htq4SzPW55In6A/PrKuTLgiI9msfiSKU+yFTnGo0NvruYG6kGMOM/qTc4qB0FAjLXFg2xB+OZmmtJy6KPLnGIY+o7fGNIgPfxHoVNF64238SuJOFoVXPsG0YAPC7OWpIN1ncJ2Uui8sLRnXfBvFOy324ydxpp18vVAllRhpDhtV1zwDSnE/kdsdFNWztjCS7hTnP5Op2tY0IBSkB8j2PliDnoJPB9in13Q7IijN7AFBax4E3jxpbLLDXze3cAAN3AiCeLKoIZgrTZBCpEEK3VDUbcImxZrDL3lnAHQaJ3w7APTBSjGk8xWdpgoWkPMt9inYZCEo0CwQzFk3cdMWyUytlRvAl2wVw2mRGs0rNVSOtGlZ/Y3RfG1wuWIovlN/zTjiy8ttFK4rfBKgvDxYH+R5pp1baa4ku3xmZNXoLtrA5ZV+G1C8svXw9xEY5i5ZyzI0jD9j/1n857exv6OklESD3f00GaOW61P/r3VqPjAe+5wF1O9nLzQTkaJhu51/+pynHB5OpVqh/Rif63 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z3-20020a05640235c300b0056101f4ecadsi2016430edc.464.2024.02.11.10.57.26; Sun, 11 Feb 2024 10:57:26 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=ALyFVueE; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 99B886807D3; Sun, 11 Feb 2024 20:57:22 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B7FAB6807D3 for ; Sun, 11 Feb 2024 20:57:16 +0200 (EET) Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1d7881b1843so23346035ad.3 for ; Sun, 11 Feb 2024 10:57:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707677834; x=1708282634; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=4XMnkyae5FOdbmfowpLwTF+7Vp9O5r+Ma8ivA6c/fh0=; b=ALyFVueEseK1jej5MGolLpYHtD6LdzlF5BZjBlUdFoIgCGguFd45KjIZu0iDZmCDMS uvejDDxQKw2MaDDOthwwjIGwwU20CdtdjoMryDKPuxasG0sF2lo7SCS6mAJwYYEAWS2w GXMrWiRzSTxVUx1mu3xcqciU7wRxa36IPpRR0/5I35Q8AZwXOA30+0BcaopGKcu1uSBQ A4Z9nEYc0v5FnrqfVv+LieI77SaHU8XMXhF7kuAhePR/6fHRi9kdbF2yk7Pcr7qVNT8P TL1hZZ6tOH+kjhw+NjZOI/jennTrA7FYezBT0pLje0D8oKnCeKAucUlNBeq86R50spde R3gA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707677834; x=1708282634; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4XMnkyae5FOdbmfowpLwTF+7Vp9O5r+Ma8ivA6c/fh0=; b=aU430Dr3oJjbze658wcUdaTq5092ikMQ0mUXj5fxiHNBrETDrVkAHujmBe2Ohw9TI4 x0/e7PYkB9Fia9i2enyrQvg7hera2oGALkYfdUsIO2l3D5/jo7rfzRJ4AgC+KLjjqDXo SZt6/BsExYV7umOufDBBt8kc7tNCSlPVhGtMMZBH2DL79zVUvma5kw6oD1Y6lkeEGE71 qGYti4XJyVing0D+RQnsADUiHHdm4wnlIy7ekuZGx5H5cbaip9givoOfb2H5pzAz7XAi oLhyUpXAKAznx/sFfzmv9MW5vdmxejwN7EWdw+sqgG6SFLgiATtgz6oCWPSbM/Ap/bVS NtOA== X-Gm-Message-State: AOJu0YyzPvdnsXA2W20tHjPsdfhDcRf7KsTSgR9Inc/GwlHfoWsI161V W3EiO86X5lAgc9MIKPEoU7FO+UrubY+58908J+Z4r/N2/80dknfx/PIS8pYA X-Received: by 2002:a17:902:ea0d:b0:1d8:fb17:a1fe with SMTP id s13-20020a170902ea0d00b001d8fb17a1femr6552880plg.48.1707677834047; Sun, 11 Feb 2024 10:57:14 -0800 (PST) Received: from localhost.localdomain (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id q13-20020a170902a3cd00b001d8accb025fsm4778166plb.197.2024.02.11.10.57.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Feb 2024 10:57:13 -0800 (PST) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Sun, 11 Feb 2024 15:56:58 -0300 Message-ID: <20240211185701.9327-1-jamrial@gmail.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/4 v10] avformat: add a Tile Grid stream group type X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: EuUOgbf2P8gP This will be used to support tiled image formats like HEIF. Signed-off-by: James Almer --- There can be more tiles than streams in the group now, which makes it possible to parse C021.heic libavformat/avformat.c | 5 ++ libavformat/avformat.h | 147 +++++++++++++++++++++++++++++++++++++++++ libavformat/dump.c | 29 ++++++++ libavformat/options.c | 34 ++++++++++ 4 files changed, 215 insertions(+) diff --git a/libavformat/avformat.c b/libavformat/avformat.c index ca31d3dc56..f53ba4ce58 100644 --- a/libavformat/avformat.c +++ b/libavformat/avformat.c @@ -99,6 +99,11 @@ void ff_free_stream_group(AVStreamGroup **pstg) av_iamf_mix_presentation_free(&stg->params.iamf_mix_presentation); break; } + case AV_STREAM_GROUP_PARAMS_TILE_GRID: + av_opt_free(stg->params.tile_grid); + av_freep(&stg->params.tile_grid->offsets); + av_freep(&stg->params.tile_grid); + break; default: break; } diff --git a/libavformat/avformat.h b/libavformat/avformat.h index 5d0fe82250..92751e5aee 100644 --- a/libavformat/avformat.h +++ b/libavformat/avformat.h @@ -1018,10 +1018,156 @@ typedef struct AVStream { int pts_wrap_bits; } AVStream; +/** + * AVStreamGroupTileGrid holds information on how to combine several + * independent images on a single canvas for presentation. + * + * The following is an example of a simple grid with 3 rows and 4 columns: + * + * +---+---+---+---+ + * | 0 | 1 | 2 | 3 | + * +---+---+---+---+ + * | 4 | 5 | 6 | 7 | + * +---+---+---+---+ + * | 8 | 9 |10 |11 | + * +---+---+---+---+ + * + * Assuming all tiles have a dimension of 512x512, the + * @ref AVStreamGroupTileGrid.offsets "offset" of the topleft pixel of + * the first @ref AVStreamGroup.streams "stream" in the group is "0,0", the + * @ref AVStreamGroupTileGrid.offsets "offset" of the topleft pixel of + * the second @ref AVStreamGroup.streams "stream" in the group is "512,0", the + * @ref AVStreamGroupTileGrid.offsets "offset" of the topleft pixel of + * the fifth @ref AVStreamGroup.streams "stream" in the group is "0,512", the + * @ref AVStreamGroupTileGrid.offsets "offset", of the topleft pixel of + * the sixth @ref AVStreamGroup.streams "stream" in the group is "512,512", + * etc. + * + * The following is an example of a canvas with overlaping tiles: + * + * +---------------+ + * |***** | + * |* 0##### | + * |***# 1 # | + * | ##### | + * | | + * | | + * | | + * +---------------+ + * + * Assuming a canvas with size 2048x2048 and both tiles with a dimension of + * 512x512, a possible @ref AVStreamGroupTileGrid.offsets "offset" for the + * topleft pixel of the first @ref AVStreamGroup.streams "stream" in the group + * would be 256x256, and the @ref AVStreamGroupTileGrid.offsets "offset" for + * the topleft pixel of the second @ref AVStreamGroup.streams "stream" in the + * group would be 512x512. + * + * sizeof(AVStreamGroupTileGrid) is not a part of the ABI and may only be + * allocated by avformat_stream_group_create(). + */ +typedef struct AVStreamGroupTileGrid { + const AVClass *av_class; + + /** + * Amount of tiles in the grid. + * + * Must be > 0. + */ + unsigned int nb_tiles; + + /** + * Width of the canvas. + * + * Must be > 0. + */ + int coded_width; + /** + * Width of the canvas. + * + * Must be > 0. + */ + int coded_height; + + /** + * An @ref nb_tiles sized array of offsets in pixels from the topleft edge + * of the canvas, indicating where each stream should be placed. + * It must be allocated with the av_malloc() family of functions. + * + * - demuxing: set by libavformat, must not be modified by the caller. + * - muxing: set by the caller before avformat_write_header(). + * + * Freed by libavformat in avformat_free_context(). + */ + struct { + /** + * Index of the stream in the group this tile references. + * + * Must be < @ref AVStreamGroup.nb_streams "nb_streams". + */ + unsigned int idx; + /** + * Offset in pixels from the left edge of the canvas where the tile + * should be placed. + */ + int horizontal; + /** + * Offset in pixels from the top edge of the canvas where the tile + * should be placed. + */ + int vertical; + } *offsets; + + /** + * The pixel value per channel in RGBA format used if no pixel of any tile + * is located at a particular pixel location. + * + * @see av_image_fill_color(). + * @see av_parse_color(). + */ + uint8_t background[4]; + + /** + * Offset in pixels from the left edge of the canvas where the actual image + * meant for presentation starts. + * + * This field must be >= 0 and < @ref coded_width. + */ + int horizontal_offset; + /** + * Offset in pixels from the top edge of the canvas where the actual image + * meant for presentation starts. + * + * This field must be >= 0 and < @ref coded_height. + */ + int vertical_offset; + + /** + * Width of the final image for presentation. + * + * Must be > 0 and <= (@ref coded_width - @ref horizontal_offset). + * When it's not equal to (@ref coded_width - @ref horizontal_offset), the + * result of (@ref coded_width - width - @ref horizontal_offset) is the + * amount amount of pixels to be cropped from the right edge of the + * final image before presentation. + */ + int width; + /** + * Height of the final image for presentation. + * + * Must be > 0 and <= (@ref coded_height - @ref vertical_offset). + * When it's not equal to (@ref coded_height - @ref vertical_offset), the + * result of (@ref coded_height - height - @ref vertical_offset) is the + * amount amount of pixels to be cropped from the bottom edge of the + * final image before presentation. + */ + int height; +} AVStreamGroupTileGrid; + enum AVStreamGroupParamsType { AV_STREAM_GROUP_PARAMS_NONE, AV_STREAM_GROUP_PARAMS_IAMF_AUDIO_ELEMENT, AV_STREAM_GROUP_PARAMS_IAMF_MIX_PRESENTATION, + AV_STREAM_GROUP_PARAMS_TILE_GRID, }; struct AVIAMFAudioElement; @@ -1062,6 +1208,7 @@ typedef struct AVStreamGroup { union { struct AVIAMFAudioElement *iamf_audio_element; struct AVIAMFMixPresentation *iamf_mix_presentation; + struct AVStreamGroupTileGrid *tile_grid; } params; /** diff --git a/libavformat/dump.c b/libavformat/dump.c index 9d37179bb7..756db8c87e 100644 --- a/libavformat/dump.c +++ b/libavformat/dump.c @@ -22,6 +22,7 @@ #include #include +#include "libavutil/avstring.h" #include "libavutil/channel_layout.h" #include "libavutil/display.h" #include "libavutil/iamf.h" @@ -738,6 +739,34 @@ static void dump_stream_group(const AVFormatContext *ic, uint8_t *printed, } break; } + case AV_STREAM_GROUP_PARAMS_TILE_GRID: { + const AVStreamGroupTileGrid *tile_grid = stg->params.tile_grid; + AVCodecContext *avctx = avcodec_alloc_context3(NULL); + const char *ptr = NULL; + av_log(NULL, AV_LOG_INFO, " Tile Grid:"); + if (avctx && stg->nb_streams && !avcodec_parameters_to_context(avctx, stg->streams[0]->codecpar)) { + avctx->width = tile_grid->width; + avctx->height = tile_grid->height; + avctx->coded_width = tile_grid->coded_width; + avctx->coded_height = tile_grid->coded_height; + if (ic->dump_separator) + av_opt_set(avctx, "dump_separator", ic->dump_separator, 0); + buf[0] = 0; + avcodec_string(buf, sizeof(buf), avctx, is_output); + ptr = av_stristr(buf, " "); + } + avcodec_free_context(&avctx); + if (ptr) + av_log(NULL, AV_LOG_INFO, "%s", ptr); + av_log(NULL, AV_LOG_INFO, "\n"); + dump_metadata(NULL, stg->metadata, " ", AV_LOG_INFO); + for (int i = 0; i < stg->nb_streams; i++) { + const AVStream *st = stg->streams[i]; + dump_stream_format(ic, st->index, i, index, is_output, AV_LOG_VERBOSE); + printed[st->index] = 1; + } + break; + } default: break; } diff --git a/libavformat/options.c b/libavformat/options.c index 03e6a2a7ff..a323aa7fe6 100644 --- a/libavformat/options.c +++ b/libavformat/options.c @@ -339,6 +339,28 @@ fail: return NULL; } +#define FLAGS AV_OPT_FLAG_ENCODING_PARAM | AV_OPT_FLAG_VIDEO_PARAM +#define OFFSET(x) offsetof(AVStreamGroupTileGrid, x) +static const AVOption tile_grid_options[] = { + { "grid_size", "size of the output canvas", OFFSET(coded_width), + AV_OPT_TYPE_IMAGE_SIZE, { .str = NULL }, 0, INT_MAX, FLAGS }, + { "output_size", "size of valid pixels in output image meant for presentation", OFFSET(width), + AV_OPT_TYPE_IMAGE_SIZE, { .str = NULL }, 0, INT_MAX, FLAGS }, + { "background_color", "set a background color for unused pixels", + OFFSET(background), AV_OPT_TYPE_COLOR, { .str = "black"}, 0, 0, FLAGS }, + { "horizontal_offset", NULL, OFFSET(horizontal_offset), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, + { "vertical_offset", NULL, OFFSET(vertical_offset), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, + { NULL }, +}; +#undef FLAGS +#undef OFFSET + +static const AVClass tile_grid_class = { + .class_name = "AVStreamGroupTileGrid", + .version = LIBAVUTIL_VERSION_INT, + .option = tile_grid_options, +}; + static void *stream_group_child_next(void *obj, void *prev) { AVStreamGroup *stg = obj; @@ -348,6 +370,8 @@ static void *stream_group_child_next(void *obj, void *prev) return stg->params.iamf_audio_element; case AV_STREAM_GROUP_PARAMS_IAMF_MIX_PRESENTATION: return stg->params.iamf_mix_presentation; + case AV_STREAM_GROUP_PARAMS_TILE_GRID: + return stg->params.tile_grid; default: break; } @@ -370,6 +394,9 @@ static const AVClass *stream_group_child_iterate(void **opaque) case AV_STREAM_GROUP_PARAMS_IAMF_MIX_PRESENTATION: ret = av_iamf_mix_presentation_get_class(); break; + case AV_STREAM_GROUP_PARAMS_TILE_GRID: + ret = &tile_grid_class; + break; default: break; } @@ -431,6 +458,13 @@ AVStreamGroup *avformat_stream_group_create(AVFormatContext *s, if (!stg->params.iamf_mix_presentation) goto fail; break; + case AV_STREAM_GROUP_PARAMS_TILE_GRID: + stg->params.tile_grid = av_mallocz(sizeof(*stg->params.tile_grid)); + if (!stg->params.tile_grid) + goto fail; + stg->params.tile_grid->av_class = &tile_grid_class; + av_opt_set_defaults(stg->params.tile_grid); + break; default: goto fail; }