From patchwork Sun Feb 4 02:29:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 46002 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:7b08:b0:19e:8a94:b663 with SMTP id s8csp26543pzh; Sat, 3 Feb 2024 18:29:35 -0800 (PST) X-Google-Smtp-Source: AGHT+IEm3laMpmAhI9xFumrUman8gm9Ot6yz/cGWZuDZLIBrDbwICUbTOC/q99yuxWH9b3+uiB7B X-Received: by 2002:a17:906:118a:b0:a35:88ad:e607 with SMTP id n10-20020a170906118a00b00a3588ade607mr9408376eja.31.1707013775064; Sat, 03 Feb 2024 18:29:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1707013775; cv=none; d=google.com; s=arc-20160816; b=gAjHD+yt+VHRfPmgBBFF1PKDQDpWRPmQZtjBaQcFJ6R9A++QPs8w020MF1wgT8Twgr RxStUg87sgqfssatogy6P2M7z3yQrEtsMCMMyS0o4csFH6rFP0Dpurfev/DfLTQQ8Tod aFs0qUEEFsRSFGrdJA1liDgh6VkfqFAn6Ny77H9BkeX0QjaOoNo7Rlf0ToVf1bzcVL1a mCJExWlJTgBovfYDCljkITAbDaNePvq1f7irUJaNEI4O1xDXYHoDNnIXRSdWM1AI8wLh 5JHu4T7q2LcWHzNSh2JYzuVeFOLoRYiMbnzwqXbARIV4RSZAA5ubDiijP1u2PMKrCqvr w+6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=laBkyeAdHZ94w8cOfGD+myZMwZVMVKBaACRY8cneF8A=; fh=a1+vDMvg3wnEwb2Oory7niKMxRamvqRQwSkr0hvr5eI=; b=L5JzSBX/3gepnHOAPCnhAIoE0Hg4iCXoVN/AB+EfoRzJm6UE4yg4IEz2HLYhVx9RMe l5sT6mC7dysTLuywEE++UtnMvPbqiuJEYawcehL8bbSH3lnYcpevELaSrws8QpyAptjf E47n/gCfapFRhIfJ1DLKBJahxnD2WGJkG4gwtDsrGWGHR7nemKuXmZIJUYeJbg/KkcMh 6ZVpzqxR0NyVw3qPupncPT+xhOQtUtnHKefkn+p5PWQhDkX6PrO4S0sIuO5lZzV1ooKr lgmQo5mSWTLn280jIa5jbAPzIrpqA1N3lEUCbvgMkFzFvTTce6DOoVUrhTEbJtsi5GNR KwYg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=fDLJGo9H; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=0; AJvYcCW5pvePnyVNC7T16gIEhqFzOmKP4jlWWvsKrgyF4tA1hYQf+ZAPaYIdLsINWKkY+aZcUPnMor7dyNV+AJUbNBpIVoyu2Y3hDL5VRwb5eHSbsgYbd8q6efkJorzC1zNY8AoqpPqgtOdtaRU1IzTf/bWzjRu2HtXDjp+l5WK9dNwMEsclPHWWuXRYoL8hduZ4TzTuGdMmLj2xW1G42rxjGTMI9W8Q6YwzqJ1XjjihkPXWEytrer/ti5NALXQvoeLD/nJCSGyQPGi1bzR1bdSw6Pxalgt6IdjCs/pqVKoTcC0JDSaKBP54kssKFU9qtABIOBR2xs7OSae657dm7AhfjogZ/87bFmMN8DkVHDPNRANe5EtGAjYhtVvKR3WE/BLsuGR59O2+4nx7flYW0NvponQfieQj8/nMm20qmpGULHaaGFEMturi6Rvi9XRS28c1nEJBFALYcvTeF54xojyZiqT5QFWdOXXcNfYr8UgxcMPu+TUrilEvgRAR2DFe9nUtUqhS4yA8QmyB2+Uh8HNs765lOE4imJy+cBVjMhDcZ6CawJNg+jdJpp6qgJhdPiwliehM4LHS59BIveeH8swFJiRQkk2qtOfSnFAeMm6MuTDfMg7O7k35XvSr9rYDpFX2XYpPZ4MV2smHxoYqEvRoR7jtiOOfdeYzSfCORa0URRmSXJ7H1BseXjdle0TZdZtrM2xQIrBV9XKURpCeb+o3ynJ+Y87zb4EdaK0CEN3q+NX53XDwwIJBXzOlaS8Z114I/mwNdHkrydMjmfv6lnVttyRf5cqPSJtmSp0ktKj6XgN7qyX0rY2XnhercDLH3niLhckPG6b/yf/U8bK+TqsoV+GCFP3mvA0XicsdADDvxrpc79N+2PB/cov7mnSL7HCwtT4e5wYatB+CkBI6+UFGlK7ey4FimrMR+mPWIGteKOflHwCTXInuSul4sNVmGRtE1hlZEl SoApkw1BrkwGfDn86H+ucXcdq8RaBZrCXoYaE+ZKd4v/lq2NIajTRwQkM81Mf4Ryrujk5DPgyeXkqIs4JItCyDTIeWp/31t8dI0c45YUZj1fuJYaV2NnC1MNLAXgcip2wzhIz67HPwYeRU9q9Lp+JuWc5pwSXEdkvmp/uDxcIXYkiyl3NI8VuBMpEyhy9iNOnC/2X1OL1zHWd/U7HuI+6f3hOoVHueCBJHPh2i71O6sVnI7nByDM5KZIJhWFWIp1zCUu5/k9uovFO++ZGces82M07Tbc62iL2PYNcCRqYs/tqEklZEZtcVnnYxDzKfzUhWBj4Lrvc5Ra/c8QqMIDRq0I6L9997sa4pjJe49iEX6kCrik/aS472H2sbBhy202zxae5OEVlT9GJ7jOWVxEe5UD5PnuUSlyDgikao8aq/M5i2rOgftdrt21WLGwCHDbDgdQV9G5gKr5pnrk5ih7fCCC1AdOiqYgegyk40N/UsSMsUUrvrKKoaT/mOEl9Y/AM8vZMSZMWOXLmJ4yt2PDvUt6Ezf5+z34Xc+sDhrsdZUd5cyK7dv0nYYtPbJCNN7e5tdg0se1vJeX+4PPjgvyBxt5HrU/dQvEz4qsJy Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f12-20020a1709067f8c00b00a35e146e3b6si2330860ejr.70.2024.02.03.18.29.29; Sat, 03 Feb 2024 18:29:35 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=fDLJGo9H; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 231ED68D13C; Sun, 4 Feb 2024 04:29:25 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 60E7C68C4F0 for ; Sun, 4 Feb 2024 04:29:18 +0200 (EET) Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1d958e0d73dso16229015ad.1 for ; Sat, 03 Feb 2024 18:29:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707013755; x=1707618555; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=gCMQrzcfzQ0EQMQBOqX4bGEyjr56FQn6egCsjc1inms=; b=fDLJGo9HtfBGJSKHg2wNxBS0dgzQLWCecij8LTY4aPdyJm2mWrKIH+VB29yTXUMOvF swpw3/7MUZUmaJTfGiBm6ahmULqz5lDAoDfmKcXCEoMeQSXSpoWUHCNXDsbfpbEGBYaq 0OKoh9yWXhVyU2nmSSSLM2JITdtGYPtSyDbjUaYw0+n/IxFIfUtDZCHC8h+BxQXwscmt QW4IG+BkPt9+xfDy6uF0gsI8reMZBfXsySxIjPmbPLlsuc8/EYyVffXe1Gf+YAo7w9nH 9AHDbWcO4OaxROntlE6eKatFWpR/X77zGM0ne6AjztAfY2yu9UpXUnAHY3CxGSQX/RUz ZfnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707013755; x=1707618555; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gCMQrzcfzQ0EQMQBOqX4bGEyjr56FQn6egCsjc1inms=; b=rOQZ05ILR7MWsANN8njhDhx3bN+vVznUaEJ2mRrXze8pBke/UWxnJQiSmxphPsMXFs +TPHkZ6Yb3U5kG207/SwQRytmDR5ryOgT3zhcZoAOgYqAS155sLckeM0aYDKFVl9JHFj S+GGPKr8zRaGTDVJbiuNyHRvhOuVet0PHQ/93hXTgywpR4ue/iEWqs2tcm8kFLrXPsJQ PFA4+D09SuJMPX24k0Q8l1Mk7nJtQzrm5gN6SCaaGAsnvxzPcLcjAxaVTTqEVsv+bX1W cWKAhN5C3sWfgpzlumObtPzWG1FKooqFKalcH3S1RnTYxx8sSn0npyWtHjkt2lOgRE+X piTg== X-Gm-Message-State: AOJu0YzuZnIERmP2nciZI0+Ynyw/BHEKmEdl+6tDfJjfEcUSpIPoh0/0 bW3ZIQiSk3BbwccZGGSv6SUdYrhv4aErlxgi1paix1zdgd/5s3iL++slJEMj X-Received: by 2002:a17:902:db01:b0:1d9:907f:636d with SMTP id m1-20020a170902db0100b001d9907f636dmr2898842plx.10.1707013755093; Sat, 03 Feb 2024 18:29:15 -0800 (PST) Received: from localhost.localdomain (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id k5-20020a170902e90500b001d7907eb711sm3785965pld.182.2024.02.03.18.29.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Feb 2024 18:29:14 -0800 (PST) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Sat, 3 Feb 2024 23:29:27 -0300 Message-ID: <20240204022930.14007-1-jamrial@gmail.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/3 v7] avformat: add a Tile Grid stream group type X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 1qiwGVfrQVaL This will be used to support tiled image formats like HEIF. Signed-off-by: James Almer --- Fixed comment about sizeof(AVStreamGroupTileGrid). libavformat/avformat.c | 5 +++ libavformat/avformat.h | 100 +++++++++++++++++++++++++++++++++++++++++ libavformat/dump.c | 29 ++++++++++++ libavformat/options.c | 32 +++++++++++++ 4 files changed, 166 insertions(+) diff --git a/libavformat/avformat.c b/libavformat/avformat.c index 8e8c6fbe55..32ef440207 100644 --- a/libavformat/avformat.c +++ b/libavformat/avformat.c @@ -100,6 +100,11 @@ void ff_free_stream_group(AVStreamGroup **pstg) av_iamf_mix_presentation_free(&stg->params.iamf_mix_presentation); break; } + case AV_STREAM_GROUP_PARAMS_TILE_GRID: + av_opt_free(stg->params.tile_grid); + av_freep(&stg->params.tile_grid->offsets); + av_freep(&stg->params.tile_grid); + break; default: break; } diff --git a/libavformat/avformat.h b/libavformat/avformat.h index 5d0fe82250..0b1c2e46b5 100644 --- a/libavformat/avformat.h +++ b/libavformat/avformat.h @@ -1018,10 +1018,109 @@ typedef struct AVStream { int pts_wrap_bits; } AVStream; +/** + * AVStreamGroupTileGrid holds information on how to combine several + * independent images on a single grid for presentation. None of the tiles may + * overlap inside the grid. + * + * The following is an example of a simple grid with 3 rows and 4 columns: + * + * +---+---+---+---+ + * | 0 | 1 | 2 | 3 | + * +---+---+---+---+ + * | 4 | 5 | 6 | 7 | + * +---+---+---+---+ + * | 8 | 9 |10 |11 | + * +---+---+---+---+ + * + * Assuming all tiles have a dimension of 512x512, the + * @ref AVStreamGroupTileGrid.offsets "offset" of the topleft pixel of + * the first @ref AVStreamGroup.streams "stream" in the group is "0,0", the + * @ref AVStreamGroupTileGrid.offsets "offset" of the topleft pixel of + * the second @ref AVStreamGroup.streams "stream" in the group is "512,0", the + * @ref AVStreamGroupTileGrid.offsets "offset" of the topleft pixel of + * the fifth @ref AVStreamGroup.streams "stream" in the group is "0,512", the + * @ref AVStreamGroupTileGrid.offsets "offset", of the topleft pixel of + * the sixth @ref AVStreamGroup.streams "stream" in the group is "512,512", + * etc. + * + * sizeof(AVStreamGroupTileGrid) is not a part of the ABI and may only be + * allocated by avformat_stream_group_create(). + */ +typedef struct AVStreamGroupTileGrid { + const AVClass *av_class; + + /** + * Width of the final image in the grid. + * + * Must be > 0. + */ + int coded_width; + /** + * Width of the final image in the grid. + * + * Must be > 0. + */ + int coded_height; + + /** + * An @ref AVStreamGroup.nb_streams "nb_streams" sized array of offsets in + * pixels from the topleft edge of the grid, indicating where each stream + * should be placed. + * It must be allocated with the av_malloc() family of functions. + * + * - demuxing: set by libavformat, must not be modified by the caller. + * - muxing: set by the caller before avformat_write_header(). + * + * Freed by libavformat in avformat_free_context(). + */ + struct { + int x; + int y; + } *offsets; + + /** + * Offset in pixels from the left edge of the grid where the actual image + * meant for presentation starts. + * + * This field must be >= 0 and <= @ref coded_width. + */ + int horizontal_offset; + /** + * Offset in pixels from the top edge of the grid where the actual image + * meant for presentation starts. + * + * This field must be >= 0 and <= @ref coded_height. + */ + int vertical_offset; + + /** + * Width of the final image for presentation. + * + * Must be > 0 and <= (@ref coded_width - @ref horizontal_offset). + * When it's not equal to (@ref coded_width - @ref horizontal_offset), the + * result of (@ref coded_width - width - @ref horizontal_offset) is the + * amount amount of pixels to be cropped from the right edge of the + * final image before presentation. + */ + int width; + /** + * Height of the final image for presentation. + * + * Must be > 0 and <= (@ref coded_height - @ref vertical_offset). + * When it's not equal to (@ref coded_height - @ref vertical_offset), the + * result of (@ref coded_height - height - @ref vertical_offset) is the + * amount amount of pixels to be cropped from the bottom edge of the + * final image before presentation. + */ + int height; +} AVStreamGroupTileGrid; + enum AVStreamGroupParamsType { AV_STREAM_GROUP_PARAMS_NONE, AV_STREAM_GROUP_PARAMS_IAMF_AUDIO_ELEMENT, AV_STREAM_GROUP_PARAMS_IAMF_MIX_PRESENTATION, + AV_STREAM_GROUP_PARAMS_TILE_GRID, }; struct AVIAMFAudioElement; @@ -1062,6 +1161,7 @@ typedef struct AVStreamGroup { union { struct AVIAMFAudioElement *iamf_audio_element; struct AVIAMFMixPresentation *iamf_mix_presentation; + struct AVStreamGroupTileGrid *tile_grid; } params; /** diff --git a/libavformat/dump.c b/libavformat/dump.c index aff51b43f6..c9b7369bcd 100644 --- a/libavformat/dump.c +++ b/libavformat/dump.c @@ -22,6 +22,7 @@ #include #include +#include "libavutil/avstring.h" #include "libavutil/channel_layout.h" #include "libavutil/display.h" #include "libavutil/iamf.h" @@ -721,6 +722,34 @@ static void dump_stream_group(const AVFormatContext *ic, uint8_t *printed, } break; } + case AV_STREAM_GROUP_PARAMS_TILE_GRID: { + const AVStreamGroupTileGrid *tile_grid = stg->params.tile_grid; + AVCodecContext *avctx = avcodec_alloc_context3(NULL); + const char *ptr = NULL; + av_log(NULL, AV_LOG_INFO, " Tile Grid:"); + if (avctx && stg->nb_streams && !avcodec_parameters_to_context(avctx, stg->streams[0]->codecpar)) { + avctx->width = tile_grid->width; + avctx->height = tile_grid->height; + avctx->coded_width = tile_grid->coded_width; + avctx->coded_height = tile_grid->coded_height; + if (ic->dump_separator) + av_opt_set(avctx, "dump_separator", ic->dump_separator, 0); + buf[0] = 0; + avcodec_string(buf, sizeof(buf), avctx, is_output); + ptr = av_stristr(buf, " "); + } + avcodec_free_context(&avctx); + if (ptr) + av_log(NULL, AV_LOG_INFO, "%s", ptr); + av_log(NULL, AV_LOG_INFO, "\n"); + dump_metadata(NULL, stg->metadata, " ", AV_LOG_INFO); + for (int i = 0; i < stg->nb_streams; i++) { + const AVStream *st = stg->streams[i]; + dump_stream_format(ic, st->index, i, index, is_output, AV_LOG_VERBOSE); + printed[st->index] = 1; + } + break; + } default: break; } diff --git a/libavformat/options.c b/libavformat/options.c index 75ec86ce05..ed96ba4afb 100644 --- a/libavformat/options.c +++ b/libavformat/options.c @@ -337,6 +337,26 @@ fail: return NULL; } +#define FLAGS AV_OPT_FLAG_ENCODING_PARAM | AV_OPT_FLAG_VIDEO_PARAM +#define OFFSET(x) offsetof(AVStreamGroupTileGrid, x) +static const AVOption tile_grid_options[] = { + { "grid_size", "size of the output image", OFFSET(coded_width), + AV_OPT_TYPE_IMAGE_SIZE, { .str = NULL }, 0, INT_MAX, FLAGS }, + { "horizontal_offset", NULL, OFFSET(horizontal_offset), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, + { "vertical_offset", NULL, OFFSET(vertical_offset), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, + { "output_size", "size of valid pixels in output image meant for presentation", OFFSET(width), + AV_OPT_TYPE_IMAGE_SIZE, { .str = NULL }, 0, INT_MAX, FLAGS }, + { NULL }, +}; +#undef FLAGS +#undef OFFSET + +static const AVClass tile_grid_class = { + .class_name = "AVStreamGroupTileGrid", + .version = LIBAVUTIL_VERSION_INT, + .option = tile_grid_options, +}; + static void *stream_group_child_next(void *obj, void *prev) { AVStreamGroup *stg = obj; @@ -346,6 +366,8 @@ static void *stream_group_child_next(void *obj, void *prev) return stg->params.iamf_audio_element; case AV_STREAM_GROUP_PARAMS_IAMF_MIX_PRESENTATION: return stg->params.iamf_mix_presentation; + case AV_STREAM_GROUP_PARAMS_TILE_GRID: + return stg->params.tile_grid; default: break; } @@ -368,6 +390,9 @@ static const AVClass *stream_group_child_iterate(void **opaque) case AV_STREAM_GROUP_PARAMS_IAMF_MIX_PRESENTATION: ret = av_iamf_mix_presentation_get_class(); break; + case AV_STREAM_GROUP_PARAMS_TILE_GRID: + ret = &tile_grid_class; + break; default: break; } @@ -429,6 +454,13 @@ AVStreamGroup *avformat_stream_group_create(AVFormatContext *s, if (!stg->params.iamf_mix_presentation) goto fail; break; + case AV_STREAM_GROUP_PARAMS_TILE_GRID: + stg->params.tile_grid = av_mallocz(sizeof(*stg->params.tile_grid)); + if (!stg->params.tile_grid) + goto fail; + stg->params.tile_grid->av_class = &tile_grid_class; + av_opt_set_defaults(stg->params.tile_grid); + break; default: goto fail; } From patchwork Sun Feb 4 02:29:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 46003 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:7b08:b0:19e:8a94:b663 with SMTP id s8csp26581pzh; Sat, 3 Feb 2024 18:29:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IHKN/kXvMigSwRIh5kv2xWH2CWeYRWfSvsKlS1BiDvhUBzmyBmz1+T8ns+dV+LEBElPZ8Ka X-Received: by 2002:a05:6512:741:b0:511:2cd6:119f with SMTP id c1-20020a056512074100b005112cd6119fmr3488480lfs.14.1707013781798; Sat, 03 Feb 2024 18:29:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1707013781; cv=none; d=google.com; s=arc-20160816; b=HuN1PlDmRf5EI7E2h8+ycmUtbKAA2s2uuyHMZr02H6zBatXOSJsU3fsrD3epJsITY0 y0faADWuFku97Wbbw1VIIXyO3SC7/gRyGwX4fkePPcHvvw8vRzyTnOMJKteQN00KZXW3 R4Hfw1bWzF5iKIaWGJgR3DuPDwpC43919LWQM8T0OUYIPYZFAheeURhNKy7i7CI2n4yX kyfTrJMlrd04xegOS+bJnw2WHbCnM5VcIuIvcLwz0II5CzmE318qeUPAq8o52mjNA9lO rrMWjaCzFVMuHv0QgnU3cY2kw7OGwrOOKN3Ild2st/c9eqrpT2fE61mkQdvDrLYhIxYT ELIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=PD1+BVhprK9IP7iqXRTdC6I62hjOxfg0O6H5rYgB334=; fh=kt0EjU+kkA8nU1KqaOcXYK6eYk4mSqYVRZt0QcvkLPs=; b=Bs8enRojjG+gsTH7TvmrlyhNDduPccIpE6WF7JL3+UnV4swxtlWkMkfhnU5bjPMxgn 9anGuG6e5tKazt0KBwIQYLZLiTr5U4Povn9jCCW7iKrWzSzSQKU6m0WZouFudHHffGS7 m0psynkBLjXmtVmg8LnZ34t9daIid+R66IoZVIc964NyBS4pjt6ySAuZfSwBvrk52L1n M6bfLukc9iImPSYejKe2xsgh8qExNvOXk4dKX6Ti4oh6R1IDPsNw10OQ1rj9fmImslLa B1Uou64GD25u+P56Y9jvo4sGpIdgn/WblZ32lhQNbKuzTbZefOYH19WjOubx2CDjre8b 6LGA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=Rs8P7Zzh; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=0; AJvYcCU8To5S3T3/cjJSzHcUwGOc34rnVTTDlGLH9qwBwqwWi0WXKfn6Io9M0evusRndAj6csXGOFUBo8/VZM2F0YAhRs4Ogno5xxgTww6c71f35/nn4hjwiiOwGN3pb+5/UYu37GsT2e3jjKlo6yNu0fmDkXs+uItekwRC0h+8f4zbMjkrf1NuId5ExXurXPNJL5DhHuZEC0Y/TDLAkIHS8H7sHxWzJ7DivxSPjjyddQVB8ferkWQPwdQ+ZmuT82ARDvyWw2ZaWhuMGsGaeQvnR3YbAoC/0VaDrBaQb8NrXN6xwbe23y+t8AUZS1pCTrp/x/x5oIRvZn6bbhu/I//Ri8OaayWgVmsE7/RZyGU9OUWGqBPtM0dE4eW7cADmgxPpjzvqg500CuZrEZeI6lG457WLQ+B3RQRgk4S0jrEYMRWrw/KH9y9MOzirn/qMbfIQ734ZobwYbosL5x5NGQ6KAJbJvbuwoHOoqTKPK0wdf0gokfDCkcFx68XHSUJhvJKS69dPcXkFfgzbbqWAGvCcQ6ftW7JTu5cbgOpcTTHQHD3MQSxDd+5z0jis7Kw2Y+3KRZXUhR0Kthxwc0yr5KCe2De1D1ZAvVENKMSH7ofgrATJUpTI6MMl2d1v3y8KNi43r835vxQ5he9FoczTH/t6eSlDSVW+FmnYMF7REk2h9yLvvY+u4rIM4KFoqKZUjUo1RNb5cq7jA9dYGUKRilnrYvdI3Ga6DiOvbirHji0f9vrHnuohqmCLcp6fsbrfg2HSNKa3Dc+/jKi7c9+999tNXT84UyfW/XzglBkU4tkDqiLkKZE2PQrCkQakA0n3IQZOeAsJAsVeHZhxSW6nXiau2mkgbkql2JD9I9PKcAvuH1UgssoaciEoVxG5WJrg0fYubZFdC0YYnIIbE4Dm3mC0WFnQLeSKrRf2ax53a3r9RIAbw89GB4PGnQiccFvhOIQI31HD9rh aurM30OPx0xo84J5pHUlwpqOF+4AG7jKhn195pXTEaqWm+kTRL/LPcSI9vQd5W/gc1UJVYuVAlVDTr780zUBuk/YHqG/O055oyaFHRpB3RMIkFBxzi7PwLndsXoF1Md2N9wdneO+D60lopfXlc/edc2u6i6+Q68WEQ6e80l0JYN+WdsW/hCeQo+cUuzfwSD7p7/qtvrugwoXTJFn5SlzEa8BrisGnoRljxSDXbUj4sVYlqEtgjcKCvczfaSrpRnUHxvPhjNH/cBW2abtLuvkOIpaCiC/tzZ0TIdFjEGH5YJ3gXfOyKQewepabuIwVtvIzcsRE4tk15h1PQdzx9Nmte6U1DGisf//siLQt3a9ueSrKNiEAOyi6o3mpIzD0JtUnW+eEbHBMweCDxv4pn3RMB1WmCJvfynR8AnK8m4NNdJTyDtZ+eYxlbzJvTvCiofvtfWe/QyMiTbH6R0QBGe737Z+lLgEMRHYQnaHVGczZl4TlR1c/qtxzryc8GC4r5FK5m9yho9VkS5dtf+qnVOIa8SrgdLqbMLtNDn9C7jOeH+9gxGGDoX//1UhvGOHXCXNRajn2kK9yhJD6uNALCkMaQGmHuqaP/ZrAq6FMQ0hkb Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id oy9-20020a170907104900b00a331d7d9135si2420918ejb.133.2024.02.03.18.29.38; Sat, 03 Feb 2024 18:29:41 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=Rs8P7Zzh; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3569668D152; Sun, 4 Feb 2024 04:29:26 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oo1-f44.google.com (mail-oo1-f44.google.com [209.85.161.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 21BBE68D125 for ; Sun, 4 Feb 2024 04:29:20 +0200 (EET) Received: by mail-oo1-f44.google.com with SMTP id 006d021491bc7-59a87156cb8so1487276eaf.2 for ; Sat, 03 Feb 2024 18:29:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707013757; x=1707618557; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=DwDNG1GwCQe0TgDZ+LeFJokdO462TWJ8Om8IvHQ0Ols=; b=Rs8P7ZzhLxihLvUOhlSy1rXUNbgR1+4rrD3YhTMKW1gRsOanG46ED1/Gj4oBBKuvs2 Ok7pbDW1Qo2a2BpU7YaEYbgkeShYzAeEudTA41/Ssx4rs4eRYrjZ8fRKBQOMujx/8IYS +ctVjha9LZNcbICZyGAkway0vjjVINebyEG2TGpY/bMViSF7lD4JKbYGFlu1pQ7Sajuk jFwbgwE0d0w20KV5NXC1e8GXpWw6zmEG+nC6fvebYnc3kPtlY18IKvbVtZGFz9+V/39M TGezY83w+EzYnf5/bZI6rGd/YeYnieXKNqt9dgIP7V7/AliKC/ovGeaoeAj+oW6GwneO /ebg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707013757; x=1707618557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DwDNG1GwCQe0TgDZ+LeFJokdO462TWJ8Om8IvHQ0Ols=; b=VsostXF5fopLL5a1OFHQ22Em7+xqteEimaKEVb9Ona6/rQa72mJJT24feyhVSVeH5y 6jOMwKG58ZR6llgzRVmjklNEDVOsvuWL1oB586zHoe9JUA26PVaPpa8Y5X4Kby0TeBa8 m0ALboUEqR4XL6i3Pk2fLDGwtBOXGisxpbDigSVsm5NbUTxbLfXlVAh2O7sDrAccJ0Ee Pd88WHNtfrBPVS9HWTHuKiLFSf6sFVtFuUUYDJOvzfD3vDJ94iDvHixjOCwROITZuu5E Zl5IoXjpjak8TIFxrhZjYdyA5l78sJxVs9hOfZWFNBva7N1Ks7dfaGM2uZW+h/dRkzGM iM3g== X-Gm-Message-State: AOJu0Yyf5U65VEf1r9d1LF3qyBBqrE0veUfIHAi9J4QS3HYyqYl9QYyk 3+9CNdgtKm/u4JSDc72z0D9JJPxTEvHLIkVYQI+U1n4JzX4xK/EZ+sVzzz5C X-Received: by 2002:a05:6359:4112:b0:178:7568:3b57 with SMTP id kh18-20020a056359411200b0017875683b57mr9320392rwc.30.1707013756857; Sat, 03 Feb 2024 18:29:16 -0800 (PST) Received: from localhost.localdomain (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id k5-20020a170902e90500b001d7907eb711sm3785965pld.182.2024.02.03.18.29.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Feb 2024 18:29:16 -0800 (PST) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Sat, 3 Feb 2024 23:29:28 -0300 Message-ID: <20240204022930.14007-2-jamrial@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240204022930.14007-1-jamrial@gmail.com> References: <20240204022930.14007-1-jamrial@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/3 v5] avformat/mov: add support for tile HEIF still images X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: KX2hwo86vgl+ Export each tile as its own stream, and the grid information as a Stream Group of type TILE_GRID. This also enables exporting other stream items like thumbnails, which may be present in non tiled HEIF images too. For those, the primary stream will be tagged with the default disposition. Based on a patch by Swaraj Hota Signed-off-by: James Almer --- Now supports exporting more than one grid if present, as well as the item name for the grid as stream group metadata. libavformat/avformat.h | 6 + libavformat/dump.c | 2 + libavformat/isom.h | 15 +- libavformat/mov.c | 343 +++++++++++++++++++++++++++++++++++++---- 4 files changed, 339 insertions(+), 27 deletions(-) diff --git a/libavformat/avformat.h b/libavformat/avformat.h index 0b1c2e46b5..ad95306efb 100644 --- a/libavformat/avformat.h +++ b/libavformat/avformat.h @@ -811,6 +811,12 @@ typedef struct AVIndexEntry { * The video stream contains still images. */ #define AV_DISPOSITION_STILL_IMAGE (1 << 20) +/** + * The video stream is intended to be merged with another stream before + * presentation. + * Used for example to signal the stream contains a tile from a HEIF grid. + */ +#define AV_DISPOSITION_TILE (1 << 21) /** * @return The AV_DISPOSITION_* flag corresponding to disp or a negative error diff --git a/libavformat/dump.c b/libavformat/dump.c index c9b7369bcd..de0e1d8b39 100644 --- a/libavformat/dump.c +++ b/libavformat/dump.c @@ -640,6 +640,8 @@ static void dump_stream_format(const AVFormatContext *ic, int i, av_log(NULL, log_level, " (still image)"); if (st->disposition & AV_DISPOSITION_NON_DIEGETIC) av_log(NULL, log_level, " (non-diegetic)"); + if (st->disposition & AV_DISPOSITION_TILE) + av_log(NULL, log_level, " (tile)"); av_log(NULL, log_level, "\n"); dump_metadata(NULL, st->metadata, extra_indent, log_level); diff --git a/libavformat/isom.h b/libavformat/isom.h index 77221d06e4..cedbd237d6 100644 --- a/libavformat/isom.h +++ b/libavformat/isom.h @@ -264,15 +264,24 @@ typedef struct MOVStreamContext { typedef struct HEIFItem { AVStream *st; + char *name; int item_id; int64_t extent_length; int64_t extent_offset; - int64_t size; + int tile_rows; + int tile_cols; int width; int height; int type; + int is_idat_relative; } HEIFItem; +typedef struct HEIFGrid { + HEIFItem *item; + int16_t *tile_id_list; + int nb_tiles; +} HEIFGrid; + typedef struct MOVContext { const AVClass *class; ///< class for private options AVFormatContext *fc; @@ -336,6 +345,10 @@ typedef struct MOVContext { int cur_item_id; HEIFItem *heif_item; int nb_heif_item; + HEIFGrid *heif_grid; + int nb_heif_grid; + int thmb_item_id; + int64_t idat_offset; int interleaved_read; } MOVContext; diff --git a/libavformat/mov.c b/libavformat/mov.c index 5fae777adb..237516ed0a 100644 --- a/libavformat/mov.c +++ b/libavformat/mov.c @@ -185,6 +185,30 @@ static int mov_read_mac_string(MOVContext *c, AVIOContext *pb, int len, return p - dst; } +static AVStream *get_curr_st(MOVContext *c) +{ + AVStream *st = NULL; + + if (c->fc->nb_streams < 1) + return NULL; + + for (int i = 0; i < c->nb_heif_item; i++) { + HEIFItem *item = &c->heif_item[i]; + + if (!item->st) + continue; + if (item->st->id != c->cur_item_id) + continue; + + st = item->st; + break; + } + if (!st) + st = c->fc->streams[c->fc->nb_streams-1]; + + return st; +} + static int mov_read_covr(MOVContext *c, AVIOContext *pb, int type, int len) { AVStream *st; @@ -1767,9 +1791,9 @@ static int mov_read_colr(MOVContext *c, AVIOContext *pb, MOVAtom atom) uint16_t color_primaries, color_trc, color_matrix; int ret; - if (c->fc->nb_streams < 1) + st = get_curr_st(c); + if (!st) return 0; - st = c->fc->streams[c->fc->nb_streams - 1]; ret = ffio_read_size(pb, color_parameter_type, 4); if (ret < 0) @@ -2117,9 +2141,9 @@ static int mov_read_glbl(MOVContext *c, AVIOContext *pb, MOVAtom atom) AVStream *st; int ret; - if (c->fc->nb_streams < 1) + st = get_curr_st(c); + if (!st) return 0; - st = c->fc->streams[c->fc->nb_streams-1]; if ((uint64_t)atom.size > (1<<30)) return AVERROR_INVALIDDATA; @@ -4951,12 +4975,10 @@ static int heif_add_stream(MOVContext *c, HEIFItem *item) st->codecpar->codec_type = AVMEDIA_TYPE_VIDEO; st->codecpar->codec_id = mov_codec_id(st, item->type); sc->ffindex = st->index; - c->trak_index = st->index; st->avg_frame_rate.num = st->avg_frame_rate.den = 1; st->time_base.num = st->time_base.den = 1; st->nb_frames = 1; sc->time_scale = 1; - sc = st->priv_data; sc->pb = c->fc->pb; sc->pb_is_copied = 1; @@ -7784,11 +7806,55 @@ static int mov_read_pitm(MOVContext *c, AVIOContext *pb, MOVAtom atom) return atom.size; } +static int mov_read_idat(MOVContext *c, AVIOContext *pb, MOVAtom atom) +{ + c->idat_offset = avio_tell(pb); + return 0; +} + +static int read_image_grid(AVFormatContext *s, AVStreamGroupTileGrid *tile_grid, + HEIFItem *item) +{ + MOVContext *c = s->priv_data; + int64_t offset = 0, pos = avio_tell(s->pb); + uint8_t flags; + + if (!(s->pb->seekable & AVIO_SEEKABLE_NORMAL)) { + av_log(c->fc, AV_LOG_INFO, "grid box with non seekable input\n"); + return AVERROR_PATCHWELCOME; + } + if (item->is_idat_relative) { + if (!c->idat_offset) { + av_log(c->fc, AV_LOG_ERROR, "missing idat box required by the image grid\n"); + return AVERROR_INVALIDDATA; + } + offset = c->idat_offset; + } + + avio_seek(s->pb, item->extent_offset + offset, SEEK_SET); + + avio_r8(s->pb); /* version */ + flags = avio_r8(s->pb); + + item->tile_rows = avio_r8(s->pb) + 1; + item->tile_cols = avio_r8(s->pb) + 1; + /* actual width and height of output image */ + tile_grid->width = (flags & 1) ? avio_rb32(s->pb) : avio_rb16(s->pb); + tile_grid->height = (flags & 1) ? avio_rb32(s->pb) : avio_rb16(s->pb); + + av_log(c->fc, AV_LOG_TRACE, "grid: grid_rows %d grid_cols %d output_width %d output_height %d\n", + item->tile_rows, item->tile_cols, tile_grid->width, tile_grid->height); + + avio_seek(s->pb, pos, SEEK_SET); + + return 0; +} + static int mov_read_iloc(MOVContext *c, AVIOContext *pb, MOVAtom atom) { int version, offset_size, length_size, base_offset_size, index_size; int item_count, extent_count; - uint64_t base_offset, extent_offset, extent_length; + int64_t base_offset, extent_offset, extent_length; uint8_t value; if (c->found_moov) { @@ -7845,6 +7911,8 @@ static int mov_read_iloc(MOVContext *c, AVIOContext *pb, MOVAtom atom) if (rb_size(pb, &extent_offset, offset_size) < 0 || rb_size(pb, &extent_length, length_size) < 0) return AVERROR_INVALIDDATA; + if (offset_type == 1) + c->heif_item[i].is_idat_relative = 1; c->heif_item[i].extent_length = extent_length; c->heif_item[i].extent_offset = base_offset + extent_offset; av_log(c->fc, AV_LOG_TRACE, "iloc: item_idx %d, offset_type %d, " @@ -7859,7 +7927,7 @@ static int mov_read_iloc(MOVContext *c, AVIOContext *pb, MOVAtom atom) static int mov_read_infe(MOVContext *c, AVIOContext *pb, MOVAtom atom) { - char item_name[128]; + AVBPrint item_name; int64_t size = atom.size; uint32_t item_type; int item_id; @@ -7869,27 +7937,32 @@ static int mov_read_infe(MOVContext *c, AVIOContext *pb, MOVAtom atom) avio_rb24(pb); // flags. size -= 4; - if (version != 2) { - av_log(c->fc, AV_LOG_ERROR, "infe: version != 2 not supported\n"); + if (version < 2) { + av_log(c->fc, AV_LOG_ERROR, "infe: version < 2 not supported\n"); return AVERROR_PATCHWELCOME; } - item_id = avio_rb16(pb); + item_id = version > 2 ? avio_rb32(pb) : avio_rb16(pb); avio_rb16(pb); // item_protection_index item_type = avio_rl32(pb); size -= 8; - size -= avio_get_str(pb, INT_MAX, item_name, sizeof(item_name)); - av_log(c->fc, AV_LOG_TRACE, "infe: item_id %d, item_type %s, item_name %s\n", - item_id, av_fourcc2str(item_type), item_name); + av_bprint_init(&item_name, 0, AV_BPRINT_SIZE_UNLIMITED); + ret = ff_read_string_to_bprint_overwrite(pb, &item_name, size); + if (ret < 0) { + av_bprint_finalize(&item_name, NULL); + return ret; + } - // Skip all but the primary item until support is added - if (item_id != c->primary_item_id) - return 0; + av_log(c->fc, AV_LOG_TRACE, "infe: item_id %d, item_type %s, item_name %s\n", + item_id, av_fourcc2str(item_type), item_name.str); + size -= ret + 1; if (size > 0) avio_skip(pb, size); + if (ret) + av_bprint_finalize(&item_name, &c->heif_item[c->cur_item_id].name); c->heif_item[c->cur_item_id].item_id = item_id; c->heif_item[c->cur_item_id].type = item_type; @@ -7900,9 +7973,6 @@ static int mov_read_infe(MOVContext *c, AVIOContext *pb, MOVAtom atom) if (ret < 0) return ret; break; - default: - av_log(c->fc, AV_LOG_TRACE, "infe: ignoring item_type %s\n", av_fourcc2str(item_type)); - break; } c->cur_item_id++; @@ -7959,6 +8029,81 @@ static int mov_read_iref(MOVContext *c, AVIOContext *pb, MOVAtom atom) return mov_read_default(c, pb, atom); } +static int mov_read_dimg(MOVContext *c, AVIOContext *pb, MOVAtom atom) +{ + HEIFItem *item = NULL; + HEIFGrid *grid; + int entries, i; + int from_item_id = avio_rb16(pb); + + for (int i = 0; i < c->nb_heif_grid; i++) { + if (c->heif_grid[i].item->item_id == from_item_id) { + av_log(c->fc, AV_LOG_ERROR, "More than one 'dimg' box referencing the same 'grid'\n"); + return AVERROR_INVALIDDATA; + } + } + for (int i = 0; i < c->nb_heif_item; i++) { + if (c->heif_item[i].item_id != from_item_id) + continue; + item = &c->heif_item[i]; + if (item->type != MKTAG('g','r','i','d')) { + avpriv_report_missing_feature(c->fc, "Derived item of type %s", av_fourcc2str(item->type)); + return 0; + } + break; + } + if (!item) { + av_log(c->fc, AV_LOG_ERROR, "Missing grid information\n"); + return AVERROR_INVALIDDATA; + } + + grid = av_realloc_array(c->heif_grid, c->nb_heif_grid + 1U, + sizeof(*c->heif_grid)); + if (!grid) + return AVERROR(ENOMEM); + c->heif_grid = grid; + grid = &grid[c->nb_heif_grid++]; + + entries = avio_rb16(pb); + grid->tile_id_list = av_malloc_array(entries, sizeof(*grid->tile_id_list)); + if (!grid->tile_id_list) + return AVERROR(ENOMEM); + /* 'to' item ids */ + for (i = 0; i < entries; i++) + grid->tile_id_list[i] = avio_rb16(pb); + grid->nb_tiles = entries; + grid->item = item; + + av_log(c->fc, AV_LOG_TRACE, "dimg: from_item_id %d, entries %d\n", + from_item_id, entries); + + return 0; +} + +static int mov_read_thmb(MOVContext *c, AVIOContext *pb, MOVAtom atom) +{ + int entries; + int to_item_id, from_item_id = avio_rb16(pb); + + entries = avio_rb16(pb); + if (entries > 1) { + avpriv_request_sample(c->fc, "More than one thmb entry"); + return AVERROR_PATCHWELCOME; + } + /* 'to' item ids */ + to_item_id = avio_rb16(pb); + + if (to_item_id != c->primary_item_id) + return 0; + + c->thmb_item_id = from_item_id; + + av_log(c->fc, AV_LOG_TRACE, "thmb: from_item_id %d, entries %d\n", + from_item_id, entries); + + return 0; +} + static int mov_read_ispe(MOVContext *c, AVIOContext *pb, MOVAtom atom) { uint32_t width, height; @@ -8079,10 +8224,6 @@ static int mov_read_iprp(MOVContext *c, AVIOContext *pb, MOVAtom atom) av_log(c->fc, AV_LOG_TRACE, "ipma: property_index %d, item_id %d, item_type %s\n", index + 1, item_id, av_fourcc2str(ref->type)); - // Skip properties referencing items other than the primary item until support is added - if (item_id != c->primary_item_id) - continue; - c->cur_item_id = item_id; ret = mov_read_default(c, &ref->b.pub, @@ -8211,6 +8352,9 @@ static const MOVParseTableEntry mov_default_parse_table[] = { { MKTAG('p','c','m','C'), mov_read_pcmc }, /* PCM configuration box */ { MKTAG('p','i','t','m'), mov_read_pitm }, { MKTAG('e','v','c','C'), mov_read_glbl }, +{ MKTAG('d','i','m','g'), mov_read_dimg }, +{ MKTAG('t','h','m','b'), mov_read_thmb }, +{ MKTAG('i','d','a','t'), mov_read_idat }, { MKTAG('i','r','e','f'), mov_read_iref }, { MKTAG('i','s','p','e'), mov_read_ispe }, { MKTAG('i','p','r','p'), mov_read_iprp }, @@ -8718,7 +8862,12 @@ static int mov_read_close(AVFormatContext *s) av_freep(&mov->aes_decrypt); av_freep(&mov->chapter_tracks); + for (i = 0; i < mov->nb_heif_item; i++) + av_freep(&mov->heif_item[i].name); av_freep(&mov->heif_item); + for (i = 0; i < mov->nb_heif_grid; i++) + av_freep(&mov->heif_grid[i].tile_id_list); + av_freep(&mov->heif_grid); return 0; } @@ -8858,6 +9007,123 @@ fail: return ret; } +static int mov_set_tile_grid_offsets(AVFormatContext *s, AVStreamGroup *stg, + const HEIFGrid *grid) +{ + AVStreamGroupTileGrid *tile_grid = stg->params.tile_grid; + int ret, x = 0, y = 0, i = 0; + + tile_grid->offsets = av_calloc(grid->nb_tiles, sizeof(*tile_grid->offsets)); + if (!tile_grid->offsets) + return AVERROR(ENOMEM); + + while (y < tile_grid->coded_height) { + int left_col = i; + + while (x < tile_grid->coded_width) { + if (i == grid->nb_tiles) { + ret = AVERROR(EINVAL); + goto fail; + } + + tile_grid->offsets[i].x = x; + tile_grid->offsets[i].y = y; + + x += stg->streams[i++]->codecpar->width; + } + + if (x > tile_grid->coded_width) { + avpriv_request_sample(s, "Non uniform HEIF tiles"); + ret = AVERROR_PATCHWELCOME; + goto fail; + } + + x = 0; + y += stg->streams[left_col]->codecpar->height; + } + + if (y > tile_grid->coded_height || i != grid->nb_tiles) { + avpriv_request_sample(s, "Non uniform HEIF tiles"); + ret = AVERROR_PATCHWELCOME; + goto fail; + } + + return 0; +fail: + av_freep(&tile_grid->offsets); + + return ret; +} + +static int mov_parse_tiles(AVFormatContext *s) +{ + MOVContext *mov = s->priv_data; + int err; + + for (int i = 0; i < mov->nb_heif_grid; i++) { + AVStreamGroup *stg = avformat_stream_group_create(s, AV_STREAM_GROUP_PARAMS_TILE_GRID, NULL); + AVStreamGroupTileGrid *tile_grid; + HEIFGrid *grid = &mov->heif_grid[i]; + int coded_width = 0, coded_height = 0; + int size; + + if (!stg) + return AVERROR(ENOMEM); + + tile_grid = stg->params.tile_grid; + err = read_image_grid(s, tile_grid, grid->item); + if (err < 0) + return err; + + for (int j = 0; j < grid->nb_tiles; j++) { + int tile_id = grid->tile_id_list[j]; + + for (int k = 0; k < mov->nb_heif_item; k++) { + const HEIFItem *item = &mov->heif_item[k]; + AVStream *st = item->st; + + if (item->item_id != tile_id) + continue; + if (!st) { + av_log(s, AV_LOG_ERROR, "HEIF tile %d doesn't reference a stream\n", tile_id); + return AVERROR_INVALIDDATA; + } + + st->codecpar->width = item->width; + st->codecpar->height = item->height; + + err = avformat_stream_group_add_stream(stg, st); + if (err == AVERROR(EEXIST)) + return AVERROR_INVALIDDATA; + else if (err < 0) + return err; + + st->disposition |= AV_DISPOSITION_TILE; + break; + } + } + + size = grid->item->tile_rows * grid->item->tile_cols; + for (int i = 0; i < grid->item->tile_cols; i++) + coded_width += stg->streams[i]->codecpar->width; + for (int i = 0; i < size; i += grid->item->tile_cols) + coded_height += stg->streams[i]->codecpar->height; + + tile_grid->coded_width = coded_width; + tile_grid->coded_height = coded_height; + + err = mov_set_tile_grid_offsets(s, stg, grid); + if (err < 0) + return AVERROR_INVALIDDATA; + + if (grid->item->name) + av_dict_set(&stg->metadata, "title", grid->item->name, 0); + } + + + return 0; +} + static int mov_read_header(AVFormatContext *s) { MOVContext *mov = s->priv_data; @@ -8874,6 +9140,8 @@ static int mov_read_header(AVFormatContext *s) mov->fc = s; mov->trak_index = -1; + mov->thmb_item_id = -1; + mov->primary_item_id = -1; /* .mov and .mp4 aren't streamable anyway (only progressive download if moov is before mdat) */ if (pb->seekable & AVIO_SEEKABLE_NORMAL) atom.size = avio_size(pb); @@ -8896,20 +9164,43 @@ static int mov_read_header(AVFormatContext *s) av_log(mov->fc, AV_LOG_TRACE, "on_parse_exit_offset=%"PRId64"\n", avio_tell(pb)); if (mov->found_iloc) { + if (mov->nb_heif_grid) { + err = mov_parse_tiles(s); + if (err < 0) + return err; + } + for (i = 0; i < mov->nb_heif_item; i++) { HEIFItem *item = &mov->heif_item[i]; MOVStreamContext *sc; AVStream *st; + int64_t offset = 0; - if (!item->st) + if (!item->st) { + if (item->item_id == mov->thmb_item_id) { + av_log(s, AV_LOG_ERROR, "HEIF thumbnail doesn't reference a stream\n"); + return AVERROR_INVALIDDATA; + } continue; + } + if (item->is_idat_relative) { + if (!mov->idat_offset) { + av_log(s, AV_LOG_ERROR, "Missing idat box for item %d\n", item->item_id); + return AVERROR_INVALIDDATA; + } + offset = mov->idat_offset; + } st = item->st; sc = st->priv_data; st->codecpar->width = item->width; st->codecpar->height = item->height; + sc->sample_sizes[0] = item->extent_length; - sc->chunk_offsets[0] = item->extent_offset; + sc->chunk_offsets[0] = item->extent_offset + offset; + + if (item->item_id == mov->primary_item_id) + st->disposition |= AV_DISPOSITION_DEFAULT; mov_build_index(mov, st); } From patchwork Sun Feb 4 02:29:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 46004 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:7b08:b0:19e:8a94:b663 with SMTP id s8csp26605pzh; Sat, 3 Feb 2024 18:29:47 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCUV5rcC65DOJ1bF4VpOFxu3VJDPEYdUXqN/BIOPcb3A1OxYfyWKHFTQX+Zu879ySJxdGn+sHq9idO0A0PNwDzJUwAZh+wDLKnCT2g== X-Google-Smtp-Source: AGHT+IEMlMs8vATQkVC9GEtWQR+4an6vlsjMBbFu0ss2xBSvtDOnB07o/17MX+EIYVJZdVf3bCKy X-Received: by 2002:a05:651c:198d:b0:2d0:a1e9:6a23 with SMTP id bx13-20020a05651c198d00b002d0a1e96a23mr783369ljb.5.1707013787239; Sat, 03 Feb 2024 18:29:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1707013787; cv=none; d=google.com; s=arc-20160816; b=TA1JR7Ibb79xt8UB4hrvU7rD0EjYcRgBUekhjjLweQTzgyojplmeXi0d9XQgGcgr5c 1MqIUe99qkxEcGgDKjaZ9dFa5HVBEE2fdkXu3Gj8otibY+E+fU8U4sZTFyd9HpX2XaI9 Om2RYz7nHn87Ps9QDq4pf14NOBECnoVTpDredXnxmNjSsdLk+RbkvphiY9qgy/9egNK7 FbfG5r1iwmRyw/EZVhM4c92DaRNL2C2gT86i98kl3uDJXa3hDK53Qhlq9ggzOhDU9jZG S7In+e+sa9lCldpAhUHOq+apW/szEbvIw/R1LVd5DI+9LgoiP5jbIeSrvWOtEAHuCOKM eOGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=PyfI8YqsrEl6IBrG7hmQgTCO2H2dJr+8hLjl94CJDRI=; fh=qsIOO2Tyulru765JyCrwn4H8fCu2sI4BxYM3rAREKc8=; b=dIhfxLM04Mt5MTzhYTUB9WryUCGAKgrVFYoC8sXytBpa9ChOec8pm9SL+QxxvhM2I9 C6f6z6RuKAI/zHDcV1F6KyhYaJyS00tj+08Daff0UXqSvhblWo0vXuf8YiL+dv96Ixk0 IIFHUnY9p6vXdLkwe+aJgO//tiQFhrshrjqyotxT6iMnIojETzdSsK2wMQkcwZGOZyqr g/yoJYBoJlsm2nrkbkXFA6dUV+DcHLX2lQSlw0TswQXXhEM3mIXsCXFRx2fwosVfhJKn g+m8PkhPYFsz9nCuIrI6giokD/H/rbhTROUj5qlQ12cbXTrJqoGrUmdIfg0spArq3+H2 ZpUQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=l4tks+3t; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=0; AJvYcCVySYnRo5EfuM169JxWhepR8/SA5lfCOb1en2VX2t7pdnRyCY/rN82X8Sssx4ekVPoqWkPMhpVIVoaKIS8He7mJ78mxbK0qrPYsINqrEEbM8wpPYAYeOugsD0WTE8mp6ml+9pLPj2MtrxkF4DM510idcLwCWqQ//HG9HV+cXOs8rKDVC7CTtfMWmYwRMklu1n/rEf3ZhqhcHW4R4NaeCtn+F5Z5YOYtDhNalqJlstJlwch1TUj69uU5J48hAtnr0vJH7VayiPpz2GJ0qpeNDGn9N2ApOuc+xzVrJQ561cmOY81OB2uk77NTpj5iiEsyoYwDsCTzBR6j++2ENfJELWa7RZP8vS5cPrXzZ9WEbRriW+jaXmZzhLHvoR2XIawizG+tV7M+i6v7I3/XYPdDPQJgSTqeihtTcMYQE9fix3vZaaFFxz8lOLH4nrFR9ea7YCEfGdkbtVdFRAMAfrEQUgQKRvudJtXkwNkPw5SkJtoMyaDqCtZsDeM37JG+YhYSfUZtl6ISv7jkcPVwin3SwFqwJ19bvPrsFymFLraXFvtx22vMwzirBwbdKjTQie0vrMVLnNVRb+fev/rU94rkOFupqCVmHw8JL7pqpFbHS/wSasXDQoV/WtfIfRml0qQffWfJ4ou9HObzC6RDTYrkOkbb+BiQKar8qbeREb6wrJhIgF7Wy5fgnjCAqse8Pok0JwQ5nzr/49564PcUmi3fwhCDdI1finesLk2/hEpCzhZ7GEmo7KZy/FYJO4Y7JQuo0k/MV3VYqEXEWeyjDnve0+/ixo/UK2dqWzaM/NdKAsQAL+NPU3DvOCV9urzO0zxWiuqknS3qP61rfWmSrWwKnEYoimwoLKY6nnhl+KiHdtoZF4N5NMk4OpCaBOaZ9zgGh7pjNRSX4e54rCVeio4DVOhdbimy93HhfzNeragdzf7UMHrYcDs1L+QwY+O2sFoOwCMG5Y 9uCY8+6qm5XtxPuKMbz5n4v7AYDYV0E06ecYyDoHa4oNsjc6Uv5J0Kf1CaicjDA6xNOh5qQHHLP1ZA40CkIadNKvskeizFoS+R3FSQpM0l+m+ERbVu3YA+ciU186Xg3jpTolV1a2QNIeHkadR3QF+dYycYFOFwmmg4ZQAiG8+NKvQ1B3tlDULXIJuji/w01CsjwbXr1j2ytX6TCc94GKXJqRoGkUYxqxuUz5YvLj+DX70geL7kfAbYnV1aGRAS1dEyhPbwM5EPlYpHIzTorFBja7kBSGcer37H1zrwh0hDpiwCTR6VG9ICICxMatxvFK+Yqo2qqPkBV9WpNZJMGKgihNd+k/vIEI7gG0e+Dcok0bebuA3w7x2EjbtWgeyg37zMirWHRy1GPYTHnouYjCp+nO97vI463msZBuPuk12WBu+3QW15DwzblWRyfb//w8ZyfawP262iFXGPi0APx74FT+CI2H7TUOxDIaIMCH4+115QMWCzDqD2tjGH+vRE154Tt3d1GBwxwkqMF77Ay++z3Lu9SKtm+7Ki+lImsdZW0Tb/lSoFROdxYkY/dntEC4v67rfN5cO9l9jcdZc2cgKqmDEOGEOL8xvzSQR9 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ee1-20020a056402290100b00559c666d29esi2345968edb.64.2024.02.03.18.29.46; Sat, 03 Feb 2024 18:29:47 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=l4tks+3t; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5E9DC68D169; Sun, 4 Feb 2024 04:29:28 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 954D968D147 for ; Sun, 4 Feb 2024 04:29:21 +0200 (EET) Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-5d8df2edd29so2653593a12.2 for ; Sat, 03 Feb 2024 18:29:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707013759; x=1707618559; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=jOYggBII/NVJ4RA3zaGI32DaZjs6skcHo5rj0oqdn0E=; b=l4tks+3tofNO8s9sKyJWs91sqCXKe+jgvaXm8/bwGSIqjnXt7x5mOp/hbbu3EmRaWB SQMqvZHjLaAFUGYadm9swOHG/J1pkBE26DveDRd1Bi4ndmmJWT3lKgMMFeHQxRbp1z5Y 82jox0q1wZjPJetD52QZH6dy1sjbutIOjonizIhElSfLwapnzLzOb0VFh8l2eU8ixgFE rPxAKHWxhNgQ5LEZVk9HBvlWJ6GH2/b08PqqXprLXtryjzfeCwIHDsY4I8Ii1ZtBxYF8 m9BhqCYSOpyqG0+wQbwGEZ+YAdtLxi/HPxIvI2IZDPySwfXEu6pcS9fDjwd2cpR/85p3 1ofg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707013759; x=1707618559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jOYggBII/NVJ4RA3zaGI32DaZjs6skcHo5rj0oqdn0E=; b=s5zeJmmDKDwthyFjTrNuGa4RxUczXOoTdifFi0v6oe4ZBv8jUntqRlLWHPRQdTllvL mlTRW8QZBrGqzpjTUC7hiFfU+0N799Cg/S3Qq/GhX1KYDuhkngQPJRz07TV9srQRekdg zlwAtUp8hXDUL/Mz6KelY9Wo4bdClfFUtioPzamUycTtoAOKJhIdxQ8BhUWtrKuUQLCO /tdF8DtmujGtohzIPzhUtnjKm9QULhmzwZ0cKIoBYWXw1yI6TfQMrrlLacWsOcfMoNoh HqTDbLvnBsq9adx0ZL2Ym7+5sL5jFMVfa1zTrk84ZIwzOekUbPadtmLY1NXA+jn8rYtr oNpQ== X-Gm-Message-State: AOJu0Yy58mVVHRycPneLwgR+zyjrtLveVVOLuLtVWwkb3XBdJDUmZ1hG UPyEXdJdpCG2s148EiJUewtV0f1LrzefTzJJcpS2u+cjLYLulpfIu53hcLTG X-Received: by 2002:a17:902:cf4c:b0:1d9:8820:f413 with SMTP id e12-20020a170902cf4c00b001d98820f413mr4315163plg.38.1707013758794; Sat, 03 Feb 2024 18:29:18 -0800 (PST) Received: from localhost.localdomain (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id k5-20020a170902e90500b001d7907eb711sm3785965pld.182.2024.02.03.18.29.17 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Feb 2024 18:29:17 -0800 (PST) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Sat, 3 Feb 2024 23:29:29 -0300 Message-ID: <20240204022930.14007-3-jamrial@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240204022930.14007-1-jamrial@gmail.com> References: <20240204022930.14007-1-jamrial@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/3 v4] fate/mov: test remuxing all stream heif items X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: PdHVKIpzWB46 Signed-off-by: James Almer --- No changes since last version. tests/fate/mov.mak | 2 +- tests/ref/fate/mov-heic-demux-still-image-multiple-items | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/tests/fate/mov.mak b/tests/fate/mov.mak index f202f36d96..f549ae33d7 100644 --- a/tests/fate/mov.mak +++ b/tests/fate/mov.mak @@ -156,7 +156,7 @@ fate-mov-heic-demux-still-image-1-item: CMD = framemd5 -i $(TARGET_SAMPLES)/heif FATE_MOV_FFMPEG-$(call FRAMEMD5, MOV, HEVC, HEVC_PARSER) \ += fate-mov-heic-demux-still-image-multiple-items -fate-mov-heic-demux-still-image-multiple-items: CMD = framemd5 -i $(TARGET_SAMPLES)/heif-conformance/C003.heic -c:v copy +fate-mov-heic-demux-still-image-multiple-items: CMD = framemd5 -i $(TARGET_SAMPLES)/heif-conformance/C003.heic -c:v copy -map 0 # Resulting remux should have: # 1. first audio stream with AV_DISPOSITION_HEARING_IMPAIRED diff --git a/tests/ref/fate/mov-heic-demux-still-image-multiple-items b/tests/ref/fate/mov-heic-demux-still-image-multiple-items index c850c1ff9c..753cef267a 100644 --- a/tests/ref/fate/mov-heic-demux-still-image-multiple-items +++ b/tests/ref/fate/mov-heic-demux-still-image-multiple-items @@ -2,10 +2,17 @@ #version: 2 #hash: MD5 #extradata 0, 100, 5444bf01e03182c73ae957179d560f4d +#extradata 1, 100, 5444bf01e03182c73ae957179d560f4d #tb 0: 1/1 #media_type 0: video #codec_id 0: hevc #dimensions 0: 1280x720 #sar 0: 0/1 +#tb 1: 1/1 +#media_type 1: video +#codec_id 1: hevc +#dimensions 1: 1280x720 +#sar 1: 0/1 #stream#, dts, pts, duration, size, hash 0, 0, 0, 1, 111554, 03ceabfab39afd2e2e796b9362111f32 +1, 0, 0, 1, 112393, daa001d351c088a5bc328459e2501c95