From patchwork Wed Feb 7 12:49:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 46094 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:a38a:b0:19e:8a94:b663 with SMTP id w10csp503035pzk; Wed, 7 Feb 2024 04:49:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IEIQelQKIGJZBe+3fzpSTUkSPBWd78JMCY4gm2ihPwswhro9QuRqzrKTuJVVNYd1jer8a9f X-Received: by 2002:a17:907:61a7:b0:a37:bdc2:e4e6 with SMTP id mt39-20020a17090761a700b00a37bdc2e4e6mr4014485ejc.10.1707310191566; Wed, 07 Feb 2024 04:49:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1707310191; cv=none; d=google.com; s=arc-20160816; b=jN5Xw29qIV1kW6daDiaPDEf53FKNZUV/Izf5pEw8UnOpmWYcXUbzpdKqtfRL7f59J5 FCjf8ZoKiiv8RsE97EGEOwZcKSQoSmNOGFnkNFbN/yT9cU5Kok8I/5qVVneshV0FkJhX FhSjiiYYL1Cy0D/YqmzXR60aaDj3MnOTynxsWhb7cj5BLwXSBYExN0nbdDCAz0kuKMFF O1jLqaXhd81BzJ1aX/XmnwH/Xds2qbjLJmRoZCHfRB5OT2XCRaT66gboA7ALtHUnfOUb uVzFKKkp4Kqn8vozkYCjclb5THkXJT79NsYW3sYiddaElqUphIisLYPt5D0BwrZlfoTW 9kKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=sLu6gOSPmtsjjfryY5EwiRHqKDE84T4T4/bm5L9GSCI=; fh=VaTgHG5XlI4OIx3EZDE9q2f2AaSSGEvqy9g+zCg+FKw=; b=Xqxv5MK8C0gMq326FjVyC2gHHWSa8yfz8z26+gb54ryMsTigkfIGPtI/oPZNYM9hF5 ixaiYsRGhfqa0u9SdM1eH648TRkfN7B3J5HDDj8MqoJUjASKjc/BMmzd7poXg9mCV/U9 omc6Fc/0wHK6et4QvIcjPIU8WfN86QhXT7tnq8nsXvmts9AMvdr5B4h9u5X1iBheMhNa lnJ/Np8ntY+Wzg/Y0XCBwkMWuyH7dTBpKPH/9/myAMUXHAVwerKj1nYB7/4WOu7e0vq0 cb9olKjGBh1zv329p3fXsHNl5qCDAd+2KcE8DSBi+7YtvuUgyrS+kuy6EsdokSD3wu5K CzNQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=GMAmUdjJ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=1; AJvYcCUeuXcJWl/H/Dq/yhMtkwdjGvOAmHpuYt6B6MU9CEybAzXfSzlQDfc1km7Dc5nWf9D91nEYlkx527t6x5oUSGP/oZ+Zaf5iQJASFQKZzMNHeqi5z5s58rNzGCK0j2iHx1CXCvymT2zf0CuSbi810bn1zbRqdlz+oF086tJ9M58VCItSRVJVKa8nQaaCZ1rFrgwf3Zb1qlm0r9OGVyWes9cDp99CvaxvdXP6g09Nt91NFsgHbm3m+8mCm5VACG4+DSMYWFKLyeAtpSxQvdrh0XeP7sRT7DVoI6V2mZvz+6Mupjr2GwNUzxXibNqZrfJ4wPvbHvi6rXILjmOjXeApyaQiXc82lNtzttiMoTBZpUErUJ4LNXB6vEjOy+VRZm65jYldBtW/+JE6ZKIxlXxflz7m/YP5t6KUZidKX9sl4+VWpDYz+qiA2KS8Ly0TLE5XyI+D2jkzDhDwQB7a+SwPquDcXQqTuGy0p+PPXMi/lYPBFAtfoxoDTUGZSdKySCUdQKOZsvkHkVZH1Pw5Opu740QWudZE4eVWxDOuCs9GoiXEBdBG0CzkvDAUUn8ATWByQseQzCodO6RVtbh9t77XArp2gkekBBBtwqKuKYtAcaX51sPTTpwx+7ORVcqPP3VuqcyLiP5x9XC+DLgvux3DEoGTUve86fSnSBYnxP+S1hbv4q3rF4D3Zz878ed1Wjv3Q+2hKHh48B2p+ygDy4wYrCVtq0wtkHU7IM/OMBgDbDIlEr784aExNVWBjqXvkH6copMduiuCXeb7U0QbbIFnYGN3DktnnIwbgtNq5XvzWuffTwQNiHODIKfnp80zDadyx5h+FV097Ay2OwZoj3N5MLCI4g1TFtg1R+G1VBw6/DVpXCzKuDPeFyv+g9ymotXnPBwgLKISIJrF/rea46s8wLnNcjm3HtedcYXvfM6cFxLUPmGnOfpeQM6YBJGYre9bfUg5Va qsI4O0PKv4e+UuxlJ9zcoi3Nu+Ck8LI+vAs+4o0r9wMTLT0h5TAFeM6nGpAlZ74XVFZK1SEuOjUXFNWItFOOHYo4dNqjiIU1mvSSVP4UWDKA6PRSG8tzSRSmSrJBR6OdmQ88DL5WXaeAwzL/86Ehbojod0cuwwb2t+K4quVBJQfzIqiKXsTgDAlRgUP0PHkckELYKo4FQXowFaUQJa/B0QqiTmyr8wofQy7yiY+FYrqZgHiHEbGmSEhJ36X5ePMQIs8xulz9acWjKXDy9g5CKMc8+CQ5OM6QL8zD3q27f2dtKeHA2i6C7LGPm4Vuwzs9sMbmd60whNbb8fIefLx9bj457V/fXy4U0yaMG24/LH3Y4kP0b3Y9p4k5e8j5Rom0bh2OMoHyWaU6St0ToInVE9FRQn685e6POKTx7LjL6upG58v7OpbkA8FYz7s398lHzxxE2oba0J1glNQFK/ZDBeAyO3HTsHch3f10Yk0WAfxSCi74GANBDSJXywBfakcu2WyjsJcqjm8hSQzPokHo3gVFTlzFAerMLVfA4q3jfBKDd5nLpS2JHwqP9AQKqvK8Dcq0+Y9u+EAE/ENbs41SbmAtLCXAxVtOXFXO2mqFwePGXKII7ECvXUXHzNDlc= Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id lo27-20020a170906fa1b00b00a388c552722si353696ejb.696.2024.02.07.04.49.51; Wed, 07 Feb 2024 04:49:51 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=GMAmUdjJ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D98CA68D14E; Wed, 7 Feb 2024 14:49:39 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6946168D147 for ; Wed, 7 Feb 2024 14:49:32 +0200 (EET) Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-296d24631c0so401513a91.3 for ; Wed, 07 Feb 2024 04:49:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707310169; x=1707914969; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=LEE/oThqcwARbXu9xsEf7Qs5wf0sTkKTQW3l199PIi0=; b=GMAmUdjJxgyNcUjDzYGQJK97hwfSIU/b931c83ITijT1fabHu/FsvsaL2D4F2pCu2f zttU5wpVQK7580FH/62xWOYq2s1yd1EiqK10f8rFLMf30EB11qhCz61Mohx8LolJ1c9q gLh0PwSx+MJyZiyjIEW8BqSCNN3fr5g+5bTq/8znNl21hv6t+ij+IJeiI94o28Iaz2DE Z/hRLffWQWXk8N5dxIU2zb3NGpOnH6ALr945mUhw+O04ts14tYLfOMVTNcEGRHjOQCn3 qtDp/KFTDLC1NVmbQkEKJ5VYcO4wVTHWNx5eOarchqitDufMgMQT7D6MzPyJ0YojkMVc zLcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707310169; x=1707914969; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LEE/oThqcwARbXu9xsEf7Qs5wf0sTkKTQW3l199PIi0=; b=pwdzeqhMgAmveoZ+nVcxGchUKMcJi8V2Plbv/SM/4CMRb3AZ8+pcrt4AxoWIBajBio 6WnOL2mRoVr0TO7VxpyMIKrl137oMG6z8xaALjdsf2XmU7jhxszwAP/M1wVBDDNkJez7 toWDjD5DN5uyCESX2Tj09F5aYcUeByT8KM3KmeWSS0KgAl5J8PqI9yWPbzSU7hB80aTx d0KTufsNMkV41D6vh5H0hbh6fBZQ5SE8kbcC+9NnDR2IP/dcp1tur2NNNty00WnBUTQj GgiB5svTPDbDxeKYphTNK5KiqzyUL06OyF0AVo2hK5CBFIHjQxWL41FOXEcviLq11kkE oeKQ== X-Gm-Message-State: AOJu0Yx/QrRJja6gw38IX5q2kiYncMVm2azfpWEMvKOieWV5WWOVLtGi f5NjJ4wEdEgo8fK4vYDaXUDuq+N3IT2mT7iN7kpWLbLIZ6kXeuDvh/XzX97q X-Received: by 2002:a17:90a:2ecb:b0:296:e77f:2a36 with SMTP id h11-20020a17090a2ecb00b00296e77f2a36mr842236pjs.38.1707310169016; Wed, 07 Feb 2024 04:49:29 -0800 (PST) Received: from localhost.localdomain (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id nh12-20020a17090b364c00b002969d5db646sm3872987pjb.9.2024.02.07.04.49.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Feb 2024 04:49:28 -0800 (PST) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Wed, 7 Feb 2024 09:49:12 -0300 Message-ID: <20240207124913.4081-2-jamrial@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207124913.4081-1-jamrial@gmail.com> References: <20240207124913.4081-1-jamrial@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/3 v6] avformat/mov: add support for tile HEIF still images X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: LY6tZsCTrTKz Export each tile as its own stream, and the grid information as a Stream Group of type TILE_GRID. This also enables exporting other stream items like thumbnails, which may be present in non tiled HEIF images too. For those, the primary stream will be tagged with the default disposition. Based on a patch by Swaraj Hota Signed-off-by: James Almer --- Now supporting more samples from the heif conformance set. libavformat/avformat.c | 8 + libavformat/avformat.h | 6 + libavformat/dump.c | 2 + libavformat/internal.h | 5 + libavformat/isom.h | 15 +- libavformat/mov.c | 427 ++++++++++++++++++++++++++++++++++++----- 6 files changed, 418 insertions(+), 45 deletions(-) diff --git a/libavformat/avformat.c b/libavformat/avformat.c index 32ef440207..c60e3fcb9d 100644 --- a/libavformat/avformat.c +++ b/libavformat/avformat.c @@ -120,6 +120,14 @@ void ff_remove_stream(AVFormatContext *s, AVStream *st) ff_free_stream(&s->streams[ --s->nb_streams ]); } +void ff_remove_stream_group(AVFormatContext *s, AVStreamGroup *stg) +{ + av_assert0(s->nb_stream_groups > 0); + av_assert0(s->stream_groups[ s->nb_stream_groups - 1 ] == stg); + + ff_free_stream_group(&s->stream_groups[ --s->nb_stream_groups ]); +} + /* XXX: suppress the packet queue */ void ff_flush_packet_queue(AVFormatContext *s) { diff --git a/libavformat/avformat.h b/libavformat/avformat.h index 0b1c2e46b5..ad95306efb 100644 --- a/libavformat/avformat.h +++ b/libavformat/avformat.h @@ -811,6 +811,12 @@ typedef struct AVIndexEntry { * The video stream contains still images. */ #define AV_DISPOSITION_STILL_IMAGE (1 << 20) +/** + * The video stream is intended to be merged with another stream before + * presentation. + * Used for example to signal the stream contains a tile from a HEIF grid. + */ +#define AV_DISPOSITION_TILE (1 << 21) /** * @return The AV_DISPOSITION_* flag corresponding to disp or a negative error diff --git a/libavformat/dump.c b/libavformat/dump.c index 2c019c36e4..6f6a67d7bc 100644 --- a/libavformat/dump.c +++ b/libavformat/dump.c @@ -655,6 +655,8 @@ static void dump_stream_format(const AVFormatContext *ic, int i, av_log(NULL, log_level, " (still image)"); if (st->disposition & AV_DISPOSITION_NON_DIEGETIC) av_log(NULL, log_level, " (non-diegetic)"); + if (st->disposition & AV_DISPOSITION_TILE) + av_log(NULL, log_level, " (tile)"); av_log(NULL, log_level, "\n"); dump_metadata(NULL, st->metadata, extra_indent, log_level); diff --git a/libavformat/internal.h b/libavformat/internal.h index f93832b3c4..9fb5397f27 100644 --- a/libavformat/internal.h +++ b/libavformat/internal.h @@ -635,6 +635,11 @@ void ff_remove_stream(AVFormatContext *s, AVStream *st); * is not yet attached to an AVFormatContext. */ void ff_free_stream_group(AVStreamGroup **pstg); +/** + * Remove a stream group from its AVFormatContext and free it. + * The stream group must be the last stream group of the AVFormatContext. + */ +void ff_remove_stream_group(AVFormatContext *s, AVStreamGroup *stg); unsigned int ff_codec_get_tag(const AVCodecTag *tags, enum AVCodecID id); diff --git a/libavformat/isom.h b/libavformat/isom.h index a4cca4c798..ade6d88faf 100644 --- a/libavformat/isom.h +++ b/libavformat/isom.h @@ -267,15 +267,24 @@ typedef struct MOVStreamContext { typedef struct HEIFItem { AVStream *st; + char *name; int item_id; int64_t extent_length; int64_t extent_offset; - int64_t size; + int tile_rows; + int tile_cols; int width; int height; int type; + int is_idat_relative; } HEIFItem; +typedef struct HEIFGrid { + HEIFItem *item; + int16_t *tile_id_list; + int nb_tiles; +} HEIFGrid; + typedef struct MOVContext { const AVClass *class; ///< class for private options AVFormatContext *fc; @@ -339,6 +348,10 @@ typedef struct MOVContext { int cur_item_id; HEIFItem *heif_item; int nb_heif_item; + HEIFGrid *heif_grid; + int nb_heif_grid; + int thmb_item_id; + int64_t idat_offset; int interleaved_read; } MOVContext; diff --git a/libavformat/mov.c b/libavformat/mov.c index 42b0135987..01d6a0543b 100644 --- a/libavformat/mov.c +++ b/libavformat/mov.c @@ -185,6 +185,30 @@ static int mov_read_mac_string(MOVContext *c, AVIOContext *pb, int len, return p - dst; } +static AVStream *get_curr_st(MOVContext *c) +{ + AVStream *st = NULL; + + if (c->fc->nb_streams < 1) + return NULL; + + for (int i = 0; i < c->nb_heif_item; i++) { + HEIFItem *item = &c->heif_item[i]; + + if (!item->st) + continue; + if (item->st->id != c->cur_item_id) + continue; + + st = item->st; + break; + } + if (!st) + st = c->fc->streams[c->fc->nb_streams-1]; + + return st; +} + static int mov_read_covr(MOVContext *c, AVIOContext *pb, int type, int len) { AVStream *st; @@ -1767,9 +1791,9 @@ static int mov_read_colr(MOVContext *c, AVIOContext *pb, MOVAtom atom) uint16_t color_primaries, color_trc, color_matrix; int ret; - if (c->fc->nb_streams < 1) + st = get_curr_st(c); + if (!st) return 0; - st = c->fc->streams[c->fc->nb_streams - 1]; ret = ffio_read_size(pb, color_parameter_type, 4); if (ret < 0) @@ -2117,9 +2141,9 @@ static int mov_read_glbl(MOVContext *c, AVIOContext *pb, MOVAtom atom) AVStream *st; int ret; - if (c->fc->nb_streams < 1) + st = get_curr_st(c); + if (!st) return 0; - st = c->fc->streams[c->fc->nb_streams-1]; if ((uint64_t)atom.size > (1<<30)) return AVERROR_INVALIDDATA; @@ -4951,12 +4975,10 @@ static int heif_add_stream(MOVContext *c, HEIFItem *item) st->codecpar->codec_type = AVMEDIA_TYPE_VIDEO; st->codecpar->codec_id = mov_codec_id(st, item->type); sc->ffindex = st->index; - c->trak_index = st->index; st->avg_frame_rate.num = st->avg_frame_rate.den = 1; st->time_base.num = st->time_base.den = 1; st->nb_frames = 1; sc->time_scale = 1; - sc = st->priv_data; sc->pb = c->fc->pb; sc->pb_is_copied = 1; @@ -7809,11 +7831,56 @@ static int mov_read_pitm(MOVContext *c, AVIOContext *pb, MOVAtom atom) return atom.size; } +static int mov_read_idat(MOVContext *c, AVIOContext *pb, MOVAtom atom) +{ + c->idat_offset = avio_tell(pb); + return 0; +} + +static int read_image_grid(AVFormatContext *s, AVStreamGroupTileGrid *tile_grid, + HEIFItem *item) +{ + MOVContext *c = s->priv_data; + int64_t offset = 0, pos = avio_tell(s->pb); + uint8_t flags; + + if (!(s->pb->seekable & AVIO_SEEKABLE_NORMAL)) { + av_log(c->fc, AV_LOG_INFO, "grid box with non seekable input\n"); + return AVERROR_PATCHWELCOME; + } + if (item->is_idat_relative) { + if (!c->idat_offset) { + av_log(c->fc, AV_LOG_ERROR, "missing idat box required by the image grid\n"); + return AVERROR_INVALIDDATA; + } + offset = c->idat_offset; + } + + avio_seek(s->pb, item->extent_offset + offset, SEEK_SET); + + avio_r8(s->pb); /* version */ + flags = avio_r8(s->pb); + + item->tile_rows = avio_r8(s->pb) + 1; + item->tile_cols = avio_r8(s->pb) + 1; + /* actual width and height of output image */ + tile_grid->width = (flags & 1) ? avio_rb32(s->pb) : avio_rb16(s->pb); + tile_grid->height = (flags & 1) ? avio_rb32(s->pb) : avio_rb16(s->pb); + + av_log(c->fc, AV_LOG_TRACE, "grid: grid_rows %d grid_cols %d output_width %d output_height %d\n", + item->tile_rows, item->tile_cols, tile_grid->width, tile_grid->height); + + avio_seek(s->pb, pos, SEEK_SET); + + return 0; +} + static int mov_read_iloc(MOVContext *c, AVIOContext *pb, MOVAtom atom) { + HEIFItem *heif_item; int version, offset_size, length_size, base_offset_size, index_size; int item_count, extent_count; - uint64_t base_offset, extent_offset, extent_length; + int64_t base_offset, extent_offset, extent_length; uint8_t value; if (c->found_moov) { @@ -7832,17 +7899,20 @@ static int mov_read_iloc(MOVContext *c, AVIOContext *pb, MOVAtom atom) base_offset_size = (value >> 4) & 0xF; index_size = !version ? 0 : (value & 0xF); if (index_size) { - av_log(c->fc, AV_LOG_ERROR, "iloc: index_size != 0 not supported.\n"); + avpriv_report_missing_feature(c->fc, "iloc: index_size != 0"); return AVERROR_PATCHWELCOME; } item_count = (version < 2) ? avio_rb16(pb) : avio_rb32(pb); - if (!c->heif_item) { - c->heif_item = av_calloc(item_count, sizeof(*c->heif_item)); - if (!c->heif_item) - return AVERROR(ENOMEM); - c->nb_heif_item = item_count; - } + heif_item = av_realloc_array(c->heif_item, item_count, sizeof(*c->heif_item)); + if (!heif_item) + return AVERROR(ENOMEM); + c->heif_item = heif_item; + if (item_count > c->nb_heif_item) + memset(c->heif_item + c->nb_heif_item, 0, + sizeof(*c->heif_item) * (item_count - c->nb_heif_item)); + c->nb_heif_item = item_count; + c->cur_item_id = 0; av_log(c->fc, AV_LOG_TRACE, "iloc: item_count %d\n", item_count); for (int i = 0; i < item_count; i++) { @@ -7852,7 +7922,7 @@ static int mov_read_iloc(MOVContext *c, AVIOContext *pb, MOVAtom atom) if (avio_feof(pb)) return AVERROR_INVALIDDATA; if (offset_type > 1) { - avpriv_request_sample(c->fc, "iloc offset type %d", offset_type); + avpriv_report_missing_feature(c->fc, "iloc offset type %d", offset_type); return AVERROR_PATCHWELCOME; } c->heif_item[i].item_id = item_id; @@ -7863,13 +7933,15 @@ static int mov_read_iloc(MOVContext *c, AVIOContext *pb, MOVAtom atom) extent_count = avio_rb16(pb); if (extent_count > 1) { // For still AVIF images, we only support one extent item. - av_log(c->fc, AV_LOG_ERROR, "iloc: extent_count > 1 not supported\n"); + avpriv_report_missing_feature(c->fc, "iloc: extent_count > 1"); return AVERROR_PATCHWELCOME; } for (int j = 0; j < extent_count; j++) { if (rb_size(pb, &extent_offset, offset_size) < 0 || rb_size(pb, &extent_length, length_size) < 0) return AVERROR_INVALIDDATA; + if (offset_type == 1) + c->heif_item[i].is_idat_relative = 1; c->heif_item[i].extent_length = extent_length; c->heif_item[i].extent_offset = base_offset + extent_offset; av_log(c->fc, AV_LOG_TRACE, "iloc: item_idx %d, offset_type %d, " @@ -7884,7 +7956,7 @@ static int mov_read_iloc(MOVContext *c, AVIOContext *pb, MOVAtom atom) static int mov_read_infe(MOVContext *c, AVIOContext *pb, MOVAtom atom) { - char item_name[128]; + AVBPrint item_name; int64_t size = atom.size; uint32_t item_type; int item_id; @@ -7894,27 +7966,32 @@ static int mov_read_infe(MOVContext *c, AVIOContext *pb, MOVAtom atom) avio_rb24(pb); // flags. size -= 4; - if (version != 2) { - av_log(c->fc, AV_LOG_ERROR, "infe: version != 2 not supported\n"); + if (version < 2) { + av_log(c->fc, AV_LOG_ERROR, "infe: version < 2 not supported\n"); return AVERROR_PATCHWELCOME; } - item_id = avio_rb16(pb); + item_id = version > 2 ? avio_rb32(pb) : avio_rb16(pb); avio_rb16(pb); // item_protection_index item_type = avio_rl32(pb); size -= 8; - size -= avio_get_str(pb, INT_MAX, item_name, sizeof(item_name)); - av_log(c->fc, AV_LOG_TRACE, "infe: item_id %d, item_type %s, item_name %s\n", - item_id, av_fourcc2str(item_type), item_name); + av_bprint_init(&item_name, 0, AV_BPRINT_SIZE_UNLIMITED); + ret = ff_read_string_to_bprint_overwrite(pb, &item_name, size); + if (ret < 0) { + av_bprint_finalize(&item_name, NULL); + return ret; + } - // Skip all but the primary item until support is added - if (item_id != c->primary_item_id) - return 0; + av_log(c->fc, AV_LOG_TRACE, "infe: item_id %d, item_type %s, item_name %s\n", + item_id, av_fourcc2str(item_type), item_name.str); + size -= ret + 1; if (size > 0) avio_skip(pb, size); + if (ret) + av_bprint_finalize(&item_name, &c->heif_item[c->cur_item_id].name); c->heif_item[c->cur_item_id].item_id = item_id; c->heif_item[c->cur_item_id].type = item_type; @@ -7925,9 +8002,6 @@ static int mov_read_infe(MOVContext *c, AVIOContext *pb, MOVAtom atom) if (ret < 0) return ret; break; - default: - av_log(c->fc, AV_LOG_TRACE, "infe: ignoring item_type %s\n", av_fourcc2str(item_type)); - break; } c->cur_item_id++; @@ -7937,6 +8011,7 @@ static int mov_read_infe(MOVContext *c, AVIOContext *pb, MOVAtom atom) static int mov_read_iinf(MOVContext *c, AVIOContext *pb, MOVAtom atom) { + HEIFItem *heif_item; int entry_count; int version, ret; @@ -7954,13 +8029,14 @@ static int mov_read_iinf(MOVContext *c, AVIOContext *pb, MOVAtom atom) avio_rb24(pb); // flags. entry_count = version ? avio_rb32(pb) : avio_rb16(pb); - if (!c->heif_item) { - c->heif_item = av_calloc(entry_count, sizeof(*c->heif_item)); - if (!c->heif_item) - return AVERROR(ENOMEM); - c->nb_heif_item = entry_count; - } - + heif_item = av_realloc_array(c->heif_item, entry_count, sizeof(*c->heif_item)); + if (!heif_item) + return AVERROR(ENOMEM); + c->heif_item = heif_item; + if (entry_count > c->nb_heif_item) + memset(c->heif_item + c->nb_heif_item, 0, + sizeof(*c->heif_item) * (entry_count - c->nb_heif_item)); + c->nb_heif_item = entry_count; c->cur_item_id = 0; for (int i = 0; i < entry_count; i++) { @@ -7977,11 +8053,119 @@ static int mov_read_iinf(MOVContext *c, AVIOContext *pb, MOVAtom atom) return 0; } +static int mov_read_iref_dimg(MOVContext *c, AVIOContext *pb, int version) +{ + HEIFItem *item = NULL; + HEIFGrid *grid; + int entries, i; + int from_item_id = version ? avio_rb32(pb) : avio_rb16(pb); + + for (int i = 0; i < c->nb_heif_grid; i++) { + if (c->heif_grid[i].item->item_id == from_item_id) { + av_log(c->fc, AV_LOG_ERROR, "More than one 'dimg' box " + "referencing the same 'grid'\n"); + return AVERROR_INVALIDDATA; + } + } + for (int i = 0; i < c->nb_heif_item; i++) { + if (c->heif_item[i].item_id != from_item_id) + continue; + item = &c->heif_item[i]; + if (item->type != MKTAG('g','r','i','d')) { + avpriv_report_missing_feature(c->fc, "Derived item of type %s", + av_fourcc2str(item->type)); + return 0; + } + break; + } + if (!item) { + av_log(c->fc, AV_LOG_ERROR, "Missing grid information\n"); + return AVERROR_INVALIDDATA; + } + + grid = av_realloc_array(c->heif_grid, c->nb_heif_grid + 1U, + sizeof(*c->heif_grid)); + if (!grid) + return AVERROR(ENOMEM); + c->heif_grid = grid; + grid = &grid[c->nb_heif_grid++]; + + entries = avio_rb16(pb); + grid->tile_id_list = av_malloc_array(entries, sizeof(*grid->tile_id_list)); + if (!grid->tile_id_list) + return AVERROR(ENOMEM); + /* 'to' item ids */ + for (i = 0; i < entries; i++) + grid->tile_id_list[i] = version ? avio_rb32(pb) : avio_rb16(pb); + grid->nb_tiles = entries; + grid->item = item; + + av_log(c->fc, AV_LOG_TRACE, "dimg: from_item_id %d, entries %d\n", + from_item_id, entries); + + return 0; +} + +static int mov_read_iref_thmb(MOVContext *c, AVIOContext *pb, int version) +{ + int entries; + int to_item_id, from_item_id = version ? avio_rb32(pb) : avio_rb16(pb); + + entries = avio_rb16(pb); + if (entries > 1) { + avpriv_request_sample(c->fc, "thmb in iref referencing several items"); + return AVERROR_PATCHWELCOME; + } + /* 'to' item ids */ + to_item_id = version ? avio_rb32(pb) : avio_rb16(pb); + + if (to_item_id != c->primary_item_id) + return 0; + + c->thmb_item_id = from_item_id; + + av_log(c->fc, AV_LOG_TRACE, "thmb: from_item_id %d, entries %d\n", + from_item_id, entries); + + return 0; +} + static int mov_read_iref(MOVContext *c, AVIOContext *pb, MOVAtom atom) { - avio_rb32(pb); /* version and flags */ + int version = avio_r8(pb); + avio_rb24(pb); // flags atom.size -= 4; - return mov_read_default(c, pb, atom); + + if (version > 1) { + av_log(c->fc, AV_LOG_WARNING, "Unknown iref box version %d\n", version); + return 0; + } + + while (atom.size) { + uint32_t type, size = avio_rb32(pb); + int64_t next = avio_tell(pb); + + if (size < 14 || next < 0 || next > INT64_MAX - size) + return AVERROR_INVALIDDATA; + + next += size - 4; + type = avio_rl32(pb); + switch (type) { + case MKTAG('d','i','m','g'): + mov_read_iref_dimg(c, pb, version); + break; + case MKTAG('t','h','m','b'): + mov_read_iref_thmb(c, pb, version); + break; + default: + av_log(c->fc, AV_LOG_DEBUG, "Unknown iref type %s size %"PRIu32"\n", + av_fourcc2str(type), size); + } + + atom.size -= size; + avio_seek(pb, next, SEEK_SET); + } + return 0; } static int mov_read_ispe(MOVContext *c, AVIOContext *pb, MOVAtom atom) @@ -8104,10 +8288,6 @@ static int mov_read_iprp(MOVContext *c, AVIOContext *pb, MOVAtom atom) av_log(c->fc, AV_LOG_TRACE, "ipma: property_index %d, item_id %d, item_type %s\n", index + 1, item_id, av_fourcc2str(ref->type)); - // Skip properties referencing items other than the primary item until support is added - if (item_id != c->primary_item_id) - continue; - c->cur_item_id = item_id; ret = mov_read_default(c, &ref->b.pub, @@ -8236,6 +8416,7 @@ static const MOVParseTableEntry mov_default_parse_table[] = { { MKTAG('p','c','m','C'), mov_read_pcmc }, /* PCM configuration box */ { MKTAG('p','i','t','m'), mov_read_pitm }, { MKTAG('e','v','c','C'), mov_read_glbl }, +{ MKTAG('i','d','a','t'), mov_read_idat }, { MKTAG('i','r','e','f'), mov_read_iref }, { MKTAG('i','s','p','e'), mov_read_ispe }, { MKTAG('i','p','r','p'), mov_read_iprp }, @@ -8745,7 +8926,12 @@ static int mov_read_close(AVFormatContext *s) av_freep(&mov->aes_decrypt); av_freep(&mov->chapter_tracks); + for (i = 0; i < mov->nb_heif_item; i++) + av_freep(&mov->heif_item[i].name); av_freep(&mov->heif_item); + for (i = 0; i < mov->nb_heif_grid; i++) + av_freep(&mov->heif_grid[i].tile_id_list); + av_freep(&mov->heif_grid); return 0; } @@ -8885,6 +9071,134 @@ fail: return ret; } +static int mov_set_tile_grid_offsets(AVFormatContext *s, AVStreamGroup *stg, + const HEIFGrid *grid) +{ + AVStreamGroupTileGrid *tile_grid = stg->params.tile_grid; + int ret, x = 0, y = 0, i = 0; + + tile_grid->offsets = av_calloc(grid->nb_tiles, sizeof(*tile_grid->offsets)); + if (!tile_grid->offsets) + return AVERROR(ENOMEM); + + while (y < tile_grid->coded_height) { + int left_col = i; + + while (x < tile_grid->coded_width) { + if (i == grid->nb_tiles) { + ret = AVERROR(EINVAL); + goto fail; + } + + tile_grid->offsets[i].x = x; + tile_grid->offsets[i].y = y; + + x += stg->streams[i++]->codecpar->width; + } + + if (x > tile_grid->coded_width) { + avpriv_request_sample(s, "Non uniform HEIF tiles"); + ret = AVERROR_PATCHWELCOME; + goto fail; + } + + x = 0; + y += stg->streams[left_col]->codecpar->height; + } + + if (y > tile_grid->coded_height || i != grid->nb_tiles) { + avpriv_request_sample(s, "Non uniform HEIF tiles"); + ret = AVERROR_PATCHWELCOME; + goto fail; + } + + return 0; +fail: + av_freep(&tile_grid->offsets); + + return ret; +} + +static int mov_parse_tiles(AVFormatContext *s) +{ + MOVContext *mov = s->priv_data; + int err; + + for (int i = 0; i < mov->nb_heif_grid; i++) { + AVStreamGroup *stg = avformat_stream_group_create(s, AV_STREAM_GROUP_PARAMS_TILE_GRID, NULL); + AVStreamGroupTileGrid *tile_grid; + HEIFGrid *grid = &mov->heif_grid[i]; + int coded_width = 0, coded_height = 0; + int size, loop = 1; + + if (!stg) + return AVERROR(ENOMEM); + + stg->id = grid->item->item_id; + tile_grid = stg->params.tile_grid; + err = read_image_grid(s, tile_grid, grid->item); + if (err < 0) + return err; + + for (int j = 0; j < grid->nb_tiles; j++) { + int tile_id = grid->tile_id_list[j]; + + for (int k = 0; k < mov->nb_heif_item; k++) { + const HEIFItem *item = &mov->heif_item[k]; + AVStream *st = item->st; + + if (item->item_id != tile_id) + continue; + if (!st) { + av_log(s, AV_LOG_WARNING, "HEIF item id %d from grid id %d doesn't " + "reference a stream\n", + tile_id, grid->item->item_id); + ff_remove_stream_group(s, stg); + loop = 0; + break; + } + + st->codecpar->width = item->width; + st->codecpar->height = item->height; + + err = avformat_stream_group_add_stream(stg, st); + if (err == AVERROR(EEXIST)) + return AVERROR_INVALIDDATA; + else if (err < 0) + return err; + + st->disposition |= AV_DISPOSITION_TILE; + break; + } + + if (!loop) + break; + } + + if (!loop) + continue; + + size = grid->item->tile_rows * grid->item->tile_cols; + for (int i = 0; i < grid->item->tile_cols; i++) + coded_width += stg->streams[i]->codecpar->width; + for (int i = 0; i < size; i += grid->item->tile_cols) + coded_height += stg->streams[i]->codecpar->height; + + tile_grid->coded_width = coded_width; + tile_grid->coded_height = coded_height; + + err = mov_set_tile_grid_offsets(s, stg, grid); + if (err < 0) + return AVERROR_INVALIDDATA; + + if (grid->item->name) + av_dict_set(&stg->metadata, "title", grid->item->name, 0); + } + + + return 0; +} + static int mov_read_header(AVFormatContext *s) { MOVContext *mov = s->priv_data; @@ -8901,6 +9215,8 @@ static int mov_read_header(AVFormatContext *s) mov->fc = s; mov->trak_index = -1; + mov->thmb_item_id = -1; + mov->primary_item_id = -1; /* .mov and .mp4 aren't streamable anyway (only progressive download if moov is before mdat) */ if (pb->seekable & AVIO_SEEKABLE_NORMAL) atom.size = avio_size(pb); @@ -8923,20 +9239,43 @@ static int mov_read_header(AVFormatContext *s) av_log(mov->fc, AV_LOG_TRACE, "on_parse_exit_offset=%"PRId64"\n", avio_tell(pb)); if (mov->found_iloc) { + if (mov->nb_heif_grid) { + err = mov_parse_tiles(s); + if (err < 0) + return err; + } + for (i = 0; i < mov->nb_heif_item; i++) { HEIFItem *item = &mov->heif_item[i]; MOVStreamContext *sc; AVStream *st; + int64_t offset = 0; - if (!item->st) + if (!item->st) { + if (item->item_id == mov->thmb_item_id) { + av_log(s, AV_LOG_ERROR, "HEIF thumbnail doesn't reference a stream\n"); + return AVERROR_INVALIDDATA; + } continue; + } + if (item->is_idat_relative) { + if (!mov->idat_offset) { + av_log(s, AV_LOG_ERROR, "Missing idat box for item %d\n", item->item_id); + return AVERROR_INVALIDDATA; + } + offset = mov->idat_offset; + } st = item->st; sc = st->priv_data; st->codecpar->width = item->width; st->codecpar->height = item->height; + sc->sample_sizes[0] = item->extent_length; - sc->chunk_offsets[0] = item->extent_offset; + sc->chunk_offsets[0] = item->extent_offset + offset; + + if (item->item_id == mov->primary_item_id) + st->disposition |= AV_DISPOSITION_DEFAULT; mov_build_index(mov, st); }