From patchwork Mon Jun 17 19:20:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derek Buitenhuis X-Patchwork-Id: 49987 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6919:8997:b0:23b:d18a:469b with SMTP id jz23csp2078156ysc; Mon, 17 Jun 2024 12:21:24 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCX6rEKTcALNcZiE7Y/x/07DWDPp9UVSHT37F8wwUsTjdgrgV85VMttUD9CILzmUrkmZc4OzWeAzoUswAbO6p5/IApXNlde0yp8Dcw== X-Google-Smtp-Source: AGHT+IG799E7q7bBq1lpRh86igVtCOxWmIFYAJldS1L2r1lYZyt8d4AlMvHkX+LYfBh79uTkQnxT X-Received: by 2002:a17:907:9447:b0:a6f:6150:3f5c with SMTP id a640c23a62f3a-a6f61504062mr865756966b.76.1718652084398; Mon, 17 Jun 2024 12:21:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1718652084; cv=none; d=google.com; s=arc-20160816; b=Ls9D8Mf6MWr3oFVH8hjSPtA7BZosIVb0pWSweXsuAKRBvOtpohHFjtbeWXa8XXbzOn qgXdMPmtKyM1DnPcj44T2n5J34ERfDvpkh9oXVt0GWB+cg4dyvgYf4EBNMjlrpDFkwbC DcVOD1RMcx99mU/VaFVJpJHKjvYnMToCsCcD9Cdi/Lc4AHM2McsAIagxCjVmlyjYZFx6 IQAjuEvw9J8TaKvH8LGdU9vFJ5zNjyilRFJnAqgUzbZELK526lkryKBXyBeyPo5wj/kW VPrywLfueV2/mD8Xt9FoCqh8dAywpYzySf+S7VIwq7l9X8v6RHrXyhlwWj3jBUnB0cVj xnlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=MSTFPDiNAHLNves4D8ibyJc0QFk0PaKA+hsa/JNOEmU=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=W+AS44fKG82ZQ+iwffmnehjZAmYCMDaN6lfjvai8NDfgalfE4vf0NIBM6vkUYwdnvI C2F+5XuKIXnqn88p+LvJnJ7sS/zxXlwCbNyhgv5Ot3T9qv1HeigSxtu/V3LfpI7RevdH Yph4i1sHCshNWthrssfUQZrGfCD6xAybZWdIAU1KZJMxSNmwV1T9sVbrkTrLjqv8sGvF 9Z/U4XhVQ3ANyjUZTUVmd3LMjDTYsdj7rvi7tDiQnupu7XBpa1uSphMKu9tsCyPIThQ5 7J4nx8oQCbPdcGWI+y41lK4fmKSov0RpefTo+6g/F2n60qnG2dlggQps6Pm7TuZV90B7 ZThw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b="b3N7IF/Y"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a640c23a62f3a-a6f8e53ebbfsi57880466b.805.2024.06.17.12.21.19; Mon, 17 Jun 2024 12:21:24 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b="b3N7IF/Y"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1226968D7AC; Mon, 17 Jun 2024 22:20:47 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 18AA068D711 for ; Mon, 17 Jun 2024 22:20:36 +0300 (EEST) Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-356c4e926a3so4433731f8f.1 for ; Mon, 17 Jun 2024 12:20:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718652035; x=1719256835; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=zWOWjZ5f74HFO6osESaqpNjmIOWO031L/UmmQ4NxUII=; b=b3N7IF/YtATiZt9e+OBdnX3hOVzty9IMEYwwGSq4yjGnqKBebfELz9B74k+ZeYL/rt 6/1GZr246Lu8VNxhkQ9cjlZJPUkKcrV1O7Vac5ChuPWmsa32KxGYh2IlmmgL3w7tkCdf klaqYDAEm3ewn5JC6u9mefGxNLztxilLEvCwnplNW9Wr1eQCV8YN/v4ljC0fu4QafZ8k po63AbWup8LRIXVCGkzBxqPLkgcxgPUjaQACZ2JPkJurM+XzgmYn+jMSrc1a0UNcK8O0 GsubaM0PSC+RVLsGX56G4lBWrqF2Dpw2Sckd5c1NeYbfuPV3gKCUSx9i4ZFszfbPLaO6 D1lQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718652035; x=1719256835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zWOWjZ5f74HFO6osESaqpNjmIOWO031L/UmmQ4NxUII=; b=Vh+dTekTj9bjeCJyhfaa/71ycVPh5gKuhPICrQJuNCzk7vZyTFCYLrUd4NVBqFt6BS CCCQ3rSxEAZw+Z8KMuOhH1O8DmPOpGxwW+btwHLLB7Ro3slhP4g3heglUHupIVzfGema qmlFfpmOIh5dIKzDphu3KWvrI5qc4TtEsEl7NhlTBLWFdxtviWh7Gyo/JOYKKzlSugPs 205vN31sD/Mntug0OQPwRsGP80Be98rNsrHEUuQzlROCflGoSuLzRE6CLd/vqzeY7fZZ rtuwNoME3fv3tSo3BKXmNJ+UwKOip30TThTaM+qYa4eODRO8WoHVo/PSTnX0OMHYv67k gDGw== X-Gm-Message-State: AOJu0YxB/OlG8MZzqg0OAv6ATYv8YnFhbDiD8lcIb2H5FpivoXe9Fyes KBTbCWVmADGmyK9xnDjYDJIBf0ug9FqNA5/XrM/RJp3XGD0YDvdyNzkVhQ== X-Received: by 2002:adf:f003:0:b0:360:9cd3:c496 with SMTP id ffacd0b85a97d-3609cd3c612mr1455450f8f.54.1718652034323; Mon, 17 Jun 2024 12:20:34 -0700 (PDT) Received: from localhost.localdomain (0540dce9.skybroadband.com. [5.64.220.233]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3607509353csm12531797f8f.22.2024.06.17.12.20.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Jun 2024 12:20:33 -0700 (PDT) From: Derek Buitenhuis To: ffmpeg-devel@ffmpeg.org Date: Mon, 17 Jun 2024 20:20:18 +0100 Message-ID: <20240617192019.512769-5-derek.buitenhuis@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240617192019.512769-1-derek.buitenhuis@gmail.com> References: <20240617192019.512769-1-derek.buitenhuis@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 4/5] avformat/mov: Add support for exporting Video Extension Usage info X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ENoEoeUYOtl3 This box is provided by files created by the Apple Vision Pro, as well as the iPhone 15+ when capture for Vision Pro is enabled. The boxes are a mix of things documented by Apple in some PDFs, their API docs, and reverse engineering. Ideally we will have a real spec one day. Links: * https://developer.apple.com/av-foundation/Stereo-Video-ISOBMFF-Extensions.pdf * https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_horizontaldisparityadjustment * https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_stereocamerabaseline * https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_heroeye Signed-off-by: Derek Buitenhuis --- libavformat/mov.c | 283 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 283 insertions(+) diff --git a/libavformat/mov.c b/libavformat/mov.c index 9016cd5ad0..5724b4ef93 100644 --- a/libavformat/mov.c +++ b/libavformat/mov.c @@ -6477,6 +6477,288 @@ static int mov_read_sv3d(MOVContext *c, AVIOContext *pb, MOVAtom atom) return 0; } +static int mov_read_vexu_proj(MOVContext *c, AVIOContext *pb, MOVAtom atom) +{ + AVStream *st; + MOVStreamContext *sc; + int size; + uint32_t tag; + enum AVSphericalProjection projection; + + if (c->fc->nb_streams < 1) + return 0; + + st = c->fc->streams[c->fc->nb_streams - 1]; + sc = st->priv_data; + + if (atom.size != 16) { + av_log(c->fc, AV_LOG_ERROR, "Invalid size for proj box: %"PRIu64"\n", atom.size); + return AVERROR_INVALIDDATA; + } + + size = avio_rb32(pb); + if (size != 16) { + av_log(c->fc, AV_LOG_ERROR, "Invalid size for prji box: %d\n", size); + return AVERROR_INVALIDDATA; + } + + tag = avio_rl32(pb); + if (tag != MKTAG('p','r','j','i')) { + av_log(c->fc, AV_LOG_ERROR, "Invalid child box of proj box: 0x%08X\n", tag); + return AVERROR_INVALIDDATA; + } + + avio_skip(pb, 1); // version + avio_skip(pb, 3); // flags + + tag = avio_rl32(pb); + switch (tag) { + case MKTAG('r','e','c','t'): + projection = AV_SPHERICAL_RECTILINEAR; + break; + case MKTAG('e','q','u','i'): + projection = AV_SPHERICAL_EQUIRECTANGULAR; + break; + case MKTAG('h','e','q','u'): + projection = AV_SPHERICAL_HALF_EQUIRECTANGULAR; + break; + case MKTAG('f','i','s','h'): + projection = AV_SPHERICAL_FISHEYE; + break; + default: + av_log(c->fc, AV_LOG_ERROR, "Invalid projection type in prji box: 0x%08X\n", tag); + return AVERROR_INVALIDDATA; + } + + sc->spherical = av_spherical_alloc(&sc->spherical_size); + if (!sc->spherical) + return AVERROR(ENOMEM); + + sc->spherical->projection = projection; + + return 0; +} + +static int mov_read_eyes(MOVContext *c, AVIOContext *pb, MOVAtom atom) +{ + AVStream *st; + MOVStreamContext *sc; + int size, flags = 0; + int64_t remaining; + uint32_t tag, baseline = 0; + enum AVStereo3DView view = AV_STEREO3D_VIEW_PACKED; + enum AVStereo3DPrimaryEye primary_eye = AV_PRIMARY_EYE_NONE; + AVRational horizontal_disparity_adjustment = { 0, 0 }; + + if (c->fc->nb_streams < 1) + return 0; + + st = c->fc->streams[c->fc->nb_streams - 1]; + sc = st->priv_data; + + remaining = atom.size; + while (remaining > 0) { + size = avio_rb32(pb); + if (size < 8 || size > remaining ) { + av_log(c->fc, AV_LOG_ERROR, "Invalid child size in eyes box\n"); + return AVERROR_INVALIDDATA; + } + + tag = avio_rl32(pb); + switch (tag) { + case MKTAG('s','t','r','i'): { + int has_right, has_left; + uint8_t tmp; + if (size != 13) { + av_log(c->fc, AV_LOG_ERROR, "Invalid size of stri box: %d\n", size); + return AVERROR_INVALIDDATA; + } + avio_skip(pb, 1); // version + avio_skip(pb, 3); // flags + + tmp = avio_r8(pb); + + // eye_views_reversed + if (tmp & 8) { + flags |= AV_STEREO3D_FLAG_INVERT; + } + // has_additional_views + if (tmp & 4) { + // skip... + } + + has_right = tmp & 2; // has_right_eye_view + has_left = tmp & 1; // has_left_eye_view + + if (has_left && has_right) + view = AV_STEREO3D_VIEW_PACKED; + else if (has_left) + view = AV_STEREO3D_VIEW_LEFT; + else if (has_right) + view = AV_STEREO3D_VIEW_RIGHT; + break; + } + case MKTAG('h','e','r','o'): { + int tmp; + if (size != 13) { + av_log(c->fc, AV_LOG_ERROR, "Invalid size of hero box: %d\n", size); + return AVERROR_INVALIDDATA; + } + avio_skip(pb, 1); // version + avio_skip(pb, 3); // flags + + tmp = avio_r8(pb); + if (tmp == 0) + primary_eye = AV_PRIMARY_EYE_NONE; + else if (tmp == 1) + primary_eye = AV_PRIMARY_EYE_LEFT; + else if (tmp == 2) + primary_eye = AV_PRIMARY_EYE_RIGHT; + else + av_log(c->fc, AV_LOG_WARNING, "Unknown hero eye type: %d\n", tmp); + + break; + } + case MKTAG('c','a','m','s'): { + uint32_t subtag; + int subsize; + if (size != 24) { + av_log(c->fc, AV_LOG_ERROR, "Invalid size of cams box: %d\n", size); + return AVERROR_INVALIDDATA; + } + + subsize = avio_rb32(pb); + if (subsize != 16) { + av_log(c->fc, AV_LOG_ERROR, "Invalid size of blin box: %d\n", size); + return AVERROR_INVALIDDATA; + } + + subtag = avio_rl32(pb); + if (subtag != MKTAG('b','l','i','n')) { + av_log(c->fc, AV_LOG_ERROR, "Expected blin box, got 0x%08X\n", subtag); + return AVERROR_INVALIDDATA; + } + + avio_skip(pb, 1); // version + avio_skip(pb, 3); // flags + + baseline = avio_rb32(pb); + + break; + } + case MKTAG('c','m','f','y'): { + uint32_t subtag; + int subsize; + int32_t adjustment; + if (size != 24) { + av_log(c->fc, AV_LOG_ERROR, "Invalid size of cmfy box: %d\n", size); + return AVERROR_INVALIDDATA; + } + + subsize = avio_rb32(pb); + if (subsize != 16) { + av_log(c->fc, AV_LOG_ERROR, "Invalid size of dadj box: %d\n", size); + return AVERROR_INVALIDDATA; + } + + subtag = avio_rl32(pb); + if (subtag != MKTAG('d','a','d','j')) { + av_log(c->fc, AV_LOG_ERROR, "Expected dadj box, got 0x%08X\n", subtag); + return AVERROR_INVALIDDATA; + } + + avio_skip(pb, 1); // version + avio_skip(pb, 3); // flags + + adjustment = (int32_t) avio_rb32(pb); + + horizontal_disparity_adjustment.num = (int) adjustment; + horizontal_disparity_adjustment.den = 10000; + + break; + } + default: + av_log(c->fc, AV_LOG_WARNING, "Unknown tag in eyes: 0x%08X\n", tag); + avio_skip(pb, size - 8); + break; + } + remaining -= size; + } + + if (remaining != 0) { + av_log(c->fc, AV_LOG_ERROR, "Broken eyes box\n"); + return AVERROR_INVALIDDATA; + } + + if (!sc->stereo3d) { + sc->stereo3d = av_stereo3d_alloc(); + if (!sc->stereo3d) + return AVERROR(ENOMEM); + } + + sc->stereo3d->flags = flags; + sc->stereo3d->view = view; + sc->stereo3d->primary_eye = primary_eye; + sc->stereo3d->baseline = baseline; + sc->stereo3d->horizontal_disparity_adjustment = horizontal_disparity_adjustment; + + return 0; +} + +static int mov_read_vexu(MOVContext *c, AVIOContext *pb, MOVAtom atom) +{ + int size; + int64_t remaining; + uint32_t tag; + + if (c->fc->nb_streams < 1) + return 0; + + if (atom.size < 8) { + av_log(c->fc, AV_LOG_ERROR, "Empty video extension usage box\n"); + return AVERROR_INVALIDDATA; + } + + remaining = atom.size; + while (remaining > 0) { + size = avio_rb32(pb); + if (size < 8 || size > remaining ) { + av_log(c->fc, AV_LOG_ERROR, "Invalid child size in vexu box\n"); + return AVERROR_INVALIDDATA; + } + + tag = avio_rl32(pb); + switch (tag) { + case MKTAG('p','r','o','j'): { + MOVAtom proj = { tag, size - 8 }; + int ret = mov_read_vexu_proj(c, pb, proj); + if (ret < 0) + return ret; + break; + } + case MKTAG('e','y','e','s'): { + MOVAtom eyes = { tag, size - 8 }; + int ret = mov_read_eyes(c, pb, eyes); + if (ret < 0) + return ret; + break; + } + default: + av_log(c->fc, AV_LOG_WARNING, "Unknown tag in vexu: 0x%08X\n", tag); + avio_skip(pb, size - 8); + break; + } + remaining -= size; + } + + if (remaining != 0) { + av_log(c->fc, AV_LOG_ERROR, "Broken vexu box\n"); + return AVERROR_INVALIDDATA; + } + + return 0; +} + static int mov_parse_uuid_spherical(MOVStreamContext *sc, AVIOContext *pb, size_t len) { int ret = 0; @@ -8595,6 +8877,7 @@ static const MOVParseTableEntry mov_default_parse_table[] = { { MKTAG('d','f','L','a'), mov_read_dfla }, { MKTAG('s','t','3','d'), mov_read_st3d }, /* stereoscopic 3D video box */ { MKTAG('s','v','3','d'), mov_read_sv3d }, /* spherical video box */ +{ MKTAG('v','e','x','u'), mov_read_vexu }, /* video extension usage */ { MKTAG('d','O','p','s'), mov_read_dops }, { MKTAG('d','m','l','p'), mov_read_dmlp }, { MKTAG('S','m','D','m'), mov_read_smdm },