From patchwork Thu Nov 17 15:53:02 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vittorio Giovara X-Patchwork-Id: 1456 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.90.1 with SMTP id o1csp841381vsb; Thu, 17 Nov 2016 07:53:13 -0800 (PST) X-Received: by 10.194.43.73 with SMTP id u9mr2569174wjl.109.1479397993845; Thu, 17 Nov 2016 07:53:13 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d75si14478328wmd.3.2016.11.17.07.53.13; Thu, 17 Nov 2016 07:53:13 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 431E5689F4F; Thu, 17 Nov 2016 17:53:10 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt0-f194.google.com (mail-qt0-f194.google.com [209.85.216.194]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3045E689E4B for ; Thu, 17 Nov 2016 17:53:04 +0200 (EET) Received: by mail-qt0-f194.google.com with SMTP id n6so14579156qtd.0 for ; Thu, 17 Nov 2016 07:53:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8l9VV8D0M66fe6cGI4kBz5fJS8ozoANtiLEVwT816S0=; b=jAHZcJYp55s59y0fcYQOVnEsQN49GnQkwoc4K4enyuBQk7+NMTjZTA73CV3I5yuc2y X+FKzCjmB55wqB6VcAB+gj6hYc/A7/BTHg/OulFqsqbwINGO0As1WEKyldv8H69fNsRg RxISTVpVQw2zp0DUpk+fOyKo6cu5Uyi89sIc6lG9bXKcvcURhzGYFfj7m0yI2Dpu09fr Ou60l4MmsqxhaS4tGYtHp+ChT3UG18Bh9aG1s+OrD4goajZTakb8RBm+5ph+e2oIkFhq nZKk6f3xDMky5XSLnJqw39VIP7v2dk9uK4qDTX2LZ0ElUpsV8jCMaicUEmIfmcGffe32 tXVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8l9VV8D0M66fe6cGI4kBz5fJS8ozoANtiLEVwT816S0=; b=IwvWSiiSaFRdpyQzLaxy2GBNBeaROYLOehjSHyceqSEvU477RPMKGDHidYyIV6zt7Z c1szjZngvq6Tno4JeZ0gKm6HyK2/qsUt1IfsQRBMlkpj0uzBWxG4eyTFchqnlOBrZECr 4i5dGbTruiqDPgS2fqK4bsaR4jjevPOROEwC51y0mGsWIZs4dlV8nl6f/zdLCf3B5a7j YkHQiTEZKhRl65nlZ43XzLLO1ZDG4rBrpdZIsrOXDl8X4D/eYd2HPlvDpzlcvhAVzYej Vw5d/0rFc0XyhDvOec2ZJ2AwEuwiDsSF+gMvOOqriH4OzifYnieTOMSDk+2p81PKVWhe g5yA== X-Gm-Message-State: AKaTC025NRvAVNYJC1H0ESSLxtf7z5yUZLNxaP9WZW7eX4+jSLxp02kQGdYnJc8k++uAoA== X-Received: by 10.200.47.38 with SMTP id j35mr2555305qta.136.1479397984145; Thu, 17 Nov 2016 07:53:04 -0800 (PST) Received: from vimacbookpro.vimeows.com (nyv-exweb.iac.com. [216.112.252.10]) by smtp.gmail.com with ESMTPSA id f21sm1739429qkh.17.2016.11.17.07.53.03 (version=TLS1 cipher=AES128-SHA bits=128/128); Thu, 17 Nov 2016 07:53:03 -0800 (PST) From: Vittorio Giovara To: ffmpeg-devel@ffmpeg.org Date: Thu, 17 Nov 2016 10:53:02 -0500 Message-Id: <20161117155302.89338-1-vittorio.giovara@gmail.com> X-Mailer: git-send-email 2.10.0 In-Reply-To: References: Subject: [FFmpeg-devel] [PATCH v2] mov: Export spherical information X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: michael@niedermayer.cc, jamrial@gmail.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This implements Spherical Video V1 and V2, as described in the spatial-media collection by Google. Signed-off-by: Vittorio Giovara --- Updated following James' review. Please CC. Vittorio libavformat/isom.h | 7 ++ libavformat/mov.c | 309 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 315 insertions(+), 1 deletion(-) diff --git a/libavformat/isom.h b/libavformat/isom.h index 02bfedd..0fd9eb0 100644 --- a/libavformat/isom.h +++ b/libavformat/isom.h @@ -24,6 +24,9 @@ #ifndef AVFORMAT_ISOM_H #define AVFORMAT_ISOM_H +#include "libavutil/spherical.h" +#include "libavutil/stereo3d.h" + #include "avio.h" #include "internal.h" #include "dv.h" @@ -177,6 +180,10 @@ typedef struct MOVStreamContext { int stsd_count; int32_t *display_matrix; + AVStereo3D *stereo3d; + AVSphericalMapping *spherical; + size_t spherical_size; + uint32_t format; int has_sidx; // If there is an sidx entry for this stream. diff --git a/libavformat/mov.c b/libavformat/mov.c index 6beb027..7d02fe0 100644 --- a/libavformat/mov.c +++ b/libavformat/mov.c @@ -42,6 +42,8 @@ #include "libavutil/aes.h" #include "libavutil/aes_ctr.h" #include "libavutil/sha.h" +#include "libavutil/spherical.h" +#include "libavutil/stereo3d.h" #include "libavutil/timecode.h" #include "libavcodec/ac3tab.h" #include "libavcodec/mpegaudiodecheader.h" @@ -4497,8 +4499,258 @@ static int mov_read_tmcd(MOVContext *c, AVIOContext *pb, MOVAtom atom) return 0; } +static int mov_read_st3d(MOVContext *c, AVIOContext *pb, MOVAtom atom) +{ + AVStream *st; + MOVStreamContext *sc; + enum AVStereo3DType type; + int mode; + + if (c->fc->nb_streams < 1) + return 0; + + st = c->fc->streams[c->fc->nb_streams - 1]; + sc = st->priv_data; + + if (atom.size < 5) { + av_log(c->fc, AV_LOG_ERROR, "Empty stereoscopic video box\n"); + return AVERROR_INVALIDDATA; + } + + mode = avio_r8(pb); + switch (mode) { + case 0: + type = AV_STEREO3D_2D; + break; + case 1: + type = AV_STEREO3D_TOPBOTTOM; + break; + case 2: + type = AV_STEREO3D_SIDEBYSIDE; + break; + default: + av_log(c->fc, AV_LOG_WARNING, "Unknown st3d mode value %d\n", mode); + return 0; + } + + sc->stereo3d = av_stereo3d_alloc(); + if (!sc->stereo3d) + return AVERROR(ENOMEM); + + sc->stereo3d->type = type; + return 0; +} + +static int mov_read_sv3d(MOVContext *c, AVIOContext *pb, MOVAtom atom) +{ + AVStream *st; + MOVStreamContext *sc; + int version, size; + int32_t yaw, pitch, roll; + uint32_t tag; + unsigned l, t, r, b; + enum AVSphericalProjection projection; + + if (c->fc->nb_streams < 1) + return 0; + + st = c->fc->streams[c->fc->nb_streams - 1]; + sc = st->priv_data; + + if (atom.size < 8) { + av_log(c->fc, AV_LOG_ERROR, "Empty spherical video box\n"); + return AVERROR_INVALIDDATA; + } + + version = avio_r8(pb); + if (version) { + av_log(c->fc, AV_LOG_WARNING, "Unsupported svhd box version (%d)\n", + version); + return 0; + } + + size = avio_rb24(pb); + if (size > atom.size) + return AVERROR_INVALIDDATA; + + tag = avio_rl32(pb); + if (tag != MKTAG('s','v','h','d')) { + av_log(c->fc, AV_LOG_ERROR, "Missing spherical video header\n"); + return 0; + } + avio_skip(pb, size - 8); /* metadata_source */ + + version = avio_r8(pb); + if (version) { + av_log(c->fc, AV_LOG_WARNING, "Unsupported proj box version (%d)\n", + version); + return 0; + } + + size = avio_rb24(pb); + if (size > atom.size) + return AVERROR_INVALIDDATA; + + tag = avio_rl32(pb); + if (tag != MKTAG('p','r','o','j')) { + av_log(c->fc, AV_LOG_ERROR, "Missing projection box\n"); + return 0; + } + + version = avio_r8(pb); + if (version) { + av_log(c->fc, AV_LOG_WARNING, "Unsupported prhd box version (%d)\n", + version); + return 0; + } + + size = avio_rb24(pb); + if (size > atom.size) + return AVERROR_INVALIDDATA; + + tag = avio_rl32(pb); + if (tag != MKTAG('p','r','h','d')) { + av_log(c->fc, AV_LOG_ERROR, "Missing projection header box\n"); + return 0; + } + + /* 16.16 fixed point */ + yaw = avio_rb32(pb); + pitch = avio_rb32(pb); + roll = avio_rb32(pb); + + avio_skip(pb, size - 20); + + version = avio_r8(pb); + if (version) { + av_log(c->fc, AV_LOG_WARNING, "Unsupported box version (%d)\n", + version); + return 0; + } + + size = avio_rb24(pb); + if (size > atom.size) + return AVERROR_INVALIDDATA; + + tag = avio_rl32(pb); + switch (tag) { + case MKTAG('c','b','m','p'): + projection = AV_SPHERICAL_CUBEMAP; + avio_skip(pb, 4); /* layout */ + l = t = r = b = avio_rb32(pb); + break; + case MKTAG('e','q','u','i'): + projection = AV_SPHERICAL_EQUIRECTANGULAR; + t = avio_rb32(pb); + b = avio_rb32(pb); + l = avio_rb32(pb); + r = avio_rb32(pb); + break; + default: + av_log(c->fc, AV_LOG_ERROR, "Unknown projection type\n"); + return 0; + } + + sc->spherical = av_spherical_alloc(&sc->spherical_size); + if (!sc->spherical) + return AVERROR(ENOMEM); + + sc->spherical->projection = projection; + + sc->spherical->yaw = yaw; + sc->spherical->pitch = pitch; + sc->spherical->roll = roll; + + sc->spherical->left_offset = l; + sc->spherical->top_offset = t; + sc->spherical->right_offset = r; + sc->spherical->bottom_offset = b; + + return 0; +} + +static int mov_parse_uuid_spherical(MOVStreamContext *sc, AVIOContext *pb, size_t len) +{ + int ret = 0; + uint8_t *buffer = av_malloc(len + 1); + const char *val; + + if (!buffer) + return AVERROR(ENOMEM); + buffer[len] = '\0'; + + ret = ffio_read_size(pb, buffer, len); + if (ret < 0) + goto out; + + /* Check for mandatory keys and values, try to support XML as best-effort */ + if (av_stristr(buffer, "") && + (val = av_stristr(buffer, "")) && + av_stristr(val, "true") && + (val = av_stristr(buffer, "")) && + av_stristr(val, "true") && + (val = av_stristr(buffer, "")) && + av_stristr(val, "equirectangular")) { + sc->spherical = av_spherical_alloc(&sc->spherical_size); + if (!sc->spherical) + goto out; + + sc->spherical->projection = AV_SPHERICAL_EQUIRECTANGULAR; + + if (av_stristr(buffer, "")) { + enum AVStereo3DType mode; + + if (av_stristr(buffer, "left-right")) + mode = AV_STEREO3D_SIDEBYSIDE; + else if (av_stristr(buffer, "top-bottom")) + mode = AV_STEREO3D_TOPBOTTOM; + else + mode = AV_STEREO3D_2D; + + sc->stereo3d = av_stereo3d_alloc(); + if (!sc->stereo3d) + goto out; + + sc->stereo3d->type = mode; + } + + /* orientation */ + val = av_stristr(buffer, ""); + if (val) + sc->spherical->yaw = strtol(val, NULL, 10) * (1 << 16); + val = av_stristr(buffer, ""); + if (val) + sc->spherical->pitch = strtol(val, NULL, 10) * (1 << 16); + val = av_stristr(buffer, ""); + if (val) + sc->spherical->roll = strtol(val, NULL, 10) * (1 << 16); + + /* cropping */ + val = av_stristr(buffer, ""); + if (val) + sc->spherical->left_offset = strtol(val, NULL, 10); + val = av_stristr(buffer, ""); + if (val) + sc->spherical->top_offset = strtol(val, NULL, 10); + val = av_stristr(buffer, ""); + if (val) + sc->spherical->right_offset = + sc->width - sc->spherical->left_offset - strtol(val, NULL, 10); + val = av_stristr(buffer, ""); + if (val) + sc->spherical->bottom_offset = + sc->height - sc->spherical->top_offset - strtol(val, NULL, 10); + } + +out: + av_free(buffer); + return ret; +} + static int mov_read_uuid(MOVContext *c, AVIOContext *pb, MOVAtom atom) { + AVStream *st; + MOVStreamContext *sc; int ret; uint8_t uuid[16]; static const uint8_t uuid_isml_manifest[] = { @@ -4509,10 +4761,19 @@ static int mov_read_uuid(MOVContext *c, AVIOContext *pb, MOVAtom atom) 0xbe, 0x7a, 0xcf, 0xcb, 0x97, 0xa9, 0x42, 0xe8, 0x9c, 0x71, 0x99, 0x94, 0x91, 0xe3, 0xaf, 0xac }; + static const uint8_t uuid_spherical[] = { + 0xff, 0xcc, 0x82, 0x63, 0xf8, 0x55, 0x4a, 0x93, + 0x88, 0x14, 0x58, 0x7a, 0x02, 0x52, 0x1f, 0xdd, + }; if (atom.size < sizeof(uuid) || atom.size == INT64_MAX) return AVERROR_INVALIDDATA; + if (c->fc->nb_streams < 1) + return 0; + st = c->fc->streams[c->fc->nb_streams - 1]; + sc = st->priv_data; + ret = avio_read(pb, uuid, sizeof(uuid)); if (ret < 0) { return ret; @@ -4584,7 +4845,14 @@ static int mov_read_uuid(MOVContext *c, AVIOContext *pb, MOVAtom atom) av_dict_set(&c->fc->metadata, "xmp", buffer, 0); } av_free(buffer); - } + } else if (!memcmp(uuid, uuid_spherical, sizeof(uuid))) { + size_t len = atom.size - sizeof(uuid); + ret = mov_parse_uuid_spherical(sc, pb, len); + if (ret < 0) + return ret; + if (!sc->spherical) + av_log(c->fc, AV_LOG_WARNING, "Invalid spherical metadata found\n"); } + return 0; } @@ -4934,6 +5202,8 @@ static const MOVParseTableEntry mov_default_parse_table[] = { { MKTAG('f','r','m','a'), mov_read_frma }, { MKTAG('s','e','n','c'), mov_read_senc }, { MKTAG('s','a','i','z'), mov_read_saiz }, +{ MKTAG('s','t','3','d'), mov_read_st3d }, /* stereoscopic 3D video box */ +{ MKTAG('s','v','3','d'), mov_read_sv3d }, /* spherical video box */ { 0, NULL } }; @@ -5354,6 +5624,9 @@ static int mov_read_close(AVFormatContext *s) av_freep(&sc->cenc.auxiliary_info); av_freep(&sc->cenc.auxiliary_info_sizes); av_aes_ctr_free(sc->cenc.aes_ctr); + + av_freep(&sc->stereo3d); + av_freep(&sc->spherical); } if (mov->dv_demux) { @@ -5681,6 +5954,40 @@ static int mov_read_header(AVFormatContext *s) sd->data = (uint8_t*)sc->display_matrix; sc->display_matrix = NULL; } + if (sc->stereo3d) { + AVPacketSideData *sd, *tmp; + + tmp = av_realloc_array(st->side_data, + st->nb_side_data + 1, sizeof(*tmp)); + if (!tmp) + return AVERROR(ENOMEM); + + st->side_data = tmp; + st->nb_side_data++; + + sd = &st->side_data[st->nb_side_data - 1]; + sd->type = AV_PKT_DATA_STEREO3D; + sd->size = sizeof(*sc->stereo3d); + sd->data = (uint8_t *)sc->stereo3d; + sc->stereo3d = NULL; + } + if (sc->spherical) { + AVPacketSideData *sd, *tmp; + + tmp = av_realloc_array(st->side_data, + st->nb_side_data + 1, sizeof(*tmp)); + if (!tmp) + return AVERROR(ENOMEM); + + st->side_data = tmp; + st->nb_side_data++; + + sd = &st->side_data[st->nb_side_data - 1]; + sd->type = AV_PKT_DATA_SPHERICAL; + sd->size = sc->spherical_size; + sd->data = (uint8_t *)sc->spherical; + sc->spherical = NULL; + } break; } }