From patchwork Tue Jun 22 06:54:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 28599 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2042:0:0:0:0 with SMTP id z2csp2700230iod; Mon, 21 Jun 2021 23:55:18 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwv+pv1yhQxwuV8SVwmC6UJ0jVBQsd+7jLBYDShv4GE6ySJqQlrachcn+Q8BvPUA4n3cTey X-Received: by 2002:a05:6402:2813:: with SMTP id h19mr2874611ede.39.1624344918420; Mon, 21 Jun 2021 23:55:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624344918; cv=none; d=google.com; s=arc-20160816; b=NaX/Ak7Vj2BxdFvj1TdKO1blTEGfmH4Emubw1PaQDuWWUKEN7tD8ry08jmUWwFDqnG djDLy13DRu9GXVVswJlYXBLg40XtSFTmdLg8g3t28Hnw1/91ZiG3yMtyUIWtO0YmmicJ qfsmpaaDHeNHzrRQdF0PW/bkpaxRb8hTumiW1XLD908xnb+COfIB9VZ+sGZzgVWW7Nv/ BLjcOj7kB3ecvfINJBRyqT+nWdnOA5t0+d9KP+d7gcGWZM9jRTXL3LTHvXTXvh2H0B0y oTzSpu7yDtilUYRUip4H86IjvcymK4HmFearLGKuZ9/ESMrrad/QtoFREtzPdPi463EF WSZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=RxAcTgpTIEhPgg5Iu3JkYwJeL1pdOSx3Ze4oJYqMI40=; b=fUg+zqai9lCQY3tvrcQ/mDPcN7iGdm5+SRTLiLpDiFS5r1URNyeoPsMEyg66k6pKqS 8GfbkY/iOz29N61SBE2pDuVf/9SPnPghdLk+htpBchetdL9uw/BsR8stOB/GBWfFt1IZ ksQ8ze6mpPHPKxLO7/dcdKXOEoYmIkSOzyUo4rZBYHdyeXAwt1HtsystfHrSwLCcxTIg f0n75pO6Q/sd9YehPkRlF3fXSGKj2Z1uYHoaL+bymkPX12Syau3E/+q/+m12eJhGLPr+ 2sCXsZ/16bL8SX8rD8h6Zr29vPYIKN+il/gJYtNRGUg/6ILbJsD1ii/XbVBhMqHrmOOO FkIw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=p+c2bURP; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id yc2si8963381ejb.200.2021.06.21.23.55.18; Mon, 21 Jun 2021 23:55:18 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=p+c2bURP; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1B37F689FD1; Tue, 22 Jun 2021 09:54:54 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 03FDA689B7F for ; Tue, 22 Jun 2021 09:54:46 +0300 (EEST) Received: by mail-lj1-f179.google.com with SMTP id d13so28542028ljg.12 for ; Mon, 21 Jun 2021 23:54:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=xG5XgqOybmeDnZy2hHMYIOy14BmaWtKySiOdxPU3BjQ=; b=p+c2bURPcNrHW47+aEu8+PSDil4aho5v+JWVMb5N3QxMTlAsLSLUyYeDfnrQ6VQQSQ 6m+Ml2SG/G/C5/dEaQxUq0tzedoZ0OBihcUX+3IkNYsTyEaPkVq4XR5ZWuEqNcqb1+KU J8NbSUFcfWadq83HEcJrZQQeBATFeiD7YiNSkhT+tlMVUFjCM9xKFI0qrfpYdqDgzmTu mHj2fchvu0HXJqIg/NoO6ik2+gVx1IZ9facgPNZqIf/DXi77ylApYGK2T5iuB4DfhsxZ xVOGX+3UgcY7iDEq2jevOqVt/Uy68QMGYrsZQNK6ibH3m+PmPfMr4SlGZkG/mzSHcdPt t9qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xG5XgqOybmeDnZy2hHMYIOy14BmaWtKySiOdxPU3BjQ=; b=G7qPO+wNTrRSaCozi0/cbli+XpJuyMseng1hEMK14BUWFuOY6su/k0fuUKjjzT1Eln mJlfMdFTitdR6wDY7de6LBEBP6IM5vWQO+PMgHIqQ4c/WOW4y/cTOPs7QahHAJepT1KD +/V66JRkCxlolZ1s+U+adDbV4CtvrjJd2ns18S/mgDoLbZwmjcORFKy8Rb2mUe/L20Ht EgglUMvR4WagNQtdjvL2NJJtVt//SqE1ubOPPcXkieYQYEPFwqWnDUqV0oVFanlNRwb8 7t/aaLi8ilyvcl228u4ZV755Qxz3d6WWxFJfD68I4H7xmStXm0Q0I9+lPF/14o1ZtAei Ibqw== X-Gm-Message-State: AOAM530HYkhLlJuSHzbGZoH/lX5jHC4w4bEjCkP6ooaRgveD+vrb8Zce RTtJW20bWiwhZiRzTk7SYgfLUcUJ5VI= X-Received: by 2002:a2e:bc25:: with SMTP id b37mr1974401ljf.120.1624344885324; Mon, 21 Jun 2021 23:54:45 -0700 (PDT) Received: from localhost.localdomain (91-159-194-103.elisa-laajakaista.fi. [91.159.194.103]) by smtp.gmail.com with ESMTPSA id d9sm2101417lfi.287.2021.06.21.23.54.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Jun 2021 23:54:44 -0700 (PDT) From: =?utf-8?q?Jan_Ekstr=C3=B6m?= To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Jun 2021 09:54:34 +0300 Message-Id: <20210622065434.9006-3-jeebjp@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210622065434.9006-1-jeebjp@gmail.com> References: <20210622065434.9006-1-jeebjp@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] avformat/movenc: add support for TTML muxing X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: sq6R0ejhXk+1 From: Jan Ekström Includes basic support for both the ISMV ('dfxp') and MP4 ('stpp') methods. This initial version also foregoes fragmentation support as this eases the initial review. Signed-off-by: Jan Ekström --- libavformat/Makefile | 2 +- libavformat/isom.h | 3 + libavformat/movenc.c | 180 +++++++++++++++++++++++++++- libavformat/movenc.h | 6 + libavformat/movenc_ttml.c | 243 ++++++++++++++++++++++++++++++++++++++ libavformat/movenc_ttml.h | 31 +++++ 6 files changed, 462 insertions(+), 3 deletions(-) create mode 100644 libavformat/movenc_ttml.c create mode 100644 libavformat/movenc_ttml.h diff --git a/libavformat/Makefile b/libavformat/Makefile index c9ef564523..931ad4ac45 100644 --- a/libavformat/Makefile +++ b/libavformat/Makefile @@ -337,7 +337,7 @@ OBJS-$(CONFIG_MOV_DEMUXER) += mov.o mov_chan.o mov_esds.o \ qtpalette.o replaygain.o OBJS-$(CONFIG_MOV_MUXER) += movenc.o av1.o avc.o hevc.o vpcc.o \ movenchint.o mov_chan.o rtp.o \ - movenccenc.o rawutils.o + movenccenc.o movenc_ttml.o rawutils.o OBJS-$(CONFIG_MP2_MUXER) += rawenc.o OBJS-$(CONFIG_MP3_DEMUXER) += mp3dec.o replaygain.o OBJS-$(CONFIG_MP3_MUXER) += mp3enc.o rawenc.o id3v2enc.o diff --git a/libavformat/isom.h b/libavformat/isom.h index ac1b3f3d56..34a58c79b7 100644 --- a/libavformat/isom.h +++ b/libavformat/isom.h @@ -387,4 +387,7 @@ static inline enum AVCodecID ff_mov_get_lpcm_codec_id(int bps, int flags) return ff_get_pcm_codec_id(bps, flags & 1, flags & 2, flags & 4 ? -1 : 0); } +#define MOV_ISMV_TTML_TAG MKTAG('d', 'f', 'x', 'p') +#define MOV_MP4_TTML_TAG MKTAG('s', 't', 'p', 'p') + #endif /* AVFORMAT_ISOM_H */ diff --git a/libavformat/movenc.c b/libavformat/movenc.c index 04f3e94158..d4efb6217f 100644 --- a/libavformat/movenc.c +++ b/libavformat/movenc.c @@ -56,6 +56,8 @@ #include "hevc.h" #include "rtpenc.h" #include "mov_chan.h" +#include "movenc_ttml.h" +#include "ttmlenc.h" #include "vpcc.h" static const AVOption options[] = { @@ -120,6 +122,7 @@ static const AVClass flavor ## _muxer_class = {\ }; static int get_moov_size(AVFormatContext *s); +static int mov_write_single_packet(AVFormatContext *s, AVPacket *pkt); static int utf8len(const uint8_t *b) { @@ -1788,7 +1791,29 @@ static int mov_write_subtitle_tag(AVIOContext *pb, MOVTrack *track) if (track->par->codec_id == AV_CODEC_ID_DVD_SUBTITLE) mov_write_esds_tag(pb, track); - else if (track->par->extradata_size) + else if (track->par->codec_id == AV_CODEC_ID_TTML) { + switch (track->par->codec_tag) { + case MOV_ISMV_TTML_TAG: + // ye olde ISMV dfxp requires no extradata. + break; + case MOV_MP4_TTML_TAG: + // As specified in 14496-30, XMLSubtitleSampleEntry + // Namespace + avio_put_str(pb, "http://www.w3.org/ns/ttml"); + // Empty schema_location + avio_w8(pb, 0); + // Empty auxiliary_mime_types + avio_w8(pb, 0); + break; + default: + av_log(NULL, AV_LOG_ERROR, + "Unknown codec tag '%s' utilized for TTML stream with " + "index %d (track id %d)!\n", + av_fourcc2str(track->par->codec_tag), track->st->index, + track->track_id); + return AVERROR(EINVAL); + } + } else if (track->par->extradata_size) avio_write(pb, track->par->extradata, track->par->extradata_size); if (track->mode == MODE_MP4 && @@ -5254,6 +5279,71 @@ static int mov_flush_fragment_interleaving(AVFormatContext *s, MOVTrack *track) return 0; } +static int mov_write_squashed_packet(AVFormatContext *s, MOVTrack *track) +{ + AVPacket *squashed_packet = ((MOVMuxContext *)s->priv_data)->pkt; + int ret = AVERROR_BUG; + + switch (track->st->codecpar->codec_id) { + case AV_CODEC_ID_TTML: + { + int we_had_packets = !!track->squashed_packet_queue; + + if ((ret = ff_mov_generate_squashed_ttml_packet(s, track, squashed_packet)) < 0) { + goto finish_squash; + } + + // We have generated a padding packet (no actual input packets in + // queue) and its duration is zero. Skipping writing it. + if (!we_had_packets && squashed_packet->duration == 0) { + goto finish_squash; + } + + track->end_reliable = 1; + break; + } + default: + ret = AVERROR(EINVAL); + goto finish_squash; + } + + squashed_packet->stream_index = track->st->index; + + ret = mov_write_single_packet(s, squashed_packet); + +finish_squash: + if (!track->squashed_packet_queue) { + track->packet_queue_start_ts = track->packet_queue_end_ts = AV_NOPTS_VALUE; + } + av_packet_unref(squashed_packet); + + return ret; +} + +static int mov_write_squashed_packets(AVFormatContext *s) +{ + MOVMuxContext *mov = s->priv_data; + + for (int i = 0; i < s->nb_streams; i++) { + MOVTrack *track = &mov->tracks[i]; + int ret = AVERROR_BUG; + + if (track->squash_fragment_samples_to_one && !track->entry) { + if ((ret = mov_write_squashed_packet(s, track)) < 0) { + av_log(s, AV_LOG_ERROR, + "Failed to write squashed packet for %s stream with " + " index %d and track id %d. Error: %s\n", + avcodec_get_name(track->st->codecpar->codec_id), + track->st->index, track->track_id, + av_err2str(ret)); + return ret; + } + } + } + + return 0; +} + static int mov_flush_fragment(AVFormatContext *s, int force) { MOVMuxContext *mov = s->priv_data; @@ -5265,6 +5355,11 @@ static int mov_flush_fragment(AVFormatContext *s, int force) if (!(mov->flags & FF_MOV_FLAG_FRAGMENT)) return 0; + // Check if we have any tracks that require squashing. + // In that case, we'll have to write the packet here. + if ((ret = mov_write_squashed_packets(s)) < 0) + return ret; + // Try to fill in the duration of the last packet in each stream // from queued packets in the interleave queues. If the flushing // of fragments was triggered automatically by an AVPacket, we @@ -5729,7 +5824,8 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt) trk->cluster[trk->entry].entries = samples_in_chunk; trk->cluster[trk->entry].dts = pkt->dts; trk->cluster[trk->entry].pts = pkt->pts; - if (!trk->entry && trk->start_dts != AV_NOPTS_VALUE) { + if (!trk->squash_fragment_samples_to_one && + !trk->entry && trk->start_dts != AV_NOPTS_VALUE) { if (!trk->frag_discont) { /* First packet of a new fragment. We already wrote the duration * of the last packet of the previous fragment based on track_duration, @@ -6022,6 +6118,42 @@ static int mov_write_packet(AVFormatContext *s, AVPacket *pkt) } } + if (trk->squash_fragment_samples_to_one) { + /* + * If the track has to have its samples squashed into one sample, + * we just take it into the track's queue. + * This will then be utilized as the samples get written in either + * mov_flush_fragment or when the mux is finalized in + * mov_write_trailer. + */ + int ret = AVERROR_BUG; + int64_t compared_end_ts = pkt->duration >= 0 ? + (pkt->pts + pkt->duration) : pkt->pts; + + if (pkt->pts == AV_NOPTS_VALUE) { + av_log(s, AV_LOG_ERROR, + "Packets without a valid presentation timestamp are " + "not supported with packet squashing!\n"); + return AVERROR(EINVAL); + } + + trk->packet_queue_start_ts = + trk->packet_queue_start_ts == AV_NOPTS_VALUE ? + pkt->pts : FFMIN(trk->packet_queue_start_ts, pkt->pts); + + trk->packet_queue_end_ts = + FFMAX(trk->packet_queue_end_ts, compared_end_ts); + + if ((ret = avpriv_packet_list_put(&trk->squashed_packet_queue, + &trk->squashed_packet_queue_end, + pkt, av_packet_ref, 0)) < 0) { + return ret; + } + + return 0; + } + + if (trk->mode == MODE_MOV && trk->par->codec_type == AVMEDIA_TYPE_VIDEO) { AVPacket *opkt = pkt; int reshuffle_ret, ret; @@ -6300,6 +6432,11 @@ static void mov_free(AVFormatContext *s) ff_mov_cenc_free(&mov->tracks[i].cenc); ffio_free_dyn_buf(&mov->tracks[i].mdat_buf); + + if (mov->tracks[i].squashed_packet_queue) { + avpriv_packet_list_free(&(mov->tracks[i].squashed_packet_queue), + &(mov->tracks[i].squashed_packet_queue_end)); + } } av_freep(&mov->tracks); @@ -6580,6 +6717,7 @@ static int mov_init(AVFormatContext *s) track->start_cts = AV_NOPTS_VALUE; track->end_pts = AV_NOPTS_VALUE; track->dts_shift = AV_NOPTS_VALUE; + track->packet_queue_start_ts = track->packet_queue_end_ts = AV_NOPTS_VALUE; if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) { if (track->tag == MKTAG('m','x','3','p') || track->tag == MKTAG('m','x','3','n') || track->tag == MKTAG('m','x','4','p') || track->tag == MKTAG('m','x','4','n') || @@ -6690,6 +6828,36 @@ static int mov_init(AVFormatContext *s) } } else if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) { track->timescale = st->time_base.den; + + if (track->par->codec_id == AV_CODEC_ID_TTML) { + /* 14496-30 requires us to use a single sample per fragment + for TTML, for which we define a per-track flag. + + We set the flag in case we are receiving TTML paragraphs + from the input, in other words in case we are not doing + stream copy. */ + track->squash_fragment_samples_to_one = + ff_is_ttml_stream_paragraph_based(track->par); + + if (mov->flags & FF_MOV_FLAG_FRAGMENT && + track->squash_fragment_samples_to_one) { + av_log(s, AV_LOG_ERROR, + "Fragmentation is not currently supported for " + "TTML in MP4/ISMV (track synchronization between " + "subtitles and other media is not yet implemented)!\n"); + return AVERROR(EINVAL); + } + + if (track->mode == MODE_MP4 && + track->par->codec_tag == MOV_ISMV_TTML_TAG && + s->strict_std_compliance > FF_COMPLIANCE_UNOFFICIAL) { + av_log(s, AV_LOG_ERROR, + "ISMV style TTML support with the 'dfxp' tag in MP4 " + "is not officially supported, add " + "'-strict unofficial' if you want to use it.\n"); + return AVERROR_EXPERIMENTAL; + } + } } else if (st->codecpar->codec_type == AVMEDIA_TYPE_DATA) { track->timescale = st->time_base.den; } else { @@ -7035,6 +7203,11 @@ static int mov_write_trailer(AVFormatContext *s) } } + // Check if we have any tracks that require squashing. + // In that case, we'll have to write the packet here. + if ((res = mov_write_squashed_packets(s)) < 0) + return res; + // If there were no chapters when the header was written, but there // are chapters now, write them in the trailer. This only works // when we are not doing fragments. @@ -7179,6 +7352,8 @@ static const AVCodecTag codec_mp4_tags[] = { { AV_CODEC_ID_MOV_TEXT, MKTAG('t', 'x', '3', 'g') }, { AV_CODEC_ID_BIN_DATA, MKTAG('g', 'p', 'm', 'd') }, { AV_CODEC_ID_MPEGH_3D_AUDIO, MKTAG('m', 'h', 'm', '1') }, + { AV_CODEC_ID_TTML, MOV_MP4_TTML_TAG }, + { AV_CODEC_ID_TTML, MOV_ISMV_TTML_TAG }, { AV_CODEC_ID_NONE, 0 }, }; #if CONFIG_MP4_MUXER || CONFIG_PSP_MUXER @@ -7187,6 +7362,7 @@ static const AVCodecTag *const mp4_codec_tags_list[] = { codec_mp4_tags, NULL }; static const AVCodecTag codec_ism_tags[] = { { AV_CODEC_ID_WMAPRO , MKTAG('w', 'm', 'a', ' ') }, + { AV_CODEC_ID_TTML , MOV_ISMV_TTML_TAG }, { AV_CODEC_ID_NONE , 0 }, }; diff --git a/libavformat/movenc.h b/libavformat/movenc.h index af1ea0bce6..9036e42f09 100644 --- a/libavformat/movenc.h +++ b/libavformat/movenc.h @@ -26,6 +26,7 @@ #include "avformat.h" #include "movenccenc.h" +#include "libavcodec/packet_internal.h" #define MOV_FRAG_INFO_ALLOC_INCREMENT 64 #define MOV_INDEX_CLUSTER_SIZE 1024 @@ -164,6 +165,11 @@ typedef struct MOVTrack { int pal_done; int is_unaligned_qt_rgb; + + unsigned int squash_fragment_samples_to_one; //< flag to note formats where all samples for a fragment are to be squashed + + PacketList *squashed_packet_queue, *squashed_packet_queue_end; + int64_t packet_queue_start_ts, packet_queue_end_ts; } MOVTrack; typedef enum { diff --git a/libavformat/movenc_ttml.c b/libavformat/movenc_ttml.c new file mode 100644 index 0000000000..865efbdbce --- /dev/null +++ b/libavformat/movenc_ttml.c @@ -0,0 +1,243 @@ +/* + * MP4, ISMV Muxer TTML helpers + * Copyright (c) 2021 24i + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "avformat.h" +#include "avio_internal.h" +#include "isom.h" +#include "movenc.h" +#include "movenc_ttml.h" +#include "libavcodec/packet_internal.h" + +static const unsigned char empty_ttml_document[] = + ""; + +static int mov_init_ttml_writer(MOVTrack *track, AVFormatContext **out_ctx) +{ + AVStream *movenc_stream = track->st, *ttml_stream = NULL; + AVFormatContext *ttml_ctx = NULL; + int ret = AVERROR_BUG; + if ((ret = avformat_alloc_output_context2(&ttml_ctx, NULL, + "ttml", NULL)) < 0) + goto fail; + + if ((ret = avio_open_dyn_buf(&ttml_ctx->pb)) < 0) + goto fail; + + if (!(ttml_stream = avformat_new_stream(ttml_ctx, NULL))) { + ret = AVERROR(ENOMEM); + goto fail; + } + + if ((ret = avcodec_parameters_copy(ttml_stream->codecpar, + movenc_stream->codecpar)) < 0) + goto fail; + + ttml_stream->time_base = movenc_stream->time_base; + + *out_ctx = ttml_ctx; + + return 0; + +fail: + if (ttml_ctx) { + uint8_t *buf = NULL; + avio_close_dyn_buf(ttml_ctx->pb, &buf); + av_freep(&buf); + } + + avformat_free_context(ttml_ctx); + + return ret; +} + +static void mov_calculate_start_and_end_based_on_other_tracks(AVFormatContext *s, + MOVTrack *track, + int64_t *start_ts, + int64_t *end_ts) +{ + MOVMuxContext *mov = s->priv_data; + + // initialize the end and start to the current end point of already written + // packets, or to zero if the track has not yet had any packets written. + int64_t max_track_end_ts = track->start_dts == AV_NOPTS_VALUE ? + 0 : (track->start_dts + track->track_duration); + *start_ts = max_track_end_ts; + + // Now, go through all the streams and figure out + // the furthest start/end points in this muxer instance. + for (unsigned int i = 0; i < s->nb_streams; i++) { + MOVTrack *other_track = &mov->tracks[i]; + + // Skip our own track, any other track that needs squashing, + // or any track still has its start_dts at NOPTS. + if (track == other_track || + other_track->squash_fragment_samples_to_one || + other_track->start_dts == AV_NOPTS_VALUE) { + continue; + } + + // finally, set the end timestamp to the end of the track + // that's furthest in the time line. + max_track_end_ts = FFMAX( + max_track_end_ts, + av_rescale_q((other_track->start_dts + other_track->track_duration), + other_track->st->time_base, + track->st->time_base)); + } + + *end_ts = max_track_end_ts; +} + +static int mov_write_ttml_document_from_queue(AVFormatContext *s, + AVFormatContext *ttml_ctx, + MOVTrack *track, + int64_t calculated_start_ts, + int64_t calculated_end_ts, + int64_t *out_start_ts, + int64_t *out_duration) +{ + int ret = AVERROR_BUG; + int64_t start_ts = FFMIN(track->packet_queue_start_ts, calculated_start_ts); + int64_t duration = FFMAX(track->packet_queue_end_ts, calculated_end_ts) - start_ts; + AVPacket *looped_pkt = av_packet_alloc(); + if (!looped_pkt) { + av_log(s, AV_LOG_ERROR, + "Failed to allocate AVPacket for going through packet queue!\n"); + return AVERROR(ENOMEM); + } + + if ((ret = avformat_write_header(ttml_ctx, NULL)) < 0) { + return ret; + } + + while (!avpriv_packet_list_get(&track->squashed_packet_queue, + &track->squashed_packet_queue_end, + looped_pkt)) { + // in case of the 'dfxp' muxing mode, each written document is offset + // to its containing sample's beginning. + if (track->par->codec_tag == MOV_ISMV_TTML_TAG) { + looped_pkt->dts = looped_pkt->pts = (looped_pkt->pts - start_ts); + } + + looped_pkt->stream_index = 0; + + av_packet_rescale_ts(looped_pkt, track->st->time_base, + ttml_ctx->streams[looped_pkt->stream_index]->time_base); + + if ((ret = av_write_frame(ttml_ctx, looped_pkt)) < 0) { + goto cleanup; + } + + av_packet_unref(looped_pkt); + } + + if ((ret = av_write_trailer(ttml_ctx)) < 0) + goto cleanup; + + *out_start_ts = start_ts; + *out_duration = duration; + + ret = 0; + +cleanup: + av_packet_free(&looped_pkt); + + return ret; +} + +int ff_mov_generate_squashed_ttml_packet(AVFormatContext *s, + MOVTrack *track, AVPacket *pkt) +{ + AVFormatContext *ttml_ctx = NULL; + // possible start/end points + int64_t calculated_start_ts = AV_NOPTS_VALUE; + int64_t calculated_end_ts = AV_NOPTS_VALUE; + // values for the generated AVPacket + int64_t start_ts = 0; + int64_t duration = 0; + + int ret = AVERROR_BUG; + + // calculate the possible start/end points for this packet + mov_calculate_start_and_end_based_on_other_tracks(s, track, + &calculated_start_ts, + &calculated_end_ts); + + if ((ret = mov_init_ttml_writer(track, &ttml_ctx)) < 0) { + av_log(s, AV_LOG_ERROR, "Failed to initialize the TTML writer: %s\n", + av_err2str(ret)); + goto cleanup; + } + + if (!track->squashed_packet_queue) { + // empty queue, write minimal empty document with calculated values + // based on other tracks. + avio_write(ttml_ctx->pb, empty_ttml_document, + sizeof(empty_ttml_document) - 1); + start_ts = calculated_start_ts; + duration = (calculated_end_ts - calculated_start_ts); + goto generate_packet; + } + + if ((ret = mov_write_ttml_document_from_queue(s, ttml_ctx, track, + calculated_start_ts, + calculated_end_ts, + &start_ts, + &duration)) < 0) { + av_log(s, AV_LOG_ERROR, + "Failed to generate a squashed TTML packet from the packet " + "queue: %s\n", + av_err2str(ret)); + goto cleanup; + } + +generate_packet: + { + // Generate an AVPacket from the data written into the dynamic buffer. + uint8_t *buf = NULL; + int buf_len = avio_close_dyn_buf(ttml_ctx->pb, &buf); + ttml_ctx->pb = NULL; + + if ((ret = av_packet_from_data(pkt, buf, buf_len)) < 0) { + av_log(s, AV_LOG_ERROR, + "Failed to create a TTML AVPacket from AVIO data: %s\n", + av_err2str(ret)); + av_freep(&buf); + goto cleanup; + } + + pkt->pts = pkt->dts = start_ts; + pkt->duration = duration; + pkt->flags |= AV_PKT_FLAG_KEY; + } + + ret = 0; + +cleanup: + if (ttml_ctx && ttml_ctx->pb) { + uint8_t *buf = NULL; + avio_close_dyn_buf(ttml_ctx->pb, &buf); + av_freep(&buf); + } + + avformat_free_context(ttml_ctx); + return ret; +} diff --git a/libavformat/movenc_ttml.h b/libavformat/movenc_ttml.h new file mode 100644 index 0000000000..c71ecd0997 --- /dev/null +++ b/libavformat/movenc_ttml.h @@ -0,0 +1,31 @@ +/* + * MP4, ISMV Muxer TTML helpers + * Copyright (c) 2021 24i + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVFORMAT_MOVENC_TTML_H +#define AVFORMAT_MOVENC_TTML_H + +#include "avformat.h" +#include "movenc.h" + +int ff_mov_generate_squashed_ttml_packet(AVFormatContext *s, + MOVTrack *track, AVPacket *pkt); + +#endif /* AVFORMAT_MOVENC_TTML_H */