From patchwork Mon Aug 16 12:26:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 29566 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp1992528iov; Mon, 16 Aug 2021 05:26:44 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwUFYzetE844vQnrLuM2gftkmt3Emo7cOsjwrRdu+xn+DqdqaLCZUek59jO75mW4piestIj X-Received: by 2002:a05:6402:50c9:: with SMTP id h9mr3188824edb.357.1629116804087; Mon, 16 Aug 2021 05:26:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629116804; cv=none; d=google.com; s=arc-20160816; b=hT7KeIMkAF0M5qJge6DHkKmdX5dctjGxpW5nKvqtzg69XY4L0+4OqenARVJ08RspvN YoVmiKpTctHpEP5CuG8cr7wSpzjX48cboAndGEQvkK2jBkIaJvurDF+yT4DH1n6WYL8S od7l91kFB0EsKJzBviw5zOmCmxiYN6IKkq96lBqihKBz9m6nsSuErSX0xssTODQ9RrS9 x/3xU8hFT+4cWM+EmN0oL/pAh39UpKDYHc4ifJWqkyCJ5XDzZvDF+veA0l0XFYPYd5Vi QPYU4GynfbDtBILZ9j5FTc589i09RaiZXBRO6VvXWJjMTEm9axvYJG8YxOq3FQOQsH4S LX8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=79DisVOmWmFS67rn7lUqDAOxt+AThL3D2kBmR+kImkc=; b=kEMzrnigob+jlRqiNrM4ZZlXQ1yemFrB6/2zqSPMg2i0KBfjK7V9UyubPtTg7vCTSR g3e2CCbssZ7X0Gm3RdAMsH+V0v9qRp4SZwJ0JiZAolJMX+Gr1oU7L/FCY4tTJqpjk0Np GREBVlr3ZMP1sWsb9098zjcKkEDOKElwMqcKlW9Y1bAekuNcX5KO8hv72E3ZI41DYYty PxBJa1LN3qUYSJQMwzlxLSj+JbS5B9gkJgNVN6AzhqTzWgSApSllRt0CHpCGZuE629E0 BKIM9wUXiF6nKlT21Ig7EQ+XF5QU43Lcj636ziPDx9dUfIXGDi34rPkYVl7r1i9UBgcu +EoA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=i8+cl8IT; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bx23si9793277edb.189.2021.08.16.05.26.43; Mon, 16 Aug 2021 05:26:44 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=i8+cl8IT; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E76F468A3C6; Mon, 16 Aug 2021 15:26:26 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2BE0C680C01 for ; Mon, 16 Aug 2021 15:26:20 +0300 (EEST) Received: by mail-lj1-f179.google.com with SMTP id f2so5183350ljn.1 for ; Mon, 16 Aug 2021 05:26:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=0Xybs5d52jY8OzbMXopNKNh/qiA1OwhmKwpjkhUN9ow=; b=i8+cl8ITG+17lkR8IRxbZgyBThg/uqsmW33bZo24UMG3BXWcZRm/pkBlKUW0wieR6D LxhHxE6/E63w64U3lyl7OFFMsY+4nCuwHUkbWgnlVRcVWfkc7Z1sy2Xi8aoCo8pUYB9R c9B/VkovV3MyuVpdewDfJEHjydWBLYwYPMhMUhV0PRxHgRlTloupbHAJG1EYSrnmlOeR N5SArTJnocNZgrh/ggHpN8Q/THlJOTHIIpErB8s8Y+byWKcfi5Zk6TfCpEk7RVpPYpj/ 6tMvcuiy4G/nhktyYGCLR+VTZRKnCG8EQn5GgwQNWcyuM/HOljZsni/AOC0Yeyr22uxO VKYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0Xybs5d52jY8OzbMXopNKNh/qiA1OwhmKwpjkhUN9ow=; b=tQEWDxu6SI6hom4ljuHuX/+wDRIr6IfCgCO6k54EKu5iwR4rxaYNJg+6EPIX41Mcc7 VAwVfcxvLiy81jO/YFxLFRSkmdkZ4YR+SNfuIDnwuaWlPo6/k2u0H/R/fyc8MN+pqWNt R5itS3xnPyhqSbw66kgJuedx6H1E5PAKNwKS9aFIgL2+RYcytT/duquMBg3hFyZ1kaS6 UfZPTNmhn5Ck7EL3yL59IhhBXnbmiHxkdJc6vVuSNM/o2jfrDOR2Jblw8DbeTNDSEbZg GKA99wk3VLiI7suRRLlVB273PS9S9x+brIbjGCZgZeHjbLdFSzdLK7DfzY7C9TLqWO78 ThCw== X-Gm-Message-State: AOAM531ZLt/LOtukB0U3sp0V7cufJLC9v/WSKyKm2/spgJ6/2AtocJNn ne8jg5esELqvYktr11KCGTWs7RgvMpI= X-Received: by 2002:a2e:4612:: with SMTP id t18mr12435860lja.346.1629116779497; Mon, 16 Aug 2021 05:26:19 -0700 (PDT) Received: from localhost.localdomain (91-159-194-103.elisa-laajakaista.fi. [91.159.194.103]) by smtp.gmail.com with ESMTPSA id i3sm930249lfr.217.2021.08.16.05.26.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Aug 2021 05:26:19 -0700 (PDT) From: =?utf-8?q?Jan_Ekstr=C3=B6m?= To: ffmpeg-devel@ffmpeg.org Date: Mon, 16 Aug 2021 15:26:11 +0300 Message-Id: <20210816122612.7444-2-jeebjp@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210816122612.7444-1-jeebjp@gmail.com> References: <20210816122612.7444-1-jeebjp@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 1/2] avformat/ttml: split TTML paragraph based or not check into header X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Tu687ZNDaQ1H From: Jan Ekström This way it can be re-utilized in movenc. Signed-off-by: Jan Ekström --- libavformat/ttmlenc.c | 9 ++------- libavformat/ttmlenc.h | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 7 deletions(-) create mode 100644 libavformat/ttmlenc.h diff --git a/libavformat/ttmlenc.c b/libavformat/ttmlenc.c index 8546d35196..896fc81958 100644 --- a/libavformat/ttmlenc.c +++ b/libavformat/ttmlenc.c @@ -30,6 +30,7 @@ #include "libavutil/avstring.h" #include "avformat.h" #include "internal.h" +#include "ttmlenc.h" #include "libavcodec/ttmlenc.h" #include "libavutil/internal.h" @@ -138,13 +139,7 @@ static int ttml_write_header(AVFormatContext *ctx) 0); const char *printed_lang = (lang && lang->value) ? lang->value : ""; - // Not perfect, but decide whether the packet is a document or not - // by the existence of the lavc ttmlenc extradata. - ttml_ctx->input_type = (st->codecpar->extradata && - st->codecpar->extradata_size >= TTMLENC_EXTRADATA_SIGNATURE_SIZE && - !memcmp(st->codecpar->extradata, - TTMLENC_EXTRADATA_SIGNATURE, - TTMLENC_EXTRADATA_SIGNATURE_SIZE)) ? + ttml_ctx->input_type = ff_is_ttml_stream_paragraph_based(st->codecpar) ? PACKET_TYPE_PARAGRAPH : PACKET_TYPE_DOCUMENT; diff --git a/libavformat/ttmlenc.h b/libavformat/ttmlenc.h new file mode 100644 index 0000000000..5a6954f3c1 --- /dev/null +++ b/libavformat/ttmlenc.h @@ -0,0 +1,39 @@ +/* + * Generic TTML helpers + * Copyright (c) 2021 24i + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVFORMAT_TTMLENC_H +#define AVFORMAT_TTMLENC_H + +#include "libavcodec/codec_par.h" +#include "libavcodec/ttmlenc.h" + +static inline unsigned int ff_is_ttml_stream_paragraph_based(const AVCodecParameters *codecpar) +{ + // Not perfect, but decide whether the packet is a document or not + // by the existence of the lavc ttmlenc extradata. + return (codecpar->extradata && + codecpar->extradata_size >= TTMLENC_EXTRADATA_SIGNATURE_SIZE && + !memcmp(codecpar->extradata, + TTMLENC_EXTRADATA_SIGNATURE, + TTMLENC_EXTRADATA_SIGNATURE_SIZE)); +} + +#endif /* AVFORMAT_TTMLENC_H */ From patchwork Mon Aug 16 12:26:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 29567 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp1992679iov; Mon, 16 Aug 2021 05:26:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwGPaDKFrwQU4/t9pphTesJhWSIwGZMWP/AdjHMYIQMKo6EeUsfQdPBBPZQZcxpMCCWOEhW X-Received: by 2002:aa7:c347:: with SMTP id j7mr20169580edr.316.1629116814508; Mon, 16 Aug 2021 05:26:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629116814; cv=none; d=google.com; s=arc-20160816; b=bwvhC4W61wQEJOWpxxHF3xWJYqW+z/QAeuz7MyuKmACRMvn90OWos44rT4kEK16RHZ 1JGNg6zjdOOneVCh85MwQdsLeGbCZA8sngxs9XnRWuw1szZX2S8fIWxdqtSVz2gvpcTV m56UtmJTirA6KiybH8Bia/RovCkwHcddfdJSMIKAXiYDQ2xR3+ARA8AqX5rcSKFFBgCM Kx9jkVg/uRuXUY+OCYXsPfDTLptQIjebMHHRCrvb+9ldORqllnfHRr6MvcfDVD++P+9f /R+7/i04LeZ7R6zKUttPBm3ytUws3sPrG1FH5xdUyMfr15Hkt3JRrDwnzagTILKg4zFt jazg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=pH2xg2ZBJDYmlhuBjUECC/nnIqpM6BeGcqejURBaIfY=; b=lF78GldTRXVJgUe4yaIoB31GKZcPZPzYAL2zXapda0/o+/hUOohgQbW5CA4ExHQ7cS zD3/Ij/3Z27tgo4i8Cye0dxPDXdtuGqD1ub50ie5vCxhg70JHlX5f4jSDHKzkWRXXofH xXDQbd9jdp65QVRlQB8S5uThRwLS6znyCevOkSn/VdbS7mSHuiJ79ZUleM9lOwE9pmxq ci5G9vTQHaDR7bjZFPbnoABJ35UZnUvfykkdXmviKm7JxRiqXVjV0Vnzc3H9gQop6g35 fjiTBrv31a8vkj9dHcTUe582WNeGT/0QjeBdRLB9SQlYtWMy7k7MA4POwtlD2TP4tESD Ccyg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=JIB1AKs2; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id h2si9975774ejd.254.2021.08.16.05.26.54; Mon, 16 Aug 2021 05:26:54 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=JIB1AKs2; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C9342689963; Mon, 16 Aug 2021 15:26:28 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0567968A33F for ; Mon, 16 Aug 2021 15:26:21 +0300 (EEST) Received: by mail-lj1-f177.google.com with SMTP id f2so5183508ljn.1 for ; Mon, 16 Aug 2021 05:26:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=8dftFgGqKBTPdlWmg1BH92nIyccTYRNhaQVTnK7aqWA=; b=JIB1AKs2os1v3a2mHRRht93CArHaLwEyxGjQStDygmM5j2N9DkGHoLHulueX4onrlY M31X5wBVSdYDQjL0JHIKpyp5WRiuV5ZxuYR7YwHgFO5DD0EclIwPBq2zi5K2XZWTq6zL HMbFBBFZO3IdvWfm6OovMSy7r/8qzSGsRQ9G4fVYg6VSw08QnKjrBlk63P1IbLSPOJ+P NOztGGhWNZgt/KhuNCHTSJd26/cL5U1ec+kl4OaeH9QXPlDowQI4kVOgt5Z0zFGCv8h+ sGCqAJOPTHfn8Dv+Oke12snTyytU1f1JzNAgfSacM3IuAyt6hIMPycHRF8UEIjsb77bu lRKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8dftFgGqKBTPdlWmg1BH92nIyccTYRNhaQVTnK7aqWA=; b=UimurLEi+PdoVDocDixrBdzwwOmsUUocpxvSZsvATUBziE/CptR6GK+cmsUdvGEdzc JNas0CB8uQhAYqSEnEW3E93SGmRrjbnmGK1CnEb9Xiig0+RlDu+iH4lGKGqXfhz2rYR8 CCH51IMeQBe+Y9t2mJ5NR+JsCG9cSW+K18uxYpvO/hrk2Zwr/SF6PSrhhN7EULOEqb8Y rgi8G/MnzrlVMo07wZmpy+eYVEbB4BLuPwSpw+dRK0pyD5M6bU7+QL4KEgDc70kL3CQt 6asVDpwXr1R0KHmAsKnsyh6c/RwEP3nN6ncurS6gfP/lSgO3BzNHPbk38Bn+rcWCDbI1 /TQw== X-Gm-Message-State: AOAM531zWtF+J/jB9vhMT38nabkRn3pbtQKLhbNAsAXe9oyXMimSHp0w UeluP7qPCIxQGcS84Csr2OhwqMUblJE= X-Received: by 2002:a05:651c:33b:: with SMTP id b27mr8381536ljp.314.1629116780197; Mon, 16 Aug 2021 05:26:20 -0700 (PDT) Received: from localhost.localdomain (91-159-194-103.elisa-laajakaista.fi. [91.159.194.103]) by smtp.gmail.com with ESMTPSA id i3sm930249lfr.217.2021.08.16.05.26.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Aug 2021 05:26:19 -0700 (PDT) From: =?utf-8?q?Jan_Ekstr=C3=B6m?= To: ffmpeg-devel@ffmpeg.org Date: Mon, 16 Aug 2021 15:26:12 +0300 Message-Id: <20210816122612.7444-3-jeebjp@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210816122612.7444-1-jeebjp@gmail.com> References: <20210816122612.7444-1-jeebjp@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 2/2] avformat/movenc: add support for TTML muxing X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 1YV2viZ/kWBv From: Jan Ekström Includes basic support for both the ISMV ('dfxp') and MP4 ('stpp') methods. This initial version also foregoes fragmentation support in case the built-in sample squashing is to be utilized, as this eases the initial review. Additionally, add basic tests for both muxing modes in MP4. Signed-off-by: Jan Ekström --- libavformat/Makefile | 2 +- libavformat/isom.h | 3 + libavformat/movenc.c | 179 ++++++++++++++++++++++++++++++- libavformat/movenc.h | 5 + libavformat/movenc_ttml.c | 171 +++++++++++++++++++++++++++++ libavformat/movenc_ttml.h | 31 ++++++ tests/fate/subtitles.mak | 4 + tests/ref/fate/sub-ttml-mp4-dfxp | 44 ++++++++ tests/ref/fate/sub-ttml-mp4-stpp | 44 ++++++++ 9 files changed, 480 insertions(+), 3 deletions(-) create mode 100644 libavformat/movenc_ttml.c create mode 100644 libavformat/movenc_ttml.h create mode 100644 tests/ref/fate/sub-ttml-mp4-dfxp create mode 100644 tests/ref/fate/sub-ttml-mp4-stpp diff --git a/libavformat/Makefile b/libavformat/Makefile index 813ddd3c20..7e0f587b41 100644 --- a/libavformat/Makefile +++ b/libavformat/Makefile @@ -337,7 +337,7 @@ OBJS-$(CONFIG_MOV_DEMUXER) += mov.o mov_chan.o mov_esds.o \ qtpalette.o replaygain.o OBJS-$(CONFIG_MOV_MUXER) += movenc.o av1.o avc.o hevc.o vpcc.o \ movenchint.o mov_chan.o rtp.o \ - movenccenc.o rawutils.o + movenccenc.o movenc_ttml.o rawutils.o OBJS-$(CONFIG_MP2_MUXER) += rawenc.o OBJS-$(CONFIG_MP3_DEMUXER) += mp3dec.o replaygain.o OBJS-$(CONFIG_MP3_MUXER) += mp3enc.o rawenc.o id3v2enc.o diff --git a/libavformat/isom.h b/libavformat/isom.h index ac1b3f3d56..34a58c79b7 100644 --- a/libavformat/isom.h +++ b/libavformat/isom.h @@ -387,4 +387,7 @@ static inline enum AVCodecID ff_mov_get_lpcm_codec_id(int bps, int flags) return ff_get_pcm_codec_id(bps, flags & 1, flags & 2, flags & 4 ? -1 : 0); } +#define MOV_ISMV_TTML_TAG MKTAG('d', 'f', 'x', 'p') +#define MOV_MP4_TTML_TAG MKTAG('s', 't', 'p', 'p') + #endif /* AVFORMAT_ISOM_H */ diff --git a/libavformat/movenc.c b/libavformat/movenc.c index bcc202300b..1bb948769c 100644 --- a/libavformat/movenc.c +++ b/libavformat/movenc.c @@ -57,6 +57,8 @@ #include "hevc.h" #include "rtpenc.h" #include "mov_chan.h" +#include "movenc_ttml.h" +#include "ttmlenc.h" #include "vpcc.h" static const AVOption options[] = { @@ -120,6 +122,7 @@ static const AVClass mov_isobmff_muxer_class = { }; static int get_moov_size(AVFormatContext *s); +static int mov_write_single_packet(AVFormatContext *s, AVPacket *pkt); static int utf8len(const uint8_t *b) { @@ -1788,7 +1791,29 @@ static int mov_write_subtitle_tag(AVIOContext *pb, MOVTrack *track) if (track->par->codec_id == AV_CODEC_ID_DVD_SUBTITLE) mov_write_esds_tag(pb, track); - else if (track->par->extradata_size) + else if (track->par->codec_id == AV_CODEC_ID_TTML) { + switch (track->par->codec_tag) { + case MOV_ISMV_TTML_TAG: + // ISMV dfxp requires no extradata. + break; + case MOV_MP4_TTML_TAG: + // As specified in 14496-30, XMLSubtitleSampleEntry + // Namespace + avio_put_str(pb, "http://www.w3.org/ns/ttml"); + // Empty schema_location + avio_w8(pb, 0); + // Empty auxiliary_mime_types + avio_w8(pb, 0); + break; + default: + av_log(NULL, AV_LOG_ERROR, + "Unknown codec tag '%s' utilized for TTML stream with " + "index %d (track id %d)!\n", + av_fourcc2str(track->par->codec_tag), track->st->index, + track->track_id); + return AVERROR(EINVAL); + } + } else if (track->par->extradata_size) avio_write(pb, track->par->extradata, track->par->extradata_size); if (track->mode == MODE_MP4 && @@ -2662,6 +2687,14 @@ static int mov_write_nmhd_tag(AVIOContext *pb) return 12; } +static int mov_write_sthd_tag(AVIOContext *pb) +{ + avio_wb32(pb, 12); + ffio_wfourcc(pb, "sthd"); + avio_wb32(pb, 0); + return 12; +} + static int mov_write_tcmi_tag(AVIOContext *pb, MOVTrack *track) { int64_t pos = avio_tell(pb); @@ -2788,6 +2821,8 @@ static int mov_write_hdlr_tag(AVFormatContext *s, AVIOContext *pb, MOVTrack *tra hdlr_type = "sbtl"; } else if (track->tag == MKTAG('m','p','4','s')) { hdlr_type = "subp"; + } else if (track->tag == MOV_MP4_TTML_TAG) { + hdlr_type = "subt"; } else { hdlr_type = "text"; } @@ -2866,6 +2901,8 @@ static int mov_write_minf_tag(AVFormatContext *s, AVIOContext *pb, MOVMuxContext else if (track->par->codec_type == AVMEDIA_TYPE_SUBTITLE) { if (track->tag == MKTAG('t','e','x','t') || is_clcp_track(track)) { mov_write_gmhd_tag(pb, track); + } else if (track->tag == MOV_MP4_TTML_TAG) { + mov_write_sthd_tag(pb); } else { mov_write_nmhd_tag(pb); } @@ -5254,6 +5291,68 @@ static int mov_flush_fragment_interleaving(AVFormatContext *s, MOVTrack *track) return 0; } +static int mov_write_squashed_packet(AVFormatContext *s, MOVTrack *track) +{ + MOVMuxContext *mov = s->priv_data; + AVPacket *squashed_packet = mov->pkt; + int ret = AVERROR_BUG; + + switch (track->st->codecpar->codec_id) { + case AV_CODEC_ID_TTML: { + int had_packets = !!track->squashed_packet_queue; + + if ((ret = ff_mov_generate_squashed_ttml_packet(s, track, squashed_packet)) < 0) { + goto finish_squash; + } + + // We have generated a padding packet (no actual input packets in + // queue) and its duration is zero. Skipping writing it. + if (!had_packets && squashed_packet->duration == 0) { + goto finish_squash; + } + + track->end_reliable = 1; + break; + } + default: + ret = AVERROR(EINVAL); + goto finish_squash; + } + + squashed_packet->stream_index = track->st->index; + + ret = mov_write_single_packet(s, squashed_packet); + +finish_squash: + av_packet_unref(squashed_packet); + + return ret; +} + +static int mov_write_squashed_packets(AVFormatContext *s) +{ + MOVMuxContext *mov = s->priv_data; + + for (int i = 0; i < s->nb_streams; i++) { + MOVTrack *track = &mov->tracks[i]; + int ret = AVERROR_BUG; + + if (track->squash_fragment_samples_to_one && !track->entry) { + if ((ret = mov_write_squashed_packet(s, track)) < 0) { + av_log(s, AV_LOG_ERROR, + "Failed to write squashed packet for %s stream with " + "index %d and track id %d. Error: %s\n", + avcodec_get_name(track->st->codecpar->codec_id), + track->st->index, track->track_id, + av_err2str(ret)); + return ret; + } + } + } + + return 0; +} + static int mov_flush_fragment(AVFormatContext *s, int force) { MOVMuxContext *mov = s->priv_data; @@ -5265,6 +5364,11 @@ static int mov_flush_fragment(AVFormatContext *s, int force) if (!(mov->flags & FF_MOV_FLAG_FRAGMENT)) return 0; + // Check if we have any tracks that require squashing. + // In that case, we'll have to write the packet here. + if ((ret = mov_write_squashed_packets(s)) < 0) + return ret; + // Try to fill in the duration of the last packet in each stream // from queued packets in the interleave queues. If the flushing // of fragments was triggered automatically by an AVPacket, we @@ -5740,7 +5844,8 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt) trk->cluster[trk->entry].entries = samples_in_chunk; trk->cluster[trk->entry].dts = pkt->dts; trk->cluster[trk->entry].pts = pkt->pts; - if (!trk->entry && trk->start_dts != AV_NOPTS_VALUE) { + if (!trk->squash_fragment_samples_to_one && + !trk->entry && trk->start_dts != AV_NOPTS_VALUE) { if (!trk->frag_discont) { /* First packet of a new fragment. We already wrote the duration * of the last packet of the previous fragment based on track_duration, @@ -6038,6 +6143,33 @@ static int mov_write_packet(AVFormatContext *s, AVPacket *pkt) } } + if (trk->squash_fragment_samples_to_one) { + /* + * If the track has to have its samples squashed into one sample, + * we just take it into the track's queue. + * This will then be utilized as the samples get written in either + * mov_flush_fragment or when the mux is finalized in + * mov_write_trailer. + */ + int ret = AVERROR_BUG; + + if (pkt->pts == AV_NOPTS_VALUE) { + av_log(s, AV_LOG_ERROR, + "Packets without a valid presentation timestamp are " + "not supported with packet squashing!\n"); + return AVERROR(EINVAL); + } + + if ((ret = avpriv_packet_list_put(&trk->squashed_packet_queue, + &trk->squashed_packet_queue_end, + pkt, av_packet_ref, 0)) < 0) { + return ret; + } + + return 0; + } + + if (trk->mode == MODE_MOV && trk->par->codec_type == AVMEDIA_TYPE_VIDEO) { AVPacket *opkt = pkt; int reshuffle_ret, ret; @@ -6316,6 +6448,11 @@ static void mov_free(AVFormatContext *s) ff_mov_cenc_free(&mov->tracks[i].cenc); ffio_free_dyn_buf(&mov->tracks[i].mdat_buf); + + if (mov->tracks[i].squashed_packet_queue) { + avpriv_packet_list_free(&(mov->tracks[i].squashed_packet_queue), + &(mov->tracks[i].squashed_packet_queue_end)); + } } av_freep(&mov->tracks); @@ -6706,6 +6843,36 @@ static int mov_init(AVFormatContext *s) } } else if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) { track->timescale = st->time_base.den; + + if (track->par->codec_id == AV_CODEC_ID_TTML) { + /* 14496-30 requires us to use a single sample per fragment + for TTML, for which we define a per-track flag. + + We set the flag in case we are receiving TTML paragraphs + from the input, in other words in case we are not doing + stream copy. */ + track->squash_fragment_samples_to_one = + ff_is_ttml_stream_paragraph_based(track->par); + + if (mov->flags & FF_MOV_FLAG_FRAGMENT && + track->squash_fragment_samples_to_one) { + av_log(s, AV_LOG_ERROR, + "Fragmentation is not currently supported for " + "TTML in MP4/ISMV (track synchronization between " + "subtitles and other media is not yet implemented)!\n"); + return AVERROR_PATCHWELCOME; + } + + if (track->mode != MODE_ISM && + track->par->codec_tag == MOV_ISMV_TTML_TAG && + s->strict_std_compliance > FF_COMPLIANCE_UNOFFICIAL) { + av_log(s, AV_LOG_ERROR, + "ISMV style TTML support with the 'dfxp' tag in " + "non-ISMV formats is not officially supported. Add " + "'-strict unofficial' if you want to use it.\n"); + return AVERROR_EXPERIMENTAL; + } + } } else if (st->codecpar->codec_type == AVMEDIA_TYPE_DATA) { track->timescale = st->time_base.den; } else { @@ -7052,6 +7219,11 @@ static int mov_write_trailer(AVFormatContext *s) } } + // Check if we have any tracks that require squashing. + // In that case, we'll have to write the packet here. + if ((res = mov_write_squashed_packets(s)) < 0) + return res; + // If there were no chapters when the header was written, but there // are chapters now, write them in the trailer. This only works // when we are not doing fragments. @@ -7196,6 +7368,8 @@ static const AVCodecTag codec_mp4_tags[] = { { AV_CODEC_ID_MOV_TEXT, MKTAG('t', 'x', '3', 'g') }, { AV_CODEC_ID_BIN_DATA, MKTAG('g', 'p', 'm', 'd') }, { AV_CODEC_ID_MPEGH_3D_AUDIO, MKTAG('m', 'h', 'm', '1') }, + { AV_CODEC_ID_TTML, MOV_MP4_TTML_TAG }, + { AV_CODEC_ID_TTML, MOV_ISMV_TTML_TAG }, { AV_CODEC_ID_NONE, 0 }, }; #if CONFIG_MP4_MUXER || CONFIG_PSP_MUXER @@ -7204,6 +7378,7 @@ static const AVCodecTag *const mp4_codec_tags_list[] = { codec_mp4_tags, NULL }; static const AVCodecTag codec_ism_tags[] = { { AV_CODEC_ID_WMAPRO , MKTAG('w', 'm', 'a', ' ') }, + { AV_CODEC_ID_TTML , MOV_ISMV_TTML_TAG }, { AV_CODEC_ID_NONE , 0 }, }; diff --git a/libavformat/movenc.h b/libavformat/movenc.h index af1ea0bce6..95db1bf46d 100644 --- a/libavformat/movenc.h +++ b/libavformat/movenc.h @@ -26,6 +26,7 @@ #include "avformat.h" #include "movenccenc.h" +#include "libavcodec/packet_internal.h" #define MOV_FRAG_INFO_ALLOC_INCREMENT 64 #define MOV_INDEX_CLUSTER_SIZE 1024 @@ -164,6 +165,10 @@ typedef struct MOVTrack { int pal_done; int is_unaligned_qt_rgb; + + unsigned int squash_fragment_samples_to_one; //< flag to note formats where all samples for a fragment are to be squashed + + PacketList *squashed_packet_queue, *squashed_packet_queue_end; } MOVTrack; typedef enum { diff --git a/libavformat/movenc_ttml.c b/libavformat/movenc_ttml.c new file mode 100644 index 0000000000..472572b45a --- /dev/null +++ b/libavformat/movenc_ttml.c @@ -0,0 +1,171 @@ +/* + * MP4, ISMV Muxer TTML helpers + * Copyright (c) 2021 24i + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "avformat.h" +#include "avio_internal.h" +#include "isom.h" +#include "movenc.h" +#include "movenc_ttml.h" +#include "libavcodec/packet_internal.h" + +static const unsigned char empty_ttml_document[] = + ""; + +static int mov_init_ttml_writer(MOVTrack *track, AVFormatContext **out_ctx) +{ + AVStream *movenc_stream = track->st, *ttml_stream = NULL; + int ret = AVERROR_BUG; + + if ((ret = avformat_alloc_output_context2(out_ctx, NULL, + "ttml", NULL)) < 0) + return ret; + + if ((ret = avio_open_dyn_buf(&(*out_ctx)->pb)) < 0) + return ret; + + if (!(ttml_stream = avformat_new_stream(*out_ctx, NULL))) { + return AVERROR(ENOMEM); + } + + if ((ret = avcodec_parameters_copy(ttml_stream->codecpar, + movenc_stream->codecpar)) < 0) + return ret; + + ttml_stream->time_base = movenc_stream->time_base; + + return 0; +} + +static int mov_write_ttml_document_from_queue(AVFormatContext *s, + AVFormatContext *ttml_ctx, + MOVTrack *track, + AVPacket *pkt, + int64_t *out_start_ts, + int64_t *out_duration) +{ + int ret = AVERROR_BUG; + int64_t start_ts = track->start_dts == AV_NOPTS_VALUE ? + 0 : (track->start_dts + track->track_duration); + int64_t end_ts = start_ts; + + if ((ret = avformat_write_header(ttml_ctx, NULL)) < 0) { + return ret; + } + + while (!avpriv_packet_list_get(&track->squashed_packet_queue, + &track->squashed_packet_queue_end, + pkt)) { + end_ts = FFMAX(end_ts, pkt->pts + pkt->duration); + + // in case of the 'dfxp' muxing mode, each written document is offset + // to its containing sample's beginning. + if (track->par->codec_tag == MOV_ISMV_TTML_TAG) { + pkt->dts = pkt->pts = (pkt->pts - start_ts); + } + + pkt->stream_index = 0; + + av_packet_rescale_ts(pkt, track->st->time_base, + ttml_ctx->streams[pkt->stream_index]->time_base); + + if ((ret = av_write_frame(ttml_ctx, pkt)) < 0) { + goto cleanup; + } + + av_packet_unref(pkt); + } + + if ((ret = av_write_trailer(ttml_ctx)) < 0) + goto cleanup; + + *out_start_ts = start_ts; + *out_duration = end_ts - start_ts; + + ret = 0; + +cleanup: + return ret; +} + +int ff_mov_generate_squashed_ttml_packet(AVFormatContext *s, + MOVTrack *track, AVPacket *pkt) +{ + AVFormatContext *ttml_ctx = NULL; + // values for the generated AVPacket + int64_t start_ts = 0; + int64_t duration = 0; + + int ret = AVERROR_BUG; + + if ((ret = mov_init_ttml_writer(track, &ttml_ctx)) < 0) { + av_log(s, AV_LOG_ERROR, "Failed to initialize the TTML writer: %s\n", + av_err2str(ret)); + goto cleanup; + } + + if (!track->squashed_packet_queue) { + // empty queue, write minimal empty document with zero duration + avio_write(ttml_ctx->pb, empty_ttml_document, + sizeof(empty_ttml_document) - 1); + start_ts = 0; + duration = 0; + goto generate_packet; + } + + if ((ret = mov_write_ttml_document_from_queue(s, ttml_ctx, track, pkt, + &start_ts, + &duration)) < 0) { + av_log(s, AV_LOG_ERROR, + "Failed to generate a squashed TTML packet from the packet " + "queue: %s\n", + av_err2str(ret)); + goto cleanup; + } + +generate_packet: + { + // Generate an AVPacket from the data written into the dynamic buffer. + uint8_t *buf = NULL; + int buf_len = avio_close_dyn_buf(ttml_ctx->pb, &buf); + ttml_ctx->pb = NULL; + + if ((ret = av_packet_from_data(pkt, buf, buf_len)) < 0) { + av_log(s, AV_LOG_ERROR, + "Failed to create a TTML AVPacket from AVIO data: %s\n", + av_err2str(ret)); + av_freep(&buf); + goto cleanup; + } + + pkt->pts = pkt->dts = start_ts; + pkt->duration = duration; + pkt->flags |= AV_PKT_FLAG_KEY; + } + + ret = 0; + +cleanup: + if (ttml_ctx) + ffio_free_dyn_buf(&ttml_ctx->pb); + + avformat_free_context(ttml_ctx); + return ret; +} diff --git a/libavformat/movenc_ttml.h b/libavformat/movenc_ttml.h new file mode 100644 index 0000000000..c71ecd0997 --- /dev/null +++ b/libavformat/movenc_ttml.h @@ -0,0 +1,31 @@ +/* + * MP4, ISMV Muxer TTML helpers + * Copyright (c) 2021 24i + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVFORMAT_MOVENC_TTML_H +#define AVFORMAT_MOVENC_TTML_H + +#include "avformat.h" +#include "movenc.h" + +int ff_mov_generate_squashed_ttml_packet(AVFormatContext *s, + MOVTrack *track, AVPacket *pkt); + +#endif /* AVFORMAT_MOVENC_TTML_H */ diff --git a/tests/fate/subtitles.mak b/tests/fate/subtitles.mak index ee65afe35b..880109f201 100644 --- a/tests/fate/subtitles.mak +++ b/tests/fate/subtitles.mak @@ -109,6 +109,10 @@ fate-sub-dvb: CMD = framecrc -i $(TARGET_SAMPLES)/sub/dvbsubtest_filter.ts -map FATE_SUBTITLES-$(call ALLYES, FILE_PROTOCOL PIPE_PROTOCOL SRT_DEMUXER SUBRIP_DECODER TTML_ENCODER TTML_MUXER) += fate-sub-ttmlenc fate-sub-ttmlenc: CMD = fmtstdout ttml -i $(TARGET_SAMPLES)/sub/SubRip_capability_tester.srt +FATE_SUBTITLES-$(call ALLYES, FILE_PROTOCOL SRT_DEMUXER MOV_DEMUXER SUBRIP_DECODER TTML_ENCODER TTML_MUXER MOV_MUXER) += fate-sub-ttml-mp4-stpp fate-sub-ttml-mp4-dfxp +fate-sub-ttml-mp4-stpp: CMD = transcode srt $(TARGET_SAMPLES)/sub/SubRip_capability_tester.srt mp4 "-map 0:s -c:s ttml -time_base:s 1:1000" "-map 0 -c copy" "" "-of json -show_entries packet:stream=index,codec_type,codec_tag_string,codec_tag,codec_name,time_base,start_time,duration_ts,duration,nb_frames,nb_read_packets:stream_tags" +fate-sub-ttml-mp4-dfxp: CMD = transcode srt $(TARGET_SAMPLES)/sub/SubRip_capability_tester.srt mp4 "-map 0:s -c:s ttml -time_base:s 1:1000 -tag:s dfxp -strict unofficial" "-map 0 -c copy" "" "-of json -show_entries packet:stream=index,codec_type,codec_tag_string,codec_tag,codec_name,time_base,start_time,duration_ts,duration,nb_frames,nb_read_packets:stream_tags" + FATE_SUBTITLES-$(call ENCMUX, ASS, ASS) += $(FATE_SUBTITLES_ASS-yes) FATE_SUBTITLES += $(FATE_SUBTITLES-yes) diff --git a/tests/ref/fate/sub-ttml-mp4-dfxp b/tests/ref/fate/sub-ttml-mp4-dfxp new file mode 100644 index 0000000000..0172e5b7e6 --- /dev/null +++ b/tests/ref/fate/sub-ttml-mp4-dfxp @@ -0,0 +1,44 @@ +2e7e01c821c111466e7a2844826b7f6d *tests/data/fate/sub-ttml-mp4-dfxp.mp4 +8519 tests/data/fate/sub-ttml-mp4-dfxp.mp4 +#tb 0: 1/1000 +#media_type 0: data +#codec_id 0: none +0, 0, 0, 68500, 7866, 0x456c36b7 +{ + "packets": [ + { + "codec_type": "data", + "stream_index": 0, + "pts": 0, + "pts_time": "0.000000", + "dts": 0, + "dts_time": "0.000000", + "duration": 68500, + "duration_time": "68.500000", + "size": "7866", + "pos": "44", + "flags": "K_" + } + ], + "programs": [ + + ], + "streams": [ + { + "index": 0, + "codec_type": "data", + "codec_tag_string": "dfxp", + "codec_tag": "0x70786664", + "time_base": "1/1000", + "start_time": "0.000000", + "duration_ts": 68500, + "duration": "68.500000", + "nb_frames": "1", + "nb_read_packets": "1", + "tags": { + "language": "und", + "handler_name": "SubtitleHandler" + } + } + ] +} diff --git a/tests/ref/fate/sub-ttml-mp4-stpp b/tests/ref/fate/sub-ttml-mp4-stpp new file mode 100644 index 0000000000..a5165b568d --- /dev/null +++ b/tests/ref/fate/sub-ttml-mp4-stpp @@ -0,0 +1,44 @@ +cbd2c7ff864a663b0d893deac5a0caec *tests/data/fate/sub-ttml-mp4-stpp.mp4 +8547 tests/data/fate/sub-ttml-mp4-stpp.mp4 +#tb 0: 1/1000 +#media_type 0: data +#codec_id 0: none +0, 0, 0, 68500, 7866, 0x456c36b7 +{ + "packets": [ + { + "codec_type": "data", + "stream_index": 0, + "pts": 0, + "pts_time": "0.000000", + "dts": 0, + "dts_time": "0.000000", + "duration": 68500, + "duration_time": "68.500000", + "size": "7866", + "pos": "44", + "flags": "K_" + } + ], + "programs": [ + + ], + "streams": [ + { + "index": 0, + "codec_type": "data", + "codec_tag_string": "stpp", + "codec_tag": "0x70707473", + "time_base": "1/1000", + "start_time": "0.000000", + "duration_ts": 68500, + "duration": "68.500000", + "nb_frames": "1", + "nb_read_packets": "1", + "tags": { + "language": "und", + "handler_name": "SubtitleHandler" + } + } + ] +}