From patchwork Wed Apr 14 06:57:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 26909 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 70BBA44A1CD for ; Wed, 14 Apr 2021 10:27:37 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5193968A848; Wed, 14 Apr 2021 10:27:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6A78A689D24 for ; Wed, 14 Apr 2021 10:27:31 +0300 (EEST) Received: by mail-lj1-f172.google.com with SMTP id l22so14857384ljc.9 for ; Wed, 14 Apr 2021 00:27:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=IgaAJeLyzFTfid9BajkQMpcllUF1M5KyAjay51WfXio=; b=s9Vqqs7R02mSiUYR2dSKGhKqTeAFxKwciifOoMun4msnVzMG/F5YnjF3vAfBW2GUfy GwCPOCOXFiZVxesw3YNfR2yygsWBBSuyy18UkYjMSJ6gvKbvc1loyWY3FNRcnbWh8UCC 4O+W0z3+U6vJr1b4bwueWPfaTM1+HKJjaG/Ur5WYehvFD8a64vQWmnJw5wGZu8jqBLJj GHUQa0k22fXsFwDEwT5aU6UThQzaS9mO8R5bMzBlNf5xLesMiAH4IZRVG4Pm6xiCgSsj rDFaJrBal0pO8mjAUUeIv+TFJsJhAXafxtCE9alfxXVQQMbPN8Oxr17y7RkMIGFqPPYb vBRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IgaAJeLyzFTfid9BajkQMpcllUF1M5KyAjay51WfXio=; b=A3tOY9d8Ax6Bd5qNxunLTO3o/ikpDBOJc1vJdIeEUjG83j6TppI2VZwyvh6Idclwai nZI+NcusWrTzh66JIH7CTcuqBAaUJIOcGOIzPQTH4dfgBkUqK8V4Omlz0ToJ8hGqrv0r gTLSN4rsg6SXxw/Wlun0+W81BIOGoA2UV1fvO9Mrw1hz6NIrrhZB/LDuTpqxnsqq4PI3 BNZ3kAbUS0awvw86SrREi4XpaS0wWA3tUa4MvLwX+PMkZ8wAd9rjXJbsjHX22oV7reL2 crn3Iix03KxqAImTzALhS0l+YlpHTveQb3A1nRo6Wv1JuLQbmoKt5k6h6G74S0X1hCfq 0yqQ== X-Gm-Message-State: AOAM531ddw4HmZoJFvOQtbWNnb8YYSrT0WvVIvqYkE2m6YLWnPd+bad3 qCJBzlySI9QS03eNyat99q1X4atJZGs= X-Google-Smtp-Source: ABdhPJxoUn6elmrtHZ7qCM3LeBR2BedPb4k/8ciNvaX+gJITzHLONv4Vzna9nJDYb5h1+MDBGuTICg== X-Received: by 2002:a05:6512:3d20:: with SMTP id d32mr14948663lfv.163.1618383450663; Tue, 13 Apr 2021 23:57:30 -0700 (PDT) Received: from localhost.localdomain (91-159-194-103.elisa-laajakaista.fi. [91.159.194.103]) by smtp.gmail.com with ESMTPSA id w5sm4960167ljo.11.2021.04.13.23.57.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 23:57:30 -0700 (PDT) From: =?utf-8?q?Jan_Ekstr=C3=B6m?= To: ffmpeg-devel@ffmpeg.org Date: Wed, 14 Apr 2021 09:57:24 +0300 Message-Id: <20210414065725.8828-4-jeebjp@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210414065725.8828-1-jeebjp@gmail.com> References: <20210414065725.8828-1-jeebjp@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 3/4] avcodec/ttmlenc: add initial support for regions and styles X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Jan Ekström Attempts to utilize the TTML cell resolution as a mapping to the reference resolution, and maps font size to cell size. Additionally sets the display and text alignment according to the ASS alignment number. Signed-off-by: Jan Ekström --- libavcodec/ttmlenc.c | 210 ++++++++++++++++++++++++++++++++++--- libavcodec/ttmlenc.h | 3 +- tests/ref/fate/sub-ttmlenc | 86 ++++++++------- 3 files changed, 247 insertions(+), 52 deletions(-) diff --git a/libavcodec/ttmlenc.c b/libavcodec/ttmlenc.c index e3c155fdd1..c190356683 100644 --- a/libavcodec/ttmlenc.c +++ b/libavcodec/ttmlenc.c @@ -100,20 +100,33 @@ static int ttml_encode_frame(AVCodecContext *avctx, uint8_t *buf, dialog = ff_ass_split_dialog(s->ass_ctx, ass, 0, &num); for (; dialog && num--; dialog++) { - int ret = ff_ass_split_override_codes(&ttml_callbacks, s, - dialog->text); - int log_level = (ret != AVERROR_INVALIDDATA || - avctx->err_recognition & AV_EF_EXPLODE) ? - AV_LOG_ERROR : AV_LOG_WARNING; + if (dialog->style) { + av_bprintf(&s->buffer, "buffer, dialog->style, NULL, + AV_ESCAPE_MODE_XML, + AV_ESCAPE_FLAG_XML_DOUBLE_QUOTES); + av_bprintf(&s->buffer, "\">"); + } - if (ret < 0) { - av_log(avctx, log_level, - "Splitting received ASS dialog failed: %s\n", - av_err2str(ret)); + { + int ret = ff_ass_split_override_codes(&ttml_callbacks, s, + dialog->text); + int log_level = (ret != AVERROR_INVALIDDATA || + avctx->err_recognition & AV_EF_EXPLODE) ? + AV_LOG_ERROR : AV_LOG_WARNING; - if (log_level == AV_LOG_ERROR) - return ret; + if (ret < 0) { + av_log(avctx, log_level, + "Splitting received ASS dialog failed: %s\n", + av_err2str(ret)); + + if (log_level == AV_LOG_ERROR) + return ret; + } } + + if (dialog->style) + av_bprintf(&s->buffer, ""); } } else { #endif @@ -121,6 +134,14 @@ static int ttml_encode_frame(AVCodecContext *avctx, uint8_t *buf, if (!dialog) return AVERROR(ENOMEM); + if (dialog->style) { + av_bprintf(&s->buffer, "buffer, dialog->style, NULL, + AV_ESCAPE_MODE_XML, + AV_ESCAPE_FLAG_XML_DOUBLE_QUOTES); + av_bprintf(&s->buffer, "\">"); + } + { int ret = ff_ass_split_override_codes(&ttml_callbacks, s, dialog->text); @@ -140,6 +161,9 @@ static int ttml_encode_frame(AVCodecContext *avctx, uint8_t *buf, } } + if (dialog->style) + av_bprintf(&s->buffer, ""); + ff_ass_free_dialog(&dialog); } #if FF_API_ASS_TIMING @@ -173,17 +197,175 @@ static av_cold int ttml_encode_close(AVCodecContext *avctx) return 0; } +static const char *ttml_get_display_alignment(int alignment) +{ + switch (alignment) { + case 1: + case 2: + case 3: + return "after"; + case 4: + case 5: + case 6: + return "center"; + case 7: + case 8: + case 9: + return "before"; + default: + return NULL; + } +} + +static const char *ttml_get_text_alignment(int alignment) +{ + switch (alignment) { + case 1: + case 4: + case 7: + return "left"; + case 2: + case 5: + case 8: + return "center"; + case 3: + case 6: + case 9: + return "right"; + default: + return NULL; + } +} + +static int ttml_write_region(AVCodecContext *avctx, AVBPrint *buf, + ASSStyle *style) +{ + const char *display_alignment = NULL; + const char *text_alignment = NULL; + + if (!style) + return AVERROR_INVALIDDATA; + + if (!style->name) { + av_log(avctx, AV_LOG_ERROR, "Subtitle style name not set!\n"); + return AVERROR_INVALIDDATA; + } + + if (style->font_size < 0) { + av_log(avctx, AV_LOG_ERROR, "Invalid font size for TTML: %d!\n", + style->font_size); + return AVERROR_INVALIDDATA; + } + + display_alignment = ttml_get_display_alignment(style->alignment); + text_alignment = ttml_get_text_alignment(style->alignment); + if (!display_alignment || !text_alignment) { + av_log(avctx, AV_LOG_ERROR, + "Failed to convert ASS style alignment %d of style %s to " + "TTML display and text alignment!\n", + style->alignment, + style->name); + return AVERROR_INVALIDDATA; + } + + av_bprintf(buf, " name, NULL, AV_ESCAPE_MODE_XML, + AV_ESCAPE_FLAG_XML_DOUBLE_QUOTES); + av_bprintf(buf, "\"\n"); + + av_bprintf(buf, " tts:displayAlign=\""); + av_bprint_escape(buf, display_alignment, NULL, AV_ESCAPE_MODE_XML, + AV_ESCAPE_FLAG_XML_DOUBLE_QUOTES); + av_bprintf(buf, "\"\n"); + + av_bprintf(buf, " tts:textAlign=\""); + av_bprint_escape(buf, text_alignment, NULL, AV_ESCAPE_MODE_XML, + AV_ESCAPE_FLAG_XML_DOUBLE_QUOTES); + av_bprintf(buf, "\"\n"); + + // if we set cell resolution to our script reference resolution, + // then a single line is a single "point" on our canvas. Thus, by setting + // our font size to font size in cells, we should gain a similar enough + // scale without resorting to explicit pixel based font sizing, which is + // frowned upon in the TTML community. + av_bprintf(buf, " tts:fontSize=\"%dc\"\n", + style->font_size); + + if (style->font_name) { + av_bprintf(buf, " tts:fontFamily=\""); + av_bprint_escape(buf, style->font_name, NULL, AV_ESCAPE_MODE_XML, + AV_ESCAPE_FLAG_XML_DOUBLE_QUOTES); + av_bprintf(buf, "\"\n"); + } + + av_bprintf(buf, " tts:overflow=\"visible\" />\n"); + + return 0; +} + static int ttml_write_header_content(AVCodecContext *avctx) { - if (!(avctx->extradata = av_mallocz(TTMLENC_EXTRADATA_SIGNATURE_SIZE + - 1 + AV_INPUT_BUFFER_PADDING_SIZE))) { + TTMLContext *s = avctx->priv_data; + ASS *ass = (ASS *)s->ass_ctx; + ASSScriptInfo script_info = ass->script_info; + const size_t base_extradata_size = TTMLENC_EXTRADATA_SIGNATURE_SIZE + 1 + + AV_INPUT_BUFFER_PADDING_SIZE; + size_t additional_extradata_size = 0; + + if (script_info.play_res_x <= 0 || script_info.play_res_y <= 0) { + av_log(avctx, AV_LOG_ERROR, + "Invalid subtitle reference resolution %dx%d!\n", + script_info.play_res_x, script_info.play_res_y); + return AVERROR_INVALIDDATA; + } + + // write the first string in extradata, attributes in the base "tt" element. + av_bprintf(&s->buffer, ttml_default_namespacing); + // the cell resolution is in character cells, so not exactly 1:1 against + // a pixel based resolution, but as the tts:extent in the root + // "tt" element is frowned upon (and disallowed in the EBU-TT profile), + // we mimic the reference resolution by setting it as the cell resolution. + av_bprintf(&s->buffer, " ttp:cellResolution=\"%d %d\"\n", + script_info.play_res_x, script_info.play_res_y); + av_bprint_chars(&s->buffer, '\0', 1); + + // write the second string in extradata, head element containing the styles + av_bprintf(&s->buffer, " \n"); + av_bprintf(&s->buffer, " \n"); + + for (int i = 0; i < ass->styles_count; i++) { + ASSStyle *style = &ass->styles[i]; + int ret = ttml_write_region(avctx, &s->buffer, style); + if (ret < 0) + return ret; + } + + av_bprintf(&s->buffer, " \n"); + av_bprintf(&s->buffer, " \n"); + av_bprint_chars(&s->buffer, '\0', 1); + + if (!av_bprint_is_complete(&s->buffer)) { return AVERROR(ENOMEM); } - avctx->extradata_size = TTMLENC_EXTRADATA_SIGNATURE_SIZE; + additional_extradata_size = s->buffer.len; + + if (!(avctx->extradata = + av_mallocz(base_extradata_size + additional_extradata_size))) { + return AVERROR(ENOMEM); + } + + avctx->extradata_size = + TTMLENC_EXTRADATA_SIGNATURE_SIZE + additional_extradata_size; memcpy(avctx->extradata, TTMLENC_EXTRADATA_SIGNATURE, TTMLENC_EXTRADATA_SIGNATURE_SIZE); + if (additional_extradata_size) + memcpy(avctx->extradata + TTMLENC_EXTRADATA_SIGNATURE_SIZE, + s->buffer.str, additional_extradata_size); + + av_bprint_clear(&s->buffer); + return 0; } diff --git a/libavcodec/ttmlenc.h b/libavcodec/ttmlenc.h index c3bb11478d..467f35c7a6 100644 --- a/libavcodec/ttmlenc.h +++ b/libavcodec/ttmlenc.h @@ -28,6 +28,7 @@ static const char ttml_default_namespacing[] = " xmlns=\"http://www.w3.org/ns/ttml\"\n" " xmlns:ttm=\"http://www.w3.org/ns/ttml#metadata\"\n" -" xmlns:tts=\"http://www.w3.org/ns/ttml#styling\"\n"; +" xmlns:tts=\"http://www.w3.org/ns/ttml#styling\"\n" +" xmlns:ttp=\"http://www.w3.org/ns/ttml#parameter\"\n"; #endif /* AVCODEC_TTMLENC_H */ diff --git a/tests/ref/fate/sub-ttmlenc b/tests/ref/fate/sub-ttmlenc index 51eab97817..6d0a8067fc 100644 --- a/tests/ref/fate/sub-ttmlenc +++ b/tests/ref/fate/sub-ttmlenc @@ -3,120 +3,132 @@ xmlns="http://www.w3.org/ns/ttml" xmlns:ttm="http://www.w3.org/ns/ttml#metadata" xmlns:tts="http://www.w3.org/ns/ttml#styling" + xmlns:ttp="http://www.w3.org/ns/ttml#parameter" + ttp:cellResolution="384 288" xml:lang=""> + + + + +

Don't show this text it may be used to insert hidden data

+ end="00:00:00.000">Don't show this text it may be used to insert hidden data

SubRip subtitles capability tester 1.3o by ale5000
Use VLC 1.1 or higher as reference for most things and MPC Home Cinema for others
This text should be blue
This text should be red
This text should be black
If you see this with the normal font, the player don't (fully) support font face

+ end="00:00:04.500">SubRip subtitles capability tester 1.3o by ale5000
Use VLC 1.1 or higher as reference for most things and MPC Home Cinema for others
This text should be blue
This text should be red
This text should be black
If you see this with the normal font, the player don't (fully) support font face

Hidden

+ end="00:00:04.500">Hidden

This text should be small
This text should be normal
This text should be big

+ end="00:00:07.500">This text should be small
This text should be normal
This text should be big

This should be an E with an accent: È
日本語
This text should be bold, italics and underline
This text should be small and green
This text should be small and red
This text should be big and brown

+ end="00:00:11.500">This should be an E with an accent: È
日本語
This text should be bold, italics and underline
This text should be small and green
This text should be small and red
This text should be big and brown

This line should be bold
This line should be italics
This line should be underline
This line should be strikethrough
Both lines
should be underline

+ end="00:00:14.500">This line should be bold
This line should be italics
This line should be underline
This line should be strikethrough
Both lines
should be underline

>
It would be a good thing to
hide invalid html tags that are closed and show the text in them
but show un-closed invalid html tags
Show not opened tags
<

+ end="00:00:17.500">>
It would be a good thing to
hide invalid html tags that are closed and show the text in them
but show un-closed invalid html tags
Show not opened tags
<

and also
hide invalid html tags with parameters that are closed and show the text in them
but show un-closed invalid html tags
This text should be showed underlined without problems also: 2<3,5>1,4<6
This shouldn't be underlined

+ end="00:00:20.500">and also
hide invalid html tags with parameters that are closed and show the text in them
but show un-closed invalid html tags
This text should be showed underlined without problems also: 2<3,5>1,4<6
This shouldn't be underlined

This text should be in the normal position...

+ end="00:00:21.500">This text should be in the normal position...

This text should NOT be in the normal position

+ end="00:00:22.500">This text should NOT be in the normal position

Implementation is the same of the ASS tag
This text should be at the
top and horizontally centered

+ end="00:00:24.500">Implementation is the same of the ASS tag
This text should be at the
top and horizontally centered

This text should be at the
middle and horizontally centered

+ end="00:00:24.500">This text should be at the
middle and horizontally centered

This text should be at the
bottom and horizontally centered

+ end="00:00:24.500">This text should be at the
bottom and horizontally centered

This text should be at the
top and horizontally at the left

+ end="00:00:26.500">This text should be at the
top and horizontally at the left

This text should be at the
middle and horizontally at the left
(The second position must be ignored)

+ end="00:00:26.500">This text should be at the
middle and horizontally at the left
(The second position must be ignored)

This text should be at the
bottom and horizontally at the left

+ end="00:00:26.500">This text should be at the
bottom and horizontally at the left

This text should be at the
top and horizontally at the right

+ end="00:00:28.500">This text should be at the
top and horizontally at the right

This text should be at the
middle and horizontally at the right

+ end="00:00:28.500">This text should be at the
middle and horizontally at the right

This text should be at the
bottom and horizontally at the right

+ end="00:00:28.500">This text should be at the
bottom and horizontally at the right

This could be the most difficult thing to implement

+ end="00:00:31.500">This could be the most difficult thing to implement

First text

+ end="00:00:50.500">First text

Second, it shouldn't overlap first

+ end="00:00:35.500">Second, it shouldn't overlap first

Third, it should replace second

+ end="00:00:37.500">Third, it should replace second

Fourth, it shouldn't overlap first and third

+ end="00:00:50.500">Fourth, it shouldn't overlap first and third

Fifth, it should replace third

+ end="00:00:45.500">Fifth, it should replace third

Sixth, it shouldn't be
showed overlapped

+ end="00:00:50.500">Sixth, it shouldn't be
showed overlapped

TEXT 1 (bottom)

+ end="00:00:52.500">TEXT 1 (bottom)

text 2

+ end="00:00:52.500">text 2

Hide these tags:
also hide these tags:
but show this: {normal text}

+ end="00:00:54.500">Hide these tags:
also hide these tags:
but show this: {normal text}


\ N is a forced line break
\ h is a hard space
Normal spaces at the start and at the end of the line are trimmed while hard spaces are not trimmed.
The\hline\hwill\hnever\hbreak\hautomatically\hright\hbefore\hor\hafter\ha\hhard\hspace.\h:-D

+ end="00:01:00.500">
\ N is a forced line break
\ h is a hard space
Normal spaces at the start and at the end of the line are trimmed while hard spaces are not trimmed.
The\hline\hwill\hnever\hbreak\hautomatically\hright\hbefore\hor\hafter\ha\hhard\hspace.\h:-D


\h\h\h\h\hA (05 hard spaces followed by a letter)
A (Normal spaces followed by a letter)
A (No hard spaces followed by a letter)

+ end="00:00:56.500">
\h\h\h\h\hA (05 hard spaces followed by a letter)
A (Normal spaces followed by a letter)
A (No hard spaces followed by a letter)

\h\h\h\h\hA (05 hard spaces followed by a letter)
A (Normal spaces followed by a letter)
A (No hard spaces followed by a letter)
Show this: \TEST and this: \-)

+ end="00:00:58.500">\h\h\h\h\hA (05 hard spaces followed by a letter)
A (Normal spaces followed by a letter)
A (No hard spaces followed by a letter)
Show this: \TEST and this: \-)


A letter followed by 05 hard spaces: A\h\h\h\h\h
A letter followed by normal spaces: A
A letter followed by no hard spaces: A
05 hard spaces between letters: A\h\h\h\h\hA
5 normal spaces between letters: A A

^--Forced line break

+ end="00:01:00.500">
A letter followed by 05 hard spaces: A\h\h\h\h\h
A letter followed by normal spaces: A
A letter followed by no hard spaces: A
05 hard spaces between letters: A\h\h\h\h\hA
5 normal spaces between letters: A A

^--Forced line break

Both line should be strikethrough,
yes.
Correctly closed tags
should be hidden.

+ end="00:01:02.500">Both line should be strikethrough,
yes.
Correctly closed tags
should be hidden.

It shouldn't be strikethrough,
not opened tag showed as text.
Not opened tag showed as text.

+ end="00:01:04.500">It shouldn't be strikethrough,
not opened tag showed as text.
Not opened tag showed as text.

Three lines should be strikethrough,
yes.
Not closed tags showed as text

+ end="00:01:06.500">Three lines should be strikethrough,
yes.
Not closed tags showed as text

Both line should be strikethrough but
the wrong closing tag should be showed

+ end="00:01:08.500">Both line should be strikethrough but
the wrong closing tag should be showed