From patchwork Fri Sep 24 00:31:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30536 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp988065iob; Thu, 23 Sep 2021 17:32:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwzB3O//qZdOUN+cvItYF7ZuOEWKixuXXJ7/GI9gbOX7CCPwmIiwOA4LPMq6je7G79tOj05 X-Received: by 2002:a05:6402:1808:: with SMTP id g8mr1984847edy.188.1632443540757; Thu, 23 Sep 2021 17:32:20 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r24si9411493ejy.194.2021.09.23.17.32.20; Thu, 23 Sep 2021 17:32:20 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=vPO39nEj; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 969DE6800C9; Fri, 24 Sep 2021 03:31:13 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11olkn2084.outbound.protection.outlook.com [40.92.18.84]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 97D0F68A543 for ; Fri, 24 Sep 2021 03:31:10 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Htwm3Ycq/b4lOeFMmKScYPldXNjOauAZu9B0gIzLDw2cR3QsFfm/oY7ogRfPwQJNoExsLAbD39sjIrzwkDFvYIJjm/5ZIMLIsAeCrw6kot/7IzRMsUXzm4qH8949oJArVuT/cEO5eBxOT1Cmk3WlRciQsL1Fien2sBBsX18a45E4gDsh4kZtpdogud+Xf0hdQsFT5Q/jz/hYRdUuD4v7iDUOHsehAYkpgTXLRzZWu4KDSqMzB5ua/xZ3CBnGMl1DQWW39GEFgGDJ3KGc+2oxb0kHzUWVU0dduEIwgxTOa/9FRR3sqKRVReXTgvKaS4rSjoQbC3XwJgfxE6tyoTPVDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=+dJuy4v0A+Bkddj7R39vAg1SVypa+dAjUTjxymbs10E=; b=ewGIq+b9y+aM1zJuH2e5JwdTVJCELiia6CLpIGNsXeSLx3vORvUgr7NlJYgXBKc/gKNI99bffnDrdd9IgtWp5PEt40v6/eZ//kubEenivZEEvkp7HxfOMJFtngzUo9no6nrPDrxiwVJiIaJW75cVdJgbN3CY2rX50vMhEXc9yRXvMHbgvJsN4cPnD8JAcvWNgbgYWB29Y0SJikzrILfhu2f91MxE/DkdAmfWFQYuTzSTPgNfzBOra0rkYlWluIb+dw7regjmqDDFu6c9PF5g5JEh0ZbLY6Ozj0FJRnUUaxWpS08KJqSsDHnW2e+U/2mBuBa0uW7H8hzRrrW4B57JXg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+dJuy4v0A+Bkddj7R39vAg1SVypa+dAjUTjxymbs10E=; b=vPO39nEjwy6vS+rPvwhhWfex5qkUF0xZt2j1/xSHVqjI85lUMXu0ua/1P5gXYybXjllBtLt13Wkxt1Cqi/fCwbdwGx073qw6frPcxABEB8qbQ1Nawaw1mj16QC7rJWSAJ6JvsejWt9pVHvJLDUL5KGgiqJ4hIn2TmUiWv90ep+ZKLke2Ln4px77YXsvf3bORET5OY+fNGun+inHH7BXtloek4HWtXx1fo6bYpyBhTWl1Of0aGCtdD32T5zS2W6mhFzAkE7hTfYV+JVvAtoDs6BpK8WYEM9Z4gfRGJ5FclrcguUFn5BqUf1pbxFRUVizBTlPzxRwqIEGAYsd7dmYwqA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5743.namprd04.prod.outlook.com (2603:10b6:208:38::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.14; Fri, 24 Sep 2021 00:31:06 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.015; Fri, 24 Sep 2021 00:31:06 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v9 09/13] avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters Thread-Index: AQHXsNt22gAhqvwq3ECQavIv0v0wtw== Date: Fri, 24 Sep 2021 00:31:06 +0000 Message-ID: References: <5bc443d3efba607cef8020fdacce3fbec59514b7.1632443311.git.softworkz@hotmail.com> <74b7003baacf9847f879d785c002e4a78293f57c.1632443311.git.softworkz@hotmail.com> In-Reply-To: <74b7003baacf9847f879d785c002e4a78293f57c.1632443311.git.softworkz@hotmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [wDYSfShuXAwqyqjLwAFapsfboETeCVew] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1a6b9ead-c237-47b8-d847-08d97ef298dd x-ms-traffictypediagnostic: MN2PR04MB5743: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: HRsdLbNN6LHfVk6GQI/fuJvI20I8FIFt0HBPD6ArDgBWEDq6oWbkcQKrqPMS80ixIR2cAN64UGB62tbnRkKJCq/son0aowMW4jEehRT6f6smRBKfKIOEoMAix52BIhqPsK58kqXQwoks6KgerY3LNKmIPt7pt9DoTBa3cH5Dh7Kj+AiYFJLDw5DJwjuf3P+brEgolgTw96jPJNM52hm37xA2SdN1qhQF/c3AfULzrmO38Zg4owUdBQhP9L5UPSL6HeSzvF0GV22vHu60boDAmm4V+mFz4dm5OmN7ZYJ0gQ410BdwHPrk3URI8k6GUlF5ndyAU3TaYSKTkU2Uti/iVibUjRcqguauMOZgnaXDRNHiGLfjU8Yy4yubjGtqVc+JTsrT2AFH5eEy9Cy7yegPVzl+7F03+MK8MtDDnP1OKyv2hi/FSFAcEa05UpwuWo7NihhTCPG9Kq8uB0LZ1HF6x0NmiW/Dj25qLTGKzwBrSik= x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: 40Ilj3mc95KQSR3dfFcyh+9Lz69Vl1o/o8P7qh3SyrSQL65BLzT++gV8d3IfWFzCcZZRWcm0JwMQWBVCJPF7pYob9/fQearzyfRkHWHCR9tg3ZnE8h1kVbXOUhIDXHANhXciluRwvi73k7nx6hQvjw== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 1a6b9ead-c237-47b8-d847-08d97ef298dd X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Sep 2021 00:31:06.0137 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5743 Subject: [FFmpeg-devel] [PATCH v9 09/13] avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Q0DfV5yVErjZ - overlay_textsubs {VS -> V) Overlay text subtitles onto a video stream. - textsubs2video {S -> V) Converts text subtitles to video frames Signed-off-by: softworkz --- configure | 2 + doc/filters.texi | 69 ++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 2 + libavfilter/avfilter.c | 18 +- libavfilter/vf_overlay_textsubs.c | 609 ++++++++++++++++++++++++++++++ 6 files changed, 697 insertions(+), 5 deletions(-) create mode 100644 libavfilter/vf_overlay_textsubs.c diff --git a/configure b/configure index d18bb73ed6..37fc4c20e7 100755 --- a/configure +++ b/configure @@ -3624,6 +3624,7 @@ openclsrc_filter_deps="opencl" overlay_opencl_filter_deps="opencl" overlay_qsv_filter_deps="libmfx" overlay_qsv_filter_select="qsvvpp" +overlay_textsubs_filter_deps="avcodec libass" overlay_vulkan_filter_deps="vulkan_lib libglslang" owdenoise_filter_deps="gpl" pad_opencl_filter_deps="opencl" @@ -3669,6 +3670,7 @@ superequalizer_filter_deps="avcodec" superequalizer_filter_select="rdft" surround_filter_deps="avcodec" surround_filter_select="rdft" +textsub2video_filter_deps="avcodec libass" tinterlace_filter_deps="gpl" tinterlace_merge_test_deps="tinterlace_filter" tinterlace_pad_test_deps="tinterlace_filter" diff --git a/doc/filters.texi b/doc/filters.texi index 9ce956e507..416bda5efb 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -25181,6 +25181,75 @@ The graphicsub2video is inserted automatically for compatibility with legacy com ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:0][0:1]overlay" output.mp4 @end example @end itemize + +@section overlay_textsubs + +Overlay text subtitles onto a video stream. + +This filter supersedes the classic @ref{subtitles} filter opposed to which it does no longer require to open and access the source stream separately, which is often causing problems or doesn't even work for non-local or slow sources. + +Inputs: +- 0: Video [YUV420P, YUV422P, YUV444P, ARGB, RGBA, ABGR, BGRA, RGB24, BGR24] +- 1: Subtitles [text] + +Outputs: +- 0: Video (same as input) + +It accepts the following parameters: + +@table @option + +@item alpha +Process alpha channel, by default alpha channel is untouched. + +@item fonts_dir +Set a directory path containing fonts that can be used by the filter. +These fonts will be used in addition to whatever the font provider uses. + +@item force_style +Override default style or script info parameters of the subtitles. It accepts a +string containing ASS style format @code{KEY=VALUE} couples separated by ",". + +@end table + +@section textsub2video + +Converts text subtitles to video frames. + +For overlaying text subtitles onto video frames it is recommended to use the overlay_textsubs filter. +The textsub2video is useful for for creating transparent text-frames when overlay is done via hw acceleration + +Inputs: +- 0: Subtitles [text] + +Outputs: +- 0: Video [ARGB, RGBA, ABGR, BGRA] + +It accepts the following parameters: + +@table @option + +@item rate, r +Set the framerate for updating overlay frames. +Normally, overlay frames will only be updated each time when the subtitles to display are changing. +In cases where subtitles include advanced features (like animation), this parameter determines the frequency by which the overlay frames should be updated. + +@item size, s +Set the output frame size. +Allows to override the size of output video frames. + +@item alpha +Process alpha channel, by default alpha channel is untouched. + +@item fonts_dir +Set a directory path containing fonts that can be used by the filter. +These fonts will be used in addition to whatever the font provider uses. + +@item force_style +Override default style or script info parameters of the subtitles. It accepts a +string containing ASS style format @code{KEY=VALUE} couples separated by ",". + +@end table @c man end SUBTITLE FILTERS @chapter Multimedia Filters diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 8fcc25989e..c0b1cc7001 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -365,6 +365,7 @@ OBJS-$(CONFIG_OVERLAY_OPENCL_FILTER) += vf_overlay_opencl.o opencl.o \ opencl/overlay.o framesync.o OBJS-$(CONFIG_OVERLAY_QSV_FILTER) += vf_overlay_qsv.o framesync.o OBJS-$(CONFIG_OVERLAY_GRAPHICSUBS_FILTER) += vf_overlay_graphicsubs.o framesync.o +OBJS-$(CONFIG_OVERLAY_TEXTSUBS_FILTER) += vf_overlay_textsubs.o OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) += vf_overlay_vulkan.o vulkan.o OBJS-$(CONFIG_OWDENOISE_FILTER) += vf_owdenoise.o OBJS-$(CONFIG_PAD_FILTER) += vf_pad.o @@ -453,6 +454,7 @@ OBJS-$(CONFIG_SWAPRECT_FILTER) += vf_swaprect.o OBJS-$(CONFIG_SWAPUV_FILTER) += vf_swapuv.o OBJS-$(CONFIG_TBLEND_FILTER) += vf_blend.o framesync.o OBJS-$(CONFIG_TELECINE_FILTER) += vf_telecine.o +OBJS-$(CONFIG_TEXTSUB2VIDEO_FILTER) += vf_overlay_textsubs.o OBJS-$(CONFIG_THISTOGRAM_FILTER) += vf_histogram.o OBJS-$(CONFIG_THRESHOLD_FILTER) += vf_threshold.o framesync.o OBJS-$(CONFIG_THUMBNAIL_FILTER) += vf_thumbnail.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 7a6ee174b0..88385e497d 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -346,6 +346,7 @@ extern const AVFilter ff_vf_overlay; extern const AVFilter ff_vf_overlay_opencl; extern const AVFilter ff_vf_overlay_qsv; extern const AVFilter ff_vf_overlay_graphicsubs; +extern const AVFilter ff_vf_overlay_textsubs; extern const AVFilter ff_vf_overlay_vulkan; extern const AVFilter ff_vf_overlay_cuda; extern const AVFilter ff_vf_owdenoise; @@ -526,6 +527,7 @@ extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_svf_graphicsub2video; +extern const AVFilter ff_svf_textsub2video; /* multimedia sources */ extern const AVFilter ff_avsrc_amovie; diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index c567b7893b..639a54096b 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -464,7 +464,7 @@ static int64_t guess_status_pts(AVFilterContext *ctx, int status, AVRational lin return AV_NOPTS_VALUE; } -static int ff_request_frame_to_filter(AVFilterLink *link) +static int ff_request_frame_to_filter(AVFilterLink *link, int input_index) { int ret = -1; @@ -473,8 +473,8 @@ static int ff_request_frame_to_filter(AVFilterLink *link) link->frame_blocked_in = 1; if (link->srcpad->request_frame) ret = link->srcpad->request_frame(link); - else if (link->src->inputs[0]) - ret = ff_request_frame(link->src->inputs[0]); + else if (link->src->inputs[input_index]) + ret = ff_request_frame(link->src->inputs[input_index]); if (ret < 0) { if (ret != AVERROR(EAGAIN) && ret != link->status_in) ff_avfilter_link_set_in_status(link, ret, guess_status_pts(link->src, ret, link->time_base)); @@ -1173,6 +1173,14 @@ static int forward_status_change(AVFilterContext *filter, AVFilterLink *in) { unsigned out = 0, progress = 0; int ret; + int input_index = 0; + + for (int i = 0; i < in->dst->nb_inputs; i++) { + if (&in->dst->input_pads[i] == in->dstpad) { + input_index = i; + break; + } + } av_assert0(!in->status_out); if (!filter->nb_outputs) { @@ -1182,7 +1190,7 @@ static int forward_status_change(AVFilterContext *filter, AVFilterLink *in) while (!in->status_out) { if (!filter->outputs[out]->status_in) { progress++; - ret = ff_request_frame_to_filter(filter->outputs[out]); + ret = ff_request_frame_to_filter(filter->outputs[out], input_index); if (ret < 0) return ret; } @@ -1219,7 +1227,7 @@ static int ff_filter_activate_default(AVFilterContext *filter) for (i = 0; i < filter->nb_outputs; i++) { if (filter->outputs[i]->frame_wanted_out && !filter->outputs[i]->frame_blocked_in) { - return ff_request_frame_to_filter(filter->outputs[i]); + return ff_request_frame_to_filter(filter->outputs[i], 0); } } return FFERROR_NOT_READY; diff --git a/libavfilter/vf_overlay_textsubs.c b/libavfilter/vf_overlay_textsubs.c new file mode 100644 index 0000000000..aa3409b1c9 --- /dev/null +++ b/libavfilter/vf_overlay_textsubs.c @@ -0,0 +1,609 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * overlay text subtitles on top of a video frame + */ + +#include +#include + +#include "drawutils.h" +#include "filters.h" + +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" + +typedef struct TextSubsContext { + const AVClass *class; + ASS_Library *library; + ASS_Renderer *renderer; + ASS_Track *track; + + char *default_font_path; + char *fonts_dir; + char *fc_file; + double font_scale; + double font_size; + char *force_style; + char *language; + int margin; + + int alpha; + FFDrawContext draw; + + int got_header; + int out_w, out_h; + AVRational frame_rate; + AVFrame *last_frame; + int eof; +} TextSubsContext; + +/* libass supports a log level ranging from 0 to 7 */ +static const int ass_libavfilter_log_level_map[] = { + AV_LOG_QUIET, /* 0 */ + AV_LOG_PANIC, /* 1 */ + AV_LOG_FATAL, /* 2 */ + AV_LOG_ERROR, /* 3 */ + AV_LOG_WARNING, /* 4 */ + AV_LOG_INFO, /* 5 */ + AV_LOG_VERBOSE, /* 6 */ + AV_LOG_DEBUG, /* 7 */ +}; + +static void ass_log(int ass_level, const char *fmt, va_list args, void *ctx) +{ + const int ass_level_clip = av_clip(ass_level, 0, FF_ARRAY_ELEMS(ass_libavfilter_log_level_map) - 1); + const int level = ass_libavfilter_log_level_map[ass_level_clip]; + + av_vlog(ctx, level, fmt, args); + av_log(ctx, level, "\n"); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + TextSubsContext *s = ctx->priv; + + if (s->track) + ass_free_track(s->track); + if (s->renderer) + ass_renderer_done(s->renderer); + if (s->library) + ass_library_done(s->library); + + s->track = NULL; + s->renderer = NULL; + s->library = NULL; + + av_frame_free(&s->last_frame); +} + +static int overlay_textsubs_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink0 = ctx->inputs[0]; + AVFilterLink *inlink1 = ctx->inputs[1]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input0 video formats */ + formats = ff_draw_supported_pixel_formats(0); + if ((ret = ff_formats_ref(formats, &inlink0->outcfg.formats)) < 0) + return ret; + + /* set input1 subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink1->outcfg.formats)) < 0) + return ret; + + /* set output0 video formats */ + formats = ff_draw_supported_pixel_formats(0); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + + outlink->w = ctx->inputs[0]->w; + outlink->h = ctx->inputs[0]->h; + outlink->time_base = ctx->inputs[0]->time_base; + + return 0; +} + +static int config_input_main(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = inlink->dst->priv; + int ret; + + ret = ff_draw_init(&s->draw, inlink->format, s->alpha ? FF_DRAW_PROCESS_ALPHA : 0); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize ff_draw.\n"); + return ret; + } + + ass_set_frame_size (s->renderer, inlink->w, inlink->h); + ass_set_pixel_aspect(s->renderer, av_q2d(inlink->sample_aspect_ratio)); + + av_log(ctx, AV_LOG_VERBOSE, "Subtitle screen: %dx%d\n\n\n\n", inlink->w, inlink->h); + + return 0; +} + +/* libass stores an RGBA color in the format RRGGBBTT, where TT is the transparency level */ +#define AR(c) ( (c)>>24) +#define AG(c) (((c)>>16)&0xFF) +#define AB(c) (((c)>>8) &0xFF) +#define AA(c) ((0xFF-(c)) &0xFF) + +static void overlay_ass_image(TextSubsContext *s, AVFrame *picref, + const ASS_Image *image) +{ + for (; image; image = image->next) { + uint8_t rgba_color[] = {AR(image->color), AG(image->color), AB(image->color), AA(image->color)}; + FFDrawColor color; + ff_draw_color(&s->draw, &color, rgba_color); + ff_blend_mask(&s->draw, &color, + picref->data, picref->linesize, + picref->width, picref->height, + image->bitmap, image->stride, image->w, image->h, + 3, 0, image->dst_x, image->dst_y); + } +} + +static void process_header(AVFilterContext *link, AVFrame *sub) +{ + TextSubsContext *s = link->priv; + ASS_Track *track = s->track; + ASS_Style *style; + int sid = 0; + + if (!track) + return; + + if (sub && sub->subtitle_header) { + char *subtitle_header = (char *)sub->subtitle_header->data; + ass_process_codec_private(s->track, subtitle_header, strlen(subtitle_header)); + } + else { + char* subtitle_header = av_ass_get_subtitle_header_default(0); + if (!subtitle_header) + return; + + ass_process_codec_private(s->track, subtitle_header, strlen(subtitle_header)); + av_free(subtitle_header); + } + + if (s->language) + s->track->Language = strdup(s->language); + + if (!s->track->event_format) { + s->track->event_format = strdup("ReadOrder, Layer, Style, Name, MarginL, MarginR, MarginV, Effect, Text"); + } + + if (s->track->n_styles == 0) { + sid = ass_alloc_style(track); + style = &s->track->styles[sid]; + style->Name = strdup("Default"); + style->PrimaryColour = 0xffffff00; + style->SecondaryColour = 0x00ffff00; + style->OutlineColour = 0x00000000; + style->BackColour = 0x00000080; + style->Bold = 200; + style->ScaleX = 1.0; + style->ScaleY = 1.0; + style->Spacing = 0; + style->BorderStyle = 1; + style->Outline = 2; + style->Shadow = 3; + style->Alignment = 2; + } + else + style = &s->track->styles[sid]; + + style->FontSize = s->font_size; + style->MarginL = style->MarginR = style->MarginV = s->margin; + + track->default_style = sid; + + s->got_header = 1; +} + +static int filter_video_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + int detect_change = 0; + + int64_t time_ms = frame->pts * av_q2d(inlink->time_base) * 1000; + + ASS_Image *image = ass_render_frame(s->renderer, s->track, time_ms, &detect_change); + + if (detect_change) + av_log(ctx, AV_LOG_DEBUG, "Change happened at time ms:%lld\n", time_ms); + + overlay_ass_image(s, frame, image); + + return ff_filter_frame(ctx->outputs[0], frame); +} + +static int filter_subtitle_frame(AVFilterLink *inlink, AVFrame *sub) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + + if (!s->got_header) + process_header(ctx, sub); + + const int64_t start_time = av_rescale_q(sub->subtitle_pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t duration = sub->subtitle_end_time; + + for (unsigned i = 0; i < sub->num_subtitle_areas; i++) { + char *ass_line = sub->subtitle_areas[i]->ass; + if (!ass_line) + break; + ass_process_chunk(s->track, ass_line, strlen(ass_line), start_time, duration); + } + + av_frame_free(&sub); + return 0; +} + +static av_cold int init(AVFilterContext *ctx) +{ + TextSubsContext *s = ctx->priv; + + s->library = ass_library_init(); + + if (!s->library) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize libass.\n"); + return AVERROR(EINVAL); + } + + ass_set_message_cb(s->library, ass_log, ctx); + + /* Initialize fonts */ + if (s->fonts_dir) + ass_set_fonts_dir(s->library, s->fonts_dir); + + ass_set_extract_fonts(s->library, 1); + + s->renderer = ass_renderer_init(s->library); + if (!s->renderer) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize libass renderer.\n"); + return AVERROR(EINVAL); + } + + s->track = ass_new_track(s->library); + if (!s->track) { + av_log(ctx, AV_LOG_ERROR, "ass_new_track() failed!\n"); + return AVERROR(EINVAL); + } + + ass_set_fonts(s->renderer, s->default_font_path, NULL, 1, s->fc_file, 1); + + if (s->force_style) { + char **list = NULL; + char *temp = NULL; + char *ptr = av_strtok(s->force_style, ",", &temp); + int i = 0; + while (ptr) { + av_dynarray_add(&list, &i, ptr); + if (!list) { + return AVERROR(ENOMEM); + } + ptr = av_strtok(NULL, ",", &temp); + } + av_dynarray_add(&list, &i, NULL); + if (!list) { + return AVERROR(ENOMEM); + } + ass_set_style_overrides(s->library, list); + av_free(list); + } + + return 0; +} + +static int textsub2video_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input0 subtitle format */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output0 video format */ + formats = ff_draw_supported_pixel_formats(AV_PIX_FMT_FLAG_ALPHA); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int textsub2video_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + + if (s->out_w <= 0 || s->out_h <= 0) { + s->out_w = inlink->w; + s->out_h = inlink->h; + } + + return 0; +} + +static int textsub2video_config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + TextSubsContext *s = ctx->priv; + int ret; + + ret = ff_draw_init(&s->draw, outlink->format, FF_DRAW_PROCESS_ALPHA); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize ff_draw.\n"); + return ret; + } + + if (s->out_w <= 0 || s->out_h <= 0) { + av_log(ctx, AV_LOG_ERROR, "No output image size set.\n"); + return AVERROR(EINVAL); + } + + ass_set_frame_size (s->renderer, s->out_w, s->out_h); + + outlink->w = s->out_w; + outlink->h = s->out_h; + outlink->sample_aspect_ratio = (AVRational){1,1}; + outlink->frame_rate = s->frame_rate; + + return 0; +} + +static int textsub2video_filter_frame(AVFilterLink *inlink, AVFrame *sub) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + + av_log(ctx, AV_LOG_VERBOSE, "textsub2video_filter_frame num_subtitle_rects: %d\n", sub->num_subtitle_areas); + + if (!s->got_header) + process_header(ctx, sub); + + const int64_t start_time = av_rescale_q(sub->subtitle_pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t duration = sub->subtitle_end_time; + + for (unsigned i = 0; i < sub->num_subtitle_areas; i++) { + char *ass_line = sub->subtitle_areas[i]->ass; + if (!ass_line) + break; + ass_process_chunk(s->track, ass_line, strlen(ass_line), start_time, duration); + } + + av_frame_free(&sub); + return 0; +} + +static int textsub2video_request_frame(AVFilterLink *outlink) +{ + TextSubsContext *s = outlink->src->priv; + const AVFilterLink *inlink = outlink->src->inputs[0]; + int64_t last_pts = outlink->current_pts; + int64_t next_pts; + int i, detect_change = 0; + AVFrame *out; + ASS_Image *image; + + if (last_pts == AV_NOPTS_VALUE) + last_pts = inlink->current_pts * av_q2d(inlink->time_base) / av_q2d(outlink->time_base); + + next_pts = last_pts + (1.0 / av_q2d(outlink->frame_rate) / av_q2d(outlink->time_base)); + + int64_t time_ms = next_pts * av_q2d(outlink->time_base) * 1000; + + image = ass_render_frame(s->renderer, s->track, time_ms, &detect_change); + + if (detect_change) + av_log(outlink->src, AV_LOG_DEBUG, "Change happened at time ms:%lld\n", time_ms); + else if (s->last_frame) { + out = av_frame_clone(s->last_frame); + if (!out) + return AVERROR(ENOMEM); + + out->pts = next_pts; + return ff_filter_frame(outlink, out); + } + + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) + return AVERROR(ENOMEM); + + for (i = 2; i < AV_NUM_DATA_POINTERS; i++) { + if (out->buf[i] && i != 1) + memset(out->buf[i]->data, 0, out->buf[i]->size); + } + + out->pts = next_pts; + + if (image) + overlay_ass_image(s, out, image); + + av_frame_free(&s->last_frame); + + s->last_frame = av_frame_clone(out); + + return ff_filter_frame(outlink, out); +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + TextSubsContext *s = ctx->priv; + AVFrame *in; + int ret; + + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + ret = ff_inlink_consume_frame(inlink, &in); + if (ret < 0) + return ret; + if (ret > 0) + return textsub2video_filter_frame(inlink, in); + + if (ff_outlink_frame_wanted(outlink)) { + if (!s->eof && ff_outlink_get_status(ctx->inputs[0])) { + s->eof = 1; + } + if (!s->eof && ff_inlink_queued_frames(ctx->inputs[0]) == 0) + ff_inlink_request_frame(ctx->inputs[0]); + if (s->eof && ff_inlink_queued_frames(ctx->inputs[0]) <= 0) { + ff_outlink_set_status(outlink, AVERROR_EOF, AV_NOPTS_VALUE); + } + else { + return textsub2video_request_frame(outlink); + } + + return 0; + } + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(TextSubsContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption overlay_textsubs_options[] = { + {"alpha", "enable processing of alpha channel", OFFSET(alpha), AV_OPT_TYPE_BOOL, {.i64 = 0 }, 0, 1, FLAGS}, + {"font_scale", "font scale factor", OFFSET(font_scale), AV_OPT_TYPE_DOUBLE, {.dbl = 1.0 }, 0.0, 100.0, FLAGS}, + {"font_size", "default font size", OFFSET(font_size), AV_OPT_TYPE_DOUBLE, {.dbl = 18.0}, 0.0, 100.0, FLAGS}, + {"force_style", "force subtitle style", OFFSET(force_style), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"margin", "default margin", OFFSET(margin), AV_OPT_TYPE_INT, {.i64 = 20 }, 0, INT_MAX, FLAGS}, + {"default_font_path", "path to default font", OFFSET(default_font_path), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fonts_dir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontsdir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontconfig_file", "fontconfig file to load", OFFSET(fc_file), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"language", "default language", OFFSET(language), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + { NULL } +}; + +static const AVOption textsub2video_options[] = { + {"rate", "set frame rate", OFFSET(frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="8"}, 0, INT_MAX, FLAGS}, + {"r", "set frame rate", OFFSET(frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="8"}, 0, INT_MAX, FLAGS}, + {"size", "set video size", OFFSET(out_w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, FLAGS}, + {"s", "set video size", OFFSET(out_w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, FLAGS}, + {"font_scale", "font scale factor", OFFSET(font_scale), AV_OPT_TYPE_DOUBLE, {.dbl = 1.0 }, 0.0, 100.0, FLAGS}, + {"font_size", "default font size", OFFSET(font_size), AV_OPT_TYPE_DOUBLE, {.dbl = 18.0}, 0.0, 100.0, FLAGS}, + {"force_style", "force subtitle style", OFFSET(force_style), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"margin", "default margin", OFFSET(margin), AV_OPT_TYPE_INT, {.i64 = 20 }, 0, INT_MAX, FLAGS}, + {"default_font_path", "path to default font", OFFSET(default_font_path), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fonts_dir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontsdir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontconfig_file", "fontconfig file to load", OFFSET(fc_file), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"language", "default language", OFFSET(language), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + { NULL } +}; + +#if CONFIG_OVERLAY_TEXTSUBS_FILTER + +AVFILTER_DEFINE_CLASS(overlay_textsubs); + +static const AVFilterPad overlay_textsubs_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input_main, + .flags = AVFILTERPAD_FLAG_NEEDS_WRITABLE, + .filter_frame = filter_video_frame, + }, + { + .name = "overlay", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_subtitle_frame, + }, +}; + +static const AVFilterPad overlay_textsubs_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +const AVFilter ff_vf_overlay_textsubs = { + .name = "overlay_textsubs", + .description = NULL_IF_CONFIG_SMALL("Overlay textual subtitles on top of the input."), + .init = init, + .uninit = uninit, + .priv_size = sizeof(TextSubsContext), + .priv_class = &overlay_textsubs_class, + .query_formats = overlay_textsubs_query_formats, + FILTER_INPUTS(overlay_textsubs_inputs), + FILTER_OUTPUTS(overlay_textsubs_outputs), +}; +#endif + +#if CONFIG_TEXTSUB2VIDEO_FILTER + +AVFILTER_DEFINE_CLASS(textsub2video); + +static const AVFilterPad textsub2video_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .config_props = textsub2video_config_input, + }, +}; + +static const AVFilterPad textsub2video_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = textsub2video_config_output, + }, +}; + +const AVFilter ff_svf_textsub2video = { + .name = "textsub2video", + .description = NULL_IF_CONFIG_SMALL("Convert textual subtitles to video frames"), + .init = init, + .uninit = uninit, + .query_formats = textsub2video_query_formats, + .priv_size = sizeof(TextSubsContext), + .priv_class = &textsub2video_class, + .activate = activate, + FILTER_INPUTS(textsub2video_inputs), + FILTER_OUTPUTS(textsub2video_outputs), +}; +#endif