From patchwork Sat Sep 11 08:35:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30144 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp2246023iov; Sat, 11 Sep 2021 01:35:33 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwdvX/JtQWC55c63FxSHrmrnD4sKw9yvm0u6JH5PXt7U2Zz/fZ61MiEfYr/DNP8aAtyM+CZ X-Received: by 2002:a17:907:3e05:: with SMTP id hp5mr1822354ejc.527.1631349332797; Sat, 11 Sep 2021 01:35:32 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id dm18si1403817ejc.257.2021.09.11.01.35.32; Sat, 11 Sep 2021 01:35:32 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=G78SiAUO; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3167E68A7DE; Sat, 11 Sep 2021 11:35:30 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10olkn2082.outbound.protection.outlook.com [40.92.42.82]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0C52068A776 for ; Sat, 11 Sep 2021 11:35:24 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JFJdXqycIgvQsb3WxPIzCKixaM29ud0Bi0d1HKnsOGSbASFgbf2KBRLAQX060RaUyo55mKUSa3aWO0ns3ucookhNB/X3Rsqq3vrWPiZx2xedMeqzMMc04CpLGuzFNxc46p5Mgt1df2CHdWjRLma77uNMCLc7o+sSACmvANFwsWltn0P7uxVw9vYT0mUp4AGkPgkoKP6gveXLtxScs9RWTRNmKavK2ekv1Lg/j38KKzM4Za6dhHW13GSYNpVsjKGWigjtzSHv1mdz7c+kBFc5pI3Gjn2smvYQTLeR+fSmihV/8NwUvjHqY0GHAYRzBueibc4NrAgqHZvwdk/2NeTwpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=PG8P9ijpVDjmY6FY00nDUo6nUim4NP/Zpg2rgO8gMeA=; b=FqAQBfT/EH4Un0lKWDSiItCLlApHC4ic+FRMyb6W1k4YsyUVlXUf/uP7t4WsKyCMXk5ZuU4LkntQ0bydTOj+XdhYQaGmiNrhUE74xKDZxKM3jiBpomvmPRIelesJ5p5sy60MWOR0589ESMTwiFZKUZ7TTo2LGl7RYK3jcnu85AjQz4j7gNcqYUQtr4ZEFLliQdrDFr8FxvycndDATsNc2jcY8jwyoYWvIggXIJycS+6eFPwckxHK3XNcRFQD8FixxN8Ja34+culToXhWd4nmJQsx96hFTnfEeglsvi5ETFs4hNOY3BLcQkKXWFJ9AdCQadz575ugodJtnrHEnwYxWA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PG8P9ijpVDjmY6FY00nDUo6nUim4NP/Zpg2rgO8gMeA=; b=G78SiAUOdSpN7eDVYV18abB+qjxmwMJNsqa7eV+/yV0lJcbYULSc6ipBIRA+N0ydKIGyNlFEmGmHJ33j7rDJqhHjOVwt50m60dZB/PPFEjaiEPYDx8K0krNMryAeGkff9KLdTGvehexX4ath7jo9OCsng09CzG2dcqMyS4ycbhwi2djLO7y/y+J3qSOn1Kx955AL8uPecCO2b0P7ph6Cuu9Onu8w3HxTKy6HWqJ2UHNlZ6VDQFeWBMDnPwjMMA80ZTEfUPPxthFbalB4iRrgBPTFkFgyJgMma/3nmQMFvCS0ANODFfB7KowC2u4aHuBrCYDxD+qtH4ICKX6FOt3SPA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5776.namprd04.prod.outlook.com (2603:10b6:208:a2::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.16; Sat, 11 Sep 2021 08:35:20 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4500.018; Sat, 11 Sep 2021 08:35:20 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v4 14/18] avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters Thread-Index: Adeh93YIltkvy4r5R7+roh7Coy/rvA== Date: Sat, 11 Sep 2021 08:35:20 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [JkWc4OcwX/6LVNwoyjLkJhRmnKYIuETkUfmeLsCTKs0=] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b2807722-19bf-4233-c0b0-08d974ff1779 x-ms-traffictypediagnostic: MN2PR04MB5776: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: fEmAYsdqoE6sLYUJs8tGQONjAab0oZHO0OHRctFN9C9q4EoxuPtJat0mwTMj3cY07tgCc2ixKVVQvouFmCmhprSE5Nbzj43EAHCz7hs3XCV+RlLrGDJwGFuzbowEk+1Yi2nXR+IRBZ1rIh3k2cCo1eMOzI37yaM+4XeXjbb3BmANrTTjD3LWHurwK4L0cc2qsQvMC6mAfa3sXm2Mnz9nNcaMIjf3kZfPp1p1DGC5D6IiGuAi/6ZDc0mw0nwrzWZe5OHIPiw++oejmMPPFBKM7WOPilIC7VKrqGPWb9b9QNctvUbYxUOZ4ZfxuPipMubD6U3ITI0Z13URG3BvOovAaNCTCPd82vAJii45WJyUXe16NaMTUDcWsaIpMt3foNdB74bPHbkwYQQd6GQNlPCz59WNUgMMhVEwFUiRtBtd+M1pzpoaBiqkF5OmeouDCmRW x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: Ws2VCVMzo6SWL8tR/BSkeUUaNrqNFIrg8iOxzra7bK0vHSjjj10ZRkXqTylVPgtm+3ULFIX0sQTW4fJj7WpFLZ/TypMj/ecpZeHGEhKy9y6xKGGdTJthCEUuck1ToLsW9gTXZI1GvbM9+shJGsgZIA== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: b2807722-19bf-4233-c0b0-08d974ff1779 X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2021 08:35:20.8064 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5776 Subject: [FFmpeg-devel] [PATCH v4 14/18] avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: TPbIHa97wSgO Signed-off-by: softworkz --- configure | 2 + libavfilter/Makefile | 2 + libavfilter/allfilters.c | 2 + libavfilter/avfilter.c | 18 +- libavfilter/vf_overlay_textsubs.c | 633 ++++++++++++++++++++++++++++++ 5 files changed, 652 insertions(+), 5 deletions(-) create mode 100644 libavfilter/vf_overlay_textsubs.c diff --git a/configure b/configure index 2f2777df9f..7840567d31 100755 --- a/configure +++ b/configure @@ -3624,6 +3624,7 @@ openclsrc_filter_deps="opencl" overlay_opencl_filter_deps="opencl" overlay_qsv_filter_deps="libmfx" overlay_qsv_filter_select="qsvvpp" +overlay_textsubs_filter_deps="avcodec libass" overlay_vulkan_filter_deps="vulkan_lib libglslang" owdenoise_filter_deps="gpl" pad_opencl_filter_deps="opencl" @@ -3669,6 +3670,7 @@ superequalizer_filter_deps="avcodec" superequalizer_filter_select="rdft" surround_filter_deps="avcodec" surround_filter_select="rdft" +textsub2video_filter_deps="avcodec libass" tinterlace_filter_deps="gpl" tinterlace_merge_test_deps="tinterlace_filter" tinterlace_pad_test_deps="tinterlace_filter" diff --git a/libavfilter/Makefile b/libavfilter/Makefile index f9b33d4ead..0e752c5bf9 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -364,6 +364,7 @@ OBJS-$(CONFIG_OVERLAY_OPENCL_FILTER) += vf_overlay_opencl.o opencl.o \ opencl/overlay.o framesync.o OBJS-$(CONFIG_OVERLAY_QSV_FILTER) += vf_overlay_qsv.o framesync.o OBJS-$(CONFIG_OVERLAY_GRAPHICSUBS_FILTER) += vf_overlay_graphicsubs.o framesync.o +OBJS-$(CONFIG_OVERLAY_TEXTSUBS_FILTER) += vf_overlay_textsubs.o OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) += vf_overlay_vulkan.o vulkan.o OBJS-$(CONFIG_OWDENOISE_FILTER) += vf_owdenoise.o OBJS-$(CONFIG_PAD_FILTER) += vf_pad.o @@ -451,6 +452,7 @@ OBJS-$(CONFIG_SWAPRECT_FILTER) += vf_swaprect.o OBJS-$(CONFIG_SWAPUV_FILTER) += vf_swapuv.o OBJS-$(CONFIG_TBLEND_FILTER) += vf_blend.o framesync.o OBJS-$(CONFIG_TELECINE_FILTER) += vf_telecine.o +OBJS-$(CONFIG_TEXTSUB2VIDEO_FILTER) += vf_overlay_textsubs.o OBJS-$(CONFIG_THISTOGRAM_FILTER) += vf_histogram.o OBJS-$(CONFIG_THRESHOLD_FILTER) += vf_threshold.o framesync.o OBJS-$(CONFIG_THUMBNAIL_FILTER) += vf_thumbnail.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index ca983db5b4..77463aa4c8 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -345,6 +345,7 @@ extern const AVFilter ff_vf_overlay; extern const AVFilter ff_vf_overlay_opencl; extern const AVFilter ff_vf_overlay_qsv; extern const AVFilter ff_vf_overlay_graphicsubs; +extern const AVFilter ff_vf_overlay_textsubs; extern const AVFilter ff_vf_overlay_vulkan; extern const AVFilter ff_vf_overlay_cuda; extern const AVFilter ff_vf_owdenoise; @@ -524,6 +525,7 @@ extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_svf_graphicsub2video; +extern const AVFilter ff_svf_textsub2video; /* multimedia sources */ extern const AVFilter ff_avsrc_amovie; diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index 961e55e7b4..e28171e919 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -463,7 +463,7 @@ static int64_t guess_status_pts(AVFilterContext *ctx, int status, AVRational lin return AV_NOPTS_VALUE; } -static int ff_request_frame_to_filter(AVFilterLink *link) +static int ff_request_frame_to_filter(AVFilterLink *link, int input_index) { int ret = -1; @@ -472,8 +472,8 @@ static int ff_request_frame_to_filter(AVFilterLink *link) link->frame_blocked_in = 1; if (link->srcpad->request_frame) ret = link->srcpad->request_frame(link); - else if (link->src->inputs[0]) - ret = ff_request_frame(link->src->inputs[0]); + else if (link->src->inputs[input_index]) + ret = ff_request_frame(link->src->inputs[input_index]); if (ret < 0) { if (ret != AVERROR(EAGAIN) && ret != link->status_in) ff_avfilter_link_set_in_status(link, ret, guess_status_pts(link->src, ret, link->time_base)); @@ -1171,6 +1171,14 @@ static int forward_status_change(AVFilterContext *filter, AVFilterLink *in) { unsigned out = 0, progress = 0; int ret; + int input_index = 0; + + for (int i = 0; i < in->dst->nb_inputs; i++) { + if (&in->dst->input_pads[i] == in->dstpad) { + input_index = i; + break; + } + } av_assert0(!in->status_out); if (!filter->nb_outputs) { @@ -1180,7 +1188,7 @@ static int forward_status_change(AVFilterContext *filter, AVFilterLink *in) while (!in->status_out) { if (!filter->outputs[out]->status_in) { progress++; - ret = ff_request_frame_to_filter(filter->outputs[out]); + ret = ff_request_frame_to_filter(filter->outputs[out], input_index); if (ret < 0) return ret; } @@ -1217,7 +1225,7 @@ static int ff_filter_activate_default(AVFilterContext *filter) for (i = 0; i < filter->nb_outputs; i++) { if (filter->outputs[i]->frame_wanted_out && !filter->outputs[i]->frame_blocked_in) { - return ff_request_frame_to_filter(filter->outputs[i]); + return ff_request_frame_to_filter(filter->outputs[i], 0); } } return FFERROR_NOT_READY; diff --git a/libavfilter/vf_overlay_textsubs.c b/libavfilter/vf_overlay_textsubs.c new file mode 100644 index 0000000000..6824cefaf3 --- /dev/null +++ b/libavfilter/vf_overlay_textsubs.c @@ -0,0 +1,633 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * overlay text subtitles on top of a video frame + */ + +#include +#include + +#include "avfilter.h" +#include "formats.h" +#include "libavutil/common.h" +#include "libavutil/avstring.h" +#include "libavutil/opt.h" +#include "internal.h" +#include "drawutils.h" +#include "filters.h" + +#include "libavcodec/avcodec.h" +#include "libavutil/pixdesc.h" + +typedef struct TextSubsContext { + const AVClass *class; + ASS_Library *library; + ASS_Renderer *renderer; + ASS_Track *track; + + AVSubtitle *last_sub; + + char *default_font_path; + char *fonts_dir; + char *fc_file; + double font_scale; + double font_size; + char *force_style; + char *language; + int margin; + + int alpha; + FFDrawContext draw; + + int got_header; + int out_w, out_h; + AVRational frame_rate; + AVFrame *last_frame; + int eof; +} TextSubsContext; + +/* libass supports a log level ranging from 0 to 7 */ +static const int ass_libavfilter_log_level_map[] = { + AV_LOG_QUIET, /* 0 */ + AV_LOG_PANIC, /* 1 */ + AV_LOG_FATAL, /* 2 */ + AV_LOG_ERROR, /* 3 */ + AV_LOG_WARNING, /* 4 */ + AV_LOG_INFO, /* 5 */ + AV_LOG_VERBOSE, /* 6 */ + AV_LOG_DEBUG, /* 7 */ +}; + +static void ass_log(int ass_level, const char *fmt, va_list args, void *ctx) +{ + const int ass_level_clip = av_clip(ass_level, 0, FF_ARRAY_ELEMS(ass_libavfilter_log_level_map) - 1); + const int level = ass_libavfilter_log_level_map[ass_level_clip]; + + av_vlog(ctx, level, fmt, args); + av_log(ctx, level, "\n"); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + TextSubsContext *s = ctx->priv; + + if (s->track) + ass_free_track(s->track); + if (s->renderer) + ass_renderer_done(s->renderer); + if (s->library) + ass_library_done(s->library); + + s->track = NULL; + s->renderer = NULL; + s->library = NULL; + + if (s->last_frame) + av_frame_unref(s->last_frame); +} + +static int overlay_textsubs_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink0 = ctx->inputs[0]; + AVFilterLink *inlink1 = ctx->inputs[1]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input0 video formats */ + formats = ff_draw_supported_pixel_formats(0); + if ((ret = ff_formats_ref(formats, &inlink0->outcfg.formats)) < 0) + return ret; + + /* set input1 subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink1->outcfg.formats)) < 0) + return ret; + + /* set output0 video formats */ + formats = ff_draw_supported_pixel_formats(0); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + + outlink->w = ctx->inputs[0]->w; + outlink->h = ctx->inputs[0]->h; + outlink->time_base = ctx->inputs[0]->time_base; + + return 0; +} + +static int config_input_main(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = inlink->dst->priv; + int ret; + + ret = ff_draw_init(&s->draw, inlink->format, s->alpha ? FF_DRAW_PROCESS_ALPHA : 0); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize ff_draw.\n"); + return ret; + } + + ass_set_frame_size (s->renderer, inlink->w, inlink->h); + ass_set_pixel_aspect(s->renderer, av_q2d(inlink->sample_aspect_ratio)); + + return 0; +} + +/* libass stores an RGBA color in the format RRGGBBTT, where TT is the transparency level */ +#define AR(c) ( (c)>>24) +#define AG(c) (((c)>>16)&0xFF) +#define AB(c) (((c)>>8) &0xFF) +#define AA(c) ((0xFF-(c)) &0xFF) + +static void overlay_ass_image(TextSubsContext *s, AVFrame *picref, + const ASS_Image *image) +{ + for (; image; image = image->next) { + uint8_t rgba_color[] = {AR(image->color), AG(image->color), AB(image->color), AA(image->color)}; + FFDrawColor color; + ff_draw_color(&s->draw, &color, rgba_color); + ff_blend_mask(&s->draw, &color, + picref->data, picref->linesize, + picref->width, picref->height, + image->bitmap, image->stride, image->w, image->h, + 3, 0, image->dst_x, image->dst_y); + } +} + +static void process_header(AVFilterContext *link, AVSubtitle *sub) +{ + TextSubsContext *s = link->priv; + ASS_Track *track = s->track; + ASS_Style *style; + int sid = 0; + + if (!track) + return; + + if (sub && sub->header) + ass_process_codec_private(s->track, sub->header, strlen(sub->header)); + else { + AVCodecContext temp_context; + ff_ass_subtitle_header_default(&temp_context); + if (!temp_context.subtitle_header) + return; + + ass_process_codec_private(s->track, (char*)temp_context.subtitle_header, temp_context.subtitle_header_size); + } + + if (s->language) + s->track->Language = strdup(s->language); + + if (!s->track->event_format) { + s->track->event_format = strdup("ReadOrder, Layer, Style, Name, MarginL, MarginR, MarginV, Effect, Text"); + } + + if (s->track->n_styles == 0) { + sid = ass_alloc_style(track); + style = &s->track->styles[sid]; + style->Name = strdup("Default"); + style->PrimaryColour = 0xffffff00; + style->SecondaryColour = 0x00ffff00; + style->OutlineColour = 0x00000000; + style->BackColour = 0x00000080; + style->Bold = 200; + style->ScaleX = 1.0; + style->ScaleY = 1.0; + style->Spacing = 0; + style->BorderStyle = 1; + style->Outline = 2; + style->Shadow = 3; + style->Alignment = 2; + } + else + style = &s->track->styles[sid]; + + style->FontSize = s->font_size; + style->MarginL = style->MarginR = style->MarginV = s->margin; + + track->default_style = sid; + + s->got_header = 1; +} + +static int filter_video_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + int detect_change = 0; + + int64_t time_ms = frame->pts * av_q2d(inlink->time_base) * 1000; + + ASS_Image *image = ass_render_frame(s->renderer, s->track, time_ms, &detect_change); + + if (detect_change) + av_log(ctx, AV_LOG_DEBUG, "Change happened at time ms:%lld\n", time_ms); + + overlay_ass_image(s, frame, image); + + return ff_filter_frame(ctx->outputs[0], frame); +} + +static int filter_subtitle_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + + if (frame->buf[0] && frame->buf[0]->data) { + AVSubtitle *sub = (AVSubtitle *)frame->buf[0]->data; + + if (sub != s->last_sub) { + unsigned i; + s->last_sub = sub; + + if (!s->got_header) + process_header(ctx, sub); + + const int64_t start_time = av_rescale_q(sub->pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t duration = sub->end_display_time; + + for (i = 0; i < sub->num_rects; i++) { + char *ass_line = sub->rects[i]->ass; + if (!ass_line) + break; + ass_process_chunk(s->track, ass_line, strlen(ass_line), start_time, duration); + } + } + } + + av_frame_free(&frame); + return 0; +} + +static av_cold int init(AVFilterContext *ctx) +{ + TextSubsContext *s = ctx->priv; + + s->library = ass_library_init(); + + if (!s->library) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize libass.\n"); + return AVERROR(EINVAL); + } + + ass_set_message_cb(s->library, ass_log, ctx); + + /* Initialize fonts */ + if (s->fonts_dir) + ass_set_fonts_dir(s->library, s->fonts_dir); + + ass_set_extract_fonts(s->library, 1); + + s->renderer = ass_renderer_init(s->library); + if (!s->renderer) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize libass renderer.\n"); + return AVERROR(EINVAL); + } + + s->track = ass_new_track(s->library); + if (!s->track) { + av_log(ctx, AV_LOG_ERROR, "ass_new_track() failed!\n"); + return AVERROR(EINVAL); + } + + ass_set_fonts(s->renderer, s->default_font_path, NULL, 1, s->fc_file, 1); + + if (s->force_style) { + char **list = NULL; + char *temp = NULL; + char *ptr = av_strtok(s->force_style, ",", &temp); + int i = 0; + while (ptr) { + av_dynarray_add(&list, &i, ptr); + if (!list) { + return AVERROR(ENOMEM); + } + ptr = av_strtok(NULL, ",", &temp); + } + av_dynarray_add(&list, &i, NULL); + if (!list) { + return AVERROR(ENOMEM); + } + ass_set_style_overrides(s->library, list); + av_free(list); + } + + return 0; +} + +static int textsub2video_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input0 subtitle format */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output0 video format */ + formats = ff_draw_supported_pixel_formats(AV_PIX_FMT_FLAG_ALPHA); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int textsub2video_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + + if (s->out_w <= 0 || s->out_h <= 0) { + s->out_w = inlink->w; + s->out_h = inlink->h; + } + + return 0; +} + +static int textsub2video_config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + TextSubsContext *s = ctx->priv; + int ret; + + ret = ff_draw_init(&s->draw, outlink->format, FF_DRAW_PROCESS_ALPHA); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize ff_draw.\n"); + return ret; + } + + if (s->out_w <= 0 || s->out_h <= 0) { + av_log(ctx, AV_LOG_ERROR, "No output image size set.\n"); + return AVERROR(EINVAL); + } + + ass_set_frame_size (s->renderer, s->out_w, s->out_h); + + outlink->w = s->out_w; + outlink->h = s->out_h; + outlink->sample_aspect_ratio = (AVRational){1,1}; + outlink->frame_rate = s->frame_rate; + + return 0; +} + +static int textsub2video_filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + + if (frame->buf[0] && frame->buf[0]->data) { + AVSubtitle *sub = (AVSubtitle *)frame->buf[0]->data; + + av_log(ctx, AV_LOG_VERBOSE, "textsub2video_filter_frame num_rects: %d\n", sub->num_rects); + + if (sub != s->last_sub) { + unsigned i; + s->last_sub = sub; + + if (!s->got_header) + process_header(ctx, sub); + + const int64_t start_time = av_rescale_q(sub->pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t duration = sub->end_display_time; + + for (i = 0; i < sub->num_rects; i++) { + char *ass_line = sub->rects[i]->ass; + if (!ass_line) + break; + ass_process_chunk(s->track, ass_line, strlen(ass_line), start_time, duration); + } + } + } + + av_frame_free(&frame); + return 0; +} + +static int textsub2video_request_frame(AVFilterLink *outlink) +{ + TextSubsContext *s = outlink->src->priv; + const AVFilterLink *inlink = outlink->src->inputs[0]; + int64_t last_pts = outlink->current_pts; + int64_t next_pts; + int i, detect_change = 0; + AVFrame *out; + ASS_Image *image; + + if (last_pts == AV_NOPTS_VALUE) + last_pts = inlink->current_pts * av_q2d(inlink->time_base) / av_q2d(outlink->time_base); + + next_pts = last_pts + (1.0 / av_q2d(outlink->frame_rate) / av_q2d(outlink->time_base)); + + int64_t time_ms = next_pts * av_q2d(outlink->time_base) * 1000; + + image = ass_render_frame(s->renderer, s->track, time_ms, &detect_change); + + if (detect_change) + av_log(outlink->src, AV_LOG_DEBUG, "Change happened at time ms:%lld\n", time_ms); + else if (s->last_frame) { + out = av_frame_clone(s->last_frame); + if (!out) + return AVERROR(ENOMEM); + + out->pts = next_pts; + return ff_filter_frame(outlink, out); + } + + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) + return AVERROR(ENOMEM); + + for (i = 2; i < AV_NUM_DATA_POINTERS; i++) { + if (out->buf[i] && i != 1) + memset(out->buf[i]->data, 0, out->buf[i]->size); + } + + out->pts = next_pts; + + if (image) + overlay_ass_image(s, out, image); + + if (s->last_frame) + av_frame_unref(s->last_frame); + + s->last_frame = av_frame_clone(out); + + return ff_filter_frame(outlink, out); +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + TextSubsContext *s = ctx->priv; + AVFrame *in; + int ret; + + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + ret = ff_inlink_consume_frame(inlink, &in); + if (ret < 0) + return ret; + if (ret > 0) + return textsub2video_filter_frame(inlink, in); + + if (ff_outlink_frame_wanted(outlink)) { + if (!s->eof && ff_outlink_get_status(ctx->inputs[0])) { + s->eof = 1; + } + if (!s->eof && ff_inlink_queued_frames(ctx->inputs[0]) == 0) + ff_inlink_request_frame(ctx->inputs[0]); + if (s->eof && ff_inlink_queued_frames(ctx->inputs[0]) <= 0) { + ff_outlink_set_status(outlink, AVERROR_EOF, AV_NOPTS_VALUE); + } + else { + return textsub2video_request_frame(outlink); + } + + return 0; + } + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(TextSubsContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption overlay_textsubs_options[] = { + {"alpha", "enable processing of alpha channel", OFFSET(alpha), AV_OPT_TYPE_BOOL, {.i64 = 0 }, 0, 1, FLAGS}, + {"font_scale", "font scale factor", OFFSET(font_scale), AV_OPT_TYPE_DOUBLE, {.dbl = 1.0 }, 0.0, 100.0, FLAGS}, + {"font_size", "default font size", OFFSET(font_size), AV_OPT_TYPE_DOUBLE, {.dbl = 18.0}, 0.0, 100.0, FLAGS}, + {"force_style", "force subtitle style", OFFSET(force_style), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"margin", "default margin", OFFSET(margin), AV_OPT_TYPE_INT, {.i64 = 20 }, 0, INT_MAX, FLAGS}, + {"default_font_path", "path to default font", OFFSET(default_font_path), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fonts_dir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontsdir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontconfig_file", "fontconfig file to load", OFFSET(fc_file), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"language", "default language", OFFSET(language), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + { NULL } +}; + +static const AVOption textsub2video_options[] = { + {"rate", "set frame rate", OFFSET(frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="8"}, 0, INT_MAX, FLAGS}, + {"r", "set frame rate", OFFSET(frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="8"}, 0, INT_MAX, FLAGS}, + {"size", "set video size", OFFSET(out_w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, FLAGS}, + {"s", "set video size", OFFSET(out_w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, FLAGS}, + {"font_scale", "font scale factor", OFFSET(font_scale), AV_OPT_TYPE_DOUBLE, {.dbl = 1.0 }, 0.0, 100.0, FLAGS}, + {"font_size", "default font size", OFFSET(font_size), AV_OPT_TYPE_DOUBLE, {.dbl = 18.0}, 0.0, 100.0, FLAGS}, + {"force_style", "force subtitle style", OFFSET(force_style), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"margin", "default margin", OFFSET(margin), AV_OPT_TYPE_INT, {.i64 = 20 }, 0, INT_MAX, FLAGS}, + {"default_font_path", "path to default font", OFFSET(default_font_path), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fonts_dir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontsdir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontconfig_file", "fontconfig file to load", OFFSET(fc_file), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"language", "default language", OFFSET(language), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + { NULL } +}; + +#if CONFIG_OVERLAY_TEXTSUBS_FILTER + +AVFILTER_DEFINE_CLASS(overlay_textsubs); + +static const AVFilterPad overlay_textsubs_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input_main, + .flags = AVFILTERPAD_FLAG_NEEDS_WRITABLE, + .filter_frame = filter_video_frame, + }, + { + .name = "overlay", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_subtitle_frame, + }, +}; + +static const AVFilterPad overlay_textsubs_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +const AVFilter ff_vf_overlay_textsubs = { + .name = "overlay_textsubs", + .description = NULL_IF_CONFIG_SMALL("Overlay textual subtitles on top of the input."), + .init = init, + .uninit = uninit, + .priv_size = sizeof(TextSubsContext), + .priv_class = &overlay_textsubs_class, + .query_formats = overlay_textsubs_query_formats, + FILTER_INPUTS(overlay_textsubs_inputs), + FILTER_OUTPUTS(overlay_textsubs_outputs), +}; +#endif + +#if CONFIG_TEXTSUB2VIDEO_FILTER + +AVFILTER_DEFINE_CLASS(textsub2video); + +static const AVFilterPad textsub2video_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .config_props = textsub2video_config_input, + }, +}; + +static const AVFilterPad textsub2video_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = textsub2video_config_output, + }, +}; + +AVFilter ff_svf_textsub2video = { + .name = "textsub2video", + .description = NULL_IF_CONFIG_SMALL("Convert textual subtitles to video frames"), + .init = init, + .uninit = uninit, + .query_formats = textsub2video_query_formats, + .priv_size = sizeof(TextSubsContext), + .priv_class = &textsub2video_class, + .activate = activate, + FILTER_INPUTS(textsub2video_inputs), + FILTER_OUTPUTS(textsub2video_outputs), +}; +#endif \ No newline at end of file