From patchwork Mon Aug 30 08:16:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29873 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp3737864iov; Mon, 30 Aug 2021 01:17:16 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwnse/Yv3xCrc8QUQ5mVCLv62u0v/oYRn7ckK/MEYOnuSahduVwJlPZSBeSoNWDcFhZtmEg X-Received: by 2002:a17:907:1b06:: with SMTP id mp6mr24073467ejc.188.1630311435933; Mon, 30 Aug 2021 01:17:15 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id o20si8388289eds.216.2021.08.30.01.17.15; Mon, 30 Aug 2021 01:17:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=VNOl63be; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9E40668A2CE; Mon, 30 Aug 2021 11:16:56 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08olkn2027.outbound.protection.outlook.com [40.92.46.27]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AF231689F94 for ; Mon, 30 Aug 2021 11:16:54 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VdtjbQxDs/IcE9A4fqaWfJ3z2OjCq9WSB7KuuyEdtdN8Gn9djWUjCujUJ1PBYF+cRY806d174n2Ilgtt7AMFnT4NzxFt8ckLoFKAAEt66C2lMHS4yU/8SoE1y8tOtDcHnxxVfyC4UnFXIjYWXCMs74grzBaSqr6OHmoIzqQcM0yP7KDAP8F7BnAURkueEczJORWnGhAe8vAAc5FeA9ILtqNxBnFzIIarMlnA086MBN32UOO9vkbdJvA18+HLr5q3Is8I4ZsylzTNeL4DEoV2dQEpr96LLpYPA7klUGBVMUumePut/NUgJm0ukfJL3uuM119rUm375o6VGe7rM2N4+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1l33jcmVP1veB9cu3OhyuqCY30g9BBAqlRDbDLgbTf4=; b=KRV/D7lJ1cQsGn8SZwFJ8sauqWLYe8COHJdmVHSCp6mPIXY5M2C8Y2db6ECjcpVQKirvvVvaqij/3GxTx64fUsqk1vLdTtaq7k3wW7eJgeQ/gpiVCVkq1/suQi2jyw8rZ7VyMHPw3MPQvc3Pq/olv+UZIqR3EtRCcvAqS8FQJXQAmYj91oKWe58nQ+BS/ni79ZYLzLB8WW4XRysRA59rbGigZGDMxvActI71ij8aa92BOcK0CWreo81HVKEMOwSM02DKnNQFSxKOF0HfiOQQg1K86XwBiFVlm9r6q/epyqGlFQ1iaITsVR85q4f25pkjl/BhtLMg9dUwXw6D3fct8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1l33jcmVP1veB9cu3OhyuqCY30g9BBAqlRDbDLgbTf4=; b=VNOl63beXJynzKKUAKFxPe1XPa0/ER9dmr3xxEtScCcguudp1EhSN0vLGALkX47PdK8UWz3VTPriDPMtIRu5+qvkGVoRv7KuciGnezmiC3QGTM7Nh9sFVnbtgS6ai0Loxoe/87ki7DUkNimDX822vvtQuxbAscGYa4oqJWpCkM39akqSyqj3W54L+bh1im74BPF7YevqTDQLiuto4UIS2p2MuOLKXqnf3Ttzo5XyFsVX6XNRLd642djoPLAPPOgL16ueF6IkF5qxtl4cIZkse9+FCDXmeYfNxGS9+wjdrIQNJzD4DS5EJSim9PAe/SVdjz7Y1wE3QsSE9sTL8p6XUA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5853.namprd04.prod.outlook.com (2603:10b6:208:a6::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.19; Mon, 30 Aug 2021 08:16:51 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 08:16:51 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v2 5/8] avfilter/sub2video: Add sub2video filter Thread-Index: AdeUd97C+XQyjRkXSyOJSLj3QMpn5g== Date: Mon, 30 Aug 2021 08:16:51 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [kDkD0Lz4NZhx8LUCyB8J7wTGTvNY1HlP] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: ad313395-7f4b-453d-7ee4-08d96b8e8565 x-ms-traffictypediagnostic: MN2PR04MB5853: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 5hg5dOdNcbe0DuFVt5q/5rviTqs3ZAx9zTnfIHxGIWhXEWUuzV/r9DQ7NmkZYm29kFuQF+euvt1qRQdRoqpJgxOyiLBRqRGhDOwtX55sRqTXY4ShWgkI1QiqHZ850vcZhLPPNor9ePLIUUImi9G+jMd2NBlQtMcZxsUoyaI15jiQQ8kKfhFmB9cUPenMTWSQt7L3Tntifjr4Lz70mfxtus7D6k1OM7IRYh/lKUysgcqOrGH3im0BDS9LdXHKDekoGkXabmZ0R0GcnAfhOZrLnn5YcpJ/XKICgNG3JxnR/sppcbvQaiurNqXs9MlH0C4LILm6YxHhQQ0/Ez9T6NISUrxnbZ/rz7zrZUGru4fkECgLHlGdmZaflxitrBuEp3U5gOIj6FBRZ03WPRafvCwq1RcVDpOHWBVnx2y4hpnrvA4vtvSqyoUKZFjObQOTNLjg x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: Opyre+skEh3O09yn4Z6+91SMxA4RdZW9+rMpM4ZJZwLx5ZKM0S27kfUM7qshLaNNnMYF03fg8KjlgKY2+gZhj3Q27dL5oonA9Lz2VRTH4ki1BNcmzpwsEqf5RZcKgTgFKk5oIgcFR15Rr9B/CRLEaQ== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: ad313395-7f4b-453d-7ee4-08d96b8e8565 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2021 08:16:51.6295 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5853 Subject: [FFmpeg-devel] [PATCH v2 5/8] avfilter/sub2video: Add sub2video filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: nIFL220BUUxt Signed-off-by: softworkz --- v2 Update: - Implemented Andreas' suggestions - overlay_subs filter: - removed duplicated code - implemented direct (no pre-conversion) blending of graphical subtitle rects - Supported input formats: - all packed RGB formats (with and without alpha) - yuv420p, yuv422p, yuv444p libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/svf_sub2video.c | 260 ++++++++++++++++++++++++++++++++++++ 3 files changed, 262 insertions(+) create mode 100644 libavfilter/svf_sub2video.c diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 8130f7b46c..e38c6b6f6d 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -542,6 +542,7 @@ OBJS-$(CONFIG_SHOWSPECTRUMPIC_FILTER) += avf_showspectrum.o OBJS-$(CONFIG_SHOWVOLUME_FILTER) += avf_showvolume.o OBJS-$(CONFIG_SHOWWAVES_FILTER) += avf_showwaves.o OBJS-$(CONFIG_SHOWWAVESPIC_FILTER) += avf_showwaves.o +OBJS-$(CONFIG_SUB2VIDEO_FILTER) += svf_sub2video.o OBJS-$(CONFIG_SPECTRUMSYNTH_FILTER) += vaf_spectrumsynth.o # multimedia sources diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index abd0a47750..5b631b3617 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -518,6 +518,7 @@ extern const AVFilter ff_avf_showvolume; extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; +extern const AVFilter ff_svf_sub2video; /* multimedia sources */ extern const AVFilter ff_avsrc_amovie; diff --git a/libavfilter/svf_sub2video.c b/libavfilter/svf_sub2video.c new file mode 100644 index 0000000000..689c2a565c --- /dev/null +++ b/libavfilter/svf_sub2video.c @@ -0,0 +1,260 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * graphical subtitles to video conversion, based on previous sub2video + * implementation. + */ + +#include + +#include "libavutil/audio_fifo.h" +#include "libavutil/avassert.h" +#include "libavutil/avstring.h" +#include "libavutil/channel_layout.h" +#include "libavutil/opt.h" +#include "libavutil/parseutils.h" +#include "libavutil/xga_font_data.h" +#include "avfilter.h" +#include "blend.h" +#include "filters.h" +#include "internal.h" +#include "libavcodec/avcodec.h" +typedef struct Sub2VideoContext { + const AVClass *class; + int w, h; + AVFrame *outpicref; + int pixstep; +} Sub2VideoContext; + +#define OFFSET(x) offsetof(Sub2VideoContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM + +////static int alloc_out_frame(Sub2VideoContext *s2v_ctx, const int16_t *p, +//// const AVFilterLink *inlink, AVFilterLink *outlink, +//// const AVFrame *in) +////{ +//// if (!s2v_ctx->outpicref) { +//// int j; +//// AVFrame *out = s2v_ctx->outpicref = +//// ff_get_video_buffer(outlink, outlink->w, outlink->h); +//// if (!out) +//// return AVERROR(ENOMEM); +//// out->width = outlink->w; +//// out->height = outlink->h; +//// out->pts = in->pts + av_rescale_q((p - (int16_t *)in->data[0]) / inlink->channels, +//// av_make_q(1, inlink->sample_rate), +//// outlink->time_base); +//// for (j = 0; j < outlink->h; j++) +//// memset(out->data[0] + j*out->linesize[0], 0, outlink->w * s2v_ctx->pixstep); +//// } +//// return 0; +////} + + +static const AVOption sub2video_options[] = { + { "size", "set video size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = "640x512"}, 0, 0, FLAGS }, + { "s", "set video size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = "640x512"}, 0, 0, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(sub2video); + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats = NULL; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { SUBTITLE_BITMAP, SUBTITLE_NONE }; + static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_RGB32, AV_PIX_FMT_NONE }; + int ret; + + /* set input subtitle format */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output video format */ + formats = ff_make_format_list(pix_fmts); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + Sub2VideoContext *s = ctx->priv; + + s->w = 1920; // inlink->w; + s->h = 1080; // inlink->h; + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + Sub2VideoContext *s = outlink->src->priv; + float overlap; + + outlink->w = s->w; + outlink->h = s->h; + outlink->sample_aspect_ratio = (AVRational){1,1}; + + return 0; +} + +static void sub2video_copy_rect(uint8_t *dst, int dst_linesize, int w, int h, + AVSubtitleRect *r) +{ + uint32_t *pal, *dst2; + uint8_t *src, *src2; + int x, y; + + if (r->type != SUBTITLE_BITMAP) { + av_log(NULL, AV_LOG_WARNING, "sub2video: non-bitmap subtitle\n"); + return; + } + if (r->x < 0 || r->x + r->w > w || r->y < 0 || r->y + r->h > h) { + av_log(NULL, AV_LOG_WARNING, "sub2video: rectangle (%d %d %d %d) overflowing %d %d\n", + r->x, r->y, r->w, r->h, w, h + ); + return; + } + + dst += r->y * dst_linesize + r->x * 4; + src = r->data[0]; + pal = (uint32_t *)r->data[1]; + for (y = 0; y < r->h; y++) { + dst2 = (uint32_t *)dst; + src2 = src; + for (x = 0; x < r->w; x++) + *(dst2++) = pal[*(src2++)]; + dst += dst_linesize; + src += r->linesize[0]; + } +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *src_frame) +{ + Sub2VideoContext *s = inlink->dst->priv; + AVFilterLink *outlink = inlink->dst->outputs[0]; + AVSubtitle *sub; + int ret; + int dst_linesize; + AVFrame *out; + unsigned int num_rects, i; + uint8_t *dst; + + outlink->w = 1920; + outlink->h = 1080; + + out = av_frame_alloc(); + if (!out) { + av_frame_free(&src_frame); + return AVERROR(ENOMEM); + } + + out->width = outlink->w; + out->height = outlink->h; + out->format = AV_PIX_FMT_RGB32; + + if ((ret = av_frame_get_buffer(out, 0)) < 0) + return ret; + + memset(out->data[0], 0, out->height * out->linesize[0]); + + ////out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + ////if (!out) { + //// av_frame_free(&src_frame); + //// return AVERROR(ENOMEM); + ////} + ////memset(out->data[0], 0, out->linesize[0] * out->height ); + + out->pts = src_frame->pts; + out->repeat_pict = src_frame->repeat_pict; + out->pkt_dts = src_frame->pkt_dts; + out->pkt_pos = src_frame->pkt_pos; + out->pkt_size = src_frame->pkt_size; + out->pkt_duration = src_frame->pkt_duration; + out->reordered_opaque = src_frame->reordered_opaque; + out->best_effort_timestamp = src_frame->best_effort_timestamp; + out->flags = src_frame->flags; + + sub = (AVSubtitle *)src_frame->data[0]; + + if (sub) { + num_rects = sub->num_rects; + dst = out->data [0]; + dst_linesize = out->linesize[0]; + for (i = 0; i < num_rects; i++) + sub2video_copy_rect(dst, dst_linesize, out->width, out->height, sub->rects[i]); + } + + av_frame_free(&src_frame); + return ff_filter_frame(outlink, out); +} + +////static int activate(AVFilterContext *ctx) +////{ +//// AVFilterLink *inlink = ctx->inputs[0]; +//// AVFilterLink *outlink = ctx->outputs[0]; +//// Sub2VideoContext *s = ctx->priv; +//// +//// do { +//// int ret = ff_outlink_get_status(outlink); +//// if (ret) { +//// ff_inlink_set_status(inlink, ret); +//// return 0; +//// } +//// } while (0); +//// +//// return FFERROR_NOT_READY; +////} + +static const AVFilterPad sub2video_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_frame, + .config_props = config_input, + }, + { NULL } +}; + +static const AVFilterPad sub2video_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, + { NULL } +}; + +AVFilter ff_svf_sub2video = { + .name = "sub2video", + .description = NULL_IF_CONFIG_SMALL("Convert graphical subtitles to video"), + .query_formats = query_formats, + .priv_size = sizeof(Sub2VideoContext), + .priv_class = &sub2video_class, + .inputs = sub2video_inputs, + .outputs = sub2video_outputs, +};