From patchwork Sat Sep 11 08:11:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30134 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp2233767iov; Sat, 11 Sep 2021 01:11:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxwldPen7gjtssIb7CItG58qcA3J4+r5unSw+I+ql/r5fCe1Qc8PwlIGNNgaPjDq0jK1crY X-Received: by 2002:a17:906:3798:: with SMTP id n24mr1813898ejc.116.1631347901303; Sat, 11 Sep 2021 01:11:41 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id gx6si1253407ejc.386.2021.09.11.01.11.26; Sat, 11 Sep 2021 01:11:41 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=kkJ4x6rw; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6E2D4680AA1; Sat, 11 Sep 2021 11:11:24 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1anam02olkn2035.outbound.protection.outlook.com [40.92.44.35]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 87087680AA1 for ; Sat, 11 Sep 2021 11:11:17 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WodC8eXyN5aLi92AJoqbt8Ai7eWDrKUKGl4o6MYwcuYUqYkwbKxv/iVXZakV2F6EXruEwF5Z58YoahiUmR+LkcoMjA/suykyigrIjZayBm7bEt8+xuLnFFykBWwayf6pa6i0OEge1f32vG2vl0NVpvdKsUsEuHkfCBHlZ2yDzWYzqPyCFlTvwQngi02gJmbS4yaBDIuwKW8n3YGRXmdyKZbBh3opgcMSt86oTZoJjqvfw8MclrJ5nusm12TLtgm1PYi2HI+G+Q9eLU2KoTtsQr74Ln/xvefIMtCbONxvEF3eE8wowkitX1KWfKdLc8ON6twuBTvzwrYsHLRwLqeXVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=bj3PaNt+zt/Osx7LIBLH+5cJh31LjMzZdv4uvhi2/9k=; b=fOpWNBgOTCxEWtwjbZv5GamOBpsgqMUYtyLOxjuy+HiUj+y1Hcoc4LJE2MvWu1jgFVWRe8bruUpJQr9v+sXC0Mdz7AjErBJanyXcNCbV8xNRGMnXp4ES2eQ5IFjls2ha4xbjNuveXRPgMjtJ7+cfFei0PRZS4yT6cZRWy4XUo09a4m+XUn1CNvipGE3k/IvxRLB2k10vGSE+egMDbqy0ISnqNRO+yxIj/ADYRF2lVjLsvNorsl6HhtQEbSCpPvTkcIxXtR/so+RScmIkZmYEmdNy5X9rZSFTFYbvYmJ14FnDh1pfLq7bUdRB+/Kt5DOeEJEfNDjv+9NuaiNB4OYoTw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bj3PaNt+zt/Osx7LIBLH+5cJh31LjMzZdv4uvhi2/9k=; b=kkJ4x6rwZkwXt3Vv8X36SPQefq15+A3ft/zq6h3Tmp9DP1DQj9DjF6tHWMSlXMlOJ2ihF10oGY+81m6wheqCgcjo4zZzBLSJn1YPqY+Sav89sSlZtjDBzfLkLi+3NSR8Ya5XCTN6Xp4ZCZJzg2aPmsYMJjJUMWU+Blz+LZ/Jdn7YfihXMk58ZfdBMRgOINxwKYpTgZe9vi4WDnhyXjHnV9XsYuFwE8xJ+ZFmSWnPnUK0eRjII4EoC8k1dmUsTCi4JvpAAjf898k2vEeob5LgW8mPAtafTqA8qjbvghZBqVE/3mAGTJOVtHDVh2Z8sYCcGHEO7zg89lDsdpqbxMR9WQ== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB6048.namprd04.prod.outlook.com (2603:10b6:208:e5::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.16; Sat, 11 Sep 2021 08:11:14 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4500.018; Sat, 11 Sep 2021 08:11:14 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v4 00/18] Subtitle Filtering Thread-Index: Adem498CJzqRU2xFRJGMB2AZR5oOSA== Date: Sat, 11 Sep 2021 08:11:14 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [t3+YDHCnOnGKWxhWmoAvxnR6SZ3SLgEBXJOz9BBHp78=] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 982b71c2-5e92-4f16-edde-08d974fbb98c x-ms-traffictypediagnostic: MN2PR04MB6048: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: HyE43JDp5n2g4lMHthIAg90QuivjFpBdIwd1Ci66mDeZGkPl5hffxDx2gP3jzvD4XonXpIg4hERd0FEcnIAMDsR2AB4hN800SyIJ3c8Lj2gmhqpofGcljUHxjWt2hkfP+skB5or5RxOZU7ipxPJMUYDAIoInvaF3VEMv+IKw7/B6Dt91rWRoWpAbX7rKRph/aJDV5Mx+a772UjzeLMo9EOa06LpD6dRL1c/ADXFnyGn2WLVXxOsyXr1UjLeLcnvG4+eGZTWyfL6IQDA/tn9b3DlzyIeQcky/wMl1ItmPkXySFnQpnesxqaga3B6yo+UnXn8pXqeXNew6ObxUpWM13Debp/SCxKwETOb1L4wrWMaoEusLTFg+qm621ccs640kOeHXEgmOkn2/r7iJc+RBhX1VMVqHWVoN/odn1FSORp7mQ+WgL4x9/QoDcGRclQoz x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: erYvq39Sah4zCVNUKMHipU8VMvjeI7JCRtYV39EdHkTinhFP31jEmvCzxG+eQX57UWvU6mKWN3DkgAjZ9L4gzhwB3ndZGyNJN4jmAzCZxYheam9kn+fu9kvZMSKR7w8p4Ev9Go2ddCkxBo3ejODFnw== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 982b71c2-5e92-4f16-edde-08d974fbb98c X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2021 08:11:14.7240 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB6048 Subject: [FFmpeg-devel] [PATCH v4 00/18] Subtitle Filtering X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: STY0u1kbV1sq v4 Update: - Re-sending due to Patchwork having failed to parse my patchset There seems to be a bug in Patchwork when parallel processing is enabled. This time, I'll send the e-mails slowly, one after another. v3 Update: - Reworked, revised, rebased, reorganized, refactored - No more prototype/test-style code - Additional subtitle filters added - Filter documentation added - Adjusted FATE tests This patchset is about introducing filtering support for subtitles. The current sub2video "hack" implementation is slow, ineffective and limited in capabilities. => This patchset introduces true subtitle filtering While v1 and v2 where intended for POC, RFC, and getting an impression regarding feasibility, this version is about getting serious on the subject. At a future point in time, it might make sense merge AVSubtitle into AVFrame. I haven't done that yet to avoid cluttering the patchset with too much distraction and keep focus on the relevant changes. Otherwise, I think most of these changes are pretty much straightforward and not a big thing. (within the context of the current state of code!) Details are always debatable and I'll be happy about any (productive) feedback. These new filters are included: - overlay_graphicsubs (VS -> V) Overlay graphic subtitles onto a video stream - graphicsub2video {S -> V) Converts graphic subtitles to (transparent) video frames - overlay_textsubs {VS -> V) Overlay text subtitles onto a video stream. - textsubs2video {S -> V) Converts text subtitles to video frames - textmod {S -> S) Modify subtitle text in a number of ways - stripstyles {S -> S) Remove all inline styles from subtitle events Regards, softworkz softworkz (18): avutil/frame: Subtitle Filtering - Add AVMediaType property to AVFrame avutil/subfmt: Move AVSubtitleType, AVSubtitle and AVSubtitleRect to avutil avcodec/avcodec: Remove AVSubtitleType, AVSubtitle and AVSubtitleRect from avcodec avcodec/avcodec: Add av_get_subtitle_format_from_codecdesc function avcodec/avcodec: Remove sub_text_format parameter avcodec/ass_split: Extend ass dialog parsing fftools/play,probe: Adjust for subtitle format type change fftools/ffmpeg: Replace sub2video with subtitle frame filtering avfilter/subtitles: Add subtitles.c avfilter/avfilter: Handle subtitle frames avfilter/subtitles: adjust for recent changes avfilter/sbuffer: Add sbuffersrv and sbuffersink filters avfilter/overlay_graphicsubs: Add overlay_graphicsubs and graphicsub2video filters avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters avfilter/textmod: Add textmod filter avfilter/stripstyles: Add stripstyles filter docs/filters: Add documentation for all new subtitle filters tests/fate: Update subtitle tests configure | 4 +- doc/filters.texi | 263 ++++++++ fftools/ffmpeg.c | 408 ++++++------ fftools/ffmpeg.h | 12 +- fftools/ffmpeg_filter.c | 198 ++++-- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- fftools/ffplay.c | 2 +- fftools/ffprobe.c | 23 +- libavcodec/ass.c | 2 +- libavcodec/ass.h | 6 +- libavcodec/ass_split.c | 12 +- libavcodec/ass_split.h | 2 + libavcodec/assdec.c | 2 +- libavcodec/assenc.c | 4 +- libavcodec/avcodec.c | 19 - libavcodec/avcodec.h | 81 +-- libavcodec/decode.c | 5 +- libavcodec/dvbsubdec.c | 2 +- libavcodec/dvdsubdec.c | 2 +- libavcodec/dvdsubenc.c | 2 +- libavcodec/libzvbi-teletextdec.c | 14 +- libavcodec/movtextenc.c | 4 +- libavcodec/options_table.h | 2 - libavcodec/pgssubdec.c | 4 +- libavcodec/srtenc.c | 4 +- libavcodec/ttmlenc.c | 4 +- libavcodec/utils.c | 11 + libavcodec/version.h | 4 +- libavcodec/webvttenc.c | 4 +- libavcodec/xsubdec.c | 2 +- libavfilter/Makefile | 9 + libavfilter/allfilters.c | 16 +- libavfilter/avfilter.c | 26 +- libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 63 ++ libavfilter/buffersink.h | 15 + libavfilter/buffersrc.c | 72 +++ libavfilter/buffersrc.h | 1 + libavfilter/f_interleave.c | 3 + libavfilter/formats.c | 14 + libavfilter/formats.h | 3 + libavfilter/internal.h | 1 + libavfilter/sf_stripstyles.c | 246 +++++++ libavfilter/sf_textmod.c | 409 ++++++++++++ libavfilter/subtitles.c | 61 ++ libavfilter/subtitles.h | 44 ++ libavfilter/version.h | 2 +- libavfilter/vf_overlay_graphicsubs.c | 740 ++++++++++++++++++++++ libavfilter/vf_overlay_textsubs.c | 633 ++++++++++++++++++ libavfilter/vf_subtitles.c | 14 +- libavutil/Makefile | 2 + libavutil/frame.c | 78 ++- libavutil/frame.h | 39 +- libavutil/subfmt.c | 78 +++ libavutil/subfmt.h | 125 ++++ libavutil/version.h | 2 +- tests/ref/fate/filter-overlay-dvdsub-2397 | 181 +++--- tests/ref/fate/sub-dvb | 162 ++--- tests/ref/fate/sub2video | 178 ++---- tests/ref/fate/sub2video_basic | 93 ++- tests/ref/fate/sub2video_time_limited | 6 +- 62 files changed, 3631 insertions(+), 797 deletions(-) create mode 100644 libavfilter/sf_stripstyles.c create mode 100644 libavfilter/sf_textmod.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/vf_overlay_graphicsubs.c create mode 100644 libavfilter/vf_overlay_textsubs.c create mode 100644 libavutil/subfmt.c create mode 100644 libavutil/subfmt.h