From patchwork Thu Oct 21 03:18:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 31181 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2084:0:0:0:0 with SMTP id a4csp126921ioa; Wed, 20 Oct 2021 20:19:37 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxdsFeOcFdvzJiFkKwJH37pT+FV5EGQFfZH/sv8Z5CBu8o6Bb36+bcF+0/2tScOS56GDJ0M X-Received: by 2002:a17:906:7017:: with SMTP id n23mr3954842ejj.446.1634786377558; Wed, 20 Oct 2021 20:19:37 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l21si5124996edj.302.2021.10.20.20.19.13; Wed, 20 Oct 2021 20:19:37 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=XEEamnPL; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 79D6B68A4E5; Thu, 21 Oct 2021 06:19:09 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11olkn2034.outbound.protection.outlook.com [40.92.20.34]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C7F1468818E for ; Thu, 21 Oct 2021 06:19:01 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GyRqGUiKoAMlOMJt0/f77IfZCc7kRG2A6Hv+2q4o4qvFlcMv1SEgLarrK51ZKNDzEPnZ3zf3NY1p4g5BYEDisd6kfcvCv+1l5vpJBJ66+I/lSwHDH0b2C3lCmUTO9ZlEWWNBWAm+D6DSCW4STfqF+XYUHWxh6MZcLkkdf+R/fwq0er88E4ywIenYCZWPAKhhrhjsefNIPfW6Y3hfEmZv/xLVQ4JuK5D1LYRpllKphepZFVzQnwDsjdWI72thlXAROSDj3XvT1u0IaEApPzktG9sP4QwLjgkX25qx7o/xWgFcsVz5ay82+9KTX+m0fx4hiWxvySxMlNhn9RNfrXpDCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VSF71/VMmjuo8aDmEWJboMkdUxMzqbQ8C0CObg3j55E=; b=CcBY/zuSjrIzAyEoRb8PunLxLGEOSDCipWFxVPNo/kfQDDRf15T9zVEslMTuLqBkgzfAY0uL3CJruBbscKoaPCZ6HHj5GOwpQ8LIS7Z+yo/GysnN3jfflX+wQkcfen3RAO0bVjgTw6hQ0TJ7xF4DwGl8g4ezNZemd80Wooy/QHKFjj7PVjawV73DxyVPaDFS8wO+Tteg1GUBnPMtb3DqsSRTp8NPR7o7apO6/p55ZnYr9lOHIci9FCPXEF3DTbeBIFPTMk9/D9bDSjIHWAYQo84EgXq2sGQnxws3bAteN2fdFkkTHtpEolenS+rT+X8nFRtXGPmkkSPSJLV0P71JfQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VSF71/VMmjuo8aDmEWJboMkdUxMzqbQ8C0CObg3j55E=; b=XEEamnPLZ/e3+gHHeq+2cEc2aB5btidhhiiH0pU2JgcsHEbJrD8hKDzfzHH9ozNvjWos8xmugIkLtAGhyH7NLzTwGkVkiIag4hQ0KNIPGlW8KQibxMQPRkCYUAAaEGTLad4yPZTQ9bZVVwhSfF9eRc4OglGzd4+1Hhdb5elh7LM/qOASTarUceFBiGCcyRBYpE8qk7nxnNLEvPWXK/+S4BFBNXUcSaZ+QkkAzywWa6H+RDzueVpQOAl4cQQBPaG1u5EjILtT5kP3Z9fw3g2lXhIjat+dEiXsZwlU+K31EcYtB5E2+3FBHC9FaRj/P7+JEUsSElLzkV67VaB9pPPSPw== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB6238.namprd04.prod.outlook.com (2603:10b6:208:e4::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.18; Thu, 21 Oct 2021 03:18:58 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4608.018; Thu, 21 Oct 2021 03:18:58 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v14 00/14] Subtitle Filtering Thread-Index: AQKuyWPLrzd7fniXW+llvbOmUbL0/A== Date: Thu, 21 Oct 2021 03:18:58 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [YCF3tWDkGgS8Ek6tOSko1w7VmomN42LJ] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 76402bde-74c7-4d0c-c1e1-08d9944185b1 x-ms-traffictypediagnostic: MN2PR04MB6238: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: p6P+K7uBhS9kmto0fm9ieNoxOru63y15HRyITKR/N/M/GGEDOr4Yd+ZlERGvHxEJ5upvM2NSfUhXKIVanu6Ic6pbrFpU349uAfeMcUk8b4nrgXkMD7mSbdog5B1S/NogUZ9BbjTdC+IeSyiEiSnZRii6yrst8F5AKQtbJBQTod6FyoGAHpkq6LH3wbc7e6CYXRtAoS4548AJMklnfhvm8W5rtRIgGPyQn/SCbqzu1OGqfHyjzjqOgiHKaipQ08vRAsyQTKb02XQcbnbtq1tpfPk5UL6jix0G1EfkZkY/EfU5ESyRtr9ZhFz7hgEbo70INmH8a/3aOVS42R46hMayKEYDf582WhQrrctHbveOykZU5tJQ8W8uMatVyTDiOC850rN8E2YKiuST4t1pWGe6DLdFezfVDXU43wj8Tzo3PGgFVihE6n1oBWk99Bxaq6k67AN0K0Qfd12tHn/YLTsTYCuUipR4zKwE3ePwW7V6AfyzLCXZHusqSCsRqeUPAoGkg+2jpz888xGJ0hDa8IFjwrzo44Qcik3lv6CWO8I9njk= x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: HE6dzS13x2XgZdoVn54pnTM/qhtIQRB9UdP+RPaEzmr3/c7DfwIpuWcpag4MoZeKvSbPU5nN2QjjDZAzi8tu8vJ8EZRs8E+Abn0xAo/lDSJkuvFsNYASOD8ogMna1O0rm661UfycGgUoVKk/isqmOiQjDF+VmkVkxxU6SS8FfLGOBy5xsiFa42o3PIWAN8HmW7M0/M8Rxj0tOeRc1fOb2h3ajq61UTJpEk7hTcxcQuKWEY/fLiifJMTbv7WZStw/Cw0chtWuMku8IvViMiXEqSUmr4jUAfp110GUxsd/feeqVdWaBJAU+SI5Oi2R3I3bKucrJN7HYMn40NC1ChKBNG1ixeWLzVtJIk44W3uf/c+3r2YBGbjGZfnyzJDpjkJokR/T+PZyBJ5jquCa5sSmBT3P0k23grT1yYwkNm4bvyib8taq4X1i00dd/6Cn/rwPbiet0R0S+Fb7uBUu5v7M2jfaF3aQrvXAJ+BE2HQAR70sWYCGusTSXusFPaN2Ktr79lfyumOEDkR8WWjv0DChvB7nt9J+WDHAq+efij+jY/p7XWM5MR8nSkS4GDoMzzFuh7yfMBl4CuX3X2oWZlkIltI/HHVbAF32ykV6eObynwv5ZA6O5VD5R5knHgUduBPVg4gZ009c6Qp1N68SBAquzAZAwUCCgBXHFscBB4Mr19V6lNph8N60TD2FPgRWd0j416gw8ZSx76frTdunU6vDvw== MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 76402bde-74c7-4d0c-c1e1-08d9944185b1 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Oct 2021 03:18:58.5950 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-userprincipalname: softworkz@hotmail.com X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB6238 Subject: [FFmpeg-devel] [PATCH v14 00/14] Subtitle Filtering X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: scwmG8W120KF v14 Update: - FEATURE-COMPLETE Fixes: - get_subtitle_buffer: set extended data - overlaygraphicsubs: copy time base from input - subtitle_heartbeat: Fix pts when re-sending - subtitle_heartbeat: only beat on video stream timing - subtitle_heartbeat: propagate beat over all input files - overlay textsubs: delay header processing + textsub2video fixes - textsub2video: reworked to support scenarios with hw overlay - subscale: add output caching for improved performance v13 Update: - FEATURE-COMPLETE - Adapted patchset to upstream changes regarding AVPacket handling (v12 couldn't be applied anymore) - Some adjustments for cases where hw acceleration is involved v12 Update: - FEATURE-COMPLETE No more features or filters are needed for the introduction of subtitle filtering - Adapted patchset to the changes regarding .query_formats in avfilter - Removed underscores in filter names - Expose all options from cc_dec from the splitcc filter - Added 'render_latest_only' mode to overlaytextsubs and textsubs2video filters (allows rendering closed captions without black background rectangles) v11 Update: - FEATURE-COMPLETE No more features or filters will be added to this patchset - Added subscale filter Provides high-quality scaling and rearranging functionality for graphical subtitles. This has been the last missing bit for making this a useful and versatile toolset for subtitle manipulation - The common ass utility functions remain to reside in avutil, but in this version, they are marked as avpriv_ and no longer public like in the previous version. - A number of detail fixes are included, but there haven't been significant changes to the core implementation in the past few versions. I'm quite confident about the current state. - As a final step I'll be going to create a wiki page presenting a comprehensive collection of usage examples, showing off the whole range of new capabilities and also providing proof that it is working universally in a wide range of use cases. -------------------------------------------------------------- This patchset introduces filtering support for subtitles. Besides the ground work, this patchset includes a range of new filters for subtitle processing: - subscale (S -> S) Provides high-quality scaling and rearranging functionality for graphical subtitles. - graphicsubs2text (S -> S) Convert graphic subtitles to text subtitles via OCR - censor {S -> S) Allows censoring word in subtitles by a configurable word list - showspeaker {S -> S) Prepend the speaker names to text lines (those are often available in ass subs) - splitcc (V -> VS) Splits out Closed-Caption data as a subtitle stream for filtering and encoding - overlaygraphicsubs (VS -> V) Overlay graphic subtitles onto a video stream - graphicsub2video {S -> V) Converts graphic subtitles to (transparent) video frames - overlaytextsubs {VS -> V) Overlay text subtitles onto a video stream. - textsubs2video {S -> V) Converts text subtitles to video frames - textmod {S -> S) Modify subtitle text in a number of ways - stripstyles {S -> S) Remove all inline styles from subtitle events Regards, softworkz softworkz (14): global: Prepare AVFrame for subtitle handling global: Move ass helper functions to avutil as avpriv_ and extend ass dialog parsing fftools/play,probe: Adjust for subtitle changes avfilter/subtitles: Add subtitles.c for subtitle frame allocation avfilter/avfilter: Handle subtitle frames avfilter/sbuffer: Add sbuffersrc and sbuffersink filters avfilter/overlaygraphicsubs: Add overlaygraphicsubs and graphicsub2video filters fftools/ffmpeg: Replace sub2video with subtitle frame filtering avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video filters avfilter/textmod: Add textmod, censor and show_speaker filters avfilter/stripstyles: Add stripstyles filter avfilter/splitcc: Add splitcc filter for closed caption handling avfilter/graphicsub2text: Add new graphicsub2text filter (OCR) avfilter/subscale: Add filter for scaling and/or re-arranging graphical subtitles configure | 7 +- doc/filters.texi | 747 +++++++++++++++ fftools/ffmpeg.c | 525 +++++------ fftools/ffmpeg.h | 15 +- fftools/ffmpeg_filter.c | 217 +++-- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- fftools/ffplay.c | 102 +- fftools/ffprobe.c | 48 +- libavcodec/Makefile | 56 +- libavcodec/ass.h | 129 +-- libavcodec/assdec.c | 2 +- libavcodec/assenc.c | 2 +- libavcodec/avcodec.c | 19 - libavcodec/avcodec.h | 75 +- libavcodec/ccaption_dec.c | 19 +- libavcodec/decode.c | 53 +- libavcodec/jacosubdec.c | 2 +- libavcodec/libaribb24.c | 2 +- libavcodec/libzvbi-teletextdec.c | 14 +- libavcodec/microdvddec.c | 7 +- libavcodec/movtextdec.c | 3 +- libavcodec/movtextenc.c | 20 +- libavcodec/mpl2dec.c | 2 +- libavcodec/pgssubdec.c | 1 + libavcodec/realtextdec.c | 2 +- libavcodec/samidec.c | 2 +- libavcodec/srtdec.c | 2 +- libavcodec/srtenc.c | 16 +- libavcodec/subviewerdec.c | 2 +- libavcodec/textdec.c | 4 +- libavcodec/ttmlenc.c | 15 +- libavcodec/utils.c | 11 + libavcodec/webvttdec.c | 2 +- libavcodec/webvttenc.c | 16 +- libavfilter/Makefile | 15 + libavfilter/allfilters.c | 13 + libavfilter/avfilter.c | 30 +- libavfilter/avfilter.h | 11 + libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 63 ++ libavfilter/buffersink.h | 15 + libavfilter/buffersrc.c | 72 ++ libavfilter/buffersrc.h | 1 + libavfilter/formats.c | 22 + libavfilter/formats.h | 3 + libavfilter/internal.h | 19 +- libavfilter/sf_graphicsub2text.c | 323 +++++++ libavfilter/sf_splitcc.c | 315 +++++++ libavfilter/sf_stripstyles.c | 191 ++++ libavfilter/sf_subscale.c | 883 ++++++++++++++++++ libavfilter/sf_textmod.c | 697 ++++++++++++++ libavfilter/subtitles.c | 63 ++ libavfilter/subtitles.h | 44 + libavfilter/vf_overlaygraphicsubs.c | 737 +++++++++++++++ libavfilter/vf_overlaytextsubs.c | 624 +++++++++++++ libavfilter/vf_subtitles.c | 50 +- libavformat/utils.c | 1 + libavutil/Makefile | 4 + {libavcodec => libavutil}/ass.c | 73 +- libavutil/ass_internal.h | 139 +++ {libavcodec => libavutil}/ass_split.c | 30 +- .../ass_split_internal.h | 24 +- libavutil/frame.c | 194 +++- libavutil/frame.h | 93 +- libavutil/subfmt.c | 240 +++++ libavutil/subfmt.h | 185 ++++ tests/ref/fate/filter-overlay-dvdsub-2397 | 181 ++-- tests/ref/fate/sub-dvb | 162 ++-- tests/ref/fate/sub2video | 116 +-- tests/ref/fate/sub2video_basic | 135 +-- tests/ref/fate/sub2video_time_limited | 4 +- 72 files changed, 6782 insertions(+), 1139 deletions(-) create mode 100644 libavfilter/sf_graphicsub2text.c create mode 100644 libavfilter/sf_splitcc.c create mode 100644 libavfilter/sf_stripstyles.c create mode 100644 libavfilter/sf_subscale.c create mode 100644 libavfilter/sf_textmod.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/vf_overlaygraphicsubs.c create mode 100644 libavfilter/vf_overlaytextsubs.c rename {libavcodec => libavutil}/ass.c (73%) create mode 100644 libavutil/ass_internal.h rename {libavcodec => libavutil}/ass_split.c (94%) rename libavcodec/ass_split.h => libavutil/ass_split_internal.h (89%) create mode 100644 libavutil/subfmt.c create mode 100644 libavutil/subfmt.h