From patchwork Mon Dec 13 23:40:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 32451 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp6138410iog; Mon, 13 Dec 2021 15:40:30 -0800 (PST) X-Google-Smtp-Source: ABdhPJzJQEf46lSsm9wwN/D10zlGRjUG9dUANR7pE/UyrqIiE+flzvjKqDEsjQ/Q1ur2zqQ/l01x X-Received: by 2002:a05:6402:d73:: with SMTP id ec51mr2662875edb.175.1639438830554; Mon, 13 Dec 2021 15:40:30 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l9si27198520ejo.594.2021.12.13.15.40.30; Mon, 13 Dec 2021 15:40:30 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=GuOZJnL7; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8777668AFD1; Tue, 14 Dec 2021 01:40:27 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11olkn2032.outbound.protection.outlook.com [40.92.18.32]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E7C0168AED3 for ; Tue, 14 Dec 2021 01:40:19 +0200 (EET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DeF/lpzE6RwVj0cobNeeAllNa1eMbbzsvN8mtdB4E65e+S/RS+Y3/OLevYeirkeU2cvyu+mWtCqfLNtbfLLS2DYwI32AcOk887nAu5JvJwMSR+7xq/o8T11Bj5AuL+R71aSunGKLKOKLL7j18+fgyaeVQ50N7WoHCp9vK1iRDnhO0Qh6l5O4M+XMfKxhMaS3/7aXqf7GGGSw7xgS0V4+ImI10Aq++TVaifO1tUpyUj6q460IzlnRHPw/LAQRWFcIIi72ptWlU3JDwTZi7gJcom5NDRhOIW2clmPEnrbj4xo8UhgJslb77FgVhYYPL12abR1K9WI4ACU9HqKMa48V2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gOKd1GRfD+3ggEZ4Bn68v4x/LuVCxqkDK0wQ7aGfmzA=; b=f/UHxnb0VTdfbk8PIWkk4QJlzJEfNcEzs/MAYo+JvRd/Fh/qb9ulQhDxsTb4Kc/6De7mxf1FkqfnPNt27LEZvV8wn3IUgeN2VGOT8Nwt2iZZtvpTniVyKWcEY0mfjIF9kjOzjgcHtOryRpE4eoYN57XuhzE0e3Or9FiojpVioZiI/FZk8An7/yjdEPsVPpMNb1QXME3r/PEsQTFRM0SV/rpmvELkXCYuvsQFwJ2T5k0ljzuQZjgkkuYY+g+BQQBtD86PRkNqaih0+0nEFpGWuQCGsG9FcxaIlS4K4iORTxMv5IubDZLhmct+5xB/Tqby3gvRQp4EHNHYMx+AN7b5Jw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gOKd1GRfD+3ggEZ4Bn68v4x/LuVCxqkDK0wQ7aGfmzA=; b=GuOZJnL7uX+cmQEKL0AK06oAa0JVVC8t/ck3F2IL05rMHUb54u9fE3QZxpLXnaM48aYv2+AdRP9qQpcQFcTYkd/SZfXyAsjp8na4kkjaTXjZD4cdNLgfJ60Sal/bFzN6kK+kAF75r/QvSkDBXv+Un7vP4xf+osSJQ5Z43XylseqLVEL5lC1+3p9t2Oy8GxrjZuKAckAhFi/N0OyqRQBuDsM1Mcda66AHIrFPJZFHbcMJ5Wv1B2OIhIZhTZgSWLuOfYDazXV9e68CQjqTMl2ac/WBqupSwDA2daX06xdSqpMhFLUXyzjPLZZMFZbooNxitprN0k1UXCskYUhh5Qco0A== Received: from DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM (2603:10b6:8:b::20) by DM8P223MB0303.NAMP223.PROD.OUTLOOK.COM (2603:10b6:8::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4778.12; Mon, 13 Dec 2021 23:40:16 +0000 Received: from DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM ([fe80::9c8d:fc63:9488:9775]) by DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM ([fe80::9c8d:fc63:9488:9775%7]) with mapi id 15.20.4778.018; Mon, 13 Dec 2021 23:40:16 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v24 00/21] Subtitle Filtering Thread-Index: AQHX8HrIUzG1B3z4gUSBoHnx2oyKWw== Date: Mon, 13 Dec 2021 23:40:16 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [KFtJNW9qTDe19u00FODu1GexEoh7fzLb] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3cf7454d-d908-4d4a-703b-08d9be91eac6 x-ms-traffictypediagnostic: DM8P223MB0303:EE_ x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: E+n34/YO9B+4lIhN20Kudnuc3y8957PigTlIUaXWUkVJDmj9LUiUPnPxNjkH32h0gku2NTB+kmMuI0wxVEFk2EHnKkoxsa5+G66hOBaMJ02drczUk6XdF4akivjTnzg8Pn7jlBfF2ukjzEDFwlZ62uQBCED/OSezvRtU27/yiO/vdRPu7mPIh6YU+P7IqrkqvwaGELkLfbguhLczzZa05hWb7j7r9J500GFxVXfFKaqE7LzWHi0q1Dd4Fz5yv7uJx7vA6wzBOfIX1ximMpU/GDZXyego/AL82Crc50gmNIRM5Je/u648oidwiDDSwB0Z61sy8vAQ4LI2JX6RHTB97LSi2P2M2gEbKU7YGC8lCXqhULBS93IWWRkSillHY8SEPsqH2D9/W9Mo5BkhrHs3x2h1ISi9MUOiMV37a/g/Z6TX/42j1Msnf/Rqo5apX+V54qwDZ2VYVfqBg0yy6Dvkfx6MeTUs+22W87+IsepNlUdvlIz8dOBVs1MJzzsKx2FIaS/UnisbCBwgwYIojvyFITBM7V0x6XYiZozKTvBjxaLYKNBBhmVMH4w7VPZJbgW+dJ3DArTX7Jke5yiWbj9XawvCUDeKw/qzEMrhvRqzH2uWwYQFpSNGxja6ErDZbPmM1nSO7tRmA/vSXSL7dYRdXLp2MsEMde5EnQNgb7GHxok= x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?q?G/m7GUSNntJbAGf3LScczJqtPGrS?= =?utf-8?q?/uTWPHWJ5tKNJXoN1jc84SBsqnP84B5vHu6LDd/h02WMkcLQj8YobJG4kP5hl7lHL?= =?utf-8?q?fbpBwJZBytAvc/y8tYwTzSvGASRuCWRISAX4G2iJEjtCWRynD/8FES20wdwWm/4JS?= =?utf-8?q?JubOOzLa77g/wKYOgL3OZut3kxWFEdj24CSGQU9tRIMeXj23fKjDfRjjk/mOUc3/x?= =?utf-8?q?AfFlZwAcYjnYpD8bLJaz2mQoNIz9bNhG2FuPYo+tLZaQZKhWGVRpwtbhmr1P4wr/5?= =?utf-8?q?ZPa/QkstqjBKnGrK9xR4YNBfao4d5w2zWtcsnQimr18HN4YCeOf5hD3GRtLEqsOeQ?= =?utf-8?q?TCuRqTRYplqsT7TRWPn1yRk1kSXFbP8tOXHaz6wWsbsjOVrS6/+Yz7dM7x7vEXFUN?= =?utf-8?q?EtXZIyw3TU61wDGvtkaY5oki3QxNWXjSCzPaYa+TEZCPQ5skn9/NFqnbCJpnonV4c?= =?utf-8?q?oQPQjHzYIXtwWtKLO/gwum6SCRsWlNG9+aTFqOaPRh+Gzwu4MPxuhW13b2/T9llbZ?= =?utf-8?q?s4CIoPIOWp6/+r4iRlT84ERD3bemfrA7lU6D3QinkTb2Dly8OyvQcOvCMqfpE8Z8B?= =?utf-8?q?Des71OUW50/ZSYrNJ9chWpO33iJHkwIITKea+m49a+dwLRBrBbhHwkUbCSfB7bbzI?= =?utf-8?q?a6xkPuDnj35Qpyx7H5vPO6SdulbVZrH8dg1QfVc18VaX13XoMZLQbADiSHGy9ECp+?= =?utf-8?q?6lqNjPx1h6bPvI64YksfLFZFyd02gJs5lGMCE+dDBoSG474OAYkx1Ujdnj82eYOpX?= =?utf-8?q?I2DOJSCjt/P0arRtK3EtMVZ6hCSSgUUFAmK/3E/7p+o5dLKTTpeeAiBwV7FvAlm/Q?= =?utf-8?q?JGUitJtFTOvH6pe+RWCtbFyKRBJDkNRgcP/BdXTePoFkEipl0XW86TCPxhU46KT5j?= =?utf-8?q?gu3pP9J9q0uMylLvjbjelQ6pFtYPywM+sf2U7wtunpZQ=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-1ff67.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 3cf7454d-d908-4d4a-703b-08d9be91eac6 X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Dec 2021 23:40:16.7158 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8P223MB0303 Subject: [FFmpeg-devel] [PATCH v24 00/21] Subtitle Filtering X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: UuWng/VpFoJM New in V24 - Fixes bugs as reported by Michal - graphicsub2video: use 1x1 output frame size as long as subtitle size is unknown (0x0) and no size is not explicitly set - decode: set subtitle frame size from decoding context - ffmpeg: re-init graph when subtitle size changes - ffmpeg_filter: always insert subscale filter for sub2video compatibility - vf_overlaytextsubs: ensure equal in/out video formats GITHUB Branch: https://github.com/softworkz/FFmpeg/tree/subs_v24_plus without extras: https://github.com/softworkz/FFmpeg/commit/c50eaa9307d448670f7119db4df0a2534ec4a6c5 New in V23 - subfmt.h: Fix includes, use #if rather than #ifdef - subfmt.c: Remove unused struct - move addition of ass hard_space callback to commit 5 - also add ass extensibility callback - remove bugfix commits regarding ass hard-space due to being incompatible with GIT e-mail workflow New in V22 - Two new commits, implementing parsing of hard-space ASS tags (\h} The ccaption_dec decoder emits a lot of \h tags via splitcc which the encoders (all using ass_split) didn't handle properly - Combined APIchanges and version bumps into a single commit at the end - Moved new fields to end of AVFrame - Changed version bumps as suggested => no more major bumps now! New in V21 - Rebased. Patchset was broken again. - Don't declare AVFrame.subtitle_pts as deprecated and clarify doc text - Remove AVFilter.subs_list and AVFilter.sub_fmt from APIchanges - Change include in formats.c New in V20 - Rebased. V19 didn't apply cleanly anymore New in V19 - Document API changes in all relevant commits - Move enum AVSubtitleType: Improved commit message - Put deprecated enum values under removal guards - Document AV_SUBTITLE_FMT_NB - Fixed all trailing whitespace - splitcc: Add scatter_realtime_output option - splitcc: Emit empty subtitle frames instead of repeating - New commit: Replace deprecated enum values New in V18 - rebased to latest head (to fix merge conflict in allfilters.c) - textmod: Removed two trailing blanks from sf_textmod.c - xsubdec: fix xsubdec regression found by Michael; packet size estimation as flawed - splitcc: Fix uninitialzed locals - splitcc: Do not propagate hwcontext on secondary output - testsub2video: fix output timeing, re-add mutex locking and implement render_latest_only parameter New in V17 - Applied almost all suggestions that were made (thanks everybody!) - Split the initial commit, no longer moving the legacy structs (AVSubtitle..) - Use the regular encoding API for subtitles - Updated subtitle encoders to handle packet allocation - Removed compatibility layer for accessing legacy encoders - Updated the compatibility implementation for the legacy subtitle encode api - Reordered commits and merged ffmpeg.c changes into a single commit New in V16 - Added missing reference to subfmt.h iun libzvbi-teletextdec.c - Fixed Fate error in patch 15/16 - Removed all trsiling whitespace in tilers.texi New in V15 - Rebased to upstream changes - avcodec/subtitles: Migrate subtitle encoders to frame-based API and provide compatibility shim for legacy api - fftools/ffmpeg: Use new frame-based subtitle encoding API - AVSubtitleArea: copy flags field, make params const - graphicsubs2text: Don't emit duplicate frames - graphicsubs2text: Combined OCR output into a single AVSubtitleArea (I have a prototype for detecting text colors and positions, but it's not ready at this point) - splitcc: cleanup local subtitle_header ref - stripstyles: add parameter for ass layer selection - avcodec/subtitles: deferred loading of ass header for text subtitle encoders - verified all example command lines in the docs are working, added somre more Kind regards, softworkz softworkz (21): avcodec,avutil: Move enum AVSubtitleType to avutil, add new and deprecate old values avutil/frame: Prepare AVFrame for subtitle handling avcodec/subtitles: Introduce new frame-based subtitle decoding API avfilter/subtitles: Update vf_subtitles to use new decoding api avcodec,avutil: Move ass helper functions to avutil as avpriv_ and extend ass dialog parsing avcodec/subtitles: Migrate subtitle encoders to frame-based API and provide a compatibility shim for the legacy api avcodec/subtitles: Replace deprecated enum values fftools/play,probe: Adjust for subtitle changes avfilter/subtitles: Add subtitles.c for subtitle frame allocation avfilter/avfilter: Handle subtitle frames avfilter/sbuffer: Add sbuffersrc and sbuffersink filters avfilter/overlaygraphicsubs: Add overlaygraphicsubs and graphicsub2video filters fftools/ffmpeg: Replace sub2video with subtitle frame filtering and use new frame-based subtitle encoding API avfilter/avfilter: Fix hardcoded input index avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video filters avfilter/textmod: Add textmod, censor and show_speaker filters avfilter/stripstyles: Add stripstyles filter avfilter/splitcc: Add splitcc filter for closed caption handling avfilter/graphicsub2text: Add new graphicsub2text filter (OCR) avfilter/subscale: Add filter for scaling and/or re-arranging graphical subtitles doc/APIchanges: update for subtitle filtering changes configure | 7 +- doc/APIchanges | 23 + doc/filters.texi | 756 +++++++++++++++ fftools/ffmpeg.c | 588 ++++++------ fftools/ffmpeg.h | 15 +- fftools/ffmpeg_filter.c | 236 +++-- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- fftools/ffplay.c | 102 +- fftools/ffprobe.c | 48 +- libavcodec/Makefile | 56 +- libavcodec/ass.h | 147 +-- libavcodec/assdec.c | 4 +- libavcodec/assenc.c | 92 +- libavcodec/avcodec.h | 32 +- libavcodec/ccaption_dec.c | 19 +- libavcodec/codec_desc.c | 11 + libavcodec/codec_desc.h | 8 + libavcodec/decode.c | 56 +- libavcodec/dvbsubdec.c | 2 +- libavcodec/dvbsubenc.c | 96 +- libavcodec/dvdsubdec.c | 2 +- libavcodec/dvdsubenc.c | 100 +- libavcodec/encode.c | 63 +- libavcodec/internal.h | 16 + libavcodec/jacosubdec.c | 2 +- libavcodec/libaribb24.c | 2 +- libavcodec/libzvbi-teletextdec.c | 14 +- libavcodec/microdvddec.c | 7 +- libavcodec/movtextdec.c | 3 +- libavcodec/movtextenc.c | 124 ++- libavcodec/mpl2dec.c | 2 +- libavcodec/pgssubdec.c | 2 +- libavcodec/realtextdec.c | 2 +- libavcodec/samidec.c | 2 +- libavcodec/srtdec.c | 2 +- libavcodec/srtenc.c | 112 ++- libavcodec/subviewerdec.c | 2 +- libavcodec/tests/avcodec.c | 2 - libavcodec/textdec.c | 4 +- libavcodec/ttmlenc.c | 111 ++- libavcodec/utils.c | 182 ++++ libavcodec/version.h | 2 +- libavcodec/webvttdec.c | 2 +- libavcodec/webvttenc.c | 90 +- libavcodec/xsubdec.c | 2 +- libavcodec/xsubenc.c | 87 +- libavfilter/Makefile | 15 + libavfilter/allfilters.c | 13 + libavfilter/avfilter.c | 30 +- libavfilter/avfilter.h | 11 + libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 54 ++ libavfilter/buffersink.h | 7 + libavfilter/buffersrc.c | 72 ++ libavfilter/buffersrc.h | 1 + libavfilter/formats.c | 22 + libavfilter/formats.h | 3 + libavfilter/internal.h | 19 +- libavfilter/sf_graphicsub2text.c | 354 +++++++ libavfilter/sf_splitcc.c | 378 ++++++++ libavfilter/sf_stripstyles.c | 196 ++++ libavfilter/sf_subscale.c | 883 ++++++++++++++++++ libavfilter/sf_textmod.c | 697 ++++++++++++++ libavfilter/subtitles.c | 63 ++ libavfilter/subtitles.h | 44 + libavfilter/vf_overlaygraphicsubs.c | 742 +++++++++++++++ libavfilter/vf_overlaytextsubs.c | 643 +++++++++++++ libavfilter/vf_subtitles.c | 54 +- libavutil/Makefile | 4 + {libavcodec => libavutil}/ass.c | 87 +- libavutil/ass_internal.h | 133 +++ {libavcodec => libavutil}/ass_split.c | 30 +- .../ass_split_internal.h | 32 +- libavutil/frame.c | 211 ++++- libavutil/frame.h | 78 +- libavutil/subfmt.c | 45 + libavutil/subfmt.h | 115 +++ libavutil/version.h | 3 +- tests/ref/fate/filter-overlay-dvdsub-2397 | 181 ++-- tests/ref/fate/sub-dvb | 162 ++-- tests/ref/fate/sub2video | 220 ++--- tests/ref/fate/sub2video_basic | 135 +-- tests/ref/fate/sub2video_time_limited | 4 +- 84 files changed, 7572 insertions(+), 1416 deletions(-) create mode 100644 libavfilter/sf_graphicsub2text.c create mode 100644 libavfilter/sf_splitcc.c create mode 100644 libavfilter/sf_stripstyles.c create mode 100644 libavfilter/sf_subscale.c create mode 100644 libavfilter/sf_textmod.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/vf_overlaygraphicsubs.c create mode 100644 libavfilter/vf_overlaytextsubs.c rename {libavcodec => libavutil}/ass.c (65%) create mode 100644 libavutil/ass_internal.h rename {libavcodec => libavutil}/ass_split.c (94%) rename libavcodec/ass_split.h => libavutil/ass_split_internal.h (86%) create mode 100644 libavutil/subfmt.c create mode 100644 libavutil/subfmt.h