From patchwork Thu Sep 30 19:30:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30807 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp984627iob; Thu, 30 Sep 2021 12:30:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzC3JUck6bpEQ0zIHsLPCHrzMucrf6EpB827sAPPOIvQZOtJt55klUbnkJL3yweRZRZpSV2 X-Received: by 2002:a05:6402:44c:: with SMTP id p12mr9495791edw.145.1633030241684; Thu, 30 Sep 2021 12:30:41 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id p7si4304649ejb.33.2021.09.30.12.30.41; Thu, 30 Sep 2021 12:30:41 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=RKBL0YkX; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6444D68A9FC; Thu, 30 Sep 2021 22:30:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam07olkn2080.outbound.protection.outlook.com [40.92.15.80]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AE4F868A867 for ; Thu, 30 Sep 2021 22:30:30 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=U4YGe8odwUCXflEyYaM4Dw0lyDqcGgCq5TuUeE+VvZoEe53UnAVaYJwv5pUCsrMrpWqqLAz6EKgtzRgEAGF1G91vgubXfMIc1xVMY4gNC6q9Ex/KXdAssNgIR6hhFH19kLtP6jY/F8fLh8fLG5xKJQQdEZ5kW6cDOKQSBn3Pd6PggjocfWRQcR50bMWXMx/nC3U9+JETP3cYgzAq6sklVFi4r/PvG7p6dNY72u7Tdc1quq/Sdyon5RZHxfqAX/D+IzlXf/44x/RDAqmLbLz7crw8kTixeyyGMX9Pnwb65FjWqQCCdSJBSY0MhX2bSmHW+ROeaDmiPo6MoVMYrQ73tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=rhCqlBX6GiJ/y2o4aeUJXnYV7aTLG5Q3KTogrRdHbJc=; b=ZRjIM5mfwNE+7PXUlNcT7mKqmWRBuMcwNjD6ysCKCTo+h7Zk01tSBXcKeFk461YyrzUHX0pAfzkGSd95aYMUjshe18ZUTFHHXCGWIAkn0kuFG8rlf+dCl4mveYbZs9B9i6qRxVgA1isPZhZyiFgVlIXOmj/tiQNv4Tz/xAeqoTN7Scpkjjn7TLzEZCVNItK3KOGYNG1J0UEiK59F1J5YhYwR+OcEpQ4KbmBBhMjPysveCgIsR6qJlQm512a4WmeYWhVa8gShVz/JcKIH235NZZ3bdHg1qzLPki46h/RMjEKFSAnOOqd0U5CsUAeH2OtbECDbFHzIriR2DCcTlE7hDg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rhCqlBX6GiJ/y2o4aeUJXnYV7aTLG5Q3KTogrRdHbJc=; b=RKBL0YkXQxHGQMyDk5L9H3azSflj8LUYBzRpdfEmuXAyJB+J2+4n+HAXfxW8xK5YDYs/FiE2NULb25RdijkqnleQHx4Hebq+FZSsAJqegs6nCJnQ5762YNHohZdDKUHjbGnm85n6BFvpatcgNRlALifUYmmZ0JFFbb1neOVS0cPjAX3rVsUvDEwpA9cBqAF3S/M1oMOay7u0AvLqbgSyYRNgj3cZByucYXnR+iRDLRx6xEJD08AR6cCV+Vmu/PizwHkJzfuY5pixyFFR+8zol1gKwEqvrQdGxW7bii2M4L+zhEoOfJJk8PJ7RHbKdMZ/WrgjlZRRRhRppd1fksd/Zw== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB6159.namprd04.prod.outlook.com (2603:10b6:208:e6::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.14; Thu, 30 Sep 2021 19:30:28 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:30:28 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 00/14] Subtitle Filtering Thread-Index: AQDjKBOTrk2qctGPCnRCkCoFUAc7JQ== Date: Thu, 30 Sep 2021 19:30:27 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [/NQikokGw6QbEfOlAAH1kR8NLmIerLTg] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: faa47359-482a-43c7-df95-08d98448c23e x-ms-traffictypediagnostic: MN2PR04MB6159: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: psh8rgyXrEeArOjUSczoc67a0w71h4ilabsKobHzAwXL/k2gk6oxI01GFkkW5Wr2H0B9wLEglf20YD/Kl6GEJJPyDY4RkpLANBK5eD5BX7XHtnELawajF9tA1pR2DmtBE/VWETDRrw7KxMWwLefoNJcWLia/aOiRvDYMo/et9RDD6aD2N+1nlGCfGlY85QWI2k94Qr/Fjw4rusZk70PmAfLPchYLFbnFmnDO0u8hyWuld+RMgLKYF/ChQLvvA35Yq9FxU6veREqrEmAHzKta6ROO9kIDvbtvLSqyBBw0lQWpLqJ0ypNuS6irE+UGb0rOKbWZI/BAOcH/MNxY24xO581QGbdcKjIgPX60A5gFkRuvlnTa3v3h5Qmzf4K/IxYZboGIxG+2s79WbvH2oAdRRxWaBoqDbfgzwdlHzlY99ExA9u6Go4QTNkFW/bCZg7J5 x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: SpAKaKMQ2jFgOr6HJCFknq2sA2y4CpQYzTS3NIqqlCXI7IYjX9zdFkJjTrTuHNhV2pU+9LqTXX4V7PT/UeNouJKLQfxw1em7CdIyg8RYokFGbR5hWdmBHdavN/D/hAY0s2hWcAYgqFRtAvXLM5lk4Q== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: faa47359-482a-43c7-df95-08d98448c23e X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:30:27.9878 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB6159 Subject: [FFmpeg-devel] [PATCH v10 00/14] Subtitle Filtering X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: I1fVSiAN9+Py v10 Update: - FEATURE-COMPLETE! No more features or filters will be added to this patchset - Added subscale filter Provides high-quality scaling and rearranging functionality for graphical subtitles. This has been the last missing bit for making this a useful and versatile toolset for subtitle manipulation - The common ass utility functions remain to reside in avutil, but in this version, they are marked as avpriv_ and no longer public like in the previous version. - A number of detail fixes are included, but there haven't been significant changes to the core implementation in the past few versions. I'm quite confident about the current state. - As a final step I'll be going to create a wiki page presenting a comprehensive collection of usage examples, showing off the whole range of new capabilities and also providing proof that it is working universally in a wide range of use cases. v9 Update: - Addressed all review comments (thanks Andreas!) - Move ass utility functions to avutil as public API - Handle subtitle decoding via new decode API (removed av_decode_subtitle3) - Change 'subtitle_header' to use refcounted buffers - Allow mapping subtitle streams to filter graph and direct outputs simultaneously - Added explanation and justification for test changes to commit message -------------------------------------------------------------- This patchset introduces filtering support for subtitles. Besides the ground work, this patchset includes a range of new filters for subtitle processing: - subscale (S -> S) Provides high-quality scaling and rearranging functionality for graphical subtitles. - graphicsubs2text (S -> S) Convert graphic subtitles to text subtitles via OCR - censor {S -> S) Allows censoring word in subtitles by a configurable word list - show_speaker {S -> S) Prepend the speaker names to text lines (those are often available in ass subs) - split_cc (V -> VS) Splits out Closed-Caption data as a subtitle stream for filtering and encoding - overlay_graphicsubs (VS -> V) Overlay graphic subtitles onto a video stream - graphicsub2video {S -> V) Converts graphic subtitles to (transparent) video frames - overlay_textsubs {VS -> V) Overlay text subtitles onto a video stream. - textsubs2video {S -> V) Converts text subtitles to video frames - textmod {S -> S) Modify subtitle text in a number of ways - stripstyles {S -> S) Remove all inline styles from subtitle events Regards, softworkz softworkz (14): fftools/play,probe: Adjust for subtitle changes avfilter/subtitles: Add subtitles.c for subtitle frame allocation avfilter/avfilter: Handle subtitle frames avfilter/sbuffer: Add sbuffersrc and sbuffersink filters avfilter/overlay_graphicsubs: Add overlay_graphicsubs and graphicsub2video filters fftools/ffmpeg: Replace sub2video with subtitle frame filtering avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters avfilter/textmod: Add textmod, censor and show_speaker filters avfilter/stripstyles: Add stripstyles filter avfilter/split_cc: Add split_cc filter for closed caption handling avfilter/graphicsub2text: Add new graphicsub2text filter (OCR) avfilter/subscale: Add filter for scaling and/or re-arranging graphical subtitles avformat/utils: Add av_stream_get_codec_properties() ffprobe: Fix incorrect display of closed_captions property configure | 7 +- doc/APIchanges | 3 + doc/filters.texi | 662 +++++++++++++++++ fftools/ffmpeg.c | 507 ++++++------- fftools/ffmpeg.h | 14 +- fftools/ffmpeg_filter.c | 209 ++++-- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- fftools/ffplay.c | 102 ++- fftools/ffprobe.c | 53 +- libavfilter/Makefile | 14 + libavfilter/allfilters.c | 13 + libavfilter/avfilter.c | 30 +- libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 63 ++ libavfilter/buffersink.h | 15 + libavfilter/buffersrc.c | 72 ++ libavfilter/buffersrc.h | 1 + libavfilter/formats.c | 14 + libavfilter/formats.h | 3 + libavfilter/internal.h | 1 + libavfilter/sf_graphicsub2text.c | 323 ++++++++ libavfilter/sf_split_cc.c | 298 ++++++++ libavfilter/sf_stripstyles.c | 211 ++++++ libavfilter/sf_subscale.c | 867 ++++++++++++++++++++++ libavfilter/sf_textmod.c | 719 ++++++++++++++++++ libavfilter/subtitles.c | 63 ++ libavfilter/subtitles.h | 44 ++ libavfilter/vf_overlay_graphicsubs.c | 724 ++++++++++++++++++ libavfilter/vf_overlay_textsubs.c | 609 +++++++++++++++ libavformat/avformat.h | 9 + libavformat/utils.c | 5 + libavformat/version.h | 2 +- tests/ref/fate/filter-overlay-dvdsub-2397 | 181 +++-- tests/ref/fate/sub-dvb | 162 ++-- tests/ref/fate/sub2video | 1 - tests/ref/fate/sub2video_basic | 93 ++- tests/ref/fate/sub2video_time_limited | 4 +- tests/ref/fate/ts-demux | 2 +- 39 files changed, 5472 insertions(+), 638 deletions(-) create mode 100644 libavfilter/sf_graphicsub2text.c create mode 100644 libavfilter/sf_split_cc.c create mode 100644 libavfilter/sf_stripstyles.c create mode 100644 libavfilter/sf_subscale.c create mode 100644 libavfilter/sf_textmod.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/vf_overlay_graphicsubs.c create mode 100644 libavfilter/vf_overlay_textsubs.c