From patchwork Sat Sep 18 03:52:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30340 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp2791274iov; Fri, 17 Sep 2021 20:52:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyo0sSRh5/B6ekgYCs+ak5pahnyWnq+n1X8RuuGPBTFmr3zDBmjr9Mmz9I5oCIs7/9r8g9V X-Received: by 2002:a17:906:39cb:: with SMTP id i11mr16653815eje.168.1631937156575; Fri, 17 Sep 2021 20:52:36 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f21si9034111eds.595.2021.09.17.20.52.36; Fri, 17 Sep 2021 20:52:36 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=OAeNyvsQ; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 124C668B1B7; Sat, 18 Sep 2021 06:52:33 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11olkn2070.outbound.protection.outlook.com [40.92.18.70]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 12A4068AC04 for ; Sat, 18 Sep 2021 06:52:26 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZSzCZAMjMzsDU1ItoY7NIVnFYn3FuQxfAKs8URQXEfXM7P8xvYlSIev9DbSY1NRuZpNPNLjIFGNwSxd9yh+dwesM/lZQ1YnG2QssSN2Qt3BcXmVK2J88lpTTzNLJlt9Pg9db7QdqhcQtTxVS7fFcisy9ncJWtXQUF3m42Z9xOtJPSHV70y5VGFLXzd/fQ6xdJnCU1gu6y3y6bFzhInXlu54aAhlFpEJ6CQKNIEQouDXZ751s/XeFEHKOCI/cxxEanmODng47o1a3pN32RmGpZFiCqGMm6zMMcf6w6CxMgplCYIV8RLatfE0/Ed0FVd7eEDhcCfLsFY5lsQqa7wNK2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=9IWxXCOleWhI55hNbknLmhbPZKQDM9jIaSwTAZpdudo=; b=WHpMQQEnjuxnN93JMOViTJE7AJ22jqmz0gKRi7Xs6kYSxl5q58491xtw0ltyWngTc4MYzH8gdNBbVVRKCMUYKEjz3R31Dla7hmBxp6AFwj5n0TZLbxhAdUuYNsEt/AOsvh6VkGCYaMikHqncisZMKYnH2PFBEQhzbqCr1e1NU+GyP5tBamL8upIsQu6hkbimYNtBXqKERxnKLNsvlk63Yr6LDmu0m+bGboTqG6e9mvTfxPNaQad+Doq9G0Is+1NEmJzJxs8EJst3O1jghJLd1VGnVnYeK9Tzxu1YjtA4tw4iwZidR0XLOmebhCRJ25JAouwNhXffa3sXuOBTVIgtmQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9IWxXCOleWhI55hNbknLmhbPZKQDM9jIaSwTAZpdudo=; b=OAeNyvsQTTiLyJNRjNNlSX72m411ZuoqEjjEsyCNyC4kAuz8AeDQrPMdYkJLS3LmpX/0t3bj9ejRsVFd0zJgKCMXhd+Kx8nBqjIT6uWS6v+BzdVArygdsHJGSBF5UAGDr2q+V9EwnMsgXYvbOwMqy7/a3oz0+0HtTZ/OOzs693pE9Ohb8wmhmNx9RtbHvLAO0DOaP2cQ/gVmDXz6nBUkdk/f2X9ID9zXjeETRmlv4KHVJ633Egw/A6eBfmHu1YgfSXfaLR4zeOtfZn9VSgzvJHGDY55tJRc2oLc9j6HsB4J8w6L2zbnHFUs9gr30Prw0is79vAkFoJBbdc3SbPxQfw== Received: from BYAPR04MB5976.namprd04.prod.outlook.com (2603:10b6:a03:f2::32) by BY5PR04MB6865.namprd04.prod.outlook.com (2603:10b6:a03:219::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4523.16; Sat, 18 Sep 2021 03:52:23 +0000 Received: from BYAPR04MB5976.namprd04.prod.outlook.com ([fe80::ecc6:937b:3740:1816]) by BYAPR04MB5976.namprd04.prod.outlook.com ([fe80::ecc6:937b:3740:1816%6]) with mapi id 15.20.4523.015; Sat, 18 Sep 2021 03:52:23 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v7 00/12] Subtitle Filtering Thread-Index: AdesQJUc2P1Mcyc6TqeNRwn6Xk9cqw== Date: Sat, 18 Sep 2021 03:52:23 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [8LXLIQz3V3LwYSXXJ4ooris9bqOzXP5m] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a5e54c4c-c731-48b8-1692-08d97a57b91b x-ms-traffictypediagnostic: BY5PR04MB6865: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: vSZ19J//s6hSaRfUXvhJGWMYPQB8kk7MRNyIMuNu8AjI6c9T/yBAVym/9lKEoC08ymfhsoU+nE0y2hMOCWDUOA4hAAgEg8nzhvUQ+fk/4ddLC9BQWsfMJnD7wEuczv+jcBMuK7gtx7CiV1HLb3jkPMTXQn9Y5Idw3ixe0B1YKRVRI1es0gSghdCmExYOWFpFnGnNED49+bFKHriH866L4NSoZfTKz22geYdMhFx2aaK3H+oEt7Dqw5GpX8wzqhmY8I6ltY+U9oPMvKZmTW54zD3Q1tCF6zEnBTzlFSe10SsCzzQsD1oWi1+ISNR2VRJjkMeNMzthNs0vpAgexOX+7x4Ki8BVMosKzybmM6uYtRO3X3j4D3/I07Ya6A80WI+O8C5isR1wUf1EMn//FEoTRf6TkG40xMPd9XqXnVe5qTlhkiyIHKY+rPb8e4KUm5x1 x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: fIASvVsG8GR8CVP7j9HZJO7y4wNj170t+rwCKq6easbMNaCswrgIS9XkUANCLI01Gh7AxAtF+hAJNiipRPrcwZlbsY6wDJljDFodPtMt7Dx+JrRvoq62BbZY8VhX0VKL3TrXkdbkm8snvW1ZPdjiqA== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR04MB5976.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: a5e54c4c-c731-48b8-1692-08d97a57b91b X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Sep 2021 03:52:23.3370 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR04MB6865 Subject: [FFmpeg-devel] [PATCH v7 00/12] Subtitle Filtering X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: UVX9gDZvy2Pn v7 Update: - Added new subtitle filter: split_cc This new filter fills a number of current feature holes: - The subcc hack is no longer required. - It's not required anymore to use a movie input container as input for getting caption data. - It's possible to overlay ("burn-in") closed captions onto a video in a single turn - The processed video can be a live stream (unlike the vf_subtitles filter, the new ones do not need a separate and individual connection for accessing the streams). - Patches 01/12 to 11/12: Almost unchanged, no need to re-review, only 12/12 is new. v6 Update: - Implements all given feedback from reviews (thanks!) - Add AVSubtitle fields to AVFrame, keeping AVSubtitle unchanged for compatibility and future deprecation - Use subtitle frames internally and for filtering - Retain compatibility with legacy subtitle implementation through a number of compatibility conversion methods - Reordered and reorganized commits, each one passing FATE now - Document new public API functions - Many more detail changes v5 Update: - Merge AVSubtitle into AVFrame - Move FATE test adjustments to corresponding commit - Move documentation updates to corresponding filter commits - Remove mediatype parameter from av_frame_get_buffer2 (still need some advice for splitting the commits in a way that every single one will pass FATE) v4 Update: - Re-Sending due to Patchwork having failed to parse my patchset There seems to be a bug in Patchwork when parallel processing is enabled. This time, I'll send the e-mails slowly, one after another. v3 Update: - Reworked, revised, rebased, reorganized, refactored - No more prototype/test-style code - Additional subtitle filters added - Filter documentation added - Adjusted FATE tests This patchset is about introducing filtering support for subtitles. The current sub2video "hack" implementation is slow, ineffective and limited in capabilities. => This patchset introduces true subtitle filtering These new filters are included: - overlay_graphicsubs (VS -> V) Overlay graphic subtitles onto a video stream - graphicsub2video {S -> V) Converts graphic subtitles to (transparent) video frames - overlay_textsubs {VS -> V) Overlay text subtitles onto a video stream. - textsubs2video {S -> V) Converts text subtitles to video frames - textmod {S -> S) Modify subtitle text in a number of ways - stripstyles {S -> S) Remove all inline styles from subtitle events Regards, softworkz softworkz (11): global: Prepare AVFrame for subtitle handling fftools/play,probe: Adjust for subtitle changes avfilter/subtitles: Add subtitles.c for subtitle frame allocation avfilter/avfilter: Handle subtitle frames avfilter/sbuffer: Add sbuffersrv and sbuffersink filters avfilter/overlay_graphicsubs: Add overlay_graphicsubs and graphicsub2video filters fftools/ffmpeg: Replace sub2video with subtitle frame filtering avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters avfilter/textmod: Add textmod filter avcodec/ass_split: Extend ass dialog parsing avfilter/stripstyles: Add stripstyles filter softworkz (12): global: Prepare AVFrame for subtitle handling fftools/play,probe: Adjust for subtitle changes avfilter/subtitles: Add subtitles.c for subtitle frame allocation avfilter/avfilter: Handle subtitle frames avfilter/sbuffer: Add sbuffersrv and sbuffersink filters avfilter/overlay_graphicsubs: Add overlay_graphicsubs and graphicsub2video filters fftools/ffmpeg: Replace sub2video with subtitle frame filtering avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters avfilter/textmod: Add textmod filter avcodec/ass_split: Extend ass dialog parsing avfilter/stripstyles: Add stripstyles filter avfilter/split_cc: Add split_cc filter for closed caption handling configure | 4 +- doc/filters.texi | 285 +++++++++ fftools/ffmpeg.c | 460 ++++++-------- fftools/ffmpeg.h | 14 +- fftools/ffmpeg_filter.c | 198 ++++-- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- fftools/ffplay.c | 50 +- fftools/ffprobe.c | 49 +- libavcodec/ass_split.c | 12 +- libavcodec/ass_split.h | 2 + libavcodec/avcodec.c | 19 - libavcodec/avcodec.h | 105 ++-- libavcodec/decode.c | 24 +- libavcodec/pgssubdec.c | 1 + libavcodec/utils.c | 11 + libavfilter/Makefile | 10 + libavfilter/allfilters.c | 17 +- libavfilter/avfilter.c | 30 +- libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 63 ++ libavfilter/buffersink.h | 15 + libavfilter/buffersrc.c | 72 +++ libavfilter/buffersrc.h | 1 + libavfilter/formats.c | 14 + libavfilter/formats.h | 3 + libavfilter/internal.h | 1 + libavfilter/sf_split_cc.c | 271 ++++++++ libavfilter/sf_stripstyles.c | 211 +++++++ libavfilter/sf_textmod.c | 374 +++++++++++ libavfilter/subtitles.c | 61 ++ libavfilter/subtitles.h | 44 ++ libavfilter/version.h | 2 +- libavfilter/vf_overlay_graphicsubs.c | 727 ++++++++++++++++++++++ libavfilter/vf_overlay_textsubs.c | 615 ++++++++++++++++++ libavfilter/vf_subtitles.c | 28 +- libavformat/utils.c | 1 + libavutil/Makefile | 2 + libavutil/frame.c | 185 +++++- libavutil/frame.h | 93 ++- libavutil/subfmt.c | 221 +++++++ libavutil/subfmt.h | 186 ++++++ libavutil/version.h | 2 +- tests/ref/fate/filter-overlay-dvdsub-2397 | 181 +++--- tests/ref/fate/sub-dvb | 162 ++--- tests/ref/fate/sub2video | 44 -- tests/ref/fate/sub2video_basic | 93 ++- tests/ref/fate/sub2video_time_limited | 4 +- 48 files changed, 4215 insertions(+), 762 deletions(-) create mode 100644 libavfilter/sf_split_cc.c create mode 100644 libavfilter/sf_stripstyles.c create mode 100644 libavfilter/sf_textmod.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/vf_overlay_graphicsubs.c create mode 100644 libavfilter/vf_overlay_textsubs.c create mode 100644 libavutil/subfmt.c create mode 100644 libavutil/subfmt.h