From patchwork Thu Sep 16 20:30:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30279 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp1581659iov; Thu, 16 Sep 2021 13:30:16 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwK+Dof8NCsVAq1sBAmMpLbtew1UcsHGCHL3xDVERbJ5m40fZDnebgOi7Etm08ZdGf4R2eV X-Received: by 2002:a17:906:3f95:: with SMTP id b21mr8005681ejj.368.1631824216503; Thu, 16 Sep 2021 13:30:16 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id n21si4409505edt.292.2021.09.16.13.30.15; Thu, 16 Sep 2021 13:30:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=WAUSiCfz; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 910F568B092; Thu, 16 Sep 2021 23:30:11 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11olkn2053.outbound.protection.outlook.com [40.92.18.53]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6F3F468AF55 for ; Thu, 16 Sep 2021 23:30:04 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dSnCCaoAWd/fubZhhkEEAuFQK9S3pGqPqL3aWBrXrR3rscIcEPhfQSr7Jsnviy2cAciXCXaK0YD5LDlot1+Cmx861cc622kk2Kw+zpmJxz0eFrd5vfKUBJlx8wRjoYCCOalTI2iiNTVu42HQX8ILt/sJ0uAtzUvDrunAOGi9ElqUMXNrDVNeIlwgWpBKgZmrvKAHdCalo0w0aPkyfyTT1PguJzExafGaMJlOiZTEaeQZt8HcTVc6hbnipP9eYWKcIeFwfs/MI1AW6elp7dPJ2YSox4xdA3yArNCKKxSviUXPAqY76aDnZB/8KnXGFbtYb2fDKVoF9DNaCZ96M58JTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=y8uuwizwjhwHGG3DS5577FFQMI0yFQmoHksklt4jlaI=; b=mxKYBRFY+K2WaikAsp8Xh2PAXQkNnJlZ4dt7yHDuUZVynX6s4Jigv2F/mxqp5Dw3qDWenu7beoxlGhhBvQIhAX4IaRLO4ewCTXnTNEm3fIpRBsEMkcpU8GyrB5Qc+nlFdJ3+6L5EeYLP0Sm3DYfwbVWLBVZL5Gb9LDvSkMnlBMv48M5TBET66MQHzpcMWEg5jbA3zQjemvR+kp/r8KM2ndUIq+09eIEdjce+o0s2IDBwCJ+CiBwTfv1iza9AdYcT2TZ27gtKj92+rV5O+LJz3S4GLUB7ryG4UUt3MiA62G2l8vH4RoUUq03jUn31IASrEcPY6ShqastvG3IONi9KuQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y8uuwizwjhwHGG3DS5577FFQMI0yFQmoHksklt4jlaI=; b=WAUSiCfzcw7z80t0kaS4qE3vSK9xgs/Uo5S6tTecJoCExvRoMZdcYPvC8xFAWXoMMDHCgYDXIy8Wz27tkfIhVs5A4yyE00sb5ahJWPuSz+MM1tOOWH1ZmJfCEsQgTUMDT0+GjHK65LhyWhQPl20v2e8wjLSCcYQsB2vgNZvgyBVBgRnatDG9Q1WG2n9XbZFGuLQeqPfk2ZasfWr/h9BFbhC3qG8HiY+qOpr0a/wefnc9w4gWNkxBoe47+6K7Q0JBNBrAuw6VDLuu6jyF01Kc4SoG9MmJBZUU7a18EXzgNKiw2zAm3mWXe6XI5kr56PVI9zElTDWpE82EfOcYEf6xbg== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB6240.namprd04.prod.outlook.com (2603:10b6:208:e2::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.17; Thu, 16 Sep 2021 20:30:00 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4523.016; Thu, 16 Sep 2021 20:30:00 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v6 00/11] Subtitle Filtering Thread-Index: AQHNVlqDXyQ9eYanUWQRGGneUn5kPg== Date: Thu, 16 Sep 2021 20:30:00 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [LQb8BFPah/wfBdd4kSgXVVEP2iu9+ExR] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e25f8b6f-d757-4273-5d8f-08d97950c20a x-ms-traffictypediagnostic: MN2PR04MB6240: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 4aaoMicg448Jt4j4fjHe+UqUblN+QJm6Zh8P9+rSKzRMDV0HGAiQlWKuipJWETWWKl6PyBMHrKjZ0a8TbWWOEbralO+YBA62yJ77BgACzSdfrmZq0YuzHrNU5hRbw5gakJKR6zvdhyPsF5qTtBjAL8dTnrYPF1em69+VY08PWHPRAhNk9198iwvH6eIpX0BQzi5UpTjmfkdHEArO0NHF8HFS3wlv5lgsg4RoIqxpVQKTGzvk0AgD06Z37pgNuQ9ucZRCACmFEuT47kd1zz1mxO5nKC6PY7xLEtUgioYHiAi/ORcAu0XN5gNNPa+W+TdNlYza4wkPHf6rYVUvV7ZAgCS3ifeZd6E416/scTt/xLPbTQUqJ2sL40DMWxSCRJnlgbiF7wIdQvhQmk38mN0q3btcfJ8mWqjvmABmStSmigIPuScW+MlQTyzMDcKfybA+ x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: xvGfOmOQ34HxfqsCyzZVNS1e2yivxQpMyncV0AU+J4PYaLT0NuGQW0QrD8aZQH6U8A4IijrjyyrXaFqxwRqV+e/YVpJaCwPp43lp0OykxbqI2ahBkiCL+gVwc/oBLVwuq/T+BR4q3uCkIm6cnvBryg== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: e25f8b6f-d757-4273-5d8f-08d97950c20a X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Sep 2021 20:30:00.8350 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB6240 Subject: [FFmpeg-devel] [PATCH v6 00/11] Subtitle Filtering X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: xEaVTdwmLYDg v6 Update: - Implements all given feedback from reviews (thanks!) - Add AVSubtitle fields to AVFrame, keeping AVSubtitle unchanged for compatibility and future deprecation - Use subtitle frames internally and for filtering - Retain compatibility with legacy subtitle implementation through a number of compatibility conversion methods - Reordered and reorganized commits, each one passing FATE now - Document new public API functions - Many more detail changes v5 Update: - Merge AVSubtitle into AVFrame - Move FATE test adjustments to corresponding commit - Move documentation updates to corresponding filter commits - Remove mediatype parameter from av_frame_get_buffer2 (still need some advice for splitting the commits in a way that every single one will pass FATE) v4 Update: - Re-Sending due to Patchwork having failed to parse my patchset There seems to be a bug in Patchwork when parallel processing is enabled. This time, I'll send the e-mails slowly, one after another. v3 Update: - Reworked, revised, rebased, reorganized, refactored - No more prototype/test-style code - Additional subtitle filters added - Filter documentation added - Adjusted FATE tests This patchset is about introducing filtering support for subtitles. The current sub2video "hack" implementation is slow, ineffective and limited in capabilities. => This patchset introduces true subtitle filtering These new filters are included: - overlay_graphicsubs (VS -> V) Overlay graphic subtitles onto a video stream - graphicsub2video {S -> V) Converts graphic subtitles to (transparent) video frames - overlay_textsubs {VS -> V) Overlay text subtitles onto a video stream. - textsubs2video {S -> V) Converts text subtitles to video frames - textmod {S -> S) Modify subtitle text in a number of ways - stripstyles {S -> S) Remove all inline styles from subtitle events Regards, softworkz softworkz (11): global: Prepare AVFrame for subtitle handling fftools/play,probe: Adjust for subtitle changes avfilter/subtitles: Add subtitles.c for subtitle frame allocation avfilter/avfilter: Handle subtitle frames avfilter/sbuffer: Add sbuffersrv and sbuffersink filters avfilter/overlay_graphicsubs: Add overlay_graphicsubs and graphicsub2video filters fftools/ffmpeg: Replace sub2video with subtitle frame filtering avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters avfilter/textmod: Add textmod filter avcodec/ass_split: Extend ass dialog parsing avfilter/stripstyles: Add stripstyles filter configure | 4 +- doc/filters.texi | 261 ++++++++ fftools/ffmpeg.c | 459 ++++++-------- fftools/ffmpeg.h | 14 +- fftools/ffmpeg_filter.c | 198 ++++-- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- fftools/ffplay.c | 50 +- fftools/ffprobe.c | 49 +- libavcodec/ass_split.c | 12 +- libavcodec/ass_split.h | 2 + libavcodec/avcodec.c | 19 - libavcodec/avcodec.h | 105 ++-- libavcodec/decode.c | 24 +- libavcodec/pgssubdec.c | 1 + libavcodec/utils.c | 11 + libavfilter/Makefile | 9 + libavfilter/allfilters.c | 16 +- libavfilter/avfilter.c | 30 +- libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 63 ++ libavfilter/buffersink.h | 15 + libavfilter/buffersrc.c | 72 +++ libavfilter/buffersrc.h | 1 + libavfilter/formats.c | 14 + libavfilter/formats.h | 3 + libavfilter/internal.h | 1 + libavfilter/sf_stripstyles.c | 211 +++++++ libavfilter/sf_textmod.c | 372 +++++++++++ libavfilter/subtitles.c | 61 ++ libavfilter/subtitles.h | 44 ++ libavfilter/version.h | 2 +- libavfilter/vf_overlay_graphicsubs.c | 727 ++++++++++++++++++++++ libavfilter/vf_overlay_textsubs.c | 613 ++++++++++++++++++ libavfilter/vf_subtitles.c | 28 +- libavformat/utils.c | 1 + libavutil/Makefile | 2 + libavutil/frame.c | 184 +++++- libavutil/frame.h | 93 ++- libavutil/subfmt.c | 221 +++++++ libavutil/subfmt.h | 186 ++++++ libavutil/version.h | 2 +- tests/ref/fate/filter-overlay-dvdsub-2397 | 181 +++--- tests/ref/fate/sub-dvb | 162 ++--- tests/ref/fate/sub2video | 44 -- tests/ref/fate/sub2video_basic | 93 ++- tests/ref/fate/sub2video_time_limited | 4 +- 47 files changed, 3912 insertions(+), 762 deletions(-) create mode 100644 libavfilter/sf_stripstyles.c create mode 100644 libavfilter/sf_textmod.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/vf_overlay_graphicsubs.c create mode 100644 libavfilter/vf_overlay_textsubs.c create mode 100644 libavutil/subfmt.c create mode 100644 libavutil/subfmt.h