From patchwork Thu Aug 19 07:43:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29609 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp64132iov; Thu, 19 Aug 2021 00:43:21 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzN0sujFRuSBBfckHT0QbaYOdD1N9/Dp4VtZhawsPuRwzvjOIrmlzMC3ATIwAvlrGwq/YcY X-Received: by 2002:a17:906:c182:: with SMTP id g2mr14037927ejz.507.1629359001605; Thu, 19 Aug 2021 00:43:21 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id gc22si2973494ejc.181.2021.08.19.00.43.20; Thu, 19 Aug 2021 00:43:21 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=fkmbhKki; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 42D65689963; Thu, 19 Aug 2021 10:43:17 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11olkn2099.outbound.protection.outlook.com [40.92.19.99]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EF5486801C5 for ; Thu, 19 Aug 2021 10:43:09 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=i5FR0h22HIXrGrYXkYGc8GAB4qCypVfcLRzLJAdtO5IYs/kPRZySBvO4Sl1DBLu+nMmWMvepti+llwxcMEY/YFkOMXAbavVBRVt2R5SPzVNV8qiXSc9vrAHJKjgp+/ZfFr7RxB/9ObEoMDUVtUEkqmsX8I6d9UMQydCen1/zLsxRBhwpdLBXsUgpJcJpEnHR046r4nnE5K2eJgtfgofwtToO4TImO8rLVyeVKpQAMHBcqywWD6SaMbZBrscsLx2LQL5UEITfr4rMxYoz2kqtbNQQIpMgf57E5TH73zcWMcedikdamlQhy6qQg309v2gLTmHcyM7SY98EleA3Gv2wow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9v18h+kmXyQ/VIbL+el41CpYx7FU7ZYrcKnTCSH8NJA=; b=RAqnVQMOZ14Y3IYzXP9FhgOuukDuvQ6gtJROuWmHS2Zj00zzSv/DMDRssOJsLWdHWNrTkBrNgBbd9ZYC1dY4WaDWyg35VHaI9YtB+dGngD/QmuKwssVmaxkPeqnsZOHCdtz5jj4q73KGv4ohry+1kk8EiIMLBLthsQVbWF5titYye+u0CSH1bNjmJ6ZxWMdXXWo6cHInHehXC6zaDMTHUZNtcxl9oz6lx8GUKrpVjAaMRj5hFLHqnpGgX8/IR2P0HwY2CyTvbQsRasrOyZr8kX4ebpJRTPAcS7yXA4utQgIQNOa7y4STBpNRwuh6DIuRF/G6A0bMRUaYUA73OfzMnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9v18h+kmXyQ/VIbL+el41CpYx7FU7ZYrcKnTCSH8NJA=; b=fkmbhKkibKkpsxrxc9Xq0ovUCmNgyQO46yKYIDlqTioF52CunoRPjkEwvRI0FFTiCqiwDwUA6c5YdIkz4CDXu2PKW+CtSvh6NQj3y6lcwlWzLo9MiLrEJ1bowGTnFRO2aPMA2FjqT8VhTcFKzJPWU4p8yAXHUTEyqOyLle6icSLE2F/t+9HXBO2FTf1TTCxs3kT19S9YzQK2V7LR4DNPbtJs9pWvQLN5fXk/I4feWbCjy1Tns2mXEqviN7SKBmWaKsokl3wKB+btkr9BBeenOwMV4763JHfZW7QQhq332pO+5dFGIw4og2Pgr2AKjZ23FFYtTqpNKLNmRWPaww0yAg== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB6255.namprd04.prod.outlook.com (2603:10b6:208:d5::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.22; Thu, 19 Aug 2021 07:43:06 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::5d83:1c26:c2b1:3a30]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::5d83:1c26:c2b1:3a30%6]) with mapi id 15.20.4415.024; Thu, 19 Aug 2021 07:43:06 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH 0/9] A New Take on the Immortal Sub2Video Code Thread-Index: AdeUzYRT3L5AgHOyRimghprZn0zxAw== Date: Thu, 19 Aug 2021 07:43:06 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [cqreKbasvIV8+CBy45L0HV+QacVYXW3B] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3ac3ff12-b4ba-40cd-2a98-08d962e4fbd8 x-ms-traffictypediagnostic: MN2PR04MB6255: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: FNYxkhs4oGa8mNlhboHCdO9otosxVlDhwkMypFw/6bkvxLwqng467hg0u78h6x8cFGOq+wlCd6OgOdlJ02Eh3Zq4YCtXntdiyYgemmLLNYRSU8bCYD+PZiXmBX2P6tlQdwgP+9zSDPqqHQfdVvSI2EGq6uq4uDtK3rRhOsa/yZC+0XnShKlr1RGgZJMTCJ1JgEBqSex+I2LojzEMaULpQbwEM7sGqOAIIxMKnixFow6xx6h9CAf7izQl6jsGoSNpGLtZOtXpiXO+v/c4A3c3QQYMGDDJSMpeDQuQKQfwHwvvl30Xjdld+AYsP5xWtLCIPv1A3NRpPrGJAa7Wm6P/hB562bcoXxPTv8nqxIxt8c26Y37CqyEqRGFvW4pS24yqMRgGdtEs/wpa1jkDBrm0lBXrVwp058hhbE6Q+DhDca3uyvuKM/vO2TGie1BjPeWF x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: Ye3HBvvP5BiBSw6Fk3AZzwcPOiVuBy1bWaL4ZlOwGdBm26YsGIWu0KuDCzJnfDqx2cEt6p1Gu4Xx19MRqJnoFXLLAxiSNt7e/l4Dt9nQRsNp8uWpNrD01CdwxwX04p4OdgNJltGJhdbb2wFcHCF4xQ== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 3ac3ff12-b4ba-40cd-2a98-08d962e4fbd8 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Aug 2021 07:43:06.6189 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB6255 Subject: [FFmpeg-devel] [PATCH 0/9] A New Take on the Immortal Sub2Video Code X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: O48txtmPTNgf Hi, some of you might remember that I had warmed-up the discusion about improving support for subtitle handling in ffmpeg sometime last year. A lot of different voices were heard and the unfinished work from Matthieau Buron has been in focus. My takeaway was that it's a pretty difficult subject involving complex requirements, for which there is no simple solution in sight. Consequently, I had put this on hold at my side just like all others. Few days ago I had to revisit and re-evaluate the subtitle situation once again: the current sub2video "hack" implementation is slow and ineffective: it creates a full-size video frame from the source subtitles and that full-size frame needs to be overlaid/blended with/onto every single video frame. That's a pretty expensive operation and it needs to be performed for every single video frame, even when there's nothing to overlay at all (= no subs to show in current scene). And when there are subtitles to show, those are usually limited to smaller areas on the screen; again, there's no need to overlay a full-size image onto each frame. From a performance perspective, it would be much more efficient to overlay the individual rects onto each frame only instead of a full frame blend. From a general perspective - even though it has always been considered as a 'hack' - the sub2video implementation has become a kind of standard behavior, despite of its shortcomings. Users have accommodated with the respective command lines and are expecting these to work. I'd expect any deviation from the current behavior to be filed as regression bugs. From there, I had decided to go for a small-step improvement of that situation while staying inside these corner points, and the plan was simple: - keep the current sub2video functionality alive at its core - but stop creating full video frames (with alpha) for feeding into a filtergraph (and later overlay) - instead, use refcounting for AVSubtitle and attach it to frames of (new) type AVMediaType_Subtitle - Those subtitle frames can travel through a filtergraph without needing to materialize them to images first For more efficient overlay (processing the relevant rects only), I have created a new filter: overlay_subs: - Input0: video - Input1: subtitles (format: SUBTITLE_BITMAP) - Output0: video In order to keep compatibility with existing command lines, I have added another filter: sub2video (reminescent naming) sub2video creates video frames (with alpha) from an existing subtitle stream: - Input0: subrtitles (format: SUBTITLE_BITMAP) - Output0: video This filter gets auto-inserted to retain compatibility with current sub2video command lines. As AVSubtitle can carry both, textual and graphical subtitles, the ability for sending textual subtitles through a filtergraph came more or less for free. For testing purposes, I have added another filter: sleet (translates subtitles to 'L337 speak') - Input0: subtitles (format: SUBTITLE_ASS or SUBTITLE_TEXT) - Output0: subtitles (format: same as input) Why leet? Real world use is surely questionable, but it has two advantages that make it a great choice for testing: You can see from almost every single line whether it has been filtered, and the text is still readable which allows to verify that timings are still valid. Working Command Lines Using "overlay_subs" (better performance): -y -loglevel verbose -ss 00:02:30 -i INPUT -filter_complex "[0:0]format=bgra[main];[main][0:3]overlay_subs=x=0:y=-220:format=auto,forma t=yuv420p" output.mp4 Using regular Overlay and the new sub2video filter: -y -loglevel verbose -ss 00:02:30 -filter_threads 1 -i INPUT -filter_complex "[0:0]format=bgra[main];[0:3]sub2video[subs];[main][subs]overlay=x=0:y=-220: format=auto,format=yuv420p" output.mp4 Legacy command line compatibility: (using regular overlay; sub2video filter is inserted automatically) -y -loglevel verbose -ss 00:04:30 -filter_threads 1 -i INPUT -filter_complex "[0:0][0:3]overlay=x=0:y=-220:format=auto,format=yuv420p" output.mp4 To make this work, I roughly did the following: - Added a new property 'type' to AVFrame of type enum AVMediaType - Changed code that uses width/height for checking whether a frame is video or audio and check the 'type' property instead - Added ff_ssrc_sbuffer and ff_ssink_sbuffer filters - Changed filtergraph code to support frames with AVMediaType == AVMediaType_Subtitle - Transformed the existing sub2video code accordingly, while still keeping the heartbeat mechanism in place Next Steps - Create modified version of the ASS filter that works based on (text) subtitle filter input - Create modified EIA-608/708 filter (e.g. split_eia608) which has one video input and one video output, plus a second output of type 'subtitle' (format: text or ass) This will allow to burn-in eia subs or save them in any other subtitle format without needing the weird "movie:xxx" input filter construct. - First of all: Get this patchset right... Reaching out for Help... This is a huge task where I hadn't expected to get that far. I'd consider my ffmpeg expertise as "spot knowledge": I do know certain areas very well, but there are others that I'm less familiar with. Nonetheless, I have a strategy and a working proof of concept. I'm, sure that there are quite a few things that need adjustment, and I'd be glad about any help and assistance! Thanks, softworkz --- softworkz (9): lavu/frame: avframe add type property avfilter/subtitles: Add subtitles.c avfilter/avfilter: Handle subtitle frames avfilter/overlay_subs: Add overlay_subs filter avfilter/sub2video: Add sub2video filter avfilter/sbuffer: Add sbuffersrv and sbuffersink filters avfilter/sleet: Add sleet filter fftools/ffmpeg: Replace sub2video with subtitle frame filtering Adjust sub2video Fate tests configure | 2 +- fftools/ffmpeg.c | 324 +++--- fftools/ffmpeg.h | 7 +- fftools/ffmpeg_filter.c | 198 +++- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- libavfilter/Makefile | 6 + libavfilter/allfilters.c | 13 +- libavfilter/avfilter.c | 8 +- libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 65 ++ libavfilter/buffersink.h | 15 + libavfilter/buffersrc.c | 65 ++ libavfilter/buffersrc.h | 1 + libavfilter/formats.c | 11 + libavfilter/internal.h | 8 + libavfilter/sf_sleet.c | 209 ++++ libavfilter/subtitles.c | 61 ++ libavfilter/subtitles.h | 42 + libavfilter/svf_sub2video.c | 260 +++++ libavfilter/vf_overlay_subs.c | 1173 +++++++++++++++++++++ libavfilter/vf_overlay_subs.h | 88 ++ libavutil/frame.c | 74 +- libavutil/frame.h | 39 +- libavutil/version.h | 2 +- tests/ref/fate/filter-overlay-dvdsub-2397 | 179 ++-- tests/ref/fate/sub-dvb | 162 +-- tests/ref/fate/sub2video | 49 +- tests/ref/fate/sub2video_basic | 178 ++-- tests/ref/fate/sub2video_time_limited | 4 +- 30 files changed, 2743 insertions(+), 510 deletions(-) create mode 100644 libavfilter/sf_sleet.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/svf_sub2video.c create mode 100644 libavfilter/vf_overlay_subs.c create mode 100644 libavfilter/vf_overlay_subs.h