From patchwork Mon Aug 30 08:15:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29874 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp3736892iov; Mon, 30 Aug 2021 01:15:57 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwTcn35FS2v46SLMnKtw3X6auNnebsds5C0AKjGVmLCSF4h1ZuGAttCJQ9Yo4Ht3ATqiXSA X-Received: by 2002:aa7:d859:: with SMTP id f25mr6623909eds.247.1630311357036; Mon, 30 Aug 2021 01:15:57 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f11si13839857edr.52.2021.08.30.01.15.56; Mon, 30 Aug 2021 01:15:57 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b="Hw2GU/3S"; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AB6F4689EC9; Mon, 30 Aug 2021 11:15:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10olkn2038.outbound.protection.outlook.com [40.92.40.38]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 438AB680831 for ; Mon, 30 Aug 2021 11:15:45 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RgrTaWuLGkFZtw3DVauxrG8N4aoJgVL30Kg8M+ddQdg+aZV3EPRYIesil1tk9dV1LAT291rEE1Ko2lIlcr5KcJQezqZLiYw5VBtfnZbon259Ut3Xn/8GxDgGVvtlrKWBpa0IZ/aLiOQwE6WLt+L9nD0A7hNKI1CbeIdt0URmdTCsQxUh+k1Zeh0jHKwdyjbGjrwVCJRg85P7viaTOPAJ5dvQXDLrgK32ic7wH1pY7C5NFeAwBqIGJCcNY/W0Bc60RDiGyDyubrz1UAVungNv8MgzUzidcdoXA+yvEW7pHZKbiHW3lXs17JU23u3XyVtvDeFjKmszWnkT2C8w5x60KA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wxDzCYi+drSVall45XGa2YQ4YT8gi15a4TnouVDphE8=; b=IMfzeRVP0QfrWHg7m3rBV7YuJFtSN9efJQCpNXv1xSouRr9zJb4HtTs6TzsI6DkpRTr7UyOqPPykpX7Ot5G6Ntw74wwFEo1pFDk5j83gPtLolaKRRM7+yjpwHianJNLDSMSL/7bAbwmXWpHH+SC84QDxMgDLXzmf2++8PZNFyq29HoEF59wMDW5DGdJ54OeYV1sJ4nqsCY10Gd67V4vT5dlM7plh5Bw6Uo02vNNpiY8KwQT6Ek8eYdjtDKgXHToN4AAFsQkEC1pEbBvpnVgbnZlW361Vh0qPKuWaM4cD8f1Iz41WU8/SeXVJ6wQyzfRjw5/zsFtealhPGHRrMwF4Bw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wxDzCYi+drSVall45XGa2YQ4YT8gi15a4TnouVDphE8=; b=Hw2GU/3SLk9n4IVgb5fIj57Yzx1ECHYjUfY8+eP0AJ8vr+WFhIy+2kaasPlPfJZrXXeU1kL8C5noJTDv76PyIk6DqV2yutxO2Vu0tt6ZJWe5aflTxKVte4ejfbrp9fvBlVGrj7gcJxbUZ4GwEXgEEUiO50hyp4t53yyLLOGobGt460ujApRRqm/VKy0krz/ALvkRFSQ2Az+vyEMmVQ517+FhXp1HhdstodnohuPhGo3VZGrYYw1clat+k7rugE2AKSP8nFVjtTsxZLXZwhLKPnUJCWKBQqq6a/z78ps3IAZqAt77P0XtUf/chZFLqertDA16Z3HrEbZtYf+X/JtttQ== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB6191.namprd04.prod.outlook.com (2603:10b6:208:de::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.23; Mon, 30 Aug 2021 08:15:42 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 08:15:42 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v2 0/8] A New Take on the Immortal Sub2Video Code Thread-Index: Adedc/nkuIOVogYGS52hZIdpOznl2A== Date: Mon, 30 Aug 2021 08:15:42 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [HDBSjKGLKHIFYtxjDOO0RsoOOjPguC+2] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3d2860e0-e9c4-49b5-b887-08d96b8e5c48 x-ms-traffictypediagnostic: MN2PR04MB6191: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ax62tjAB5uEqjMqHPBOWGzYUm5CFuiFn3WFQ2flOeTFyRxOjTctnq9NMM6FFhMCKa1qHaqGrmcc6evUrtONOBA29hJuR6ImjM6WIsWgqBQz//NrY5xDqifqNxywKVyEVWxD71TdKqv3KCbxmjAtvrxEUsfGlqHpIE5a9qslMW/oDTAhaKpSWsZQeQuSvJgT6BFg26Lijy8sWf9LLc1oaWc8XLDB0y8jpxndhtk/b7yKyI3/uSuL2MdW2R/5MbZ+Kn4gt3aNMtnZZ5tZa5b5cNBYW/lMFJx7fDPxC1pQYSe2hVe+LfSn66dNMf6kXWLu0QaxHHLFYuUIln5JW33ehAp0etvUu/vt79J5JP7U4nnWQDXQRIVgJ0FTGKZmtFO5E6TW0O9Pi4aLdx0OLg9YI74J6Em+s4Blp8rWX/bjxhmKtwKYoAjzfHH7R2sLA+fgt x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: nnDYymURXtBB70cQw0KbL1cQB4LYoEroRFDA6/RD3lm7icZ5AwL7iHgukBDcKQNCwwT8GoK9Zln0c9ODXQQCSsdPpBkvovT7iFdmWxFvLgaSD2WablsYGXbz9R3jgSoI3rQssdRTTiRnou59U91c8Q== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 3d2860e0-e9c4-49b5-b887-08d96b8e5c48 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2021 08:15:42.6434 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB6191 Subject: [FFmpeg-devel] [PATCH v2 0/8] A New Take on the Immortal Sub2Video Code X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: CxldmdcWQ5iX v2 Update: - Implemented Andreas' suggestions - overlay_subs filter: - removed duplicated code - implemented direct (no pre-conversion) blending of graphical subtitle rects - Supported input formats: - all packed RGB formats (with and without alpha) - yuv420p, yuv422p, yuv444p This patchset is about introducing filtering support for subtitles. The current sub2video "hack" implementation is slow and ineffective: it creates a full-size video frame from the source subtitles and that full-size frame needs to be overlaid/blended with/onto every single video frame. That's a pretty expensive operation and it needs to be performed for every single video frame, even when there's nothing to overlay at all (= no subs to show in current scene). And when there are subtitles to show, those are usually limited to smaller areas on the screen; again, there's no need to overlay a full-size image onto each frame. From a performance perspective, it would be much more efficient to overlay the individual rects onto each frame only instead of a full frame blend. From a general perspective - even though it has always been considered as a 'hack' - the sub2video implementation has become a kind of standard behavior, despite of its shortcomings. Users have accommodated with the respective command lines and are expecting these to work. I'd expect any deviation from the current behavior to be filed as regression bugs. From there, I had decided to go for a small-step improvement of that situation while staying inside these corner points, and the plan was simple: - keep the current sub2video behavior (specifically hearbeat) working to avoid regressions - but stop creating full video frames (with alpha) for feeding into a filtergraph (and later overlay) - instead, use refcounting for AVSubtitle and attach it to frames of (new) type AVMediaType_Subtitle - those subtitle frames can travel through a filtergraph without needing to materialize them to images first For more efficient overlay (processing the relevant rects only), I have created a new filter: overlay_subs: - Input0: video - Input1: subtitles (format: SUBTITLE_BITMAP) - Output0: video In order to keep compatibility with existing command lines, I have added another filter: sub2video (reminescent naming) sub2video creates video frames (with alpha) from an existing subtitle stream: - Input0: subrtitles (format: SUBTITLE_BITMAP) - Output0: video This filter gets auto-inserted to retain compatibility with current sub2video command lines. As AVSubtitle can carry both, textual and graphical subtitles, the ability for sending textual subtitles through a filtergraph came more or less for free. For testing purposes, I have added another filter: sleet (translates subtitles to 'L337 speak') - Input0: subtitles (format: SUBTITLE_ASS or SUBTITLE_TEXT) - Output0: subtitles (format: same as input) Why leet? Real world use is surely questionable, but it has two advantages that make it a great choice for testing: You can see from almost every single line whether it has been filtered, and the text is still readable which allows to verify that timings are still valid. Working Command Lines Using "overlay_subs" (better performance): -y -loglevel verbose -ss 00:02:30 -i INPUT -filter_complex "[0:0][0:3]overlay_subs=x=0:y=-220" output.mp4 Using regular Overlay and the new sub2video filter: -y -loglevel verbose -ss 00:02:30 -filter_threads 1 -i INPUT -filter_complex "[0:0]format=bgra[main];[0:3]sub2video[subs];[main][subs]overlay=x=0:y=-220: format=auto,format=yuv420p" output.mp4 Legacy command line compatibility: (using regular overlay; sub2video filter is inserted automatically) -y -loglevel verbose -ss 00:04:30 -filter_threads 1 -i INPUT -filter_complex "[0:0][0:3]overlay=x=0:y=-220:format=auto,format=yuv420p" output.mp4 To make this work, I roughly did the following: - Added a new property 'type' to AVFrame of type enum AVMediaType - Changed code that uses width/height for checking whether a frame is video or audio and check the 'type' property instead - Added ff_ssrc_sbuffer and ff_ssink_sbuffer filters - Changed filtergraph code to support frames with AVMediaType == AVMediaType_Subtitle - Transformed the existing sub2video code accordingly, while still keeping the heartbeat mechanism in place Next Steps - Create modified version of the ASS filter that works based on (text) subtitle filter input - Create modified EIA-608/708 filter (e.g. split_eia608) which has one video input and one video output, plus a second output of type 'subtitle' (format: text or ass) This will allow to burn-in eia subs or save them in any other subtitle format without needing the weird "movie:xxx" input filter construct. Regards, softworkz --- softworkz (8): lavu/frame: avframe add type property avfilter/subtitles: Add subtitles.c avfilter/avfilter: Handle subtitle frames avfilter/overlay_subs: Add overlay_subs filter avfilter/sub2video: Add sub2video filter avfilter/sbuffer: Add sbuffersrv and sbuffersink filters avfilter/sleet: Add sleet filter fftools/ffmpeg: Replace sub2video with subtitle frame filtering configure | 2 +- fftools/ffmpeg.c | 324 +++++++++++--------- fftools/ffmpeg.h | 7 +- fftools/ffmpeg_filter.c | 198 ++++++++---- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- libavfilter/Makefile | 6 + libavfilter/allfilters.c | 13 +- libavfilter/avfilter.c | 8 +- libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 65 ++++ libavfilter/buffersink.h | 15 + libavfilter/buffersrc.c | 65 ++++ libavfilter/buffersrc.h | 1 + libavfilter/f_interleave.c | 3 + libavfilter/formats.c | 11 + libavfilter/internal.h | 1 + libavfilter/sf_sleet.c | 209 +++++++++++++ libavfilter/subtitles.c | 61 ++++ libavfilter/subtitles.h | 44 +++ libavfilter/svf_sub2video.c | 260 ++++++++++++++++ libavfilter/vf_overlay_subs.c | 551 ++++++++++++++++++++++++++++++++++ libavfilter/vf_overlay_subs.h | 65 ++++ libavutil/frame.c | 74 ++++- libavutil/frame.h | 39 ++- libavutil/version.h | 2 +- 26 files changed, 1804 insertions(+), 230 deletions(-) create mode 100644 libavfilter/sf_sleet.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/svf_sub2video.c create mode 100644 libavfilter/vf_overlay_subs.c create mode 100644 libavfilter/vf_overlay_subs.h