From patchwork Sat May 28 13:25:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35956 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1365834pzj; Sat, 28 May 2022 06:25:38 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxeMF+R7bH9NvsGStTAYp7zrtgXqbE5VtRkiW3Fo/haslbWw/65AizD2kKR4NPbt2Px7H/F X-Received: by 2002:a05:6402:3582:b0:42b:32a8:912b with SMTP id y2-20020a056402358200b0042b32a8912bmr40806306edc.137.1653744338475; Sat, 28 May 2022 06:25:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744338; cv=none; d=google.com; s=arc-20160816; b=aGv0hXu1EgADnDUk94et3xXOyp0mbAQrn4JRLM3cLCyzj2xcrnvMxhTz4kZK9+99fD N3ajvT1Wvsq8wwaxkoc0/NUFtgoNFPofT9yQWyApWkv1eJGoxiiHgD9ViuQoHfqS3KpL i4MbiirplgpTtwiZrYSF34oU41qhhGS6o7NsPiIUm4nbSH8QpS7jCoDhZ0i3ToWJ23M6 dlkfF7rr2tbw/SaWW46bLRRqAG4uqpM77HIRVUhAlvdNzvVow63UUXlIOp/3d6rMbLoX X5SYxs+Jp9pVpiSZ9CA31RU6wjeYUcCgDcCwbWfJ7y0iaWb4yDS0dl5hVunViq4RLrU8 I8wA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=1jykCuDVhwaeo5L4f3kX1M8olexFwDbepwCNobL+y4o=; b=PQFarRw2XDqxiCbmcIQsxRtLMUjDAvTjjDq0RyjaEPLHoU8zcc2f8CDc0aQYvm7Zpk XgBH68DInoF7Fmw/0nE8kHLDrDaxk7Mh3hOQTk7BpaA2CZ/3mzA2ymMhSXP6br/SVOfM wtgdCOy3UNYCPl8lkmSEi0n7KiZ8ZAIB1mrmGJIt4UUHiXnv/wOLZ19OPJzstbGAyiYe fGIUnFswbUFf/sC7/nkulB/SX7QB2CJpsGtSKMDB1h+PvZGViMwt+PrEsrjUFCv4ICKM UrzB4GdGz70ExcrsahCtt8W99bswBKxugVCjZLJ6fChq8uONsti9VyS767VpNDLgacRQ zx1A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=UZ1V5z3D; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b20-20020a056402351400b0041d70951197si5576150edd.530.2022.05.28.06.25.37; Sat, 28 May 2022 06:25:38 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=UZ1V5z3D; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E57A168B5C6; Sat, 28 May 2022 16:25:34 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id ACB3D68B56B for ; Sat, 28 May 2022 16:25:28 +0300 (EEST) Received: by mail-pj1-f54.google.com with SMTP id w2-20020a17090ac98200b001e0519fe5a8so6644497pjt.4 for ; Sat, 28 May 2022 06:25:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=zrkyrcT8nnxOemP+bVsEo+8DRx2fNfP1JrF6HEN4bls=; b=UZ1V5z3DPHoqZC/oPPMYgnNzpqJ0lj/CH2HPr4BgIVUFkqXqyukGCvhpiK/rkensQv O5yRfhqRr3GsEQeImH5BeS9TIm1+vXY8cZ27rRxzusmRlNOAVFIWlwd/YbU/Cu2rIHQR zEf310Bw/zL5DdnF2RiXopUmF5PmP3RA36BORuzkv8+CGxO4RwnQOc8l0ljszrrOq9FX Rn+WGvwWxbHx91UFX6JkbOinwvuWervcMX4HUV0VnpucPpD384sDpnIYAXMjCTuOB1Zl /GYK/GHmdfkt9r81Su0nQGtwi488Et+RA2wfAiCSnJpcmsNe9yhd+Hzu8M4fdtCb/Jru 5mgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=zrkyrcT8nnxOemP+bVsEo+8DRx2fNfP1JrF6HEN4bls=; b=IQLflhbU6csPBtf4NrfKy7vgBWNkhEW2Juqgr1DVS/XfytvXqrx7v5AJ7IcgZjqkUJ i1dysTf1l0uB/boLwTgGHtfV1kTssfuXcHUUAy+Uwdl98yEjWtsJMHGvrv6C1wIhQ8iD EVKyLpHB0Ncljlk0KSgXwUv71g3+QsPTvhIus7J9N8w1U+Qpyxvd+MaO+aqQti84urIE hT63QiKt714EMR2LbTYRRkLZ6wVXJtj5XOHIZKgX2aOhHCdWi0EfSLNWYQaF+lpV+Y6Q aWctB7SkADtrJQGKchwLJuOBd1cqb1KPDn79nk7JxENmeR3ccuuuv3LJyH+sNkonbu96 SnMA== X-Gm-Message-State: AOAM5333qVlQlvLUG//k1pvn0Pk2GGBbhj9OTazqvLtC05QQLB3CBvuS 5B/Vs0S3JaioCOJGSl7TsDW2liSHo0Hc2A== X-Received: by 2002:a17:902:a9ca:b0:161:54a6:af3f with SMTP id b10-20020a170902a9ca00b0016154a6af3fmr48523387plr.48.1653744326989; Sat, 28 May 2022 06:25:26 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id jc1-20020a17090325c100b00161b50c3db4sm5615725plb.94.2022.05.28.06.25.26 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:26 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <2f3ba171f5261b1d200f396d51f9156637fe54ef.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:01 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 01/23] avcodec, avutil: Move enum AVSubtitleType to avutil, add new and deprecate old values X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: GB4fj7uX0dpM From: softworkz Signed-off-by: softworkz --- libavcodec/avcodec.h | 19 +------------ libavutil/Makefile | 1 + libavutil/subfmt.h | 68 ++++++++++++++++++++++++++++++++++++++++++++ libavutil/version.h | 1 + 4 files changed, 71 insertions(+), 18 deletions(-) create mode 100644 libavutil/subfmt.h diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h index 4dae23d06e..56d551f92d 100644 --- a/libavcodec/avcodec.h +++ b/libavcodec/avcodec.h @@ -35,6 +35,7 @@ #include "libavutil/frame.h" #include "libavutil/log.h" #include "libavutil/pixfmt.h" +#include "libavutil/subfmt.h" #include "libavutil/rational.h" #include "codec.h" @@ -2255,24 +2256,6 @@ typedef struct AVHWAccel { * @} */ -enum AVSubtitleType { - SUBTITLE_NONE, - - SUBTITLE_BITMAP, ///< A bitmap, pict will be set - - /** - * Plain text, the text field must be set by the decoder and is - * authoritative. ass and pict fields may contain approximations. - */ - SUBTITLE_TEXT, - - /** - * Formatted text, the ass field must be set by the decoder and is - * authoritative. pict and text fields may contain approximations. - */ - SUBTITLE_ASS, -}; - #define AV_SUBTITLE_FLAG_FORCED 0x00000001 typedef struct AVSubtitleRect { diff --git a/libavutil/Makefile b/libavutil/Makefile index 234de62a4b..5fcdd68ca8 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -76,6 +76,7 @@ HEADERS = adler32.h \ sha512.h \ spherical.h \ stereo3d.h \ + subfmt.h \ threadmessage.h \ time.h \ timecode.h \ diff --git a/libavutil/subfmt.h b/libavutil/subfmt.h new file mode 100644 index 0000000000..791b45519f --- /dev/null +++ b/libavutil/subfmt.h @@ -0,0 +1,68 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_SUBFMT_H +#define AVUTIL_SUBFMT_H + +#include "version.h" + +enum AVSubtitleType { + + /** + * Subtitle format unknown. + */ + AV_SUBTITLE_FMT_NONE = -1, + + /** + * Subtitle format unknown. + */ + AV_SUBTITLE_FMT_UNKNOWN = 0, +#if FF_API_OLD_SUBTITLES + SUBTITLE_NONE = 0, ///< Deprecated, use AV_SUBTITLE_FMT_NONE instead. +#endif + + /** + * Bitmap area in AVSubtitleRect.data, pixfmt AV_PIX_FMT_PAL8. + */ + AV_SUBTITLE_FMT_BITMAP = 1, +#if FF_API_OLD_SUBTITLES + SUBTITLE_BITMAP = 1, ///< Deprecated, use AV_SUBTITLE_FMT_BITMAP instead. +#endif + + /** + * Plain text in AVSubtitleRect.text. + */ + AV_SUBTITLE_FMT_TEXT = 2, +#if FF_API_OLD_SUBTITLES + SUBTITLE_TEXT = 2, ///< Deprecated, use AV_SUBTITLE_FMT_TEXT instead. +#endif + + /** + * Text Formatted as per ASS specification, contained AVSubtitleRect.ass. + */ + AV_SUBTITLE_FMT_ASS = 3, +#if FF_API_OLD_SUBTITLES + SUBTITLE_ASS = 3, ///< Deprecated, use AV_SUBTITLE_FMT_ASS instead. +#endif + + AV_SUBTITLE_FMT_NB, ///< number of subtitle formats, DO NOT USE THIS if you want to link with shared libav* because the number of formats might differ between versions. +}; + +#endif /* AVUTIL_SUBFMT_H */ diff --git a/libavutil/version.h b/libavutil/version.h index 1b4b41d81f..11baa362d3 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -114,6 +114,7 @@ #define FF_API_XVMC (LIBAVUTIL_VERSION_MAJOR < 58) #define FF_API_OLD_CHANNEL_LAYOUT (LIBAVUTIL_VERSION_MAJOR < 58) #define FF_API_AV_FOPEN_UTF8 (LIBAVUTIL_VERSION_MAJOR < 58) +#define FF_API_OLD_SUBTITLES (LIBAVUTIL_VERSION_MAJOR < 58) /** * @} From patchwork Sat May 28 13:25:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35957 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1365957pzj; Sat, 28 May 2022 06:26:00 -0700 (PDT) X-Google-Smtp-Source: ABdhPJztO6g2HgO+RC89Vxx4zjwUCMkl74UigGjH726Yrykw595yWyx17nf6v3SNDoomz5G3Lc53 X-Received: by 2002:a17:907:1c87:b0:6f0:29ea:cc01 with SMTP id nb7-20020a1709071c8700b006f029eacc01mr41858866ejc.671.1653744359866; Sat, 28 May 2022 06:25:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744359; cv=none; d=google.com; s=arc-20160816; b=aB4ym7sSLqeb4EgSHBwGEdYbt/QcOCUub3ZZKYBKhOSnhg++i9OkyPB94fujcaOCUy jEQWJjLD/mnxEtXsXSejsDgKaUs8Ebf5O/Uw0CUkFEJ1UM3XY+qcAATlPHeOgmMTRcP+ +z4n81pDNxtMeYW1RTQ1LQtWnjMULt03WNaHO17VfwMe4d6Sz012jYWk39NK/EkbTuWc vJk3NoRBeoID55PZEMuHaSyVs9vzjxBqmR/lr+WfXpFDF7p7urO/vJ0at+JWGMjqamR0 xHLKbdjrR4XYdB8jV7mhVdA66FGI73kDNKe2BPNCspGDrrbt9oN5ely4bWTyNckOlNik f/mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=RpceRPWiBeomkTpn7+TEuisf357I6gylK9D+jxWMZAI=; b=y1jYqPS5UMqwskHvXriFF0+3/2Wk6moYEb/DY1uowJIr2/O+hjex9EEb2C5Z8+GARd xGdzEHNMqV1EprKPAwY1UobYYoN42F5GOeBQMQGJRv89tsnbURAImwT7bPTwo9TDU2cA zVbJ0JtOL29n65xxSnb9E2jOaVoIEw8GSv/t0EgnJdFZD8R0DrSJFOAPlVBVSxObdZh1 8/uUnb240w9VsFkYc7+fWpO0WVWVwNwakUAs39TYBuOFfbvx250Z1TJ+3s5Z/20rKPyd Ms0SFDlE5luJV++K3LHK030/qYLoEOvqSZKg4YUaZEGYLa3/H11P5ctQpmFEt8eHRM6i Ei7A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=RMxWsfrx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ep23-20020a1709069b5700b006fe4c66b711si4693756ejc.46.2022.05.28.06.25.59; Sat, 28 May 2022 06:25:59 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=RMxWsfrx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D1CE768B5E3; Sat, 28 May 2022 16:25:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1663868B5C8 for ; Sat, 28 May 2022 16:25:30 +0300 (EEST) Received: by mail-pg1-f181.google.com with SMTP id 137so6220352pgb.5 for ; Sat, 28 May 2022 06:25:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=e+7lXbbeEiiL4PZWRt8hWWuLAbKgs/Ew8EfhL1wcoHg=; b=RMxWsfrx68DhN1E8Be0gEiRFqRIpvdb7zejjaEq1w2Ya6Cb9SnGex9zKTID+RL+iL+ QBCGpbVpndyHJ0Qdx2xuNXz8FAUcaojbdXLLMo4SleR+wHrUePhVpb/ox+EjqiQS6iza iK9AJJgZzDvl8gds9uI8Mm4VX8XUdBhVwdx4l3ZXu0Wg8fwMWjznnAOjjm8KePb3AoAN 2l00+aCvVXq9qRo8noLCvSy3JAez4XTYwelPhvBEs4FNaqKo604GpZlc1u4cCSoboICo +cL2cch4Qq87OMW9UHSMSMrWnOa64pYwsJk+DhICqie2Mz5T9O3lfL74+Ev3y53K/KEu mc2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=e+7lXbbeEiiL4PZWRt8hWWuLAbKgs/Ew8EfhL1wcoHg=; b=CZfYTYZsv+XjBN4I+S/x460sbKnV/wRej4nskINEH8BLOMby1DG8FWV6GSmhKr/Zxc 4pZlEj9B/xsxb391mnjbhhS9yaTUqFLmjq6UpW9xuTCjHyeaW24uk0s2/u0bmdCCGZYU rCHe2PQYlunrQFyWXo5R0a4lHCUf+8o80ZXUoCBDKrLDLTGGysZ3IqDvsc+S7SNHi9gB Sl7hzDFCXT6a56O9/DNAVCo4dlLZfj8m/vasK+hnTQ/6q9Yik3qmFuTco+inOUdu4Ow7 aWMxbWnyx2OpDFrOF0KyL0Mvu95kLWe7Q94B7wiF+OHIZwvv/eMDcxOyvAID0w/xiRbp Bpmg== X-Gm-Message-State: AOAM533jZYiwusep//umyb83AdUPMb/2t9fRhBowoeDL84QnW4RhUpvR 4DBm2oJY3iDt0I5GthNU8J+wMDH7OioNFA== X-Received: by 2002:a63:a06:0:b0:3c2:3345:bf99 with SMTP id 6-20020a630a06000000b003c23345bf99mr41286957pgk.477.1653744328148; Sat, 28 May 2022 06:25:28 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id u21-20020a631415000000b003c14af50626sm5094291pgl.62.2022.05.28.06.25.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:27 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: In-Reply-To: References: Date: Sat, 28 May 2022 13:25:02 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 02/23] avutil/frame: Prepare AVFrame for subtitle handling X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: z/Ubmv07wFHe From: softworkz Root commit for adding subtitle filtering capabilities. In detail: - Add type (AVMediaType) field to AVFrame Replaces previous way of distinction which was based on checking width and height to determine whether a frame is audio or video - Add subtitle fields to AVFrame - Add new struct AVSubtitleArea, similar to AVSubtitleRect, but different allocation logic. Cannot and must not be used interchangeably, hence the new struct Signed-off-by: softworkz --- libavutil/Makefile | 1 + libavutil/frame.c | 211 +++++++++++++++++++++++++++++++++++++++------ libavutil/frame.h | 85 +++++++++++++++++- libavutil/subfmt.c | 45 ++++++++++ libavutil/subfmt.h | 47 ++++++++++ 5 files changed, 362 insertions(+), 27 deletions(-) create mode 100644 libavutil/subfmt.c diff --git a/libavutil/Makefile b/libavutil/Makefile index 5fcdd68ca8..78f65c144a 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -162,6 +162,7 @@ OBJS = adler32.o \ slicethread.o \ spherical.o \ stereo3d.o \ + subfmt.o \ threadmessage.o \ time.o \ timecode.o \ diff --git a/libavutil/frame.c b/libavutil/frame.c index fbb869fffa..b24fdc5868 100644 --- a/libavutil/frame.c +++ b/libavutil/frame.c @@ -26,6 +26,7 @@ #include "imgutils.h" #include "mem.h" #include "samplefmt.h" +#include "subfmt.h" #include "hwcontext.h" #if FF_API_OLD_CHANNEL_LAYOUT @@ -52,6 +53,9 @@ const char *av_get_colorspace_name(enum AVColorSpace val) return name[val]; } #endif + +static int frame_copy_subtitles(AVFrame *dst, const AVFrame *src, int copy_data); + static void get_frame_defaults(AVFrame *frame) { memset(frame, 0, sizeof(*frame)); @@ -72,7 +76,12 @@ static void get_frame_defaults(AVFrame *frame) frame->colorspace = AVCOL_SPC_UNSPECIFIED; frame->color_range = AVCOL_RANGE_UNSPECIFIED; frame->chroma_location = AVCHROMA_LOC_UNSPECIFIED; - frame->flags = 0; + frame->num_subtitle_areas = 0; + frame->subtitle_areas = NULL; + frame->subtitle_header = NULL; + frame->repeat_sub = 0; + frame->subtitle_timing.start_pts = 0; + frame->subtitle_timing.duration = 0; } static void free_side_data(AVFrameSideData **ptr_sd) @@ -251,6 +260,23 @@ FF_ENABLE_DEPRECATION_WARNINGS } +static int get_subtitle_buffer(AVFrame *frame) +{ + // Buffers in AVFrame->buf[] are not used in case of subtitle frames. + // To accomodate with existing code, checking ->buf[0] to determine + // whether a frame is ref-counted or has data, we're adding a 1-byte + // buffer here, which marks the subtitle frame to contain data. + frame->buf[0] = av_buffer_alloc(1); + if (!frame->buf[0]) { + av_frame_unref(frame); + return AVERROR(ENOMEM); + } + + frame->extended_data = frame->data; + + return 0; +} + int av_frame_get_buffer(AVFrame *frame, int align) { if (frame->format < 0) @@ -258,23 +284,41 @@ int av_frame_get_buffer(AVFrame *frame, int align) FF_DISABLE_DEPRECATION_WARNINGS if (frame->width > 0 && frame->height > 0) - return get_video_buffer(frame, align); + frame->type = AVMEDIA_TYPE_VIDEO; else if (frame->nb_samples > 0 && (av_channel_layout_check(&frame->ch_layout) #if FF_API_OLD_CHANNEL_LAYOUT || frame->channel_layout || frame->channels > 0 #endif )) - return get_audio_buffer(frame, align); + frame->type = AVMEDIA_TYPE_AUDIO; FF_ENABLE_DEPRECATION_WARNINGS - return AVERROR(EINVAL); + return av_frame_get_buffer2(frame, align); +} + +int av_frame_get_buffer2(AVFrame *frame, int align) +{ + if (frame->format < 0) + return AVERROR(EINVAL); + + switch (frame->type) { + case AVMEDIA_TYPE_VIDEO: + return get_video_buffer(frame, align); + case AVMEDIA_TYPE_AUDIO: + return get_audio_buffer(frame, align); + case AVMEDIA_TYPE_SUBTITLE: + return get_subtitle_buffer(frame); + default: + return AVERROR(EINVAL); + } } static int frame_copy_props(AVFrame *dst, const AVFrame *src, int force_copy) { int ret, i; + dst->type = src->type; dst->key_frame = src->key_frame; dst->pict_type = src->pict_type; dst->sample_aspect_ratio = src->sample_aspect_ratio; @@ -306,6 +350,12 @@ static int frame_copy_props(AVFrame *dst, const AVFrame *src, int force_copy) dst->colorspace = src->colorspace; dst->color_range = src->color_range; dst->chroma_location = src->chroma_location; + dst->repeat_sub = src->repeat_sub; + dst->subtitle_timing.start_pts = src->subtitle_timing.start_pts; + dst->subtitle_timing.duration = src->subtitle_timing.duration; + + if ((ret = av_buffer_replace(&dst->subtitle_header, src->subtitle_header)) < 0) + return ret; av_dict_copy(&dst->metadata, src->metadata, 0); @@ -353,6 +403,7 @@ FF_ENABLE_DEPRECATION_WARNINGS av_assert1(dst->ch_layout.nb_channels == 0 && dst->ch_layout.order == AV_CHANNEL_ORDER_UNSPEC); + dst->type = src->type; dst->format = src->format; dst->width = src->width; dst->height = src->height; @@ -385,7 +436,7 @@ FF_ENABLE_DEPRECATION_WARNINGS /* duplicate the frame data if it's not refcounted */ if (!src->buf[0]) { - ret = av_frame_get_buffer(dst, 0); + ret = av_frame_get_buffer2(dst, 0); if (ret < 0) goto fail; @@ -407,6 +458,10 @@ FF_ENABLE_DEPRECATION_WARNINGS } } + /* copy subtitle areas */ + if (src->type == AVMEDIA_TYPE_SUBTITLE) + frame_copy_subtitles(dst, src, 0); + if (src->extended_buf) { dst->extended_buf = av_calloc(src->nb_extended_buf, sizeof(*dst->extended_buf)); @@ -476,7 +531,7 @@ AVFrame *av_frame_clone(const AVFrame *src) void av_frame_unref(AVFrame *frame) { - int i; + unsigned i, n; if (!frame) return; @@ -495,6 +550,21 @@ void av_frame_unref(AVFrame *frame) av_buffer_unref(&frame->opaque_ref); av_buffer_unref(&frame->private_ref); + av_buffer_unref(&frame->subtitle_header); + + for (i = 0; i < frame->num_subtitle_areas; i++) { + AVSubtitleArea *area = frame->subtitle_areas[i]; + + for (n = 0; n < FF_ARRAY_ELEMS(area->buf); n++) + av_buffer_unref(&area->buf[n]); + + av_freep(&area->text); + av_freep(&area->ass); + av_free(area); + } + + av_freep(&frame->subtitle_areas); + if (frame->extended_data != frame->data) av_freep(&frame->extended_data); @@ -522,18 +592,28 @@ FF_ENABLE_DEPRECATION_WARNINGS int av_frame_is_writable(AVFrame *frame) { - int i, ret = 1; + AVSubtitleArea *area; + int ret = 1; /* assume non-refcounted frames are not writable */ if (!frame->buf[0]) return 0; - for (i = 0; i < FF_ARRAY_ELEMS(frame->buf); i++) + for (unsigned i = 0; i < FF_ARRAY_ELEMS(frame->buf); i++) if (frame->buf[i]) ret &= !!av_buffer_is_writable(frame->buf[i]); - for (i = 0; i < frame->nb_extended_buf; i++) + for (unsigned i = 0; i < frame->nb_extended_buf; i++) ret &= !!av_buffer_is_writable(frame->extended_buf[i]); + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + area = frame->subtitle_areas[i]; + if (area) { + for (unsigned n = 0; n < FF_ARRAY_ELEMS(area->buf); n++) + if (area->buf[n]) + ret &= !!av_buffer_is_writable(area->buf[n]); + } + } + return ret; } @@ -549,6 +629,7 @@ int av_frame_make_writable(AVFrame *frame) return 0; memset(&tmp, 0, sizeof(tmp)); + tmp.type = frame->type; tmp.format = frame->format; tmp.width = frame->width; tmp.height = frame->height; @@ -568,7 +649,7 @@ FF_ENABLE_DEPRECATION_WARNINGS if (frame->hw_frames_ctx) ret = av_hwframe_get_buffer(frame->hw_frames_ctx, &tmp, 0); else - ret = av_frame_get_buffer(&tmp, 0); + ret = av_frame_get_buffer2(&tmp, 0); if (ret < 0) return ret; @@ -603,7 +684,12 @@ AVBufferRef *av_frame_get_plane_buffer(AVFrame *frame, int plane) uint8_t *data; int planes, i; - if (frame->nb_samples) { + switch(frame->type) { + case AVMEDIA_TYPE_VIDEO: + planes = 4; + break; + case AVMEDIA_TYPE_AUDIO: + { int channels = frame->ch_layout.nb_channels; #if FF_API_OLD_CHANNEL_LAYOUT @@ -617,8 +703,11 @@ FF_ENABLE_DEPRECATION_WARNINGS if (!channels) return NULL; planes = av_sample_fmt_is_planar(frame->format) ? channels : 1; - } else - planes = 4; + break; + } + default: + return NULL; + } if (plane < 0 || plane >= planes || !frame->extended_data[plane]) return NULL; @@ -752,24 +841,98 @@ FF_ENABLE_DEPRECATION_WARNINGS return 0; } +static int av_subtitle_area2area(AVSubtitleArea *dst, const AVSubtitleArea *src, int copy_data) +{ + dst->x = src->x; + dst->y = src->y; + dst->w = src->w; + dst->h = src->h; + dst->nb_colors = src->nb_colors; + dst->type = src->type; + dst->flags = src->flags; + + switch (dst->type) { + case AV_SUBTITLE_FMT_BITMAP: + + for (unsigned i = 0; i < AV_NUM_BUFFER_POINTERS; i++) { + if (src->h > 0 && src->w > 0 && src->buf[i]) { + dst->buf[0] = av_buffer_ref(src->buf[i]); + if (!dst->buf[i]) + return AVERROR(ENOMEM); + + if (copy_data) { + const int ret = av_buffer_make_writable(&dst->buf[i]); + if (ret < 0) + return ret; + } + + dst->linesize[i] = src->linesize[i]; + } + } + + memcpy(&dst->pal[0], &src->pal[0], sizeof(src->pal[0]) * 256); + + break; + case AV_SUBTITLE_FMT_TEXT: + + if (src->text) { + dst->text = av_strdup(src->text); + if (!dst->text) + return AVERROR(ENOMEM); + } + + break; + case AV_SUBTITLE_FMT_ASS: + + if (src->ass) { + dst->ass = av_strdup(src->ass); + if (!dst->ass) + return AVERROR(ENOMEM); + } + + break; + default: + + av_log(NULL, AV_LOG_ERROR, "Subtitle area has invalid format: %d", dst->type); + return AVERROR(ENOMEM); + } + + return 0; +} + +static int frame_copy_subtitles(AVFrame *dst, const AVFrame *src, int copy_data) +{ + dst->format = src->format; + + dst->num_subtitle_areas = src->num_subtitle_areas; + + if (src->num_subtitle_areas) { + dst->subtitle_areas = av_malloc_array(src->num_subtitle_areas, sizeof(AVSubtitleArea *)); + + for (unsigned i = 0; i < src->num_subtitle_areas; i++) { + dst->subtitle_areas[i] = av_mallocz(sizeof(AVSubtitleArea)); + av_subtitle_area2area(dst->subtitle_areas[i], src->subtitle_areas[i], copy_data); + } + } + + return 0; +} + int av_frame_copy(AVFrame *dst, const AVFrame *src) { if (dst->format != src->format || dst->format < 0) return AVERROR(EINVAL); -FF_DISABLE_DEPRECATION_WARNINGS - if (dst->width > 0 && dst->height > 0) + switch(dst->type) { + case AVMEDIA_TYPE_VIDEO: return frame_copy_video(dst, src); - else if (dst->nb_samples > 0 && - (av_channel_layout_check(&dst->ch_layout) -#if FF_API_OLD_CHANNEL_LAYOUT - || dst->channel_layout || dst->channels -#endif - )) + case AVMEDIA_TYPE_AUDIO: return frame_copy_audio(dst, src); -FF_ENABLE_DEPRECATION_WARNINGS - - return AVERROR(EINVAL); + case AVMEDIA_TYPE_SUBTITLE: + return frame_copy_subtitles(dst, src, 1); + default: + return AVERROR(EINVAL); + } } void av_frame_remove_side_data(AVFrame *frame, enum AVFrameSideDataType type) diff --git a/libavutil/frame.h b/libavutil/frame.h index 33fac2054c..dd36f5b27c 100644 --- a/libavutil/frame.h +++ b/libavutil/frame.h @@ -25,7 +25,6 @@ #ifndef AVUTIL_FRAME_H #define AVUTIL_FRAME_H -#include #include #include "avutil.h" @@ -35,6 +34,7 @@ #include "rational.h" #include "samplefmt.h" #include "pixfmt.h" +#include "subfmt.h" #include "version.h" @@ -293,7 +293,7 @@ typedef struct AVRegionOfInterest { } AVRegionOfInterest; /** - * This structure describes decoded (raw) audio or video data. + * This structure describes decoded (raw) audio, video or subtitle data. * * AVFrame must be allocated using av_frame_alloc(). Note that this only * allocates the AVFrame itself, the buffers for the data must be managed @@ -407,7 +407,7 @@ typedef struct AVFrame { /** * format of the frame, -1 if unknown or unset * Values correspond to enum AVPixelFormat for video frames, - * enum AVSampleFormat for audio) + * enum AVSampleFormat for audio, AVSubtitleType for subtitles) */ int format; @@ -702,6 +702,53 @@ typedef struct AVFrame { * Channel layout of the audio data. */ AVChannelLayout ch_layout; + + /** + * Media type of the frame (audio, video, subtitles..) + * + * See AVMEDIA_TYPE_xxx + */ + enum AVMediaType type; + + /** + * Number of items in the @ref subtitle_areas array. + */ + unsigned num_subtitle_areas; + + /** + * Array of subtitle areas, may be empty. + */ + AVSubtitleArea **subtitle_areas; + + /** + * Header containing style information for text subtitles. + */ + AVBufferRef *subtitle_header; + + /** + * Indicates that a subtitle frame is a repeated frame for arbitrating flow + * in a filter graph. + * The field subtitle_timing.start_pts always indicates the original presentation + * time, while the frame's pts field may be different. + */ + int repeat_sub; + + struct SubtitleTiming + { + /** + * The display start time, in AV_TIME_BASE. + * + * For subtitle frames, AVFrame.pts is populated from the packet pts value, + * which is not always the same as this value. + */ + int64_t start_pts; + + /** + * Display duration, in AV_TIME_BASE. + */ + int64_t duration; + + } subtitle_timing; } AVFrame; @@ -778,6 +825,8 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src); /** * Allocate new buffer(s) for audio or video data. * + * Note: For subtitle data, use av_frame_get_buffer2 + * * The following fields must be set on frame before calling this function: * - format (pixel format for video, sample format for audio) * - width and height for video @@ -797,9 +846,39 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src); * recommended to pass 0 here unless you know what you are doing. * * @return 0 on success, a negative AVERROR on error. + * + * @deprecated Use @ref av_frame_get_buffer2 instead and set @ref AVFrame.type + * before calling. */ +attribute_deprecated int av_frame_get_buffer(AVFrame *frame, int align); +/** + * Allocate new buffer(s) for audio, video or subtitle data. + * + * The following fields must be set on frame before calling this function: + * - format (pixel format for video, sample format for audio) + * - width and height for video + * - nb_samples and channel_layout for audio + * - type (AVMediaType) + * + * This function will fill AVFrame.data and AVFrame.buf arrays and, if + * necessary, allocate and fill AVFrame.extended_data and AVFrame.extended_buf. + * For planar formats, one buffer will be allocated for each plane. + * + * @warning: if frame already has been allocated, calling this function will + * leak memory. In addition, undefined behavior can occur in certain + * cases. + * + * @param frame frame in which to store the new buffers. + * @param align Required buffer size alignment. If equal to 0, alignment will be + * chosen automatically for the current CPU. It is highly + * recommended to pass 0 here unless you know what you are doing. + * + * @return 0 on success, a negative AVERROR on error. + */ +int av_frame_get_buffer2(AVFrame *frame, int align); + /** * Check if the frame data is writable. * diff --git a/libavutil/subfmt.c b/libavutil/subfmt.c new file mode 100644 index 0000000000..c72ebe2a43 --- /dev/null +++ b/libavutil/subfmt.c @@ -0,0 +1,45 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include +#include "common.h" +#include "subfmt.h" + +static const char sub_fmt_info[AV_SUBTITLE_FMT_NB][24] = { + [AV_SUBTITLE_FMT_UNKNOWN] = "Unknown subtitle format", + [AV_SUBTITLE_FMT_BITMAP] = "Graphical subtitles", + [AV_SUBTITLE_FMT_TEXT] = "Text subtitles (plain)", + [AV_SUBTITLE_FMT_ASS] = "Text subtitles (ass)", +}; + +const char *av_get_subtitle_fmt_name(enum AVSubtitleType sub_fmt) +{ + if (sub_fmt < 0 || sub_fmt >= AV_SUBTITLE_FMT_NB) + return NULL; + return sub_fmt_info[sub_fmt]; +} + +enum AVSubtitleType av_get_subtitle_fmt(const char *name) +{ + for (int i = 0; i < AV_SUBTITLE_FMT_NB; i++) + if (!strcmp(sub_fmt_info[i], name)) + return i; + return AV_SUBTITLE_FMT_NONE; +} diff --git a/libavutil/subfmt.h b/libavutil/subfmt.h index 791b45519f..3b8a0613b2 100644 --- a/libavutil/subfmt.h +++ b/libavutil/subfmt.h @@ -21,6 +21,9 @@ #ifndef AVUTIL_SUBFMT_H #define AVUTIL_SUBFMT_H +#include + +#include "buffer.h" #include "version.h" enum AVSubtitleType { @@ -65,4 +68,48 @@ enum AVSubtitleType { AV_SUBTITLE_FMT_NB, ///< number of subtitle formats, DO NOT USE THIS if you want to link with shared libav* because the number of formats might differ between versions. }; +typedef struct AVSubtitleArea { +#define AV_NUM_BUFFER_POINTERS 1 + + enum AVSubtitleType type; + int flags; + + int x; ///< top left corner of area. + int y; ///< top left corner of area. + int w; ///< width of area. + int h; ///< height of area. + int nb_colors; ///< number of colors in bitmap palette (@ref pal). + + /** + * Buffers and line sizes for the bitmap of this subtitle. + * + * @{ + */ + AVBufferRef *buf[AV_NUM_BUFFER_POINTERS]; + int linesize[AV_NUM_BUFFER_POINTERS]; + /** + * @} + */ + + uint32_t pal[256]; ///< RGBA palette for the bitmap. + + char *text; ///< 0-terminated plain UTF-8 text + char *ass; ///< 0-terminated ASS/SSA compatible event line. + +} AVSubtitleArea; + +/** + * Return the name of sub_fmt, or NULL if sub_fmt is not + * recognized. + */ +const char *av_get_subtitle_fmt_name(enum AVSubtitleType sub_fmt); + +/** + * Return a subtitle format corresponding to name, or AV_SUBTITLE_FMT_NONE + * on error. + * + * @param name Subtitle format name. + */ +enum AVSubtitleType av_get_subtitle_fmt(const char *name); + #endif /* AVUTIL_SUBFMT_H */ From patchwork Sat May 28 13:25:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35958 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1365994pzj; Sat, 28 May 2022 06:26:10 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxkR7H6/IDVduY+CqhnCDf1goXF3ET9RjVsh+n/+vFa/0M6CpUzXLVSQi7TvW2tRi7pam73 X-Received: by 2002:a17:907:94d2:b0:6ff:2690:eae9 with SMTP id dn18-20020a17090794d200b006ff2690eae9mr10726987ejc.307.1653744370597; Sat, 28 May 2022 06:26:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744370; cv=none; d=google.com; s=arc-20160816; b=QT7T3bl/b9g8uQ6JL6hjO4q+/ivZDc8+tky0s8hI6kXafK4UQDMnK8hstimpFNMIxo ZvzW917TD4PdJBVDyflFycvBo3Am6b6TMX7BvHsFmQ29pVcIPTnMbfK3BAqyPrzA9PQo Y9hNO+nFfRcjSczqN9yTEaw8XvVJfo+7y6S62hi6dchl1tKH2Ee9BV12DqkdqGmPP69T AxHbAYYnu11GXdTLLiNlRJW8RTDW3gbJB9F/Ta8a2ERDoAoTTU3plPUTNSLW2Py2Q+b+ Kty0xjjpGa1p8RLurXLtF8Or9I+powp8ruseJuC1IPfUmVZfhHrVWAbDPoQN4PyxEdYZ O3pA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=B0r2vPMQZt8RfF5nEHAnPaeaoa1wdV9HiTjt890TW24=; b=UFNrkwTXm7Xkhgj/nisA9FCGA8ntz1pjhT2QyNFpNsFhppk9TXM8naeSNMCWlO5xbV 1gwGcbCB7n3wUow4gM6EhNgJHwiAPKlJS+bYeS4qMMOn5M44Qk+zKaJeBFS4TTd0+WoA 0cAJlXaut+wcf+gEuecNhlnyA/XTK2KaOAk6LBtXlzpC2hFrHekJBCE1LQMM6RcxHhmQ 1q0osLayqqMHuHVwdpvhERia2V6kPXYeflNx2uKj1fEwoZvDH+aETEnLzmNdPB9Sti9h UgnHVB4ZnnFV4pvjlmX+Sve03zDWiqLsIX52YO7UCvRxhMxbCCWEo0Qk4vqaA7I9Yhu2 mhJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b="lZkvrP/z"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id dp22-20020a170906c15600b006ff17356288si3094617ejc.803.2022.05.28.06.26.10; Sat, 28 May 2022 06:26:10 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b="lZkvrP/z"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E985568B56B; Sat, 28 May 2022 16:25:38 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CC1FF68B5D0 for ; Sat, 28 May 2022 16:25:30 +0300 (EEST) Received: by mail-pg1-f175.google.com with SMTP id e66so6207794pgc.8 for ; Sat, 28 May 2022 06:25:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/skisbii8ze4D/1K0NRg9Syg6zDsEmnSxaGfph/8Ux0=; b=lZkvrP/zJeQVI06MspaB5ESliYLp2HfYSO77phawPWUt99XnB2HM28HeFZ90KfJUtT IBxbQAo9GyvgC7Jc04+sjnjCxVUP7jtLHdLSQy4+T7xtnjInJ4TYjluRUDe4fuA7utTz FfOnnAeabpbqMP71OMZLcD5czf/CN40RoFpW2GIgSgmyGCtmAVhw4XUORDoTQm9ybeBV 3R2wX0x+YuAEAcrF54tqM7sv3QcoqbwGLQIBkv21tUgUtxuKjQ5LjEo0n9iitWacKrOU +eOUTfLJgjPDL5UgUN9IZAKCApuNXDouGKf3f6tiUNJHFKWvZL++gf06bED1SkVFZ7vj vUzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/skisbii8ze4D/1K0NRg9Syg6zDsEmnSxaGfph/8Ux0=; b=Tmq7SRTKbOKBIIJcdeTA81FoP6/tM9pVjuu7AKwSFeoEoKJAjYKEa0QjjWWd4Xy+sz vFnKDmOfMWppHp3EEfhYqWN73OOeJx5/Y8Uxf5GEIXY8fETZidWJ9lfDx5CouGkHFDcc n4rqGoWTad7KOJ9e5En6DRV/CPTNq+Gizyr+pE1Mw1rAaCSsMjL4J0nBWNP96rq7gMjO TdrWcvsF0ChpFQYo7rLDt8NyW10t2WFEDPt3zwZH1n7cpNSDR5xBSqSwIYNRpoVvIeRN Ci3QaIBZ0NQKTikSAXSSQr1YU13FS1/12OKwCPS6FKBgNn8QOYr92yaK4/ZOiJagMhrn 2SJg== X-Gm-Message-State: AOAM5332kHAZrlRz/QEq9kFvA3jStqutoM1pXEuBz52IKvBIkVpLWzEK pgkrX2/WOuc4RkNZ1fksQ9XxelJytGN3uA== X-Received: by 2002:a05:6a00:1f0d:b0:518:3c8d:78b1 with SMTP id be13-20020a056a001f0d00b005183c8d78b1mr47765855pfb.23.1653744329187; Sat, 28 May 2022 06:25:29 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id k11-20020a170902ba8b00b0015f0dcd1579sm5641877pls.9.2022.05.28.06.25.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:28 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: In-Reply-To: References: Date: Sat, 28 May 2022 13:25:03 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 03/23] avcodec/subtitles: Introduce new frame-based subtitle decoding API X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: O/1HiWboNV08 From: softworkz - Modify avcodec_send_packet() to support subtitles via the regular frame based decoding API - Add decode_subtitle_shim() which takes subtitle frames, and serves as a compatibility shim to the legacy subtitle decoding API until all subtitle decoders are migrated to the frame-based API - Add additional methods for conversion between old and new API Signed-off-by: softworkz --- libavcodec/avcodec.c | 8 ++ libavcodec/avcodec.h | 10 ++- libavcodec/decode.c | 60 ++++++++++++-- libavcodec/internal.h | 16 ++++ libavcodec/utils.c | 184 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 269 insertions(+), 9 deletions(-) diff --git a/libavcodec/avcodec.c b/libavcodec/avcodec.c index 5f6e71a39e..0a1d961fc6 100644 --- a/libavcodec/avcodec.c +++ b/libavcodec/avcodec.c @@ -358,6 +358,14 @@ FF_DISABLE_DEPRECATION_WARNINGS FF_ENABLE_DEPRECATION_WARNINGS #endif + // Set the subtitle type from the codec descriptor in case the decoder hasn't done itself + if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE && avctx->subtitle_type == AV_SUBTITLE_FMT_UNKNOWN) { + if(avctx->codec_descriptor->props & AV_CODEC_PROP_BITMAP_SUB) + avctx->subtitle_type = AV_SUBTITLE_FMT_BITMAP; + if(avctx->codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB) + avctx->subtitle_type = AV_SUBTITLE_FMT_ASS; + } + #if FF_API_AVCTX_TIMEBASE if (avctx->framerate.num > 0 && avctx->framerate.den > 0) avctx->time_base = av_inv_q(av_mul_q(avctx->framerate, (AVRational){avctx->ticks_per_frame, 1})); diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h index 56d551f92d..de87b0406b 100644 --- a/libavcodec/avcodec.h +++ b/libavcodec/avcodec.h @@ -1698,7 +1698,7 @@ typedef struct AVCodecContext { /** * Header containing style information for text subtitles. - * For SUBTITLE_ASS subtitle type, it should contain the whole ASS + * For AV_SUBTITLE_FMT_ASS subtitle type, it should contain the whole ASS * [Script Info] and [V4+ Styles] section, plus the [Events] line and * the Format line following. It shouldn't include any Dialogue line. * - encoding: Set/allocated/freed by user (before avcodec_open2()) @@ -2056,6 +2056,8 @@ typedef struct AVCodecContext { * The decoder can then override during decoding as needed. */ AVChannelLayout ch_layout; + + enum AVSubtitleType subtitle_type; } AVCodecContext; /** @@ -2432,7 +2434,10 @@ int avcodec_close(AVCodecContext *avctx); * Free all allocated data in the given subtitle struct. * * @param sub AVSubtitle to free. + * + * @deprecated Use the regular frame based encode and decode APIs instead. */ +attribute_deprecated void avsubtitle_free(AVSubtitle *sub); /** @@ -2525,7 +2530,10 @@ enum AVChromaLocation avcodec_chroma_pos_to_enum(int xpos, int ypos); * must be freed with avsubtitle_free if *got_sub_ptr is set. * @param[in,out] got_sub_ptr Zero if no subtitle could be decompressed, otherwise, it is nonzero. * @param[in] avpkt The input AVPacket containing the input buffer. + * + * @deprecated Use the new decode API (avcodec_send_packet, avcodec_receive_frame) instead. */ +attribute_deprecated int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, int *got_sub_ptr, AVPacket *avpkt); diff --git a/libavcodec/decode.c b/libavcodec/decode.c index 1893caa6a6..e8ca7b6da4 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -573,6 +573,39 @@ static int decode_receive_frame_internal(AVCodecContext *avctx, AVFrame *frame) return ret; } +static int decode_subtitle2_priv(AVCodecContext *avctx, AVSubtitle *sub, + int *got_sub_ptr, AVPacket *avpkt); + +static int decode_subtitle_shim(AVCodecContext *avctx, AVFrame *frame, AVPacket *avpkt) +{ + int ret, got_sub_ptr = 0; + AVSubtitle subtitle = { 0 }; + + if (frame->buf[0]) + return AVERROR(EAGAIN); + + av_frame_unref(frame); + + ret = decode_subtitle2_priv(avctx, &subtitle, &got_sub_ptr, avpkt); + + if (ret >= 0 && got_sub_ptr) { + frame->type = AVMEDIA_TYPE_SUBTITLE; + frame->format = subtitle.format; + ret = av_frame_get_buffer2(frame, 0); + + if (ret >= 0) + ret = ff_frame_put_subtitle(frame, &subtitle); + + frame->width = avctx->width; + frame->height = avctx->height; + frame->pkt_dts = avpkt->dts; + } + + avsubtitle_free(&subtitle); + + return ret; +} + int attribute_align_arg avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt) { AVCodecInternal *avci = avctx->internal; @@ -587,6 +620,13 @@ int attribute_align_arg avcodec_send_packet(AVCodecContext *avctx, const AVPacke if (avpkt && !avpkt->size && avpkt->data) return AVERROR(EINVAL); + if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE) + // this does not exactly implement the avcodec_send_packet/avcodec_receive_frame API + // but we know that no subtitle decoder produces multiple AVSubtitles per packet through + // the legacy API, and this will be changed when migrating the subtitle decoders + // to the frame based decoding api + return decode_subtitle_shim(avctx, avci->buffer_frame, avpkt); + av_packet_unref(avci->buffer_pkt); if (avpkt && (avpkt->data || avpkt->side_data_elems)) { ret = av_packet_ref(avci->buffer_pkt, avpkt); @@ -648,7 +688,9 @@ int attribute_align_arg avcodec_receive_frame(AVCodecContext *avctx, AVFrame *fr if (avci->buffer_frame->buf[0]) { av_frame_move_ref(frame, avci->buffer_frame); - } else { + } else if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE) + return AVERROR(EAGAIN); + else { ret = decode_receive_frame_internal(avctx, frame); if (ret < 0) return ret; @@ -813,9 +855,8 @@ static int utf8_check(const uint8_t *str) return 1; } -int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, - int *got_sub_ptr, - AVPacket *avpkt) +static int decode_subtitle2_priv(AVCodecContext *avctx, AVSubtitle *sub, + int *got_sub_ptr, AVPacket *avpkt) { int ret = 0; @@ -861,10 +902,7 @@ int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, avctx->pkt_timebase, ms); } - if (avctx->codec_descriptor->props & AV_CODEC_PROP_BITMAP_SUB) - sub->format = 0; - else if (avctx->codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB) - sub->format = 1; + sub->format = (uint16_t)avctx->subtitle_type; for (unsigned i = 0; i < sub->num_rects; i++) { if (avctx->sub_charenc_mode != FF_SUB_CHARENC_MODE_IGNORE && @@ -885,6 +923,12 @@ int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, return ret; } +int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, + int *got_sub_ptr, AVPacket *avpkt) +{ + return decode_subtitle2_priv(avctx, sub, got_sub_ptr, avpkt); +} + enum AVPixelFormat avcodec_default_get_format(struct AVCodecContext *avctx, const enum AVPixelFormat *fmt) { diff --git a/libavcodec/internal.h b/libavcodec/internal.h index 17e1de8127..69656729d8 100644 --- a/libavcodec/internal.h +++ b/libavcodec/internal.h @@ -290,4 +290,20 @@ int ff_int_from_list_or_default(void *ctx, const char * val_name, int val, void ff_dvdsub_parse_palette(uint32_t *palette, const char *p); +/** + * Copies subtitle data from AVSubtitle to AVFrame. + * + * @deprecated This is a compatibility method for interoperability with + * the legacy subtitle API. + */ +int ff_frame_put_subtitle(AVFrame* frame, const AVSubtitle* sub); + +/** + * Copies subtitle data from AVFrame to AVSubtitle. + * + * @deprecated This is a compatibility method for interoperability with + * the legacy subtitle API. + */ +int ff_frame_get_subtitle(AVSubtitle* sub, AVFrame* frame); + #endif /* AVCODEC_INTERNAL_H */ diff --git a/libavcodec/utils.c b/libavcodec/utils.c index eb7e505a62..b67b6b6122 100644 --- a/libavcodec/utils.c +++ b/libavcodec/utils.c @@ -827,6 +827,190 @@ FF_ENABLE_DEPRECATION_WARNINGS return FFMAX(0, duration); } +static int subtitle_area2rect(AVSubtitleRect *dst, const AVSubtitleArea *src) +{ + dst->x = src->x; + dst->y = src->y; + dst->w = src->w; + dst->h = src->h; + dst->nb_colors = src->nb_colors; + dst->type = src->type; + dst->flags = src->flags; + + switch (dst->type) { + case AV_SUBTITLE_FMT_BITMAP: + + if (src->h > 0 && src->w > 0 && src->buf[0]) { + uint32_t *pal; + AVBufferRef *buf = src->buf[0]; + dst->data[0] = av_mallocz(buf->size); + memcpy(dst->data[0], buf->data, buf->size); + dst->linesize[0] = src->linesize[0]; + + dst->data[1] = av_mallocz(256 * 4); + pal = (uint32_t *)dst->data[1]; + + for (unsigned i = 0; i < 256; i++) { + pal[i] = src->pal[i]; + } + } + + break; + case AV_SUBTITLE_FMT_TEXT: + + if (src->text) + dst->text = av_strdup(src->text); + else + dst->text = av_strdup(""); + + if (!dst->text) + return AVERROR(ENOMEM); + + break; + case AV_SUBTITLE_FMT_ASS: + + if (src->ass) + dst->ass = av_strdup(src->ass); + else + dst->ass = av_strdup(""); + + if (!dst->ass) + return AVERROR(ENOMEM); + + break; + default: + + av_log(NULL, AV_LOG_ERROR, "Subtitle rect has invalid format: %d", dst->type); + return AVERROR(EINVAL); + } + + return 0; +} + +static int subtitle_rect2area(AVSubtitleArea *dst, const AVSubtitleRect *src) +{ + dst->x = src->x; + dst->y = src->y; + dst->w = src->w; + dst->h = src->h; + dst->nb_colors = src->nb_colors; + dst->type = src->type; + dst->flags = src->flags; + + switch (dst->type) { + case AV_SUBTITLE_FMT_BITMAP: + + if (src->h > 0 && src->w > 0 && src->data[0]) { + AVBufferRef *buf = av_buffer_allocz(src->h * src->linesize[0]); + memcpy(buf->data, src->data[0], buf->size); + + dst->buf[0] = buf; + dst->linesize[0] = src->linesize[0]; + } + + if (src->data[1]) { + uint32_t *pal = (uint32_t *)src->data[1]; + + for (unsigned i = 0; i < 256; i++) { + dst->pal[i] = pal[i]; + } + } + + break; + case AV_SUBTITLE_FMT_TEXT: + + if (src->text) { + dst->text = av_strdup(src->text); + if (!dst->text) + return AVERROR(ENOMEM); + } + + break; + case AV_SUBTITLE_FMT_ASS: + + if (src->ass) { + dst->ass = av_strdup(src->ass); + if (!dst->ass) + return AVERROR(ENOMEM); + } + + break; + default: + + av_log(NULL, AV_LOG_ERROR, "Subtitle area has invalid format: %d", dst->type); + return AVERROR(EINVAL); + } + + return 0; +} + +/** + * Copies subtitle data from AVSubtitle (deprecated) to AVFrame + * + * @note This is a compatibility method for conversion to the legacy API + */ +int ff_frame_put_subtitle(AVFrame *frame, const AVSubtitle *sub) +{ + frame->format = sub->format; + frame->subtitle_timing.start_pts = sub->pts; + frame->subtitle_timing.start_pts += av_rescale_q(sub->start_display_time, (AVRational){ 1, 1000 }, AV_TIME_BASE_Q); + frame->subtitle_timing.duration = av_rescale_q(sub->end_display_time - sub->start_display_time, (AVRational){ 1, 1000 }, AV_TIME_BASE_Q); + + if (sub->num_rects) { + frame->subtitle_areas = av_malloc_array(sub->num_rects, sizeof(AVSubtitleArea*)); + if (!frame->subtitle_areas) + return AVERROR(ENOMEM); + + for (unsigned i = 0; i < sub->num_rects; i++) { + int ret; + frame->subtitle_areas[i] = av_mallocz(sizeof(AVSubtitleArea)); + if (!frame->subtitle_areas[i]) + return AVERROR(ENOMEM); + ret = subtitle_rect2area(frame->subtitle_areas[i], sub->rects[i]); + if (ret < 0) { + frame->num_subtitle_areas = i; + return ret; + } + } + } + + frame->num_subtitle_areas = sub->num_rects; + return 0; +} + +/** + * Copies subtitle data from AVFrame to AVSubtitle (deprecated) + * + * @note This is a compatibility method for conversion to the legacy API + */ +int ff_frame_get_subtitle(AVSubtitle *sub, AVFrame *frame) +{ + const int64_t duration_ms = av_rescale_q(frame->subtitle_timing.duration, AV_TIME_BASE_Q, (AVRational){ 1, 1000 }); + + sub->start_display_time = 0; + sub->end_display_time = (int32_t)duration_ms; + sub->pts = frame->subtitle_timing.start_pts; + + if (frame->num_subtitle_areas) { + sub->rects = av_malloc_array(frame->num_subtitle_areas, sizeof(AVSubtitleRect*)); + if (!sub->rects) + return AVERROR(ENOMEM); + + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + int ret; + sub->rects[i] = av_mallocz(sizeof(AVSubtitleRect)); + ret = subtitle_area2rect(sub->rects[i], frame->subtitle_areas[i]); + if (ret < 0) { + sub->num_rects = i; + return ret; + } + } + } + + sub->num_rects = frame->num_subtitle_areas; + return 0; +} + int av_get_audio_frame_duration2(AVCodecParameters *par, int frame_bytes) { int channels = par->ch_layout.nb_channels; From patchwork Sat May 28 13:25:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35959 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366035pzj; Sat, 28 May 2022 06:26:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyTdPpLyLE+iUBcpws1eLH26PGKG74RmqZZjNGxwKyizonx3WHXNggIyIidSws9gAOoUzqf X-Received: by 2002:aa7:d851:0:b0:42b:d6f4:bc0e with SMTP id f17-20020aa7d851000000b0042bd6f4bc0emr14252221eds.21.1653744379930; Sat, 28 May 2022 06:26:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744379; cv=none; d=google.com; s=arc-20160816; b=iYjIOD+GBvl9o15QudQ6Xtr+Od2o5QrwI7s4g1b0uFANfsBvxG10uTOZGxT9ihKFjS kBjEXIkXjRSRL7FT0/iRyMdoWvG3qSxpf3IzWb2y9IGh/EKQrcmqEeltJaP/oQmqRGic VGyUWZTohgOkKTD9zF1tt5GMLQsjaxkyfmqNaa2Wn1GT9+R+rwLJZWZ/rvhKiVMMTIE9 Goi2uzbXRJAWMGBF+/xziH8dJvDOzyjiG1ZdbL2IbB/wKaWv+0Jgg9du7LQbPtT+Fbek z8dZNqDcgKS/kcAHYYmGZHQwIP0beIP83E3OCy3kSaC9q6VEA1cWF+RrZw9IjcKo1mv4 t8YA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=bgO2FILsiUaA9+DuYBhV5+Tp3oqJK/J1s227fb/Pp94=; b=ZiedccFATq+bDBSxEmQZxHNUQHgsbsWaxn3ahGT0YCqWSts6g5FAdIaTvQrKYVgRaK 0nksLAIBVmQrh47wIkevxjWIghpxcTrPKC9LDCCY/tfFLOG+TeoXA7LmZ0od60hUNUjo hruwbAP8GhJoetNBq4aVOLPAt+22hCcHi0/ateRZ/MIcWaIGvuNzewnjr+Fnl1QdKCVW FzDlCQGcF8NQ77+xq5hS7q3UMdeQn8aO3DBLPPGvdYJFJDtO9+xASywBgftkRjuFLGQQ HE86jIIUIDDyK9UyVFGpdPSHbNcACPrvdkgxY0VkPPjwAsbE+ji772pU7KiAsiG27BRj HHmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=ZfQTUUrb; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t34-20020a056402242200b0042dbcd63aabsi1276396eda.587.2022.05.28.06.26.19; Sat, 28 May 2022 06:26:19 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=ZfQTUUrb; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E8DF868B608; Sat, 28 May 2022 16:25:39 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CDE6E68B5A1 for ; Sat, 28 May 2022 16:25:31 +0300 (EEST) Received: by mail-pg1-f176.google.com with SMTP id x12so6207844pgj.7 for ; Sat, 28 May 2022 06:25:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Xms+Nl1RCwtH/NgtY1GhF/TxEMmjHnLFrVwEyH/IDWM=; b=ZfQTUUrbUVh2N/FOPh17NyAwdn0cbwiqdWHdpMuRBTrvYL28QhG8Da5g4bZMfxm5zU zvnfIxbmTpO3KnuByFdC5YsTguaccTvqNQn2JlWowGg937rhRdnJ3qWkYo3ikfNxciC1 /lw4jUzWB2fGN7VvYOeu6wURixNeVMA3wXhwjdgScT8s69z/D7vNUvBtazNlIMuc4FuF IfJjZcfBDoiMwwTOTIEDyGYQpfm/XcLfoqS8ogFCIZ59SfZbtHdIrR8p8EbCnIMMOepk tL14KWmkT89+ChGQS2qu9lGBHp5MFo8Y6beY7B3qyJLrfboQ0C1xjLHwWgfCZXLN1s33 VrOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Xms+Nl1RCwtH/NgtY1GhF/TxEMmjHnLFrVwEyH/IDWM=; b=eVDSWLQzzuuNtDPMfeHuypIriOOHttsAA2SZ9g8MZKc5/xV+1EmhRmREYdo2AlQSYg dnotMYrK/dU31cYhVLEl9aFsFy2hPM4afN/nmf2vvMIB91ba9sMddBoyMiXxMZfRs3xo T4NWp1imaLw7hBhwCRcgsGjhfUuaJl+l6rLDDAyCF2/XAmsuo2PQW9saUE+ghk/Aamjf rby0hWXzYl6QX4CpiytAjA0TEuubET5Or/dDRzPs0kscaTEwPKgumu8LM2JkQhyFYmK/ K3hvHvsShnDp9k6df9SbxZq8DcHIrnWIlGOpoXqDNjJ0BksSBaILYBvsZp7Xlxh6EbKD a+Jg== X-Gm-Message-State: AOAM532J0WjBvCmHS+FKfoYFpmFKeW+L7NhXwuEBGKXrwWqA8wCMQax5 VljrxlvFJ0KOWh9IV28GA4iJHiSTlN4tYw== X-Received: by 2002:a63:f158:0:b0:3db:8563:e8f5 with SMTP id o24-20020a63f158000000b003db8563e8f5mr41921400pgk.191.1653744330307; Sat, 28 May 2022 06:25:30 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id y25-20020aa78059000000b0051850716942sm5550225pfm.140.2022.05.28.06.25.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:29 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <4b44732e072d508ce76d5c9fb77ace424f1b5007.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:04 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 04/23] avcodec/libzvbi: set subtitle type X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: vT2sCXFhdEKt From: softworkz Signed-off-by: softworkz --- libavcodec/libzvbi-teletextdec.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/libavcodec/libzvbi-teletextdec.c b/libavcodec/libzvbi-teletextdec.c index 92466cc11e..2aab10a548 100644 --- a/libavcodec/libzvbi-teletextdec.c +++ b/libavcodec/libzvbi-teletextdec.c @@ -751,10 +751,13 @@ static int teletext_init_decoder(AVCodecContext *avctx) switch (ctx->format_id) { case 0: + avctx->subtitle_type = AV_SUBTITLE_FMT_BITMAP; return 0; case 1: + avctx->subtitle_type = AV_SUBTITLE_FMT_ASS; return ff_ass_subtitle_header_default(avctx); case 2: + avctx->subtitle_type = AV_SUBTITLE_FMT_ASS; return my_ass_subtitle_header(avctx); } return AVERROR_BUG; From patchwork Sat May 28 13:25:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35962 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366181pzj; Sat, 28 May 2022 06:26:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx0mqvUAbiDz1tN/5Ob3xggEXKeNrNW+cX43xJtrTZJue9gKHIa7GJmbbWfWX/DYK8wpW1I X-Received: by 2002:a17:906:3a45:b0:6f4:e9e7:4ff with SMTP id a5-20020a1709063a4500b006f4e9e704ffmr41323454ejf.100.1653744399632; Sat, 28 May 2022 06:26:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744399; cv=none; d=google.com; s=arc-20160816; b=lL10N0aD2CuQdcPO1vJI1wEwtzI3nFY4t6BKUW+svRYZRCsGytZfZ4F24r0lX5zZfy TIJgELjDvn9CF4j52Z3txwQ8W6nd5gmhxaQ8tKAYd2wCm9Z6lIUhEzAYuTFj8pBSOw3g rEwaFh633HgkBheR3BpHRa9Kbe9Vt3xBktb/G70Og2AIhSnjY2nAZrNaRQ+ysARLkoXt eo+speKBBoUyikRWiIc+9gfvZs967CKomZCeRbF3Hfm4qjN4DbYLMAhK/DQwbALrvwQX qQnogXMLjAnE0AOKwbCG4fH28v0FZ4PIfcZYVmhV8d2pUR0X9xrFBOQrFtwVPjVWiwGS D7qA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=77pS+/jtHJYjMI6CvOVvrbGk3iHeJem4CBImfLKA2gQ=; b=cp21F2sHvVfU6tjWxRWEwsMTRlUpIEavYYoSA2eCgBZjH18d9uvqxOxed+VcQzM2Cy +fAz50XlVaA1/M+XxS6UZLvOsN9vU/dQ7B/PlsZkagYQsNzqwZSdf3TsvG3oCwGfGZub td5z5md4uc+fvfkCSu2FCBD295oKlMelb2R8dqebQzYtfar56rYvbhgiZ5kuu1WcU1BT Xpa9dfAvkyzQt8HTgYTXVd3mID6y+pOwu87kYOF2ujrETiXqXdCSmp7rxyiznyoZX22p TkSYGWbjLxLved4SN7Wcnev9SdWJ3GGoDXbBHgDV4w8lSUCCM9kU+sy10JpkmK4TGVkK 67YQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=GSh7OlKz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id oz16-20020a1709077d9000b006fef1a9ef80si7425953ejc.397.2022.05.28.06.26.39; Sat, 28 May 2022 06:26:39 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=GSh7OlKz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 98DE568B5C9; Sat, 28 May 2022 16:25:41 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E46A368B5E5 for ; Sat, 28 May 2022 16:25:32 +0300 (EEST) Received: by mail-pf1-f177.google.com with SMTP id c14so6649580pfn.2 for ; Sat, 28 May 2022 06:25:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=OwAbOvynqRhMS61zCf3wNHPYflVi3DT6Sol+N98Ic8k=; b=GSh7OlKz+WHHdyVoSsgjr0z4vzMkgnHV+CeGf+ova26hw6oQAzrGc13NiGZhFFs5wl ltlcc1IkO5hC4nxkd84G5+JOhB6immY8Bq9CQeugsYtZ9uqmOMthhyNhiZ2D/OPYybt3 wpx+FOBy1te/CdsHTYSBDuZfnVgJj37amMCKNsKaGNhqo0kjJGRtxPAkMKdCZyjlWrie dOEYCZOlQoI0oUB/TnUBYBUhVrfQmu/yl6sWavEvQEbOsG+TQaZyiEUlcnsel0DCx3DH yG/zMD3GkhM0H4n5mYVpIq7y34To7fI5atPyXfQPRASefJtXmK2kwvrAE5WTy0e0BUlm IA8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=OwAbOvynqRhMS61zCf3wNHPYflVi3DT6Sol+N98Ic8k=; b=WqF4WMOh8QzO/Xq51kxhGjQwdZTmbva5NCWrf+0n/HPRbup9iO9d6EDFMDIY2gr4gO 6XYgi18mu2uIme5DAmRHVosZ8sK3FVIxcExjbROTt3B3I0V0Yr2eESq8Ic3pCG5PIIQ+ KVwq80pfcPGkkIF7jFNO7TRsmOPIWbQYxCukqUCHTTwQsIUi1mHkg+JzIGqfZR6OWb4h oAsKlkJ2D+9nKo4hkI4vOZYAdk7cHtQi/x+on7+mvrBqpo29uc7nQOefuMkpfvDEJjOU XrWSvjLdR34VpaScFNFv17K1K7Ziypa9mA9qimVJebVhK8NQtGsCSPa6TAjdiK5rsrnc EiiA== X-Gm-Message-State: AOAM533bK5g78ImFe3xg8jOtiJ4Wg87RrC2Vwa2MglmEprCLZ2g3D+0p wQ+SfcWS+H5iSNAoPOVfiDhdUIHSTW6fig== X-Received: by 2002:a63:4853:0:b0:3fa:dc6:7ac2 with SMTP id x19-20020a634853000000b003fa0dc67ac2mr29673671pgk.298.1653744331333; Sat, 28 May 2022 06:25:31 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id b14-20020aa7810e000000b005183c6a21c8sm5286565pfi.165.2022.05.28.06.25.30 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:30 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <8faa7a404341796e9497adbd645e20358a06d785.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:05 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 05/23] avfilter/subtitles: Update vf_subtitles to use new decoding api X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: I3gYoo4oqg6x From: softworkz Signed-off-by: softworkz --- libavfilter/vf_subtitles.c | 67 ++++++++++++++++++++++++++++++-------- 1 file changed, 54 insertions(+), 13 deletions(-) diff --git a/libavfilter/vf_subtitles.c b/libavfilter/vf_subtitles.c index 82e140e986..0ae156ad07 100644 --- a/libavfilter/vf_subtitles.c +++ b/libavfilter/vf_subtitles.c @@ -36,14 +36,12 @@ # include "libavformat/avformat.h" #endif #include "libavutil/avstring.h" -#include "libavutil/imgutils.h" #include "libavutil/opt.h" #include "libavutil/parseutils.h" #include "drawutils.h" #include "avfilter.h" #include "internal.h" #include "formats.h" -#include "video.h" typedef struct AssContext { const AVClass *class; @@ -304,8 +302,42 @@ static int attachment_is_font(AVStream * st) return 0; } +static int decode(AVCodecContext *avctx, AVFrame *frame, int *got_frame, AVPacket *pkt) +{ + int ret; + + *got_frame = 0; + + if (pkt) { + ret = avcodec_send_packet(avctx, pkt); + // In particular, we don't expect AVERROR(EAGAIN), because we read all + // decoded frames with avcodec_receive_frame() until done. + if (ret < 0 && ret != AVERROR_EOF) + return ret; + } + + ret = avcodec_receive_frame(avctx, frame); + if (ret < 0 && ret != AVERROR(EAGAIN)) + return ret; + if (ret >= 0) + *got_frame = 1; + + return 0; +} + AVFILTER_DEFINE_CLASS(subtitles); +static enum AVSubtitleType get_subtitle_format(const AVCodecDescriptor *codec_descriptor) +{ + if(codec_descriptor->props & AV_CODEC_PROP_BITMAP_SUB) + return AV_SUBTITLE_FMT_BITMAP; + + if(codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB) + return AV_SUBTITLE_FMT_ASS; + + return AV_SUBTITLE_FMT_UNKNOWN; +} + static av_cold int init_subtitles(AVFilterContext *ctx) { int j, ret, sid; @@ -318,6 +350,7 @@ static av_cold int init_subtitles(AVFilterContext *ctx) AVStream *st; AVPacket pkt; AssContext *ass = ctx->priv; + enum AVSubtitleType subtitle_format; /* Init libass */ ret = init(ctx); @@ -398,13 +431,17 @@ static av_cold int init_subtitles(AVFilterContext *ctx) ret = AVERROR_DECODER_NOT_FOUND; goto end; } + dec_desc = avcodec_descriptor_get(st->codecpar->codec_id); - if (dec_desc && !(dec_desc->props & AV_CODEC_PROP_TEXT_SUB)) { + subtitle_format = get_subtitle_format(dec_desc); + + if (subtitle_format != AV_SUBTITLE_FMT_ASS) { av_log(ctx, AV_LOG_ERROR, - "Only text based subtitles are currently supported\n"); - ret = AVERROR_PATCHWELCOME; + "Only text based subtitles are supported by this filter\n"); + ret = AVERROR_INVALIDDATA; goto end; } + if (ass->charenc) av_dict_set(&codec_opts, "sub_charenc", ass->charenc, 0); @@ -460,27 +497,31 @@ static av_cold int init_subtitles(AVFilterContext *ctx) dec_ctx->subtitle_header_size); while (av_read_frame(fmt, &pkt) >= 0) { int i, got_subtitle; - AVSubtitle sub = {0}; + AVFrame *sub = av_frame_alloc(); + if (!sub) { + ret = AVERROR(ENOMEM); + goto end; + } if (pkt.stream_index == sid) { - ret = avcodec_decode_subtitle2(dec_ctx, &sub, &got_subtitle, &pkt); + ret = decode(dec_ctx, sub, &got_subtitle, &pkt); if (ret < 0) { av_log(ctx, AV_LOG_WARNING, "Error decoding: %s (ignored)\n", av_err2str(ret)); } else if (got_subtitle) { - const int64_t start_time = av_rescale_q(sub.pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); - const int64_t duration = sub.end_display_time; - for (i = 0; i < sub.num_rects; i++) { - char *ass_line = sub.rects[i]->ass; + const int64_t start_time = av_rescale_q(sub->subtitle_timing.start_pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t duration = av_rescale_q(sub->subtitle_timing.duration, AV_TIME_BASE_Q, av_make_q(1, 1000)); + for (i = 0; i < sub->num_subtitle_areas; i++) { + char *ass_line = sub->subtitle_areas[i]->ass; if (!ass_line) - break; + continue; ass_process_chunk(ass->track, ass_line, strlen(ass_line), start_time, duration); } } } av_packet_unref(&pkt); - avsubtitle_free(&sub); + av_frame_free(&sub); } end: From patchwork Sat May 28 13:25:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35964 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366270pzj; Sat, 28 May 2022 06:27:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyoUkUabGnCl4mKAFSNjFM+smN12xm9Ah6YJwFwnCKI25KgcWquocLaz3UU3bVV6GHuRlzk X-Received: by 2002:a17:907:6eab:b0:6fe:b5e2:7b0e with SMTP id sh43-20020a1709076eab00b006feb5e27b0emr35803479ejc.736.1653744428691; Sat, 28 May 2022 06:27:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744428; cv=none; d=google.com; s=arc-20160816; b=oGQWCM5NoKYiV/HHt/cl+8zLqz7Sx3rct7IhRNDWEFY0PyUb2lOdOrUVq6xkHcxZZ2 4fWgRhXMeZbzC4up6+NdW19W7Y9sgOIDOTBv8XnLy+k/3mNdwR76aHKiJsqZkL+wDAGE xtow9f7txfNqK/qBF5USEkBpE98iWzm06KS/hy2hjcaNCtZk7dVqLKBn8uSoj72z7zc5 zFaXjR9e4yQKlNB0VFwsa/Cmfs2gs8KdJ0np9d+3/GVzQnV9sroXRzOEUsa5YSOFP18Y OEnXM6AeGNafl88Ug4Oi/xv6f/8BmurtIEXvx14hbpaGH0KKT3Ob8lsXPuZ3sgcnUxnp 6tPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=7Wkod7KVWhhDiYKbNp+NHGVnaSb8ImOuE3jMS5TFdrg=; b=e2fRPei+Co4EtQNXci8njWWEJcUYIkjDVERmqDE0Un1AeyGiLTPw7AcYHrFUS/4/Lo dUolnnIVCk6+9fFoO7r2EgMwWVxV0hl0RHAvUb1bGClo7+EtocZtu1nT5LRb61rfA95M vHQxClXoQ+3O9U9cHc/bs/++L4jLUdwr7Fmp7AbdBpOmuWauI1xQc0jUloyMsOHJDB5a 4nBqRS4Ew880O0vYXVua40j9kiVezg/yAXyu59Wrp2J6CqwwOVtxU3CRJ0YbhcxYbCSW oZr9n8epF+DC57AdaElKtuAA4QE2SKMhbS9LzQNQqeyvNgFXduebPvxgI2s2xKMdYUqy HCgA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b="ib+UO/Fh"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c8-20020a056402120800b0042b55aa0742si4965411edw.281.2022.05.28.06.27.08; Sat, 28 May 2022 06:27:08 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b="ib+UO/Fh"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2CCDA68B634; Sat, 28 May 2022 16:25:44 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7E1F068B5D4 for ; Sat, 28 May 2022 16:25:34 +0300 (EEST) Received: by mail-pl1-f171.google.com with SMTP id s14so6447668plk.8 for ; Sat, 28 May 2022 06:25:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=FxDa04BbsIhlmM4n/XTwJ75S48ZDS+yz7CjCphxgYuI=; b=ib+UO/Fhsbvge7gIQ4Vm/fgPm4xnv3VmKVTP40GxNmi1OASepSNqw3n3u3AyYD+p0z lSkz/jY4YFbmpJRcGHdZZUtBIXNy2P7xr2X7P64UV5m4lVnl5yE8kiM6U9FwOaWfWoDV ebKnYQOIfADNnfCHX4ANzHKPvTd3mqEioL1WXp+8zd6x1mnPW4yR/6L7Xf5VG28MCevi AwYq5Z+N+RrqzuroulU+pS6s/AMnAHkdt+4gWZS8DBF6mxNeegDwItRt/4eIMD3L4VvU vBnMYVES4imziZPiN6EW+10NjjXbH1nJ1v0BzF0krw+idfcehrvaynSR+D2Qv/BIHu95 IUeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=FxDa04BbsIhlmM4n/XTwJ75S48ZDS+yz7CjCphxgYuI=; b=x/X5445SAcWq0P07cCrs1Z6OMwBJeRDgtfisINWdyZceMW+gWjpYCBp1lHe8HVWeDc /4SMT9bxwK4mVObGZ/FeMzHFrMWFlgEDGF2NmeLGyKKp4Bmo9w8qHqRhmpRZw9qLaHz3 UAGHy8iRMibxcb6S/aLeQouMIzmDH4mkWLpsNJDUNdSySwHmIe3SST4oG65rGV7Z7a6M t5VdsrmgdhUnkI9F0NSc/5rUn3EmxecVCPSyz7k3tY+5b3AE/AIovmwa+qXM6+oO5dv3 8suu1AN5EGR0oj41/tD7nYTqugZhQMB8yWtga0TX7XbVfo1LiHz5X3VQHqSqrw4zWXQL VN4w== X-Gm-Message-State: AOAM5319D8dht5F87aCpTlU77wVpc9wi1kyI4HpIwbpEt3zRjmuJkgI3 H5yQdpg3BaMoglg6WQE3PNpki//ie5IXpg== X-Received: by 2002:a17:902:aa07:b0:162:467:db94 with SMTP id be7-20020a170902aa0700b001620467db94mr36572900plb.26.1653744332457; Sat, 28 May 2022 06:25:32 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id b2-20020aa78ec2000000b0051868418b06sm5288151pfr.35.2022.05.28.06.25.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:31 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <1664026d7c73df7ee7dce777e39609cd822c3162.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:06 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 06/23] avcodec, avutil: Move ass helper functions to avutil as avpriv_ and extend ass dialog parsing X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Wk92bxbPtsl0 From: softworkz Also add - hard_space callback (for upcoming fix) - extensible callback (for future extension) Signed-off-by: softworkz --- libavcodec/Makefile | 56 +++---- libavcodec/ass.h | 151 ++++++------------ libavcodec/assdec.c | 2 +- libavcodec/assenc.c | 2 +- libavcodec/ccaption_dec.c | 20 +-- libavcodec/jacosubdec.c | 2 +- libavcodec/libaribb24.c | 2 +- libavcodec/libzvbi-teletextdec.c | 14 +- libavcodec/microdvddec.c | 7 +- libavcodec/movtextdec.c | 3 +- libavcodec/movtextenc.c | 20 +-- libavcodec/mpl2dec.c | 2 +- libavcodec/realtextdec.c | 2 +- libavcodec/samidec.c | 2 +- libavcodec/srtdec.c | 2 +- libavcodec/srtenc.c | 16 +- libavcodec/subviewerdec.c | 2 +- libavcodec/textdec.c | 4 +- libavcodec/ttmlenc.c | 15 +- libavcodec/webvttdec.c | 2 +- libavcodec/webvttenc.c | 16 +- libavutil/Makefile | 2 + {libavcodec => libavutil}/ass.c | 115 ++++--------- libavutil/ass_internal.h | 135 ++++++++++++++++ {libavcodec => libavutil}/ass_split.c | 30 ++-- .../ass_split_internal.h | 32 ++-- 26 files changed, 355 insertions(+), 301 deletions(-) rename {libavcodec => libavutil}/ass.c (59%) create mode 100644 libavutil/ass_internal.h rename {libavcodec => libavutil}/ass_split.c (94%) rename libavcodec/ass_split.h => libavutil/ass_split_internal.h (86%) diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 38425d2f22..1864f06cbb 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -221,10 +221,10 @@ OBJS-$(CONFIG_APNG_DECODER) += png.o pngdec.o pngdsp.o OBJS-$(CONFIG_APNG_ENCODER) += png.o pngenc.o OBJS-$(CONFIG_ARBC_DECODER) += arbc.o OBJS-$(CONFIG_ARGO_DECODER) += argo.o -OBJS-$(CONFIG_SSA_DECODER) += assdec.o ass.o -OBJS-$(CONFIG_SSA_ENCODER) += assenc.o ass.o -OBJS-$(CONFIG_ASS_DECODER) += assdec.o ass.o -OBJS-$(CONFIG_ASS_ENCODER) += assenc.o ass.o +OBJS-$(CONFIG_SSA_DECODER) += assdec.o +OBJS-$(CONFIG_SSA_ENCODER) += assenc.o +OBJS-$(CONFIG_ASS_DECODER) += assdec.o +OBJS-$(CONFIG_ASS_ENCODER) += assenc.o OBJS-$(CONFIG_ASV1_DECODER) += asvdec.o asv.o mpeg12data.o OBJS-$(CONFIG_ASV1_ENCODER) += asvenc.o asv.o mpeg12data.o OBJS-$(CONFIG_ASV2_DECODER) += asvdec.o asv.o mpeg12data.o @@ -265,7 +265,7 @@ OBJS-$(CONFIG_BRENDER_PIX_DECODER) += brenderpix.o OBJS-$(CONFIG_C93_DECODER) += c93.o OBJS-$(CONFIG_CAVS_DECODER) += cavs.o cavsdec.o cavsdsp.o \ cavsdata.o -OBJS-$(CONFIG_CCAPTION_DECODER) += ccaption_dec.o ass.o +OBJS-$(CONFIG_CCAPTION_DECODER) += ccaption_dec.o OBJS-$(CONFIG_CDGRAPHICS_DECODER) += cdgraphics.o OBJS-$(CONFIG_CDTOONS_DECODER) += cdtoons.o OBJS-$(CONFIG_CDXL_DECODER) += cdxl.o @@ -442,7 +442,7 @@ OBJS-$(CONFIG_INTERPLAY_ACM_DECODER) += interplayacm.o OBJS-$(CONFIG_INTERPLAY_DPCM_DECODER) += dpcm.o OBJS-$(CONFIG_INTERPLAY_VIDEO_DECODER) += interplayvideo.o OBJS-$(CONFIG_IPU_DECODER) += mpeg12dec.o mpeg12.o mpeg12data.o -OBJS-$(CONFIG_JACOSUB_DECODER) += jacosubdec.o ass.o +OBJS-$(CONFIG_JACOSUB_DECODER) += jacosubdec.o OBJS-$(CONFIG_JPEG2000_ENCODER) += j2kenc.o mqcenc.o mqc.o jpeg2000.o \ jpeg2000dwt.o OBJS-$(CONFIG_JPEG2000_DECODER) += jpeg2000dec.o jpeg2000.o jpeg2000dsp.o \ @@ -464,7 +464,7 @@ OBJS-$(CONFIG_MAGICYUV_ENCODER) += magicyuvenc.o OBJS-$(CONFIG_MDEC_DECODER) += mdec.o mpeg12.o mpeg12data.o OBJS-$(CONFIG_METASOUND_DECODER) += metasound.o metasound_data.o \ twinvq.o -OBJS-$(CONFIG_MICRODVD_DECODER) += microdvddec.o ass.o +OBJS-$(CONFIG_MICRODVD_DECODER) += microdvddec.o OBJS-$(CONFIG_MIMIC_DECODER) += mimic.o OBJS-$(CONFIG_MJPEG_DECODER) += mjpegdec.o mjpegdec_common.o OBJS-$(CONFIG_MJPEG_QSV_DECODER) += qsvdec.o @@ -479,8 +479,8 @@ OBJS-$(CONFIG_MLP_ENCODER) += mlpenc.o mlp.o OBJS-$(CONFIG_MMVIDEO_DECODER) += mmvideo.o OBJS-$(CONFIG_MOBICLIP_DECODER) += mobiclip.o OBJS-$(CONFIG_MOTIONPIXELS_DECODER) += motionpixels.o -OBJS-$(CONFIG_MOVTEXT_DECODER) += movtextdec.o ass.o -OBJS-$(CONFIG_MOVTEXT_ENCODER) += movtextenc.o ass_split.o +OBJS-$(CONFIG_MOVTEXT_DECODER) += movtextdec.o +OBJS-$(CONFIG_MOVTEXT_ENCODER) += movtextenc.o OBJS-$(CONFIG_MP1_DECODER) += mpegaudiodec_fixed.o OBJS-$(CONFIG_MP1FLOAT_DECODER) += mpegaudiodec_float.o OBJS-$(CONFIG_MP2_DECODER) += mpegaudiodec_fixed.o @@ -521,7 +521,7 @@ OBJS-$(CONFIG_MPEG4_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_MPEG4_OMX_ENCODER) += omx.o OBJS-$(CONFIG_MPEG4_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_MPEG4_V4L2M2M_ENCODER) += v4l2_m2m_enc.o -OBJS-$(CONFIG_MPL2_DECODER) += mpl2dec.o ass.o +OBJS-$(CONFIG_MPL2_DECODER) += mpl2dec.o OBJS-$(CONFIG_MSA1_DECODER) += mss3.o OBJS-$(CONFIG_MSCC_DECODER) += mscc.o OBJS-$(CONFIG_MSMPEG4V1_DECODER) += msmpeg4dec.o msmpeg4.o msmpeg4data.o @@ -574,7 +574,7 @@ OBJS-$(CONFIG_PGX_DECODER) += pgxdec.o OBJS-$(CONFIG_PHOTOCD_DECODER) += photocd.o OBJS-$(CONFIG_PICTOR_DECODER) += pictordec.o cga_data.o OBJS-$(CONFIG_PIXLET_DECODER) += pixlet.o -OBJS-$(CONFIG_PJS_DECODER) += textdec.o ass.o +OBJS-$(CONFIG_PJS_DECODER) += textdec.o OBJS-$(CONFIG_PNG_DECODER) += png.o pngdec.o pngdsp.o OBJS-$(CONFIG_PNG_ENCODER) += png.o pngenc.o OBJS-$(CONFIG_PPM_DECODER) += pnmdec.o pnm.o @@ -607,7 +607,7 @@ OBJS-$(CONFIG_RALF_DECODER) += ralf.o OBJS-$(CONFIG_RASC_DECODER) += rasc.o OBJS-$(CONFIG_RAWVIDEO_DECODER) += rawdec.o OBJS-$(CONFIG_RAWVIDEO_ENCODER) += rawenc.o -OBJS-$(CONFIG_REALTEXT_DECODER) += realtextdec.o ass.o +OBJS-$(CONFIG_REALTEXT_DECODER) += realtextdec.o OBJS-$(CONFIG_RL2_DECODER) += rl2.o OBJS-$(CONFIG_ROQ_DECODER) += roqvideodec.o roqvideo.o OBJS-$(CONFIG_ROQ_ENCODER) += roqvideoenc.o roqvideo.o elbg.o @@ -622,7 +622,7 @@ OBJS-$(CONFIG_RV20_DECODER) += rv10.o OBJS-$(CONFIG_RV20_ENCODER) += rv20enc.o OBJS-$(CONFIG_RV30_DECODER) += rv30.o rv34.o rv30dsp.o OBJS-$(CONFIG_RV40_DECODER) += rv40.o rv34.o rv40dsp.o -OBJS-$(CONFIG_SAMI_DECODER) += samidec.o ass.o htmlsubtitles.o +OBJS-$(CONFIG_SAMI_DECODER) += samidec.o htmlsubtitles.o OBJS-$(CONFIG_S302M_DECODER) += s302m.o OBJS-$(CONFIG_S302M_ENCODER) += s302menc.o OBJS-$(CONFIG_SANM_DECODER) += sanm.o @@ -657,13 +657,13 @@ OBJS-$(CONFIG_SPEEDHQ_ENCODER) += speedhq.o mpeg12data.o mpeg12enc.o spe OBJS-$(CONFIG_SPEEX_DECODER) += speexdec.o OBJS-$(CONFIG_SP5X_DECODER) += sp5xdec.o OBJS-$(CONFIG_SRGC_DECODER) += mscc.o -OBJS-$(CONFIG_SRT_DECODER) += srtdec.o ass.o htmlsubtitles.o -OBJS-$(CONFIG_SRT_ENCODER) += srtenc.o ass_split.o -OBJS-$(CONFIG_STL_DECODER) += textdec.o ass.o -OBJS-$(CONFIG_SUBRIP_DECODER) += srtdec.o ass.o htmlsubtitles.o -OBJS-$(CONFIG_SUBRIP_ENCODER) += srtenc.o ass_split.o -OBJS-$(CONFIG_SUBVIEWER1_DECODER) += textdec.o ass.o -OBJS-$(CONFIG_SUBVIEWER_DECODER) += subviewerdec.o ass.o +OBJS-$(CONFIG_SRT_DECODER) += srtdec.o htmlsubtitles.o +OBJS-$(CONFIG_SRT_ENCODER) += srtenc.o +OBJS-$(CONFIG_STL_DECODER) += textdec.o +OBJS-$(CONFIG_SUBRIP_DECODER) += srtdec.o htmlsubtitles.o +OBJS-$(CONFIG_SUBRIP_ENCODER) += srtenc.o +OBJS-$(CONFIG_SUBVIEWER1_DECODER) += textdec.o +OBJS-$(CONFIG_SUBVIEWER_DECODER) += subviewerdec.o OBJS-$(CONFIG_SUNRAST_DECODER) += sunrast.o OBJS-$(CONFIG_SUNRAST_ENCODER) += sunrastenc.o OBJS-$(CONFIG_LIBRSVG_DECODER) += librsvgdec.o @@ -673,8 +673,8 @@ OBJS-$(CONFIG_SVQ1_DECODER) += svq1dec.o svq1.o h263data.o OBJS-$(CONFIG_SVQ1_ENCODER) += svq1enc.o svq1.o h263data.o \ h263.o ituh263enc.o OBJS-$(CONFIG_SVQ3_DECODER) += svq3.o mpegutils.o h264data.o -OBJS-$(CONFIG_TEXT_DECODER) += textdec.o ass.o -OBJS-$(CONFIG_TEXT_ENCODER) += srtenc.o ass_split.o +OBJS-$(CONFIG_TEXT_DECODER) += textdec.o +OBJS-$(CONFIG_TEXT_ENCODER) += srtenc.o OBJS-$(CONFIG_TAK_DECODER) += takdec.o tak.o takdsp.o OBJS-$(CONFIG_TARGA_DECODER) += targa.o OBJS-$(CONFIG_TARGA_ENCODER) += targaenc.o rle.o @@ -694,7 +694,7 @@ OBJS-$(CONFIG_TSCC_DECODER) += tscc.o msrledec.o OBJS-$(CONFIG_TSCC2_DECODER) += tscc2.o OBJS-$(CONFIG_TTA_DECODER) += tta.o ttadata.o ttadsp.o OBJS-$(CONFIG_TTA_ENCODER) += ttaenc.o ttaencdsp.o ttadata.o -OBJS-$(CONFIG_TTML_ENCODER) += ttmlenc.o ass_split.o +OBJS-$(CONFIG_TTML_ENCODER) += ttmlenc.o OBJS-$(CONFIG_TWINVQ_DECODER) += twinvqdec.o twinvq.o metasound_data.o OBJS-$(CONFIG_TXD_DECODER) += txd.o OBJS-$(CONFIG_ULTI_DECODER) += ulti.o @@ -751,15 +751,15 @@ OBJS-$(CONFIG_VP9_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_VP9_RKMPP_DECODER) += rkmppdec.o OBJS-$(CONFIG_VP9_VAAPI_ENCODER) += vaapi_encode_vp9.o OBJS-$(CONFIG_VP9_QSV_ENCODER) += qsvenc_vp9.o -OBJS-$(CONFIG_VPLAYER_DECODER) += textdec.o ass.o +OBJS-$(CONFIG_VPLAYER_DECODER) += textdec.o OBJS-$(CONFIG_VP9_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_VQA_DECODER) += vqavideo.o OBJS-$(CONFIG_WAVPACK_DECODER) += wavpack.o wavpackdata.o dsd.o OBJS-$(CONFIG_WAVPACK_ENCODER) += wavpackdata.o wavpackenc.o OBJS-$(CONFIG_WCMV_DECODER) += wcmv.o OBJS-$(CONFIG_WEBP_DECODER) += webp.o -OBJS-$(CONFIG_WEBVTT_DECODER) += webvttdec.o ass.o -OBJS-$(CONFIG_WEBVTT_ENCODER) += webvttenc.o ass_split.o +OBJS-$(CONFIG_WEBVTT_DECODER) += webvttdec.o +OBJS-$(CONFIG_WEBVTT_ENCODER) += webvttenc.o OBJS-$(CONFIG_WMALOSSLESS_DECODER) += wmalosslessdec.o wma_common.o OBJS-$(CONFIG_WMAPRO_DECODER) += wmaprodec.o wma.o wma_common.o OBJS-$(CONFIG_WMAV1_DECODER) += wmadec.o wma.o wma_common.o aactab.o @@ -1049,7 +1049,7 @@ OBJS-$(CONFIG_PCM_ALAW_AT_ENCODER) += audiotoolboxenc.o OBJS-$(CONFIG_PCM_MULAW_AT_ENCODER) += audiotoolboxenc.o OBJS-$(CONFIG_LIBAOM_AV1_DECODER) += libaomdec.o OBJS-$(CONFIG_LIBAOM_AV1_ENCODER) += libaomenc.o -OBJS-$(CONFIG_LIBARIBB24_DECODER) += libaribb24.o ass.o +OBJS-$(CONFIG_LIBARIBB24_DECODER) += libaribb24.o OBJS-$(CONFIG_LIBCELT_DECODER) += libcelt_dec.o OBJS-$(CONFIG_LIBCODEC2_DECODER) += libcodec2.o OBJS-$(CONFIG_LIBCODEC2_ENCODER) += libcodec2.o @@ -1102,7 +1102,7 @@ OBJS-$(CONFIG_LIBX265_ENCODER) += libx265.o OBJS-$(CONFIG_LIBXAVS_ENCODER) += libxavs.o OBJS-$(CONFIG_LIBXAVS2_ENCODER) += libxavs2.o OBJS-$(CONFIG_LIBXVID_ENCODER) += libxvid.o -OBJS-$(CONFIG_LIBZVBI_TELETEXT_DECODER) += libzvbi-teletextdec.o ass.o +OBJS-$(CONFIG_LIBZVBI_TELETEXT_DECODER) += libzvbi-teletextdec.o # parsers OBJS-$(CONFIG_AAC_LATM_PARSER) += latm_parser.o diff --git a/libavcodec/ass.h b/libavcodec/ass.h index 4dffe923d9..8bc13d7ab8 100644 --- a/libavcodec/ass.h +++ b/libavcodec/ass.h @@ -23,124 +23,73 @@ #define AVCODEC_ASS_H #include "avcodec.h" -#include "libavutil/bprint.h" - -#define ASS_DEFAULT_PLAYRESX 384 -#define ASS_DEFAULT_PLAYRESY 288 - -/** - * @name Default values for ASS style - * @{ - */ -#define ASS_DEFAULT_FONT "Arial" -#define ASS_DEFAULT_FONT_SIZE 16 -#define ASS_DEFAULT_COLOR 0xffffff -#define ASS_DEFAULT_BACK_COLOR 0 -#define ASS_DEFAULT_BOLD 0 -#define ASS_DEFAULT_ITALIC 0 -#define ASS_DEFAULT_UNDERLINE 0 -#define ASS_DEFAULT_ALIGNMENT 2 -#define ASS_DEFAULT_BORDERSTYLE 1 -/** @} */ +#include "libavutil/ass_internal.h" typedef struct FFASSDecoderContext { int readorder; } FFASSDecoderContext; -/** - * Generate a suitable AVCodecContext.subtitle_header for SUBTITLE_ASS. - * Can specify all fields explicitly - * - * @param avctx pointer to the AVCodecContext - * @param play_res_x subtitle frame width - * @param play_res_y subtitle frame height - * @param font name of the default font face to use - * @param font_size default font size to use - * @param primary_color default text color to use (ABGR) - * @param secondary_color default secondary text color to use (ABGR) - * @param outline_color default outline color to use (ABGR) - * @param back_color default background color to use (ABGR) - * @param bold 1 for bold text, 0 for normal text - * @param italic 1 for italic text, 0 for normal text - * @param underline 1 for underline text, 0 for normal text - * @param border_style 1 for outline, 3 for opaque box - * @param alignment position of the text (left, center, top...), defined after - * the layout of the numpad (1-3 sub, 4-6 mid, 7-9 top) - * @return >= 0 on success otherwise an error code <0 - */ -int ff_ass_subtitle_header_full(AVCodecContext *avctx, +static inline int ff_ass_subtitle_header_full(AVCodecContext *avctx, int play_res_x, int play_res_y, const char *font, int font_size, int primary_color, int secondary_color, int outline_color, int back_color, int bold, int italic, int underline, - int border_style, int alignment); -/** - * Generate a suitable AVCodecContext.subtitle_header for SUBTITLE_ASS. - * - * @param avctx pointer to the AVCodecContext - * @param font name of the default font face to use - * @param font_size default font size to use - * @param color default text color to use (ABGR) - * @param back_color default background color to use (ABGR) - * @param bold 1 for bold text, 0 for normal text - * @param italic 1 for italic text, 0 for normal text - * @param underline 1 for underline text, 0 for normal text - * @param alignment position of the text (left, center, top...), defined after - * the layout of the numpad (1-3 sub, 4-6 mid, 7-9 top) - * @return >= 0 on success otherwise an error code <0 - */ -int ff_ass_subtitle_header(AVCodecContext *avctx, - const char *font, int font_size, - int color, int back_color, - int bold, int italic, int underline, - int border_style, int alignment); + int border_style, int alignment) +{ + avctx->subtitle_header = (uint8_t *)avpriv_ass_get_subtitle_header_full( + play_res_x, play_res_y, font, font_size, + primary_color, secondary_color, outline_color, + back_color, bold,italic,underline,border_style,alignment, + !(avctx->flags & AV_CODEC_FLAG_BITEXACT)); -/** - * Generate a suitable AVCodecContext.subtitle_header for SUBTITLE_ASS - * with default style. - * - * @param avctx pointer to the AVCodecContext - * @return >= 0 on success otherwise an error code <0 - */ -int ff_ass_subtitle_header_default(AVCodecContext *avctx); + if (!avctx->subtitle_header) + return AVERROR(ENOMEM); + avctx->subtitle_header_size = strlen((char *)avctx->subtitle_header); + return 0; +} -/** - * Craft an ASS dialog string. - */ -char *ff_ass_get_dialog(int readorder, int layer, const char *style, - const char *speaker, const char *text); +static inline int ff_ass_subtitle_header_default(AVCodecContext *avctx) +{ + avctx->subtitle_header = (uint8_t *)avpriv_ass_get_subtitle_header_default(!(avctx->flags & AV_CODEC_FLAG_BITEXACT)); -/** - * Add an ASS dialog to a subtitle. - */ -int ff_ass_add_rect(AVSubtitle *sub, const char *dialog, - int readorder, int layer, const char *style, - const char *speaker); + if (!avctx->subtitle_header) + return AVERROR(ENOMEM); + avctx->subtitle_header_size = strlen((char *)avctx->subtitle_header); + return 0; +} + +static inline void ff_ass_decoder_flush(AVCodecContext *avctx) +{ + FFASSDecoderContext *s = avctx->priv_data; + if (!(avctx->flags2 & AV_CODEC_FLAG2_RO_FLUSH_NOOP)) + s->readorder = 0; +} /** * Add an ASS dialog to a subtitle. */ -int ff_ass_add_rect2(AVSubtitle *sub, const char *dialog, - int readorder, int layer, const char *style, - const char *speaker, unsigned *nb_rect_allocated); +static inline int avpriv_ass_add_rect(AVSubtitle *sub, const char *dialog, + int readorder, int layer, const char *style, + const char *speaker) +{ + char *ass_str; + AVSubtitleRect **rects; -/** - * Helper to flush a text subtitles decoder making use of the - * FFASSDecoderContext. - */ -void ff_ass_decoder_flush(AVCodecContext *avctx); + rects = av_realloc_array(sub->rects, sub->num_rects+1, sizeof(*sub->rects)); + if (!rects) + return AVERROR(ENOMEM); + sub->rects = rects; + rects[sub->num_rects] = av_mallocz(sizeof(*rects[0])); + if (!rects[sub->num_rects]) + return AVERROR(ENOMEM); + rects[sub->num_rects]->type = SUBTITLE_ASS; + ass_str = avpriv_ass_get_dialog(readorder, layer, style, speaker, dialog); + if (!ass_str) + return AVERROR(ENOMEM); + rects[sub->num_rects]->ass = ass_str; + sub->num_rects++; + return 0; +} -/** - * Escape a text subtitle using ASS syntax into an AVBPrint buffer. - * Newline characters will be escaped to \N. - * - * @param buf pointer to an initialized AVBPrint buffer - * @param p source text - * @param size size of the source text - * @param linebreaks additional newline chars, which will be escaped to \N - * @param keep_ass_markup braces and backslash will not be escaped if set - */ -void ff_ass_bprint_text_event(AVBPrint *buf, const char *p, int size, - const char *linebreaks, int keep_ass_markup); #endif /* AVCODEC_ASS_H */ diff --git a/libavcodec/assdec.c b/libavcodec/assdec.c index f43b500aa7..1a1363471d 100644 --- a/libavcodec/assdec.c +++ b/libavcodec/assdec.c @@ -22,9 +22,9 @@ #include #include "avcodec.h" -#include "ass.h" #include "codec_internal.h" #include "config_components.h" +#include "libavutil/ass_internal.h" #include "libavutil/internal.h" #include "libavutil/mem.h" diff --git a/libavcodec/assenc.c b/libavcodec/assenc.c index 2ac40d5afe..391d690133 100644 --- a/libavcodec/assenc.c +++ b/libavcodec/assenc.c @@ -24,8 +24,8 @@ #include #include "avcodec.h" -#include "ass.h" #include "codec_internal.h" +#include "libavutil/ass_internal.h" #include "libavutil/avstring.h" #include "libavutil/internal.h" #include "libavutil/mem.h" diff --git a/libavcodec/ccaption_dec.c b/libavcodec/ccaption_dec.c index 34f0513b1a..5f706d985f 100644 --- a/libavcodec/ccaption_dec.c +++ b/libavcodec/ccaption_dec.c @@ -272,15 +272,12 @@ static av_cold int init_decoder(AVCodecContext *avctx) ctx->bg_color = CCCOL_BLACK; ctx->rollup = 2; ctx->cursor_row = 10; - ret = ff_ass_subtitle_header(avctx, "Monospace", + ret = ff_ass_subtitle_header_full(avctx, ASS_DEFAULT_PLAYRESX, ASS_DEFAULT_PLAYRESY, "Monospace", ASS_DEFAULT_FONT_SIZE, - ASS_DEFAULT_COLOR, - ASS_DEFAULT_BACK_COLOR, - ASS_DEFAULT_BOLD, - ASS_DEFAULT_ITALIC, - ASS_DEFAULT_UNDERLINE, - 3, - ASS_DEFAULT_ALIGNMENT); + ASS_DEFAULT_COLOR, ASS_DEFAULT_COLOR, + ASS_DEFAULT_BACK_COLOR, ASS_DEFAULT_BACK_COLOR, + ASS_DEFAULT_BOLD, ASS_DEFAULT_ITALIC, ASS_DEFAULT_UNDERLINE, + 3, ASS_DEFAULT_ALIGNMENT); if (ret < 0) { return ret; } @@ -850,7 +847,6 @@ static int decode(AVCodecContext *avctx, AVSubtitle *sub, int len = avpkt->size; int ret = 0; int i; - unsigned nb_rect_allocated = 0; for (i = 0; i < len; i += 3) { uint8_t hi, cc_type = bptr[i] & 1; @@ -887,7 +883,7 @@ static int decode(AVCodecContext *avctx, AVSubtitle *sub, AV_TIME_BASE_Q, ms_tb); else sub->end_display_time = -1; - ret = ff_ass_add_rect2(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL, &nb_rect_allocated); + ret = avpriv_ass_add_rect(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL); if (ret < 0) return ret; ctx->last_real_time = sub->pts; @@ -897,7 +893,7 @@ static int decode(AVCodecContext *avctx, AVSubtitle *sub, if (!bptr && !ctx->real_time && ctx->buffer[!ctx->buffer_index].str[0]) { bidx = !ctx->buffer_index; - ret = ff_ass_add_rect2(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL, &nb_rect_allocated); + ret = avpriv_ass_add_rect(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL); if (ret < 0) return ret; sub->pts = ctx->buffer_time[1]; @@ -915,7 +911,7 @@ static int decode(AVCodecContext *avctx, AVSubtitle *sub, capture_screen(ctx); ctx->buffer_changed = 0; - ret = ff_ass_add_rect2(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL, &nb_rect_allocated); + ret = avpriv_ass_add_rect(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL); if (ret < 0) return ret; sub->end_display_time = -1; diff --git a/libavcodec/jacosubdec.c b/libavcodec/jacosubdec.c index e3bf9f4226..40abdebcc6 100644 --- a/libavcodec/jacosubdec.c +++ b/libavcodec/jacosubdec.c @@ -182,7 +182,7 @@ static int jacosub_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, av_bprint_init(&buffer, JSS_MAX_LINESIZE, JSS_MAX_LINESIZE); jacosub_to_ass(avctx, &buffer, ptr); - ret = ff_ass_add_rect(sub, buffer.str, s->readorder++, 0, NULL, NULL); + ret = avpriv_ass_add_rect(sub, buffer.str, s->readorder++, 0, NULL, NULL); av_bprint_finalize(&buffer, NULL); if (ret < 0) return ret; diff --git a/libavcodec/libaribb24.c b/libavcodec/libaribb24.c index 9658e1d5ac..360e20834b 100644 --- a/libavcodec/libaribb24.c +++ b/libavcodec/libaribb24.c @@ -274,7 +274,7 @@ next_region: av_log(avctx, AV_LOG_DEBUG, "Styled ASS line: %s\n", buf.str); - ret = ff_ass_add_rect(sub, buf.str, b24->read_order++, + ret = avpriv_ass_add_rect(sub, buf.str, b24->read_order++, 0, NULL, NULL); } diff --git a/libavcodec/libzvbi-teletextdec.c b/libavcodec/libzvbi-teletextdec.c index 2aab10a548..54a78342f2 100644 --- a/libavcodec/libzvbi-teletextdec.c +++ b/libavcodec/libzvbi-teletextdec.c @@ -153,12 +153,12 @@ static char *create_ass_text(TeletextContext *ctx, const char *text) AVBPrint buf; av_bprint_init(&buf, 0, AV_BPRINT_SIZE_UNLIMITED); - ff_ass_bprint_text_event(&buf, text, strlen(text), "", 0); + avpriv_ass_bprint_text_event(&buf, text, strlen(text), "", 0); if (!av_bprint_is_complete(&buf)) { av_bprint_finalize(&buf, NULL); return NULL; } - dialog = ff_ass_get_dialog(ctx->readorder++, 0, NULL, NULL, buf.str); + dialog = avpriv_ass_get_dialog(ctx->readorder++, 0, NULL, NULL, buf.str); av_bprint_finalize(&buf, NULL); return dialog; } @@ -225,7 +225,7 @@ static int gen_sub_text(TeletextContext *ctx, AVSubtitleRect *sub_rect, vbi_page } av_log(ctx, AV_LOG_DEBUG, "subtext:%s:txetbus\n", sub_rect->ass); } else { - sub_rect->type = SUBTITLE_NONE; + sub_rect->type = AV_SUBTITLE_FMT_NONE; } av_bprint_finalize(&buf, NULL); return 0; @@ -395,7 +395,7 @@ static int gen_sub_ass(TeletextContext *ctx, AVSubtitleRect *sub_rect, vbi_page if (buf.len) { sub_rect->type = SUBTITLE_ASS; - sub_rect->ass = ff_ass_get_dialog(ctx->readorder++, 0, is_subtitle_page ? "Subtitle" : "Teletext", NULL, buf.str); + sub_rect->ass = avpriv_ass_get_dialog(ctx->readorder++, 0, is_subtitle_page ? "Subtitle" : "Teletext", NULL, buf.str); if (!sub_rect->ass) { av_bprint_finalize(&buf, NULL); @@ -403,7 +403,7 @@ static int gen_sub_ass(TeletextContext *ctx, AVSubtitleRect *sub_rect, vbi_page } av_log(ctx, AV_LOG_DEBUG, "subtext:%s:txetbus\n", sub_rect->ass); } else { - sub_rect->type = SUBTITLE_NONE; + sub_rect->type = AV_SUBTITLE_FMT_NONE; } av_bprint_finalize(&buf, NULL); return 0; @@ -463,7 +463,7 @@ static int gen_sub_bitmap(TeletextContext *ctx, AVSubtitleRect *sub_rect, vbi_pa if (vc >= vcend) { av_log(ctx, AV_LOG_DEBUG, "dropping empty page %3x\n", page->pgno); - sub_rect->type = SUBTITLE_NONE; + sub_rect->type = AV_SUBTITLE_FMT_NONE; return 0; } @@ -696,7 +696,7 @@ static int teletext_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, sub->num_rects = 0; sub->pts = ctx->pages->pts; - if (ctx->pages->sub_rect->type != SUBTITLE_NONE) { + if (ctx->pages->sub_rect->type != AV_SUBTITLE_FMT_NONE) { sub->rects = av_malloc(sizeof(*sub->rects)); if (sub->rects) { sub->num_rects = 1; diff --git a/libavcodec/microdvddec.c b/libavcodec/microdvddec.c index f36ad51468..de17400edd 100644 --- a/libavcodec/microdvddec.c +++ b/libavcodec/microdvddec.c @@ -309,7 +309,7 @@ static int microdvd_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, } } if (new_line.len) { - int ret = ff_ass_add_rect(sub, new_line.str, s->readorder++, 0, NULL, NULL); + int ret = avpriv_ass_add_rect(sub, new_line.str, s->readorder++, 0, NULL, NULL); av_bprint_finalize(&new_line, NULL); if (ret < 0) return ret; @@ -362,8 +362,9 @@ static int microdvd_init(AVCodecContext *avctx) } } } - return ff_ass_subtitle_header(avctx, font_buf.str, font_size, color, - ASS_DEFAULT_BACK_COLOR, bold, italic, + return ff_ass_subtitle_header_full(avctx, ASS_DEFAULT_PLAYRESX, ASS_DEFAULT_PLAYRESY, + font_buf.str, font_size, color, color, + ASS_DEFAULT_BACK_COLOR, ASS_DEFAULT_BACK_COLOR, bold, italic, underline, ASS_DEFAULT_BORDERSTYLE, alignment); } diff --git a/libavcodec/movtextdec.c b/libavcodec/movtextdec.c index 70162b4888..8f2459d45b 100644 --- a/libavcodec/movtextdec.c +++ b/libavcodec/movtextdec.c @@ -22,7 +22,6 @@ #include "avcodec.h" #include "ass.h" #include "libavutil/opt.h" -#include "libavutil/avstring.h" #include "libavutil/common.h" #include "libavutil/bprint.h" #include "libavutil/intreadwrite.h" @@ -553,7 +552,7 @@ static int mov_text_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, } else text_to_ass(&buf, ptr, end, avctx); - ret = ff_ass_add_rect(sub, buf.str, m->readorder++, 0, NULL, NULL); + ret = avpriv_ass_add_rect(sub, buf.str, m->readorder++, 0, NULL, NULL); av_bprint_finalize(&buf, NULL); if (ret < 0) return ret; diff --git a/libavcodec/movtextenc.c b/libavcodec/movtextenc.c index 728338f2cc..6f0b7a8a5c 100644 --- a/libavcodec/movtextenc.c +++ b/libavcodec/movtextenc.c @@ -26,8 +26,8 @@ #include "libavutil/intreadwrite.h" #include "libavutil/mem.h" #include "libavutil/common.h" -#include "ass_split.h" -#include "ass.h" +#include "libavutil/ass_split_internal.h" +#include "libavutil/ass_internal.h" #include "bytestream.h" #include "codec_internal.h" @@ -167,7 +167,7 @@ static int mov_text_encode_close(AVCodecContext *avctx) { MovTextContext *s = avctx->priv_data; - ff_ass_split_free(s->ass_ctx); + avpriv_ass_split_free(s->ass_ctx); av_freep(&s->style_attributes); av_freep(&s->fonts); av_bprint_finalize(&s->buffer, NULL); @@ -222,7 +222,7 @@ static int encode_sample_description(AVCodecContext *avctx) else s->font_scale_factor = 1; - style = ff_ass_style_get(s->ass_ctx, "Default"); + style = avpriv_ass_style_get(s->ass_ctx, "Default"); if (!style && ass->styles_count) { style = &ass->styles[0]; } @@ -329,7 +329,7 @@ static av_cold int mov_text_encode_init(AVCodecContext *avctx) av_bprint_init(&s->buffer, 0, AV_BPRINT_SIZE_UNLIMITED); - s->ass_ctx = ff_ass_split(avctx->subtitle_header); + s->ass_ctx = avpriv_ass_split(avctx->subtitle_header); if (!s->ass_ctx) return AVERROR_INVALIDDATA; ret = encode_sample_description(avctx); @@ -566,7 +566,7 @@ static void mov_text_ass_style_set(MovTextContext *s, ASSStyle *style) static void mov_text_dialog(MovTextContext *s, ASSDialog *dialog) { - ASSStyle *style = ff_ass_style_get(s->ass_ctx, dialog->style); + ASSStyle *style = avpriv_ass_style_get(s->ass_ctx, dialog->style); s->ass_dialog_style = style; mov_text_ass_style_set(s, style); @@ -580,7 +580,7 @@ static void mov_text_cancel_overrides_cb(void *priv, const char *style_name) if (!style_name || !*style_name) style = s->ass_dialog_style; else - style= ff_ass_style_get(s->ass_ctx, style_name); + style= avpriv_ass_style_get(s->ass_ctx, style_name); mov_text_ass_style_set(s, style); } @@ -652,12 +652,12 @@ static int mov_text_encode_frame(AVCodecContext *avctx, unsigned char *buf, return AVERROR(EINVAL); } - dialog = ff_ass_split_dialog(s->ass_ctx, ass); + dialog = avpriv_ass_split_dialog(s->ass_ctx, ass); if (!dialog) return AVERROR(ENOMEM); mov_text_dialog(s, dialog); - ff_ass_split_override_codes(&mov_text_callbacks, s, dialog->text); - ff_ass_free_dialog(&dialog); + avpriv_ass_split_override_codes(&mov_text_callbacks, s, dialog->text); + avpriv_ass_free_dialog(&dialog); } if (s->buffer.len > UINT16_MAX) diff --git a/libavcodec/mpl2dec.c b/libavcodec/mpl2dec.c index 56f008b65c..175bd319e1 100644 --- a/libavcodec/mpl2dec.c +++ b/libavcodec/mpl2dec.c @@ -73,7 +73,7 @@ static int mpl2_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, av_bprint_init(&buf, 0, AV_BPRINT_SIZE_UNLIMITED); if (ptr && avpkt->size > 0 && *ptr && !mpl2_event_to_ass(&buf, ptr)) - ret = ff_ass_add_rect(sub, buf.str, s->readorder++, 0, NULL, NULL); + ret = avpriv_ass_add_rect(sub, buf.str, s->readorder++, 0, NULL, NULL); av_bprint_finalize(&buf, NULL); if (ret < 0) return ret; diff --git a/libavcodec/realtextdec.c b/libavcodec/realtextdec.c index c3e138a7ba..49d42a1d4d 100644 --- a/libavcodec/realtextdec.c +++ b/libavcodec/realtextdec.c @@ -66,7 +66,7 @@ static int realtext_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, av_bprint_init(&buf, 0, 4096); if (ptr && avpkt->size > 0 && !rt_event_to_ass(&buf, ptr)) - ret = ff_ass_add_rect(sub, buf.str, s->readorder++, 0, NULL, NULL); + ret = avpriv_ass_add_rect(sub, buf.str, s->readorder++, 0, NULL, NULL); av_bprint_finalize(&buf, NULL); if (ret < 0) return ret; diff --git a/libavcodec/samidec.c b/libavcodec/samidec.c index cf5dec955b..f35d312c2b 100644 --- a/libavcodec/samidec.c +++ b/libavcodec/samidec.c @@ -143,7 +143,7 @@ static int sami_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, if (ret < 0) return ret; // TODO: pass escaped sami->encoded_source.str as source - ret = ff_ass_add_rect(sub, sami->full.str, sami->readorder++, 0, NULL, NULL); + ret = avpriv_ass_add_rect(sub, sami->full.str, sami->readorder++, 0, NULL, NULL); if (ret < 0) return ret; } diff --git a/libavcodec/srtdec.c b/libavcodec/srtdec.c index b2df34474e..ccf5981263 100644 --- a/libavcodec/srtdec.c +++ b/libavcodec/srtdec.c @@ -79,7 +79,7 @@ static int srt_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, ret = srt_to_ass(avctx, &buffer, avpkt->data, x1, y1, x2, y2); if (ret >= 0) - ret = ff_ass_add_rect(sub, buffer.str, s->readorder++, 0, NULL, NULL); + ret = avpriv_ass_add_rect(sub, buffer.str, s->readorder++, 0, NULL, NULL); av_bprint_finalize(&buffer, NULL); if (ret < 0) return ret; diff --git a/libavcodec/srtenc.c b/libavcodec/srtenc.c index 51456c8b9d..2baa6e70ad 100644 --- a/libavcodec/srtenc.c +++ b/libavcodec/srtenc.c @@ -25,9 +25,9 @@ #include "avcodec.h" #include "libavutil/avstring.h" #include "libavutil/bprint.h" -#include "ass_split.h" -#include "ass.h" #include "codec_internal.h" +#include "libavutil/ass_split_internal.h" +#include "libavutil/ass_internal.h" #define SRT_STACK_SIZE 64 @@ -96,7 +96,7 @@ static void srt_stack_push_pop(SRTContext *s, const char c, int close) static void srt_style_apply(SRTContext *s, const char *style) { - ASSStyle *st = ff_ass_style_get(s->ass_ctx, style); + ASSStyle *st = avpriv_ass_style_get(s->ass_ctx, style); if (st) { int c = st->primary_color & 0xFFFFFF; if (st->font_name && strcmp(st->font_name, ASS_DEFAULT_FONT) || @@ -137,7 +137,7 @@ static av_cold int srt_encode_init(AVCodecContext *avctx) { SRTContext *s = avctx->priv_data; s->avctx = avctx; - s->ass_ctx = ff_ass_split(avctx->subtitle_header); + s->ass_ctx = avpriv_ass_split(avctx->subtitle_header); av_bprint_init(&s->buffer, 0, AV_BPRINT_SIZE_UNLIMITED); return s->ass_ctx ? 0 : AVERROR_INVALIDDATA; } @@ -247,14 +247,14 @@ static int encode_frame(AVCodecContext *avctx, return AVERROR(EINVAL); } - dialog = ff_ass_split_dialog(s->ass_ctx, ass); + dialog = avpriv_ass_split_dialog(s->ass_ctx, ass); if (!dialog) return AVERROR(ENOMEM); s->alignment_applied = 0; if (avctx->codec_id == AV_CODEC_ID_SUBRIP) srt_style_apply(s, dialog->style); - ff_ass_split_override_codes(cb, s, dialog->text); - ff_ass_free_dialog(&dialog); + avpriv_ass_split_override_codes(cb, s, dialog->text); + avpriv_ass_free_dialog(&dialog); } if (!av_bprint_is_complete(&s->buffer)) @@ -286,7 +286,7 @@ static int text_encode_frame(AVCodecContext *avctx, static int srt_encode_close(AVCodecContext *avctx) { SRTContext *s = avctx->priv_data; - ff_ass_split_free(s->ass_ctx); + avpriv_ass_split_free(s->ass_ctx); av_bprint_finalize(&s->buffer, NULL); return 0; } diff --git a/libavcodec/subviewerdec.c b/libavcodec/subviewerdec.c index 2bda5fa5c1..8651453e3f 100644 --- a/libavcodec/subviewerdec.c +++ b/libavcodec/subviewerdec.c @@ -57,7 +57,7 @@ static int subviewer_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, av_bprint_init(&buf, 0, AV_BPRINT_SIZE_UNLIMITED); if (ptr && avpkt->size > 0 && !subviewer_event_to_ass(&buf, ptr)) - ret = ff_ass_add_rect(sub, buf.str, s->readorder++, 0, NULL, NULL); + ret = avpriv_ass_add_rect(sub, buf.str, s->readorder++, 0, NULL, NULL); av_bprint_finalize(&buf, NULL); if (ret < 0) return ret; diff --git a/libavcodec/textdec.c b/libavcodec/textdec.c index d509452391..06a25e7128 100644 --- a/libavcodec/textdec.c +++ b/libavcodec/textdec.c @@ -55,8 +55,8 @@ static int text_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, av_bprint_init(&buf, 0, AV_BPRINT_SIZE_UNLIMITED); if (ptr && avpkt->size > 0 && *ptr) { - ff_ass_bprint_text_event(&buf, ptr, avpkt->size, text->linebreaks, text->keep_ass_markup); - ret = ff_ass_add_rect(sub, buf.str, text->readorder++, 0, NULL, NULL); + avpriv_ass_bprint_text_event(&buf, ptr, avpkt->size, text->linebreaks, text->keep_ass_markup); + ret = avpriv_ass_add_rect(sub, buf.str, text->readorder++, 0, NULL, NULL); } av_bprint_finalize(&buf, NULL); if (ret < 0) diff --git a/libavcodec/ttmlenc.c b/libavcodec/ttmlenc.c index be1d8fb2e8..d4f11a87d2 100644 --- a/libavcodec/ttmlenc.c +++ b/libavcodec/ttmlenc.c @@ -32,8 +32,7 @@ #include "libavutil/avstring.h" #include "libavutil/bprint.h" #include "libavutil/internal.h" -#include "ass_split.h" -#include "ass.h" +#include "libavutil/ass_split_internal.h" #include "ttmlenc.h" typedef struct { @@ -95,7 +94,7 @@ static int ttml_encode_frame(AVCodecContext *avctx, uint8_t *buf, return AVERROR(EINVAL); } - dialog = ff_ass_split_dialog(s->ass_ctx, ass); + dialog = avpriv_ass_split_dialog(s->ass_ctx, ass); if (!dialog) return AVERROR(ENOMEM); @@ -107,7 +106,7 @@ static int ttml_encode_frame(AVCodecContext *avctx, uint8_t *buf, av_bprintf(&s->buffer, "\">"); } - ret = ff_ass_split_override_codes(&ttml_callbacks, s, dialog->text); + ret = avpriv_ass_split_override_codes(&ttml_callbacks, s, dialog->text); if (ret < 0) { int log_level = (ret != AVERROR_INVALIDDATA || avctx->err_recognition & AV_EF_EXPLODE) ? @@ -118,7 +117,7 @@ static int ttml_encode_frame(AVCodecContext *avctx, uint8_t *buf, av_err2str(ret)); if (log_level == AV_LOG_ERROR) { - ff_ass_free_dialog(&dialog); + avpriv_ass_free_dialog(&dialog); return ret; } } @@ -126,7 +125,7 @@ static int ttml_encode_frame(AVCodecContext *avctx, uint8_t *buf, if (dialog->style) av_bprintf(&s->buffer, ""); - ff_ass_free_dialog(&dialog); + avpriv_ass_free_dialog(&dialog); } if (!av_bprint_is_complete(&s->buffer)) @@ -148,7 +147,7 @@ static av_cold int ttml_encode_close(AVCodecContext *avctx) { TTMLContext *s = avctx->priv_data; - ff_ass_split_free(s->ass_ctx); + avpriv_ass_split_free(s->ass_ctx); av_bprint_finalize(&s->buffer, NULL); @@ -372,7 +371,7 @@ static av_cold int ttml_encode_init(AVCodecContext *avctx) av_bprint_init(&s->buffer, 0, AV_BPRINT_SIZE_UNLIMITED); - if (!(s->ass_ctx = ff_ass_split(avctx->subtitle_header))) { + if (!(s->ass_ctx = avpriv_ass_split(avctx->subtitle_header))) { return AVERROR_INVALIDDATA; } diff --git a/libavcodec/webvttdec.c b/libavcodec/webvttdec.c index fcf1062d86..549e60fe62 100644 --- a/libavcodec/webvttdec.c +++ b/libavcodec/webvttdec.c @@ -90,7 +90,7 @@ static int webvtt_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, av_bprint_init(&buf, 0, AV_BPRINT_SIZE_UNLIMITED); if (ptr && avpkt->size > 0 && !webvtt_event_to_ass(&buf, ptr)) - ret = ff_ass_add_rect(sub, buf.str, s->readorder++, 0, NULL, NULL); + ret = avpriv_ass_add_rect(sub, buf.str, s->readorder++, 0, NULL, NULL); av_bprint_finalize(&buf, NULL); if (ret < 0) return ret; diff --git a/libavcodec/webvttenc.c b/libavcodec/webvttenc.c index e433bb4579..24d60c5dc1 100644 --- a/libavcodec/webvttenc.c +++ b/libavcodec/webvttenc.c @@ -24,9 +24,9 @@ #include "avcodec.h" #include "libavutil/avstring.h" #include "libavutil/bprint.h" -#include "ass_split.h" -#include "ass.h" #include "codec_internal.h" +#include "libavutil/ass_split_internal.h" +#include "libavutil/ass_internal.h" #define WEBVTT_STACK_SIZE 64 typedef struct { @@ -93,7 +93,7 @@ static void webvtt_stack_push_pop(WebVTTContext *s, const char c, int close) static void webvtt_style_apply(WebVTTContext *s, const char *style) { - ASSStyle *st = ff_ass_style_get(s->ass_ctx, style); + ASSStyle *st = avpriv_ass_style_get(s->ass_ctx, style); if (st) { if (st->bold != ASS_DEFAULT_BOLD) { webvtt_print(s, ""); @@ -172,12 +172,12 @@ static int webvtt_encode_frame(AVCodecContext *avctx, return AVERROR(EINVAL); } - dialog = ff_ass_split_dialog(s->ass_ctx, ass); + dialog = avpriv_ass_split_dialog(s->ass_ctx, ass); if (!dialog) return AVERROR(ENOMEM); webvtt_style_apply(s, dialog->style); - ff_ass_split_override_codes(&webvtt_callbacks, s, dialog->text); - ff_ass_free_dialog(&dialog); + avpriv_ass_split_override_codes(&webvtt_callbacks, s, dialog->text); + avpriv_ass_free_dialog(&dialog); } if (!av_bprint_is_complete(&s->buffer)) @@ -197,7 +197,7 @@ static int webvtt_encode_frame(AVCodecContext *avctx, static int webvtt_encode_close(AVCodecContext *avctx) { WebVTTContext *s = avctx->priv_data; - ff_ass_split_free(s->ass_ctx); + avpriv_ass_split_free(s->ass_ctx); av_bprint_finalize(&s->buffer, NULL); return 0; } @@ -206,7 +206,7 @@ static av_cold int webvtt_encode_init(AVCodecContext *avctx) { WebVTTContext *s = avctx->priv_data; s->avctx = avctx; - s->ass_ctx = ff_ass_split(avctx->subtitle_header); + s->ass_ctx = avpriv_ass_split(avctx->subtitle_header); av_bprint_init(&s->buffer, 0, AV_BPRINT_SIZE_UNLIMITED); return s->ass_ctx ? 0 : AVERROR_INVALIDDATA; } diff --git a/libavutil/Makefile b/libavutil/Makefile index 78f65c144a..a803add1e0 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -101,6 +101,8 @@ BUILT_HEADERS = avconfig.h \ OBJS = adler32.o \ aes.o \ aes_ctr.o \ + ass.o \ + ass_split.o \ audio_fifo.o \ avstring.o \ avsscanf.o \ diff --git a/libavcodec/ass.c b/libavutil/ass.c similarity index 59% rename from libavcodec/ass.c rename to libavutil/ass.c index a1e560d7ff..9eeaa38ba9 100644 --- a/libavcodec/ass.c +++ b/libavutil/ass.c @@ -19,21 +19,22 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ -#include "avcodec.h" -#include "ass.h" +#include "ass_internal.h" + +#include "subfmt.h" #include "libavutil/avstring.h" #include "libavutil/bprint.h" #include "libavutil/common.h" -int ff_ass_subtitle_header_full(AVCodecContext *avctx, - int play_res_x, int play_res_y, - const char *font, int font_size, - int primary_color, int secondary_color, - int outline_color, int back_color, - int bold, int italic, int underline, - int border_style, int alignment) +char* avpriv_ass_get_subtitle_header_full(int play_res_x, int play_res_y, + const char *font, int font_size, + int primary_color, int secondary_color, + int outline_color, int back_color, + int bold, int italic, int underline, + int border_style, int alignment, + int print_av_version) { - avctx->subtitle_header = av_asprintf( + char* header = av_asprintf( "[Script Info]\r\n" "; Script generated by FFmpeg/Lavc%s\r\n" "ScriptType: v4.00+\r\n" @@ -68,34 +69,31 @@ int ff_ass_subtitle_header_full(AVCodecContext *avctx, "\r\n" "[Events]\r\n" "Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text\r\n", - !(avctx->flags & AV_CODEC_FLAG_BITEXACT) ? AV_STRINGIFY(LIBAVCODEC_VERSION) : "", + print_av_version ? AV_STRINGIFY(LIBAVCODEC_VERSION) : "", play_res_x, play_res_y, font, font_size, primary_color, secondary_color, outline_color, back_color, -bold, -italic, -underline, border_style, alignment); - if (!avctx->subtitle_header) - return AVERROR(ENOMEM); - avctx->subtitle_header_size = strlen(avctx->subtitle_header); - return 0; + return header; } -int ff_ass_subtitle_header(AVCodecContext *avctx, - const char *font, int font_size, +char* avpriv_ass_get_subtitle_header(const char *font, int font_size, int color, int back_color, int bold, int italic, int underline, - int border_style, int alignment) + int border_style, int alignment, + int print_av_version) { - return ff_ass_subtitle_header_full(avctx, - ASS_DEFAULT_PLAYRESX, ASS_DEFAULT_PLAYRESY, - font, font_size, color, color, - back_color, back_color, - bold, italic, underline, - border_style, alignment); + return avpriv_ass_get_subtitle_header_full(ASS_DEFAULT_PLAYRESX, ASS_DEFAULT_PLAYRESY, + font, font_size, color, color, + back_color, back_color, + bold, italic, underline, + border_style, alignment, + print_av_version); } -int ff_ass_subtitle_header_default(AVCodecContext *avctx) +char* avpriv_ass_get_subtitle_header_default(int print_av_version) { - return ff_ass_subtitle_header(avctx, ASS_DEFAULT_FONT, + return avpriv_ass_get_subtitle_header(ASS_DEFAULT_FONT, ASS_DEFAULT_FONT_SIZE, ASS_DEFAULT_COLOR, ASS_DEFAULT_BACK_COLOR, @@ -103,10 +101,11 @@ int ff_ass_subtitle_header_default(AVCodecContext *avctx) ASS_DEFAULT_ITALIC, ASS_DEFAULT_UNDERLINE, ASS_DEFAULT_BORDERSTYLE, - ASS_DEFAULT_ALIGNMENT); + ASS_DEFAULT_ALIGNMENT, + print_av_version); } -char *ff_ass_get_dialog(int readorder, int layer, const char *style, +char *avpriv_ass_get_dialog(int readorder, int layer, const char *style, const char *speaker, const char *text) { return av_asprintf("%d,%d,%s,%s,0,0,0,,%s", @@ -114,61 +113,17 @@ char *ff_ass_get_dialog(int readorder, int layer, const char *style, speaker ? speaker : "", text); } -int ff_ass_add_rect2(AVSubtitle *sub, const char *dialog, - int readorder, int layer, const char *style, - const char *speaker, unsigned *nb_rect_allocated) -{ - AVSubtitleRect **rects = sub->rects, *rect; - char *ass_str; - uint64_t new_nb = 0; - - if (sub->num_rects >= UINT_MAX) - return AVERROR(ENOMEM); - - if (nb_rect_allocated && *nb_rect_allocated <= sub->num_rects) { - if (sub->num_rects < UINT_MAX / 17 * 16) { - new_nb = sub->num_rects + sub->num_rects/16 + 1; - } else - new_nb = UINT_MAX; - } else if (!nb_rect_allocated) - new_nb = sub->num_rects + 1; - - if (new_nb) { - rects = av_realloc_array(rects, new_nb, sizeof(*sub->rects)); - if (!rects) - return AVERROR(ENOMEM); - if (nb_rect_allocated) - *nb_rect_allocated = new_nb; - sub->rects = rects; - } - - rect = av_mallocz(sizeof(*rect)); - if (!rect) - return AVERROR(ENOMEM); - rects[sub->num_rects++] = rect; - rect->type = SUBTITLE_ASS; - ass_str = ff_ass_get_dialog(readorder, layer, style, speaker, dialog); - if (!ass_str) - return AVERROR(ENOMEM); - rect->ass = ass_str; - return 0; -} - -int ff_ass_add_rect(AVSubtitle *sub, const char *dialog, - int readorder, int layer, const char *style, - const char *speaker) +char *avpriv_ass_get_dialog_ex(int readorder, int layer, const char *style, + const char *speaker, int margin_l, int margin_r, + int margin_v, const char *text) { - return ff_ass_add_rect2(sub, dialog, readorder, layer, style, speaker, NULL); -} - -void ff_ass_decoder_flush(AVCodecContext *avctx) -{ - FFASSDecoderContext *s = avctx->priv_data; - if (!(avctx->flags2 & AV_CODEC_FLAG2_RO_FLUSH_NOOP)) - s->readorder = 0; + return av_asprintf("%d,%d,%s,%s,%d,%d,%d,,%s", + readorder, layer, style ? style : "Default", + speaker ? speaker : "", margin_l, margin_r, + margin_v, text); } -void ff_ass_bprint_text_event(AVBPrint *buf, const char *p, int size, +void avpriv_ass_bprint_text_event(AVBPrint *buf, const char *p, int size, const char *linebreaks, int keep_ass_markup) { const char *p_end = p + size; diff --git a/libavutil/ass_internal.h b/libavutil/ass_internal.h new file mode 100644 index 0000000000..cde5561cd3 --- /dev/null +++ b/libavutil/ass_internal.h @@ -0,0 +1,135 @@ +/* + * SSA/ASS common functions + * Copyright (c) 2010 Aurelien Jacobs + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_ASS_INTERNAL_H +#define AVUTIL_ASS_INTERNAL_H + +#include "subfmt.h" +#include "libavutil/bprint.h" + +#define ASS_DEFAULT_PLAYRESX 384 +#define ASS_DEFAULT_PLAYRESY 288 + +/** + * @name Default values for ASS style + * @{ + */ +#define ASS_DEFAULT_FONT "Arial" +#define ASS_DEFAULT_FONT_SIZE 16 +#define ASS_DEFAULT_COLOR 0xffffff +#define ASS_DEFAULT_BACK_COLOR 0 +#define ASS_DEFAULT_BOLD 0 +#define ASS_DEFAULT_ITALIC 0 +#define ASS_DEFAULT_UNDERLINE 0 +#define ASS_DEFAULT_ALIGNMENT 2 +#define ASS_DEFAULT_BORDERSTYLE 1 +/** @} */ + +/** + * Generate a suitable AVCodecContext.subtitle_header for SUBTITLE_ASS. + * Can specify all fields explicitly + * + * @param play_res_x subtitle frame width + * @param play_res_y subtitle frame height + * @param font name of the default font face to use + * @param font_size default font size to use + * @param primary_color default text color to use (ABGR) + * @param secondary_color default secondary text color to use (ABGR) + * @param outline_color default outline color to use (ABGR) + * @param back_color default background color to use (ABGR) + * @param bold 1 for bold text, 0 for normal text + * @param italic 1 for italic text, 0 for normal text + * @param underline 1 for underline text, 0 for normal text + * @param border_style 1 for outline, 3 for opaque box + * @param alignment position of the text (left, center, top...), defined after + * the layout of the numpad (1-3 sub, 4-6 mid, 7-9 top) + * @param print_av_version include library version in header + * @return a string containing the subtitle header that needs + * to be released via av_free() + */ +char* avpriv_ass_get_subtitle_header_full(int play_res_x, int play_res_y, + const char *font, int font_size, + int primary_color, int secondary_color, + int outline_color, int back_color, + int bold, int italic, int underline, + int border_style, int alignment, + int print_av_version); + +/** + * Generate a suitable AVCodecContext.subtitle_header for SUBTITLE_ASS. + * + * @param font name of the default font face to use + * @param font_size default font size to use + * @param color default text color to use (ABGR) + * @param back_color default background color to use (ABGR) + * @param bold 1 for bold text, 0 for normal text + * @param italic 1 for italic text, 0 for normal text + * @param underline 1 for underline text, 0 for normal text + * @param border_style 1 for outline, 3 for opaque box + * @param alignment position of the text (left, center, top...), defined after + * the layout of the numpad (1-3 sub, 4-6 mid, 7-9 top) + * @param print_av_version include library version in header + * @return a string containing the subtitle header that needs + * to be released via av_free() + */ +char* avpriv_ass_get_subtitle_header(const char *font, int font_size, + int color, int back_color, + int bold, int italic, int underline, + int border_style, int alignment, + int print_av_version); + +/** + * Generate a suitable AVCodecContext.subtitle_header for SUBTITLE_ASS + * with default style. + * + * @param print_av_version include library version in header + * @return a string containing the subtitle header that needs + * to be released via av_free() + */ +char* avpriv_ass_get_subtitle_header_default(int print_av_version); + +/** + * Craft an ASS dialog string. + */ +char *avpriv_ass_get_dialog(int readorder, int layer, const char *style, + const char *speaker, const char *text); + +/** + * Craft an ASS dialog string. + */ +char *avpriv_ass_get_dialog_ex(int readorder, int layer, const char *style, + const char *speaker, int margin_l, int margin_r, + int margin_v, const char *text); + +/** + * Escape a text subtitle using ASS syntax into an AVBPrint buffer. + * Newline characters will be escaped to \N. + * + * @param buf pointer to an initialized AVBPrint buffer + * @param p source text + * @param size size of the source text + * @param linebreaks additional newline chars, which will be escaped to \N + * @param keep_ass_markup braces and backslash will not be escaped if set + */ +void avpriv_ass_bprint_text_event(AVBPrint *buf, const char *p, int size, + const char *linebreaks, int keep_ass_markup); + +#endif /* AVUTIL_ASS_INTERNAL_H */ diff --git a/libavcodec/ass_split.c b/libavutil/ass_split.c similarity index 94% rename from libavcodec/ass_split.c rename to libavutil/ass_split.c index 73ef6196c5..deb8b2e07f 100644 --- a/libavcodec/ass_split.c +++ b/libavutil/ass_split.c @@ -28,7 +28,7 @@ #include "libavutil/error.h" #include "libavutil/macros.h" #include "libavutil/mem.h" -#include "ass_split.h" +#include "ass_split_internal.h" typedef enum { ASS_STR, @@ -379,7 +379,7 @@ static int ass_split(ASSSplitContext *ctx, const char *buf) return buf ? 0 : AVERROR_INVALIDDATA; } -ASSSplitContext *ff_ass_split(const char *buf) +ASSSplitContext *avpriv_ass_split(const char *buf) { ASSSplitContext *ctx = av_mallocz(sizeof(*ctx)); if (!ctx) @@ -388,7 +388,7 @@ ASSSplitContext *ff_ass_split(const char *buf) buf += 3; ctx->current_section = -1; if (ass_split(ctx, buf) < 0) { - ff_ass_split_free(ctx); + avpriv_ass_split_free(ctx); return NULL; } return ctx; @@ -418,7 +418,7 @@ static void free_section(ASSSplitContext *ctx, const ASSSection *section) av_freep((uint8_t *)&ctx->ass + section->offset); } -void ff_ass_free_dialog(ASSDialog **dialogp) +void avpriv_ass_free_dialog(ASSDialog **dialogp) { ASSDialog *dialog = *dialogp; if (!dialog) @@ -430,7 +430,7 @@ void ff_ass_free_dialog(ASSDialog **dialogp) av_freep(dialogp); } -ASSDialog *ff_ass_split_dialog(ASSSplitContext *ctx, const char *buf) +ASSDialog *avpriv_ass_split_dialog(ASSSplitContext *ctx, const char *buf) { int i; static const ASSFields fields[] = { @@ -457,7 +457,7 @@ ASSDialog *ff_ass_split_dialog(ASSSplitContext *ctx, const char *buf) buf = skip_space(buf); len = last ? strlen(buf) : strcspn(buf, ","); if (len >= INT_MAX) { - ff_ass_free_dialog(&dialog); + avpriv_ass_free_dialog(&dialog); return NULL; } convert_func[type](ptr, buf, len); @@ -467,7 +467,7 @@ ASSDialog *ff_ass_split_dialog(ASSSplitContext *ctx, const char *buf) return dialog; } -void ff_ass_split_free(ASSSplitContext *ctx) +void avpriv_ass_split_free(ASSSplitContext *ctx) { if (ctx) { int i; @@ -480,7 +480,7 @@ void ff_ass_split_free(ASSSplitContext *ctx) } -int ff_ass_split_override_codes(const ASSCodesCallbacks *callbacks, void *priv, +int avpriv_ass_split_override_codes(const ASSCodesCallbacks *callbacks, void *priv, const char *buf) { const char *text = NULL; @@ -503,8 +503,8 @@ int ff_ass_split_override_codes(const ASSCodesCallbacks *callbacks, void *priv, while (*buf == '\\') { char style[2], c[2], sep[2], c_num[2] = "0", tmp[128] = {0}; unsigned int color = 0xFFFFFFFF; - int len, size = -1, an = -1, alpha = -1; - int x1, y1, x2, y2, t1 = -1, t2 = -1; + int len, size = -1, an = -1, alpha = -1, scale = 0; + int x1, y1, x2, y2, t1 = -1, t2 = -1, accel = 1; if (sscanf(buf, "\\%1[bisu]%1[01\\}]%n", style, c, &len) > 1) { int close = c[0] == '0' ? 1 : c[0] == '1' ? 0 : -1; len += close != -1; @@ -552,6 +552,14 @@ int ff_ass_split_override_codes(const ASSCodesCallbacks *callbacks, void *priv, } else if (sscanf(buf, "\\org(%d,%d)%1[\\}]%n", &x1, &y1, sep, &len) > 2) { if (callbacks->origin) callbacks->origin(priv, x1, y1); + } else if (sscanf(buf, "\\t(%d,%d,%1[\\}]%n", &t1, &t2, sep, &len) > 2 || + sscanf(buf, "\\t(%d,%d,%d,%1[\\}]%n", &t1, &t2, &accel, sep, &len) > 3) { + if (callbacks->animate) + callbacks->animate(priv, t1, t2, accel, tmp); + } else if (sscanf(buf, "\\p%1[\\}]%n", sep, &len) > 0 || + sscanf(buf, "\\p%u%1[\\}]%n", &scale, sep, &len) > 1) { + if (callbacks->drawing_mode) + callbacks->drawing_mode(priv, scale); } else { len = strcspn(buf+1, "\\}") + 2; /* skip unknown code */ } @@ -575,7 +583,7 @@ int ff_ass_split_override_codes(const ASSCodesCallbacks *callbacks, void *priv, return 0; } -ASSStyle *ff_ass_style_get(ASSSplitContext *ctx, const char *style) +ASSStyle *avpriv_ass_style_get(ASSSplitContext *ctx, const char *style) { ASS *ass = &ctx->ass; int i; diff --git a/libavcodec/ass_split.h b/libavutil/ass_split_internal.h similarity index 86% rename from libavcodec/ass_split.h rename to libavutil/ass_split_internal.h index a45fb9b8a1..eee49ef0f5 100644 --- a/libavcodec/ass_split.h +++ b/libavutil/ass_split_internal.h @@ -19,8 +19,8 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ -#ifndef AVCODEC_ASS_SPLIT_H -#define AVCODEC_ASS_SPLIT_H +#ifndef AVUTIL_ASS_SPLIT_INTERNAL_H +#define AVUTIL_ASS_SPLIT_INTERNAL_H /** * fields extracted from the [Script Info] section @@ -81,7 +81,7 @@ typedef struct { char *effect; char *text; /**< actual text which will be displayed as a subtitle, can include style override control codes (see - ff_ass_split_override_codes()) */ + avpriv_ass_split_override_codes()) */ } ASSDialog; /** @@ -107,12 +107,12 @@ typedef struct ASSSplitContext ASSSplitContext; * @param buf String containing the ASS formatted data. * @return Newly allocated struct containing split data. */ -ASSSplitContext *ff_ass_split(const char *buf); +ASSSplitContext *avpriv_ass_split(const char *buf); /** - * Free a dialogue obtained from ff_ass_split_dialog(). + * Free a dialogue obtained from avpriv_ass_split_dialog(). */ -void ff_ass_free_dialog(ASSDialog **dialogp); +void avpriv_ass_free_dialog(ASSDialog **dialogp); /** * Split one ASS Dialogue line from a string buffer. @@ -121,14 +121,14 @@ void ff_ass_free_dialog(ASSDialog **dialogp); * @param buf String containing the ASS "Dialogue" line. * @return Pointer to the split ASSDialog. Must be freed with ff_ass_free_dialog() */ -ASSDialog *ff_ass_split_dialog(ASSSplitContext *ctx, const char *buf); +ASSDialog *avpriv_ass_split_dialog(ASSSplitContext *ctx, const char *buf); /** * Free all the memory allocated for an ASSSplitContext. * * @param ctx Context previously initialized by ff_ass_split(). */ -void ff_ass_split_free(ASSSplitContext *ctx); +void avpriv_ass_split_free(ASSSplitContext *ctx); /** @@ -141,6 +141,7 @@ typedef struct { * @{ */ void (*text)(void *priv, const char *text, int len); + void (*hard_space)(void *priv); void (*new_line)(void *priv, int forced); void (*style)(void *priv, char style, int close); void (*color)(void *priv, unsigned int /* color */, unsigned int color_id); @@ -156,7 +157,16 @@ typedef struct { * @{ */ void (*move)(void *priv, int x1, int y1, int x2, int y2, int t1, int t2); + void (*animate)(void *priv, int t1, int t2, int accel, char *style); void (*origin)(void *priv, int x, int y); + void (*drawing_mode)(void *priv, int scale); + /** @} */ + + /** + * @defgroup ass_ext ASS extensible parsing callback + * @{ + */ + void (*ext)(void *priv, int ext_id, const char *text, int p1, int p2); /** @} */ /** @@ -176,7 +186,7 @@ typedef struct { * @param buf The ASS "Dialogue" Text field to split. * @return >= 0 on success otherwise an error code <0 */ -int ff_ass_split_override_codes(const ASSCodesCallbacks *callbacks, void *priv, +int avpriv_ass_split_override_codes(const ASSCodesCallbacks *callbacks, void *priv, const char *buf); /** @@ -186,6 +196,6 @@ int ff_ass_split_override_codes(const ASSCodesCallbacks *callbacks, void *priv, * @param style name of the style to search for. * @return the ASSStyle corresponding to style, or NULL if style can't be found */ -ASSStyle *ff_ass_style_get(ASSSplitContext *ctx, const char *style); +ASSStyle *avpriv_ass_style_get(ASSSplitContext *ctx, const char *style); -#endif /* AVCODEC_ASS_SPLIT_H */ +#endif /* AVUTIL_ASS_SPLIT_INTERNAL_H */ From patchwork Sat May 28 13:25:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35965 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366340pzj; Sat, 28 May 2022 06:27:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwK4O5amZBmL9HePsNMFaL46EIpv3fKvJ6rM+VkCK7uCB8FwOCdy/UtSMGNHKXGibq1V7jF X-Received: by 2002:aa7:cad4:0:b0:428:715f:5ce4 with SMTP id l20-20020aa7cad4000000b00428715f5ce4mr50994829edt.124.1653744439341; Sat, 28 May 2022 06:27:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744439; cv=none; d=google.com; s=arc-20160816; b=yNhCiWD5XCQNO2kjd1mEBVUsFnFuLEMOE4azg0V5OUI1+P/AeMucPqjz9/mXohQNCb S8pBtXDyu9vVDaBWGBFFU490VuDPoXaH/BysfxXMYuoMpLESsw+diWfbbcqhJZt9BJGK zYtqnb7hyAQAKKnloOrIfSpF9NOLWuqm28hfw6hFByfhtMJsvOkwSM6rld/mwKaML6kE VFkS94nLG9YW4kbLjM9UaKyvSsFXEs4doLKMgG/1kZEccpJqSWhB4hfme2Z6OMeMGCXy ngq0jN7jjQi0Rlq66lgHOnChcTCIVGnsWTpyxZc/rTG7hc9w6yfx4SCkkpeqKWLnIMaz oybw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=Wjr+VY02jmx8JL2dBALmz8QEjJFtTlhRDu2iSBVqCC0=; b=BsUkhyxHW0tucqdTmyl/WLAJt5eCGKPtPRwwcQ75sTG7aaoDif6MF2hjmU4333Kr9K AG4yJV0M5Aeff6QIvKABwBzjqpInWPXO2S4m1V4j3Cu3uCMZwDfuB02YpsIrWel1oCR4 9SnrgMyKHcCwAxFCYxPSIzWVREjQE6ST5TgwyKniOx210g7biEGK7M5O0M738azrvU11 ttn+klliKUxEx1a9q3fJbSKd+ZzlbN1WHdve+KfGTEt59+4FZtqLiM7aoB9hY1jQ3/CY Vm0yhEeHYXwTUoXZgEIeGPOScmlmh2AYflbjErK3r1Ie7XYTN0FbO0635E+yDEcuAtAa d2xg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=j5hJUNgy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id go6-20020a1709070d8600b006df76385e32si8092687ejc.722.2022.05.28.06.27.18; Sat, 28 May 2022 06:27:19 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=j5hJUNgy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 09D4068B63E; Sat, 28 May 2022 16:25:45 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1595068B5D4 for ; Sat, 28 May 2022 16:25:35 +0300 (EEST) Received: by mail-pl1-f182.google.com with SMTP id d22so6444794plr.9 for ; Sat, 28 May 2022 06:25:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jl7XH25H8r0mb+5TSAgSCoRFYVVG3H9Vh0iwMC9ANFA=; b=j5hJUNgyB+Th1ojUjVy8UFcPBqyKI0pSiTNq1Ud+UuEqw8tAq+iIRGqVLJJt38wh/f Mxb049sasQ9tL2/gFs0FpR+i6afAk5rYkzH/U/NqrcPuX91n8P2Or21fyvpTG/HBkXiD 0hkW/GOvvVL/KA+J+j+GP/LmR73L3XeXT2HuYB6IxvkWy6TxF9nDaRN8iywuX5DDbaP7 VMGRFA5fKbIGGg6iK/nC9gv+b13Ds0aDoYemd2tZ4qihh7MbNluypPtU++lTc0UAjpeI FGoJWO/AECPh03+DMlob2ZKrt6iGcOkq+Qen1WDgfNKmbBC266hXfPIo7zcPi+3jKWvp P7vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jl7XH25H8r0mb+5TSAgSCoRFYVVG3H9Vh0iwMC9ANFA=; b=zNU+pToxyQTlI0N3+eDRzZ2XHhA75Kiy52+MVAtJ4nV1pWavO57lpwzd4UX8O7I8qz aDxnD//Wn8p+ehd/RVUfPujlvCySdXBGdkCP2Ndc9u6TEVYsayAH8Bcx5vupD+BWijHo CYVjc+ta5t9Cse7i1j6NVTXzhX6/RUAvzMJqKuLKMBgpk9meVSoqes2AjPZ1fhab0coM XJysYKawBYhWNxYjlImIfAlktOeDiAgw4Zda/YOTeLicLqaGg9ddUrBGWKKh/pHyVVSL xgBDY5Zlk/78Xz0giPjZ5567dz+ZDP+WAO+izK0Jwyzcw/ON4nV7MLzwHQOpHe2+ktmY Wpqw== X-Gm-Message-State: AOAM533XFFGYJK2zEZOTTwTkAiNC33wUXj/gB8JFQP3sE+tShDBala46 jLsleB5Y6i6TC6PEIsLOoopgTInEZAQ59g== X-Received: by 2002:a17:90b:1a8a:b0:1e0:3630:19f0 with SMTP id ng10-20020a17090b1a8a00b001e0363019f0mr13298690pjb.89.1653744333514; Sat, 28 May 2022 06:25:33 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id l10-20020a654c4a000000b003fa120ba003sm5312690pgr.81.2022.05.28.06.25.32 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:33 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <09d8cf7880efc9a244fb338e812176dab620181b.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:07 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 07/23] avcodec/subtitles: Replace deprecated enum values X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: hP5tArrjpRG7 From: softworkz Signed-off-by: softworkz --- libavcodec/ass.h | 2 +- libavcodec/assdec.c | 2 +- libavcodec/dvbsubdec.c | 2 +- libavcodec/dvdsubdec.c | 2 +- libavcodec/dvdsubenc.c | 2 +- libavcodec/pgssubdec.c | 2 +- libavcodec/xsubdec.c | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/libavcodec/ass.h b/libavcodec/ass.h index 8bc13d7ab8..43c5ad651a 100644 --- a/libavcodec/ass.h +++ b/libavcodec/ass.h @@ -83,7 +83,7 @@ static inline int avpriv_ass_add_rect(AVSubtitle *sub, const char *dialog, rects[sub->num_rects] = av_mallocz(sizeof(*rects[0])); if (!rects[sub->num_rects]) return AVERROR(ENOMEM); - rects[sub->num_rects]->type = SUBTITLE_ASS; + rects[sub->num_rects]->type = AV_SUBTITLE_FMT_ASS; ass_str = avpriv_ass_get_dialog(readorder, layer, style, speaker, dialog); if (!ass_str) return AVERROR(ENOMEM); diff --git a/libavcodec/assdec.c b/libavcodec/assdec.c index 1a1363471d..7bb60c9b26 100644 --- a/libavcodec/assdec.c +++ b/libavcodec/assdec.c @@ -53,7 +53,7 @@ static int ass_decode_frame(AVCodecContext *avctx, AVSubtitle *sub, if (!sub->rects[0]) return AVERROR(ENOMEM); sub->num_rects = 1; - sub->rects[0]->type = SUBTITLE_ASS; + sub->rects[0]->type = AV_SUBTITLE_FMT_ASS; sub->rects[0]->ass = av_strdup(avpkt->data); if (!sub->rects[0]->ass) return AVERROR(ENOMEM); diff --git a/libavcodec/dvbsubdec.c b/libavcodec/dvbsubdec.c index 6e510d12c7..fb54cf3f33 100644 --- a/libavcodec/dvbsubdec.c +++ b/libavcodec/dvbsubdec.c @@ -796,7 +796,7 @@ static int save_subtitle_set(AVCodecContext *avctx, AVSubtitle *sub, int *got_ou rect->w = region->width; rect->h = region->height; rect->nb_colors = (1 << region->depth); - rect->type = SUBTITLE_BITMAP; + rect->type = AV_SUBTITLE_FMT_BITMAP; rect->linesize[0] = region->width; clut = get_clut(ctx, region->clut); diff --git a/libavcodec/dvdsubdec.c b/libavcodec/dvdsubdec.c index b54073393e..058f4e22c5 100644 --- a/libavcodec/dvdsubdec.c +++ b/libavcodec/dvdsubdec.c @@ -407,7 +407,7 @@ static int decode_dvd_subtitles(DVDSubContext *ctx, AVSubtitle *sub_header, sub_header->rects[0]->y = y1; sub_header->rects[0]->w = w; sub_header->rects[0]->h = h; - sub_header->rects[0]->type = SUBTITLE_BITMAP; + sub_header->rects[0]->type = AV_SUBTITLE_FMT_BITMAP; sub_header->rects[0]->linesize[0] = w; sub_header->rects[0]->flags = is_menu ? AV_SUBTITLE_FLAG_FORCED : 0; } diff --git a/libavcodec/dvdsubenc.c b/libavcodec/dvdsubenc.c index d29db7d49c..24da94faee 100644 --- a/libavcodec/dvdsubenc.c +++ b/libavcodec/dvdsubenc.c @@ -269,7 +269,7 @@ static int encode_dvd_subtitles(AVCodecContext *avctx, if (rects == 0 || !h->rects) return AVERROR(EINVAL); for (i = 0; i < rects; i++) - if (h->rects[i]->type != SUBTITLE_BITMAP) { + if (h->rects[i]->type != AV_SUBTITLE_FMT_BITMAP) { av_log(avctx, AV_LOG_ERROR, "Bitmap subtitle required\n"); return AVERROR(EINVAL); } diff --git a/libavcodec/pgssubdec.c b/libavcodec/pgssubdec.c index e50c6766c5..05399863b6 100644 --- a/libavcodec/pgssubdec.c +++ b/libavcodec/pgssubdec.c @@ -535,7 +535,7 @@ static int display_end_segment(AVCodecContext *avctx, AVSubtitle *sub, if (!rect) return AVERROR(ENOMEM); sub->rects[sub->num_rects++] = rect; - rect->type = SUBTITLE_BITMAP; + rect->type = AV_SUBTITLE_FMT_BITMAP; /* Process bitmap */ object = find_object(ctx->presentation.objects[i].id, &ctx->objects); diff --git a/libavcodec/xsubdec.c b/libavcodec/xsubdec.c index d62fa164a5..30c3595c97 100644 --- a/libavcodec/xsubdec.c +++ b/libavcodec/xsubdec.c @@ -107,7 +107,7 @@ static int decode_frame(AVCodecContext *avctx, AVSubtitle *sub, sub->num_rects = 1; rect->x = x; rect->y = y; rect->w = w; rect->h = h; - rect->type = SUBTITLE_BITMAP; + rect->type = AV_SUBTITLE_FMT_BITMAP; rect->linesize[0] = w; rect->data[0] = av_malloc(w * h); rect->nb_colors = 4; From patchwork Sat May 28 13:25:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35967 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366496pzj; Sat, 28 May 2022 06:27:38 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzoOuVMLiHP3KqaSAaC5jQy6HDhxNsaPemSc/XhQcVW8LL72BLC/eq0HafpJ8xCgR4KZce2 X-Received: by 2002:a17:907:72ca:b0:6f9:8675:6a2a with SMTP id du10-20020a17090772ca00b006f986756a2amr43161845ejc.98.1653744458543; Sat, 28 May 2022 06:27:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744458; cv=none; d=google.com; s=arc-20160816; b=WufZA6YyZExFQL9eVH/zxUuUy6UWDVoCOJ+VOrDfjQN6edtw7Dng/OiGTXGX8arZqF UZuzKyQ6rN+5IF/kKIqxgBzzBLn/n+hJmZbaFc4mARDUWaV3QLxJrCq6MAMSH75g31Pe Tq74z4tYjttzAlzn+5ISI2a3DWH3tCa7e7tapL8RcQxeh5Qvrqplhu1NEJN3v5VsFcUf G5/6TT2J+m9kkneXFmRMEXhjMXaifAgQd5RoszasLQDXvDjLzUKGOFWLl4WAC0MzJsA9 pAMCQv6clBbZodDLcwCPdb72F9X3JAtphoJQpyahqcTTjIAMkaxg8zJK/9Kc1JZmMuEU p/ag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=kHRW9qQpLVrj2PLXh+IHZE94+ORvnfiZfDHhi0bz+DQ=; b=YId/WToJqTApuGxyafpj032TBjw7jYYs4xnYsV5ekrZ9JpCEGfCesyUSy3llXHm8JL 5Qim5lAEoO8G/S2bNFjqJCPEGlzTAbCAGfOb5R7za+g53IR1TFFXvDhCp4iKa6JOMfyC CwZ/bTV67+E0oGDEOdugi0JkAdTUG2VOkrjfxn+5OBUr47H/QL+fZf4/+K/25e9SqBBy WX8JI08RXmzV8uoX89UycUsPEyMowpomql46xntbBXgmUcdQkV8huoQCus9A//Np/BbX AnwfwEcSNTt+tF3AdEVw2YLBtQto6KGNJ3BTyMkyfjC+Sv1VHlagRco9Sz/KpE7vp8Eu /leA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=HaF34Bzo; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id u23-20020a1709060b1700b006f363ca1755si7371367ejg.921.2022.05.28.06.27.38; Sat, 28 May 2022 06:27:38 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=HaF34Bzo; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0DB0F68B60D; Sat, 28 May 2022 16:25:47 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0C14568B354 for ; Sat, 28 May 2022 16:25:36 +0300 (EEST) Received: by mail-pf1-f169.google.com with SMTP id 202so6680174pfu.0 for ; Sat, 28 May 2022 06:25:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XpeR+5t8lhg+ZuhU1t8oKuS2DyOXTKzpQzVQ4gP1INQ=; b=HaF34Bzowd6fVYzgq+KHlZExRqzXdYxjerWiFhoM9e/iDgQvKe3YuQXE40nUaxisVO 8jJFn0WmcrJ3G5FjSv2ykKY/t1CYA6XZIBPkZCBvdZb+rp+OXKrdUMRuPNihSs805LsI JtC3nZFjYx/ELhgiPmfpyX7/+2dKEqAGVtDc7McLg1xidSwWow/6Opf5kXF2u5gbPbL4 fiz2WmKY7QyfbH6xssyvYg2K1xSnePRXD0pTOpsAS73dASj80GJ5HPKQH2p0YJ3BdgNZ b87YQO5eqCgZskWwaxr05m2clgnGCQBtDS6+yZl5JuEL9ikVDrx9I53Xo09oAUtE18TX oI5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XpeR+5t8lhg+ZuhU1t8oKuS2DyOXTKzpQzVQ4gP1INQ=; b=Oyocy2NCYVwkNQQb1KHhc7BUy7kTMnZPda/9vdqeMgJVYJHhTPHp48impJqFR5NFko bJuUoIEB/fk25GokxCz4xI+j7XbDsFJijNk/+4q4H/mrtz7fgICxfQzXbG9sRS7TBTMi gIgHZ9qIKPLUcmKAtQnz7dWh7u+X8DdQAX97JGSjTREnDSAi4lOT4quve6YSmdjXrXZm IYllhXaAeuw3cFC/2voOt9bH61ogs2yMxHO+dPSQ3p09UxmRyYlT0t5/F77Z8bGtEqO/ CO4K3kd1PXI+I6IubWA3O5wwefrGZrvPuQ766qJW/JMC9mEGBD1K6MKXzRQl+JUxEsac rSbQ== X-Gm-Message-State: AOAM531qs7nrJaUEc2R6kk4+fTlZbaN8OPSjqMuKhIwUe7+3j7qyM4cq u4ueCVRCmVMGXyTehDC/X4wU7e/TGJ9w3Q== X-Received: by 2002:a65:64c8:0:b0:3fa:91bf:a5d8 with SMTP id t8-20020a6564c8000000b003fa91bfa5d8mr22432128pgv.473.1653744334449; Sat, 28 May 2022 06:25:34 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id x11-20020a170902b40b00b00162496617b9sm5577470plr.286.2022.05.28.06.25.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:34 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <897299bf7f87fed909b7421cd479140f0d35a8b8.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:08 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 08/23] fftools/play, probe: Adjust for subtitle changes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: pPFxMzJ1fs6U From: softworkz Signed-off-by: softworkz --- fftools/ffplay.c | 102 +++++++++++++++++++++------------------------- fftools/ffprobe.c | 47 +++++++++++++-------- 2 files changed, 77 insertions(+), 72 deletions(-) diff --git a/fftools/ffplay.c b/fftools/ffplay.c index 040afa0189..111e157979 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -153,7 +153,6 @@ typedef struct Clock { /* Common struct for handling all types of decoded data and allocated render buffers. */ typedef struct Frame { AVFrame *frame; - AVSubtitle sub; int serial; double pts; /* presentation timestamp for the frame */ double duration; /* estimated duration of the frame */ @@ -574,7 +573,7 @@ static int decoder_init(Decoder *d, AVCodecContext *avctx, PacketQueue *queue, S return 0; } -static int decoder_decode_frame(Decoder *d, AVFrame *frame, AVSubtitle *sub) { +static int decoder_decode_frame(Decoder *d, AVFrame *frame) { int ret = AVERROR(EAGAIN); for (;;) { @@ -608,6 +607,9 @@ static int decoder_decode_frame(Decoder *d, AVFrame *frame, AVSubtitle *sub) { } } break; + case AVMEDIA_TYPE_SUBTITLE: + ret = avcodec_receive_frame(d->avctx, frame); + break; } if (ret == AVERROR_EOF) { d->finished = d->pkt_serial; @@ -640,25 +642,11 @@ static int decoder_decode_frame(Decoder *d, AVFrame *frame, AVSubtitle *sub) { av_packet_unref(d->pkt); } while (1); - if (d->avctx->codec_type == AVMEDIA_TYPE_SUBTITLE) { - int got_frame = 0; - ret = avcodec_decode_subtitle2(d->avctx, sub, &got_frame, d->pkt); - if (ret < 0) { - ret = AVERROR(EAGAIN); - } else { - if (got_frame && !d->pkt->data) { - d->packet_pending = 1; - } - ret = got_frame ? 0 : (d->pkt->data ? AVERROR(EAGAIN) : AVERROR_EOF); - } - av_packet_unref(d->pkt); + if (avcodec_send_packet(d->avctx, d->pkt) == AVERROR(EAGAIN)) { + av_log(d->avctx, AV_LOG_ERROR, "Receive_frame and send_packet both returned EAGAIN, which is an API violation.\n"); + d->packet_pending = 1; } else { - if (avcodec_send_packet(d->avctx, d->pkt) == AVERROR(EAGAIN)) { - av_log(d->avctx, AV_LOG_ERROR, "Receive_frame and send_packet both returned EAGAIN, which is an API violation.\n"); - d->packet_pending = 1; - } else { - av_packet_unref(d->pkt); - } + av_packet_unref(d->pkt); } } } @@ -671,7 +659,6 @@ static void decoder_destroy(Decoder *d) { static void frame_queue_unref_item(Frame *vp) { av_frame_unref(vp->frame); - avsubtitle_free(&vp->sub); } static int frame_queue_init(FrameQueue *f, PacketQueue *pktq, int max_size, int keep_last) @@ -969,7 +956,7 @@ static void video_image_display(VideoState *is) if (frame_queue_nb_remaining(&is->subpq) > 0) { sp = frame_queue_peek(&is->subpq); - if (vp->pts >= sp->pts + ((float) sp->sub.start_display_time / 1000)) { + if (vp->pts >= sp->pts) { if (!sp->uploaded) { uint8_t* pixels[4]; int pitch[4]; @@ -981,25 +968,27 @@ static void video_image_display(VideoState *is) if (realloc_texture(&is->sub_texture, SDL_PIXELFORMAT_ARGB8888, sp->width, sp->height, SDL_BLENDMODE_BLEND, 1) < 0) return; - for (i = 0; i < sp->sub.num_rects; i++) { - AVSubtitleRect *sub_rect = sp->sub.rects[i]; + for (i = 0; i < sp->frame->num_subtitle_areas; i++) { + AVSubtitleArea *area = sp->frame->subtitle_areas[i]; + SDL_Rect sdl_rect = { .x = area->x, .y = area->y, .w = area->w, .h = area->h }; - sub_rect->x = av_clip(sub_rect->x, 0, sp->width ); - sub_rect->y = av_clip(sub_rect->y, 0, sp->height); - sub_rect->w = av_clip(sub_rect->w, 0, sp->width - sub_rect->x); - sub_rect->h = av_clip(sub_rect->h, 0, sp->height - sub_rect->y); + area->x = av_clip(area->x, 0, sp->width ); + area->y = av_clip(area->y, 0, sp->height); + area->w = av_clip(area->w, 0, sp->width - area->x); + area->h = av_clip(area->h, 0, sp->height - area->y); is->sub_convert_ctx = sws_getCachedContext(is->sub_convert_ctx, - sub_rect->w, sub_rect->h, AV_PIX_FMT_PAL8, - sub_rect->w, sub_rect->h, AV_PIX_FMT_BGRA, + area->w, area->h, AV_PIX_FMT_PAL8, + area->w, area->h, AV_PIX_FMT_BGRA, 0, NULL, NULL, NULL); if (!is->sub_convert_ctx) { av_log(NULL, AV_LOG_FATAL, "Cannot initialize the conversion context\n"); return; } - if (!SDL_LockTexture(is->sub_texture, (SDL_Rect *)sub_rect, (void **)pixels, pitch)) { - sws_scale(is->sub_convert_ctx, (const uint8_t * const *)sub_rect->data, sub_rect->linesize, - 0, sub_rect->h, pixels, pitch); + if (!SDL_LockTexture(is->sub_texture, &sdl_rect, (void **)pixels, pitch)) { + const uint8_t* data[2] = { area->buf[0]->data, (uint8_t *)&area->pal }; + sws_scale(is->sub_convert_ctx, data, area->linesize, + 0, area->h, pixels, pitch); SDL_UnlockTexture(is->sub_texture); } } @@ -1026,16 +1015,18 @@ static void video_image_display(VideoState *is) #if USE_ONEPASS_SUBTITLE_RENDER SDL_RenderCopy(renderer, is->sub_texture, NULL, &rect); #else - int i; + unsigned i; double xratio = (double)rect.w / (double)sp->width; double yratio = (double)rect.h / (double)sp->height; - for (i = 0; i < sp->sub.num_rects; i++) { - SDL_Rect *sub_rect = (SDL_Rect*)sp->sub.rects[i]; - SDL_Rect target = {.x = rect.x + sub_rect->x * xratio, - .y = rect.y + sub_rect->y * yratio, - .w = sub_rect->w * xratio, - .h = sub_rect->h * yratio}; - SDL_RenderCopy(renderer, is->sub_texture, sub_rect, &target); + for (i = 0; i < sp->frame->num_subtitle_areas; i++) { + AVSubtitleArea *area = sp->frame->subtitle_areas[i]; + SDL_Rect sub_rect = { .x = area->x, .y = area->y, + .w = area->w, .h = area->h}; + SDL_Rect target = {.x = rect.x + sub_rect.x * xratio, + .y = rect.y + sub_rect.y * yratio, + .w = sub_rect.w * xratio, + .h = sub_rect.h * yratio}; + SDL_RenderCopy(renderer, is->sub_texture, &sub_rect, &target); } #endif } @@ -1639,19 +1630,20 @@ retry: sp2 = NULL; if (sp->serial != is->subtitleq.serial - || (is->vidclk.pts > (sp->pts + ((float) sp->sub.end_display_time / 1000))) - || (sp2 && is->vidclk.pts > (sp2->pts + ((float) sp2->sub.start_display_time / 1000)))) + || (is->vidclk.pts > (sp->pts + ((float) sp->frame->subtitle_timing.duration / AV_TIME_BASE))) + || (sp2 && is->vidclk.pts > (sp2->pts))) { if (sp->uploaded) { int i; - for (i = 0; i < sp->sub.num_rects; i++) { - AVSubtitleRect *sub_rect = sp->sub.rects[i]; + for (i = 0; i < sp->frame->num_subtitle_areas; i++) { + AVSubtitleArea *area = sp->frame->subtitle_areas[i]; + SDL_Rect sdl_rect = { .x = area->x, .y = area->y, .w = area->w, .h = area->h }; uint8_t *pixels; int pitch, j; - if (!SDL_LockTexture(is->sub_texture, (SDL_Rect *)sub_rect, (void **)&pixels, &pitch)) { - for (j = 0; j < sub_rect->h; j++, pixels += pitch) - memset(pixels, 0, sub_rect->w << 2); + if (!SDL_LockTexture(is->sub_texture, &sdl_rect, (void **)&pixels, &pitch)) { + for (j = 0; j < area->h; j++, pixels += pitch) + memset(pixels, 0, area->w << 2); SDL_UnlockTexture(is->sub_texture); } } @@ -1762,7 +1754,7 @@ static int get_video_frame(VideoState *is, AVFrame *frame) { int got_picture; - if ((got_picture = decoder_decode_frame(&is->viddec, frame, NULL)) < 0) + if ((got_picture = decoder_decode_frame(&is->viddec, frame)) < 0) return -1; if (got_picture) { @@ -2032,7 +2024,7 @@ static int audio_thread(void *arg) return AVERROR(ENOMEM); do { - if ((got_frame = decoder_decode_frame(&is->auddec, frame, NULL)) < 0) + if ((got_frame = decoder_decode_frame(&is->auddec, frame)) < 0) goto the_end; if (got_frame) { @@ -2229,14 +2221,14 @@ static int subtitle_thread(void *arg) if (!(sp = frame_queue_peek_writable(&is->subpq))) return 0; - if ((got_subtitle = decoder_decode_frame(&is->subdec, NULL, &sp->sub)) < 0) + if ((got_subtitle = decoder_decode_frame(&is->subdec, sp->frame)) < 0) break; pts = 0; - if (got_subtitle && sp->sub.format == 0) { - if (sp->sub.pts != AV_NOPTS_VALUE) - pts = sp->sub.pts / (double)AV_TIME_BASE; + if (got_subtitle && sp->frame->format == AV_SUBTITLE_FMT_BITMAP) { + if (sp->frame->subtitle_timing.start_pts != AV_NOPTS_VALUE) + pts = (double)sp->frame->subtitle_timing.start_pts / (double)AV_TIME_BASE; sp->pts = pts; sp->serial = is->subdec.pkt_serial; sp->width = is->subdec.avctx->width; @@ -2246,7 +2238,7 @@ static int subtitle_thread(void *arg) /* now we can update the picture count */ frame_queue_push(&is->subpq); } else if (got_subtitle) { - avsubtitle_free(&sp->sub); + av_frame_free(&sp->frame); } } return 0; diff --git a/fftools/ffprobe.c b/fftools/ffprobe.c index c51c82ff65..d8e63c6df4 100644 --- a/fftools/ffprobe.c +++ b/fftools/ffprobe.c @@ -2452,22 +2452,42 @@ static void show_packet(WriterContext *w, InputFile *ifile, AVPacket *pkt, int p fflush(stdout); } -static void show_subtitle(WriterContext *w, AVSubtitle *sub, AVStream *stream, +static void show_subtitle(WriterContext *w, AVFrame *sub, AVStream *stream, AVFormatContext *fmt_ctx) { AVBPrint pbuf; + const char *s; av_bprint_init(&pbuf, 1, AV_BPRINT_SIZE_UNLIMITED); writer_print_section_header(w, SECTION_ID_SUBTITLE); print_str ("media_type", "subtitle"); - print_ts ("pts", sub->pts); - print_time("pts_time", sub->pts, &AV_TIME_BASE_Q); - print_int ("format", sub->format); - print_int ("start_display_time", sub->start_display_time); - print_int ("end_display_time", sub->end_display_time); - print_int ("num_rects", sub->num_rects); + print_ts ("pts", sub->subtitle_timing.start_pts); + print_time("pts_time", sub->subtitle_timing.start_pts, &AV_TIME_BASE_Q); + print_time("duration", sub->subtitle_timing.duration, &AV_TIME_BASE_Q); + + // Remain compatible with previous outputs + switch (sub->format) { + case AV_SUBTITLE_FMT_BITMAP: + print_int ("format", 0); + break; + case AV_SUBTITLE_FMT_TEXT: + print_int ("format", 1); + break; + case AV_SUBTITLE_FMT_ASS: + print_int ("format", 1); + break; + default: + print_int ("format", -1); + break; + } + + s = av_get_subtitle_fmt_name(sub->format); + if (s) print_str ("format_str", s); + else print_str_opt("format_str", "unknown"); + + print_int ("num_subtitle_rects", sub->num_subtitle_areas); writer_print_section_footer(w); @@ -2635,7 +2655,6 @@ static av_always_inline int process_frame(WriterContext *w, AVFormatContext *fmt_ctx = ifile->fmt_ctx; AVCodecContext *dec_ctx = ifile->streams[pkt->stream_index].dec_ctx; AVCodecParameters *par = ifile->streams[pkt->stream_index].st->codecpar; - AVSubtitle sub; int ret = 0, got_frame = 0; clear_log(1); @@ -2643,6 +2662,7 @@ static av_always_inline int process_frame(WriterContext *w, switch (par->codec_type) { case AVMEDIA_TYPE_VIDEO: case AVMEDIA_TYPE_AUDIO: + case AVMEDIA_TYPE_SUBTITLE: if (*packet_new) { ret = avcodec_send_packet(dec_ctx, pkt); if (ret == AVERROR(EAGAIN)) { @@ -2661,12 +2681,6 @@ static av_always_inline int process_frame(WriterContext *w, } } break; - - case AVMEDIA_TYPE_SUBTITLE: - if (*packet_new) - ret = avcodec_decode_subtitle2(dec_ctx, &sub, &got_frame, pkt); - *packet_new = 0; - break; default: *packet_new = 0; } @@ -2681,12 +2695,11 @@ static av_always_inline int process_frame(WriterContext *w, nb_streams_frames[pkt->stream_index]++; if (do_show_frames) if (is_sub) - show_subtitle(w, &sub, ifile->streams[pkt->stream_index].st, fmt_ctx); + show_subtitle(w, frame, ifile->streams[pkt->stream_index].st, fmt_ctx); else show_frame(w, frame, ifile->streams[pkt->stream_index].st, fmt_ctx); - if (is_sub) - avsubtitle_free(&sub); } + return got_frame || *packet_new; } From patchwork Sat May 28 13:25:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35960 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366089pzj; Sat, 28 May 2022 06:26:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyPRvh4ni/8rBh3QekNbocQvkHh7JldstlVQkhN8Mye5NQHLm1420yJgMponTKRZAdCcPys X-Received: by 2002:a17:907:9714:b0:6ff:338f:b4d7 with SMTP id jg20-20020a170907971400b006ff338fb4d7mr7573796ejc.718.1653744389726; Sat, 28 May 2022 06:26:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744389; cv=none; d=google.com; s=arc-20160816; b=c0U4DMdKtx7KVWtmPpgki3qbS+Ix9Zk9RKqEi0r/GmiPIfrWQI2yqYP+C6iq4SnLCm NLNM8RSccJCLqYpLocUodUJ4iQ8ee0i7Vp5T3/7uJfucaX2GiLCskTN1R/B1iN8ohBP8 xFbcHnHqwEjReb124MrdJjPhR5S7ILExgj876nXEJWxdX6gmQuupHaix4kT5l6FsbZNi QLQnFWqzWPd+WoMEmb+Z90TMuPCkC2aBOcuHYMQH3cbiPhohtk2mSzZjB0GE8Dsl4miQ mfMP0UV5awqQ3ThBUCGLKggv/ypJ32HfWJ6LWDD6x+DFraSIJmk1aAVfEms1KCiHmV4a Ojew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=4hwz+FrrjTq9nLEp/f5U8ktKcLiM8tGH8maZD+d8crE=; b=jsrmlKKhI/5pWaBh5AWUqj/6DdRsMaonjpLbYuw9XVV/vUMNTTKmRCVsPmQeCz8us9 mmJImAR0JQ0jOSgBXq/DNXWqNvjrIGHkkRYB/Z8JXiXnnuWKBh0obFa2FH5XoCRK4jJF Ay+c8fKnr6VrrfxKZxVx9E+9Gc4HsEj12dEUGvrGFB9cv4z1aqL2YLtNeEFMtAehRq3U gq3fy75hI3+zt4PS1yeSLgcBFDUgWZMzvT+rSzvj8dCqzjZaBRZkdYc09J4XIvmJsY0V 3AlZSgwMtdwhYv7lts2ruLzY8Tb7fjI/FV/gb+8hntVA0YUiUmRFViLyjCRDX0ECp/Uo X7Pg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=SmwF1Po2; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id kj24-20020a170907765800b006f39e0407e1si6469807ejc.342.2022.05.28.06.26.29; Sat, 28 May 2022 06:26:29 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=SmwF1Po2; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CD56B68B611; Sat, 28 May 2022 16:25:40 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id F3F3968B5A1 for ; Sat, 28 May 2022 16:25:36 +0300 (EEST) Received: by mail-pg1-f181.google.com with SMTP id a63so38491pge.12 for ; Sat, 28 May 2022 06:25:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wxzzJKpeRGnGC+l6EWLTROQmIRm+MeCE5MMiTQ1xCx8=; b=SmwF1Po2+9WumMK+/j36qe75NXissuKchvKzjSEwXhQZcan+ckGTWVnN9xTPVkQ4a2 5A1NRj4tzIXmjMwq+4e2qaHyKPHxk0u7znrEI3aCVMJvwY+lnl0DjWKBknB3SEWyuCHu R55c7zRLJxE00vdMXxnR8eezn60nvMk+ikEp55/wNSj43NEoAlT/a8HBFiYNVMiu0wfL wJT8yn5ds33t0EJLmpCf7DeIMA1MdAADalNAsvb0LEapYaxJV93ZlCdNNLRY5OZ21j2i ogcuiUtOz75Ciy1IwxxT+3bMcdZuVAPTolJNg9VGshyqdp+eyIx+36VEwe0OJLxr2Ji4 zohA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wxzzJKpeRGnGC+l6EWLTROQmIRm+MeCE5MMiTQ1xCx8=; b=0qyD0M6NnggFysIbjnfj7Ps32fIbJ9O9LcPY/pYqCMrgsxqOiI3+K+Zx1KrXK8aR5y 8Iw3FzUsCbZ5cyEnK7o9YtPu28foDT5edlE2fdRZHi73RUo8nrx+vrYTTQGLeXTRp9Np HiSqUw1VkVVFUblXk8hNcQ8SLjyyqkpjEZnG4BtebdrXioTeAj9dmR9aZ9cDH7Fj9E0s UEH4Pyrpn7gBzYhHBXQHgB35M4Z005gfArjRNgQ+OMfgEVvlaE82nm1Lz1qupN6erpWn JmToWdnJOjMLLN+VrRuR+MJwVSznTNKdFr7Ba8CE8jUxX8tL92U0LXIZmczzJBW3wWRY nNXQ== X-Gm-Message-State: AOAM532jTyJWo32u4WJd0LACYtVZMjJMm7VRL15mWItT5ufUmd1AXotk ZgeT//ToIxVEokyN0ntjVy3YRGApxPLJKw== X-Received: by 2002:a65:694d:0:b0:3fa:34df:cdc6 with SMTP id w13-20020a65694d000000b003fa34dfcdc6mr26826403pgq.439.1653744335384; Sat, 28 May 2022 06:25:35 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id 10-20020aa7924a000000b00518950bfc82sm1972521pfp.10.2022.05.28.06.25.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:34 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: In-Reply-To: References: Date: Sat, 28 May 2022 13:25:09 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 09/23] avfilter/subtitles: Add subtitles.c for subtitle frame allocation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: oMt+rns3DJlU From: softworkz Analog to avfilter/video.c and avfilter/audio.c Signed-off-by: softworkz --- libavfilter/Makefile | 1 + libavfilter/avfilter.c | 4 +++ libavfilter/internal.h | 1 + libavfilter/subtitles.c | 63 +++++++++++++++++++++++++++++++++++++++++ libavfilter/subtitles.h | 44 ++++++++++++++++++++++++++++ 5 files changed, 113 insertions(+) create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h diff --git a/libavfilter/Makefile b/libavfilter/Makefile index e0e4d0de2c..842068e902 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -20,6 +20,7 @@ OBJS = allfilters.o \ framequeue.o \ graphdump.o \ graphparser.o \ + subtitles.o \ version.o \ video.o \ diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index 965f5d0f63..28c5430c3e 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -42,6 +42,7 @@ #include "formats.h" #include "framepool.h" #include "internal.h" +#include "subtitles.h" static void tlog_ref(void *ctx, AVFrame *ref, int end) { @@ -1452,6 +1453,9 @@ int ff_inlink_make_frame_writable(AVFilterLink *link, AVFrame **rframe) case AVMEDIA_TYPE_AUDIO: out = ff_get_audio_buffer(link, frame->nb_samples); break; + case AVMEDIA_TYPE_SUBTITLE: + out = ff_get_subtitles_buffer(link, link->format); + break; default: return AVERROR(EINVAL); } diff --git a/libavfilter/internal.h b/libavfilter/internal.h index 0f8da367d0..6c8496879a 100644 --- a/libavfilter/internal.h +++ b/libavfilter/internal.h @@ -89,6 +89,7 @@ struct AVFilterPad { union { AVFrame *(*video)(AVFilterLink *link, int w, int h); AVFrame *(*audio)(AVFilterLink *link, int nb_samples); + AVFrame *(*subtitle)(AVFilterLink *link, int format); } get_buffer; /** diff --git a/libavfilter/subtitles.c b/libavfilter/subtitles.c new file mode 100644 index 0000000000..951bfd612c --- /dev/null +++ b/libavfilter/subtitles.c @@ -0,0 +1,63 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/common.h" + +#include "subtitles.h" +#include "avfilter.h" +#include "internal.h" + + +AVFrame *ff_null_get_subtitles_buffer(AVFilterLink *link, int format) +{ + return ff_get_subtitles_buffer(link->dst->outputs[0], format); +} + +AVFrame *ff_default_get_subtitles_buffer(AVFilterLink *link, int format) +{ + AVFrame *frame; + + frame = av_frame_alloc(); + if (!frame) + return NULL; + + frame->format = format; + frame->type = AVMEDIA_TYPE_SUBTITLE; + + if (av_frame_get_buffer2(frame, 0) < 0) { + av_frame_free(&frame); + return NULL; + } + + return frame; +} + +AVFrame *ff_get_subtitles_buffer(AVFilterLink *link, int format) +{ + AVFrame *ret = NULL; + + if (link->dstpad->get_buffer.subtitle) + ret = link->dstpad->get_buffer.subtitle(link, format); + + if (!ret) + ret = ff_default_get_subtitles_buffer(link, format); + + return ret; +} diff --git a/libavfilter/subtitles.h b/libavfilter/subtitles.h new file mode 100644 index 0000000000..4a9115126e --- /dev/null +++ b/libavfilter/subtitles.h @@ -0,0 +1,44 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVFILTER_SUBTITLES_H +#define AVFILTER_SUBTITLES_H + +#include "avfilter.h" +#include "internal.h" + +/** default handler for get_subtitles_buffer() for subtitle inputs */ +AVFrame *ff_default_get_subtitles_buffer(AVFilterLink *link, int format); + +/** get_subtitles_buffer() handler for filters which simply pass subtitles along */ +AVFrame *ff_null_get_subtitles_buffer(AVFilterLink *link, int format); + +/** + * Request a subtitles frame with a specific set of permissions. + * + * @param link the output link to the filter from which the buffer will + * be requested + * @param format The subtitles format. + * @return A reference to the frame. This must be unreferenced with + * av_frame_free when you are finished with it. +*/ +AVFrame *ff_get_subtitles_buffer(AVFilterLink *link, int format); + +#endif /* AVFILTER_SUBTITLES_H */ From patchwork Sat May 28 13:25:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35961 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366175pzj; Sat, 28 May 2022 06:26:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwClyo6tjEVjypLVYiz+kZN64WMO1RUUkKwmWodaJCU1uA0AXBF4qzbeJFacuY+OcFFDDd1 X-Received: by 2002:a17:907:a0ca:b0:6f8:5bef:b9c with SMTP id hw10-20020a170907a0ca00b006f85bef0b9cmr42678436ejc.630.1653744409015; Sat, 28 May 2022 06:26:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744409; cv=none; d=google.com; s=arc-20160816; b=FgqbHFzMoZTc76yk1i8YXGQ/6oOjPS5t/CT4jiZ1zd+Bf4/yKFgh7PC6ozSgGJr75Q mdivPFyP8LAaAAFQqDjxAEehai0L31PUbqU8/O1SfM14T/NYIWBrC4bNHzKqYWuvKskI B9ms7w4puNUMjihrjyupvK0LXIKFKsoMOnuwCY/5sm2o+q/WKpjL6trp7eVwyXNWNTAT kJe3kXOT1LbAmq6QWq4PpsVMJu0ipOp5p06sj9A+ieG4SAXkvWb5BtzjhTmDZjgsCiBh ichkA/vtQ2XI4r6jiemQSA11Xt0LmExtRmT+uYxq2iEc84jyaxQR/TcjEM2jfG2/bcdu N6fQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=6wqfjOCZN/DLVbp8V9Nqyhg5ilk9fIwTZDNqS60LX8U=; b=XAKOVenQRBtQU4F+TmVNodFY3U9TO9+v01N2RrRwhWs7NPBe6SvFnL2Ahr3VFGZ/hC S4finafOWwCaAd8dhkVrqvlWr+0y30RRK9+UQibolUUWmh18D4EatSIpFcJnjm5AWHlX NiINwby/aPP/JpLcuC7JlwIwyQxbhPccTj0YzlxPAJ3Ngup1gQ61i4x+RNs5Y+Hdt8V/ h1CsCSe+zsTP8ZvoHDyAwCB+fLZpTLgfeOz7lqgkxWT/8YC3CmP5aOtEH+ZZi51CLuty Y4hQVMs0rQaaPhIAj2vP6ZzN1TTiJ8o6Yr6+1b1D+dfUtqi8prqI/8GLwH5Kdj+Bo9gY BdDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=J3dzdTMi; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z73-20020a509e4f000000b00428727e4425si6026621ede.408.2022.05.28.06.26.48; Sat, 28 May 2022 06:26:48 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=J3dzdTMi; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6884968B5F8; Sat, 28 May 2022 16:25:42 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 386BA68B5DB for ; Sat, 28 May 2022 16:25:37 +0300 (EEST) Received: by mail-pl1-f176.google.com with SMTP id q4so6442612plr.11 for ; Sat, 28 May 2022 06:25:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sudeUR61eBYI+Fyp+5hQtywz0i+fvmHVnQPMuFmx5q0=; b=J3dzdTMisVr7Tmqv7rJh0psigDfYf8N1pG0jYSZzbRVq+5+vlJRcy3D5fX9ptB4EFL 5zVGY3FbKtVSaqVJ97rttpN1AvyDAgDMsiPU2APJC7MxSuKN8EcMvfvV95f+CMGHpcCo 6PJA+tH7MZ3bJwN2rJS600SKOXsX2uYN3lnwwX1Z5Z1SQEV8WJAaHoAHzdFp90SiwaB7 P+vkkkYWImckcpEeHTrhru/tQaN5cTp2wCoLhzoQtbYH6wpqk+39+VCd0XlW6gV14+vy Q2gYSMOmscVuInSUYO1EDU+C0VZXoy5G7pfeGeImSLvFe6RmW/s5mFF8VtpNz0KGjpu4 nQFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sudeUR61eBYI+Fyp+5hQtywz0i+fvmHVnQPMuFmx5q0=; b=cOiCThgxxnuWySh562PYyXBMBdkSWnRU7PmxcdF2q+Ta1cwDMLFykNV/zpKmRFD1+A dfasnZ5WELSBIZmWDxoFHdCbE6Cm/5l87nmJYQ9Vzftohyr68nrKdek2T79/C0wh7vvp diV2ZcEh8ANDUuCkFsiAnOwqVKwwSVFdMkqu5qDh0ziA4nJx+2LNrdfh0ezjxhhSfyQ0 GTgeSIK3R9/ow2dN3rHFE0MUw0TTl+gwXHYbZJUo3JkfhjRCCWrNNm9Usn6EpzSI8tp0 LCJEUbW0Y3Qu1qTzoTh9NcMQW6U3Ja+Ly6xFFOYgSKYLagt/ZdyuPjhv7+GpY1/ixQ90 6UPg== X-Gm-Message-State: AOAM530+FM1bDGIuojBMzRTBuQTWRw0XLlJ6QUXA/5Tx1KUqE9I7HvWK hu0AbY39l6AWMQU/REbRJDJmxSKlnZuGSQ== X-Received: by 2002:a17:90b:1b48:b0:1e0:ab0:df00 with SMTP id nv8-20020a17090b1b4800b001e00ab0df00mr13362230pjb.52.1653744336366; Sat, 28 May 2022 06:25:36 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id p11-20020a1709028a8b00b00163247b64bfsm5580789plo.115.2022.05.28.06.25.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:35 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <0781e974a256763f0d7d1be811dc1039efee37f2.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:10 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 10/23] avfilter/avfilter: Handle subtitle frames X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: PcI8AXtNq/I5 From: softworkz Signed-off-by: softworkz --- libavfilter/avfilter.c | 12 +++++++++--- libavfilter/avfilter.h | 11 +++++++++++ libavfilter/avfiltergraph.c | 5 +++++ libavfilter/formats.c | 16 ++++++++++++++++ libavfilter/formats.h | 3 +++ libavfilter/internal.h | 18 +++++++++++++++--- 6 files changed, 59 insertions(+), 6 deletions(-) diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index 28c5430c3e..30745f41cb 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -52,7 +52,8 @@ static void tlog_ref(void *ctx, AVFrame *ref, int end) ref->linesize[0], ref->linesize[1], ref->linesize[2], ref->linesize[3], ref->pts, ref->pkt_pos); - if (ref->width) { + switch(ref->type) { + case AVMEDIA_TYPE_VIDEO: ff_tlog(ctx, " a:%d/%d s:%dx%d i:%c iskey:%d type:%c", ref->sample_aspect_ratio.num, ref->sample_aspect_ratio.den, ref->width, ref->height, @@ -60,12 +61,13 @@ static void tlog_ref(void *ctx, AVFrame *ref, int end) ref->top_field_first ? 'T' : 'B', /* Top / Bottom */ ref->key_frame, av_get_picture_type_char(ref->pict_type)); - } - if (ref->nb_samples) { + break; + case AVMEDIA_TYPE_AUDIO: ff_tlog(ctx, " cl:%"PRId64"d n:%d r:%d", ref->channel_layout, ref->nb_samples, ref->sample_rate); + break; } ff_tlog(ctx, "]%s", end ? "\n" : ""); @@ -1013,6 +1015,10 @@ int ff_filter_frame(AVFilterLink *link, AVFrame *frame) av_assert1(frame->width == link->w); av_assert1(frame->height == link->h); } + } else if (link->type == AVMEDIA_TYPE_SUBTITLE) { + if (frame->format != link->format) { + av_log(link->dst, AV_LOG_WARNING, "Subtitle format change from %d to %d\n", link->format, frame->format); + } } else { if (frame->format != link->format) { av_log(link->dst, AV_LOG_ERROR, "Format change is not supported\n"); diff --git a/libavfilter/avfilter.h b/libavfilter/avfilter.h index 2e8197c9a6..c436f304bc 100644 --- a/libavfilter/avfilter.h +++ b/libavfilter/avfilter.h @@ -45,6 +45,7 @@ #include "libavutil/log.h" #include "libavutil/samplefmt.h" #include "libavutil/pixfmt.h" +#include "libavutil/subfmt.h" #include "libavutil/rational.h" #include "libavfilter/version_major.h" @@ -349,6 +350,12 @@ typedef struct AVFilter { * and outputs use the same sample rate and channel count/layout. */ const enum AVSampleFormat *samples_list; + /** + * Analogous to pixels, but delimited by AV_SUBTITLE_FMT_NONE + * and restricted to filters that only have AVMEDIA_TYPE_SUBTITLE + * inputs and outputs. + */ + const enum AVSubtitleType *subs_list; /** * Equivalent to { pix_fmt, AV_PIX_FMT_NONE } as pixels_list. */ @@ -357,6 +364,10 @@ typedef struct AVFilter { * Equivalent to { sample_fmt, AV_SAMPLE_FMT_NONE } as samples_list. */ enum AVSampleFormat sample_fmt; + /** + * Equivalent to { sub_fmt, AV_SUBTITLE_FMT_NONE } as subs_list. + */ + enum AVSubtitleType sub_fmt; } formats; int priv_size; ///< size of private data to allocate for the filter diff --git a/libavfilter/avfiltergraph.c b/libavfilter/avfiltergraph.c index b7dbfc063b..f37be0203b 100644 --- a/libavfilter/avfiltergraph.c +++ b/libavfilter/avfiltergraph.c @@ -309,6 +309,8 @@ static int filter_link_check_formats(void *log, AVFilterLink *link, AVFilterForm return ret; break; + case AVMEDIA_TYPE_SUBTITLE: + return 0; default: av_assert0(!"reached"); } @@ -439,6 +441,9 @@ static int query_formats(AVFilterGraph *graph, void *log_ctx) if (!link) continue; + if (link->type == AVMEDIA_TYPE_SUBTITLE) + continue; + neg = ff_filter_get_negotiation(link); av_assert0(neg); for (neg_step = 1; neg_step < neg->nb_mergers; neg_step++) { diff --git a/libavfilter/formats.c b/libavfilter/formats.c index e8c2888c0c..12585ed428 100644 --- a/libavfilter/formats.c +++ b/libavfilter/formats.c @@ -19,6 +19,7 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include "libavutil/subfmt.h" #include "libavutil/avassert.h" #include "libavutil/channel_layout.h" #include "libavutil/common.h" @@ -491,6 +492,13 @@ AVFilterFormats *ff_all_formats(enum AVMediaType type) return NULL; fmt++; } + } else if (type == AVMEDIA_TYPE_SUBTITLE) { + if (ff_add_format(&ret, AV_SUBTITLE_FMT_BITMAP) < 0) + return NULL; + if (ff_add_format(&ret, AV_SUBTITLE_FMT_ASS) < 0) + return NULL; + if (ff_add_format(&ret, AV_SUBTITLE_FMT_TEXT) < 0) + return NULL; } return ret; @@ -774,6 +782,10 @@ int ff_default_query_formats(AVFilterContext *ctx) type = AVMEDIA_TYPE_AUDIO; formats = ff_make_format_list(f->formats.samples_list); break; + case FF_FILTER_FORMATS_SUBFMTS_LIST: + type = AVMEDIA_TYPE_SUBTITLE; + formats = ff_make_format_list(f->formats.subs_list); + break; case FF_FILTER_FORMATS_SINGLE_PIXFMT: type = AVMEDIA_TYPE_VIDEO; formats = ff_make_formats_list_singleton(f->formats.pix_fmt); @@ -782,6 +794,10 @@ int ff_default_query_formats(AVFilterContext *ctx) type = AVMEDIA_TYPE_AUDIO; formats = ff_make_formats_list_singleton(f->formats.sample_fmt); break; + case FF_FILTER_FORMATS_SINGLE_SUBFMT: + type = AVMEDIA_TYPE_SUBTITLE; + formats = ff_make_formats_list_singleton(f->formats.sub_fmt); + break; default: av_assert2(!"Unreachable"); /* Intended fallthrough */ diff --git a/libavfilter/formats.h b/libavfilter/formats.h index 22224dce2d..6cf952a059 100644 --- a/libavfilter/formats.h +++ b/libavfilter/formats.h @@ -183,6 +183,9 @@ av_warn_unused_result int ff_add_channel_layout(AVFilterChannelLayouts **l, const AVChannelLayout *channel_layout); +av_warn_unused_result +int ff_add_subtitle_type(AVFilterFormats **avff, int64_t fmt); + /** * Add *ref as a new reference to f. */ diff --git a/libavfilter/internal.h b/libavfilter/internal.h index 6c8496879a..7972c5c6de 100644 --- a/libavfilter/internal.h +++ b/libavfilter/internal.h @@ -148,9 +148,11 @@ static av_always_inline int ff_filter_execute(AVFilterContext *ctx, avfilter_act enum FilterFormatsState { /** - * The default value meaning that this filter supports all formats - * and (for audio) sample rates and channel layouts/counts as long - * as these properties agree for all inputs and outputs. + * The default value meaning that this filter supports + * - For video: all formats + * - For audio: all sample rates and channel layouts/counts + * - For subtitles: all subtitle formats + * as long as these properties agree for all inputs and outputs. * This state is only allowed in case all inputs and outputs actually * have the same type. * The union is unused in this state. @@ -161,8 +163,10 @@ enum FilterFormatsState { FF_FILTER_FORMATS_QUERY_FUNC, ///< formats.query active. FF_FILTER_FORMATS_PIXFMT_LIST, ///< formats.pixels_list active. FF_FILTER_FORMATS_SAMPLEFMTS_LIST, ///< formats.samples_list active. + FF_FILTER_FORMATS_SUBFMTS_LIST, ///< formats.subs_list active. FF_FILTER_FORMATS_SINGLE_PIXFMT, ///< formats.pix_fmt active FF_FILTER_FORMATS_SINGLE_SAMPLEFMT, ///< formats.sample_fmt active. + FF_FILTER_FORMATS_SINGLE_SUBFMT, ///< formats.sub_fmt active. }; #define FILTER_QUERY_FUNC(func) \ @@ -174,16 +178,24 @@ enum FilterFormatsState { #define FILTER_SAMPLEFMTS_ARRAY(array) \ .formats.samples_list = array, \ .formats_state = FF_FILTER_FORMATS_SAMPLEFMTS_LIST +#define FILTER_SUBFMTS_ARRAY(array) \ + .formats.subs_list = array, \ + .formats_state = FF_FILTER_FORMATS_SUBFMTS_LIST #define FILTER_PIXFMTS(...) \ FILTER_PIXFMTS_ARRAY(((const enum AVPixelFormat []) { __VA_ARGS__, AV_PIX_FMT_NONE })) #define FILTER_SAMPLEFMTS(...) \ FILTER_SAMPLEFMTS_ARRAY(((const enum AVSampleFormat[]) { __VA_ARGS__, AV_SAMPLE_FMT_NONE })) +#define FILTER_SUBFMTS(...) \ + FILTER_SUBFMTS_ARRAY(((const enum AVSubtitleType[]) { __VA_ARGS__, AV_SUBTITLE_FMT_NONE })) #define FILTER_SINGLE_PIXFMT(pix_fmt_) \ .formats.pix_fmt = pix_fmt_, \ .formats_state = FF_FILTER_FORMATS_SINGLE_PIXFMT #define FILTER_SINGLE_SAMPLEFMT(sample_fmt_) \ .formats.sample_fmt = sample_fmt_, \ .formats_state = FF_FILTER_FORMATS_SINGLE_SAMPLEFMT +#define FILTER_SINGLE_SUBFMT(sub_fmt_) \ + .formats.sub_fmt = sub_fmt_, \ + .formats_state = FF_FILTER_FORMATS_SINGLE_SUBFMT #define FILTER_INOUTPADS(inout, array) \ .inout = array, \ From patchwork Sat May 28 13:25:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35963 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366211pzj; Sat, 28 May 2022 06:26:58 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxT/KK59InAD+6Zy9ZUhYE337VsC6z0dApbxPhp4NYJWoKO/EBqyEn4e+ToFfjN/9drq6g1 X-Received: by 2002:a05:6402:1150:b0:42a:9dbf:8860 with SMTP id g16-20020a056402115000b0042a9dbf8860mr49558769edw.350.1653744418490; Sat, 28 May 2022 06:26:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744418; cv=none; d=google.com; s=arc-20160816; b=WstZYLhyd080WII0mTBGuZrv08sq4uPXehriClwIvjE9aJ1uavOY4IKMKIfezFqd+U 9eAj3rNKvq4LyVUh/K0z/EkHSHMj19ZMAFm5nQugxboi8JseZXQErI4fPoWz4e0ftCCC 0yZNpbHEpDe1UEuc00szaKxW9FxG33Hvoduygza733OjWvy0TLAgA8VCRKOK/p3H+u/m 6mi76uVJpLI4ZYnBUVE1FJUVEguTXMkC7J0HWsEQyxrsaCRC2IAt13OcO7rSU8YQK7GY nHdQQj5Fed68S0ZbE06oA2yyFc2aqYlUv051HyxGhqOa/k9A5yiCrc8UfVJiHefTib87 5apg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=UtqzexoT4Ezmtfi57f7x4tIuzPiZzyGh8hkddPuZOVE=; b=Jsv1mAFsWEILKUtLwx5oFVINtFA5fV09B0ZWlGLUFzJ9eDnpWZELmSaEwAZwOiGpAB /UCvm89/EX0cHCFuG1OpJSem2Cz6vsLva7Ib1l6xO9O1fkwvTkm+ezlheRo1/xfzQxSw S2fR4vmuC9eI4l84qN8EQqmy/M3WbR2sMDgQZP89RdaZO0uOvunUnvO04sHzeXMkkPTb JcpUnR5Y9rx7DB2xDUle0e5m8efHi55p4HBkxnHaGqwJci7jW3w6YM0x5SRah62wYtKF 2N9LlQG/NjAg7NwDCF9WBul1Bvd9uzSqb+iWMYUDvxiyRe15Hh6RjyT0NuDVQFl0YAjB QmqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=G6TJNm47; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id qk32-20020a1709077fa000b006fe58b19736si6883559ejc.510.2022.05.28.06.26.58; Sat, 28 May 2022 06:26:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=G6TJNm47; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 54D5568B5D4; Sat, 28 May 2022 16:25:43 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C8FC668B56B for ; Sat, 28 May 2022 16:25:38 +0300 (EEST) Received: by mail-pl1-f176.google.com with SMTP id n8so6473020plh.1 for ; Sat, 28 May 2022 06:25:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HvaSrUNkdYU7ZKtqQZac5tap7Ome60MMPRJc94yc6OY=; b=G6TJNm47Pco2JGKBYdmE+WMZw1jzmW+tzko4xPB8TXZfA9QGYwR+atu3cZny6K3Slq zLHRRhtqSWobwvNtvTIUMBYotkJOrZKy8JFxWhpXsyyjdHKxfOENUbwLduL1gssVdVdx wp8cmmI5d2ksISl2D0S/K29FzQ+HPxoF+1QHt51esvpB54/HN4BGmCji0rm0nH7NPF4/ PywwC1Y4Hv8grjXDnnecamdi8+PLgcUGhg+gQEIueKQJ8MK62cHRfK6gaVK8EpHbM1JD z/vBYJmKnOJVoZWz3UaV2W14OW/RGcpUDYuMShSZvyahVCQ0OjNbL/ID7REBSNNE2XZb CagA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HvaSrUNkdYU7ZKtqQZac5tap7Ome60MMPRJc94yc6OY=; b=jqXcdKTYxUw6TAJe7A/0TGOi9zbPH7rffjKdtv3hxBk+s0lxMJ8vl3mauXZdA6ue4w TxmqGio6ejKkdvD7LrnrMhRtzz5sdn+rUS91t4uJaCDBrNmEap8Xd6M1DeY37X6wRsmI +RBKHN29a2MvUL2LEGY/wo4iUDUeAhFR/YeifabjxwfW8oar1n96H04ga4BlAvGB2XRG vc3r+Uvqs+LhKicA14iKtVOE12dhfCgdb0sYS7ttAiI4kDed5G4tjaJWOVXngBOT+4s9 sc/Od6sPdnaoNi+JIeORS0U8s5qkJzWiBvPAE3fjw+lZEH2KNgfq8A8E2j7C3K1pnZhL 81vA== X-Gm-Message-State: AOAM533jAJHJ71hKp1nEv/ofQ6NAJDQ8kiBglKsz5UHTLeO8fbmE/EAM wVAk6h80ohFIEGh/px0O4HZxcR1yLOzGvA== X-Received: by 2002:a17:90b:1a88:b0:1dc:e4a1:a81e with SMTP id ng8-20020a17090b1a8800b001dce4a1a81emr13504549pjb.96.1653744337313; Sat, 28 May 2022 06:25:37 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id n15-20020a170902968f00b001617ffc6d25sm3993106plp.19.2022.05.28.06.25.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:36 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: In-Reply-To: References: Date: Sat, 28 May 2022 13:25:11 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 11/23] avfilter/avfilter: Fix hardcoded input index X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: f0dsTSk1BBiO From: softworkz This fix targets (rare) cases where multiple input pads have a .filter_frame function. ff_request_frame_to_filter needs to call ff_request_frame with the correct input pad instead of the hardcoded first one. Signed-off-by: softworkz --- libavfilter/avfilter.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index 30745f41cb..6cbfc54a85 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -443,7 +443,7 @@ static int64_t guess_status_pts(AVFilterContext *ctx, int status, AVRational lin return AV_NOPTS_VALUE; } -static int ff_request_frame_to_filter(AVFilterLink *link) +static int ff_request_frame_to_filter(AVFilterLink *link, int input_index) { int ret = -1; @@ -452,8 +452,8 @@ static int ff_request_frame_to_filter(AVFilterLink *link) link->frame_blocked_in = 1; if (link->srcpad->request_frame) ret = link->srcpad->request_frame(link); - else if (link->src->inputs[0]) - ret = ff_request_frame(link->src->inputs[0]); + else if (link->src->inputs[input_index]) + ret = ff_request_frame(link->src->inputs[input_index]); if (ret < 0) { if (ret != AVERROR(EAGAIN) && ret != link->status_in) ff_avfilter_link_set_in_status(link, ret, guess_status_pts(link->src, ret, link->time_base)); @@ -1153,6 +1153,14 @@ static int forward_status_change(AVFilterContext *filter, AVFilterLink *in) { unsigned out = 0, progress = 0; int ret; + int input_index = 0; + + for (int i = 0; i < in->dst->nb_inputs; i++) { + if (&in->dst->input_pads[i] == in->dstpad) { + input_index = i; + break; + } + } av_assert0(!in->status_out); if (!filter->nb_outputs) { @@ -1162,7 +1170,7 @@ static int forward_status_change(AVFilterContext *filter, AVFilterLink *in) while (!in->status_out) { if (!filter->outputs[out]->status_in) { progress++; - ret = ff_request_frame_to_filter(filter->outputs[out]); + ret = ff_request_frame_to_filter(filter->outputs[out], input_index); if (ret < 0) return ret; } @@ -1199,7 +1207,7 @@ static int ff_filter_activate_default(AVFilterContext *filter) for (i = 0; i < filter->nb_outputs; i++) { if (filter->outputs[i]->frame_wanted_out && !filter->outputs[i]->frame_blocked_in) { - return ff_request_frame_to_filter(filter->outputs[i]); + return ff_request_frame_to_filter(filter->outputs[i], 0); } } return FFERROR_NOT_READY; From patchwork Sat May 28 13:25:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35966 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366441pzj; Sat, 28 May 2022 06:27:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzEvqflq40mMTvoAcm5fXoFu4TIXu6D3uTPyHL4msIKPv1/XrUKnAiXCQwfseX7eM5v/U+7 X-Received: by 2002:aa7:df0c:0:b0:42b:dc11:3f7c with SMTP id c12-20020aa7df0c000000b0042bdc113f7cmr12694759edy.283.1653744449657; Sat, 28 May 2022 06:27:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744449; cv=none; d=google.com; s=arc-20160816; b=kVfoF3Dtxab8OWDIZsMdJ5aA+YpariLgw1p9oaUe6IiutbGDuCBbETWF0J52Ofa6bT AUoe+tjqTF+je+nmRx3HSryLhHSZXTGjezuCMsuo2qkTJpZ8ike/zlHBE9PF4VzDi8u5 vfhkvnuHl3FkkC7DWPNIesjGX7Uw9vlo26oThzT2DCLhbQrR0UXB/usA1iSsfI9KClMv mCnABOdlHsZzHZ+tYX8J81yhd7zALjJFbl099hWgoFe88ZwDZ1/zuhxjh4UmAM7WztB8 II0g54A9HQklriYBGTsMIwKLEraam+oIVjKpFjUqRzGEalKpGRoYmFSDYNQyojg9TrSB UMAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=I9tHvesGg0ctUo0WFd4lTuwdue7VCM7wsWHwA5UfIKA=; b=ywJju0zYTriNMYclpe8quF51b0mBhwvULXVDYH4IxN9m14aX3T09scj/Dx8QxMtcb2 GxHXrO5S1mCleDaw/eATXLe4Ia2U2h/MuLyXhpvFxEVhC/BpTa4FLvvG+211yk7KI/t/ Lip6z8fMS+iSAco3VKV3R+/sY+bYqaAust803mdujFkkay5j+0hUR3Wexz82n11NgVed XEF3Qu0SJQ5seReOnfnHq3GNWnMYB3mlNcKY2RI6Jb6WN1TNnS9J5OL/3E8Yxc7cCTsN CrhDqcfpLy9WLwVvYlotiOyhgondKFAlO5i4A0+GCjDqkISxxn87+ubDGRmTXLrMvzAc SRnA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=P0lPUJDU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cy25-20020a0564021c9900b0042dc645b651si518615edb.166.2022.05.28.06.27.28; Sat, 28 May 2022 06:27:29 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=P0lPUJDU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E051A68B5F6; Sat, 28 May 2022 16:25:45 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1ED6C68B609 for ; Sat, 28 May 2022 16:25:39 +0300 (EEST) Received: by mail-pg1-f169.google.com with SMTP id j21so6198394pga.13 for ; Sat, 28 May 2022 06:25:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7+2NOdpXS5f47BJOvy49xBFHt4Y/tu4/n0HMu0J0+tI=; b=P0lPUJDUuBa8cLVg9ydcInz2VKUPX49IeBCn8rS4nA72jok8zGYVNlYj+SX/LtSnKF j/atX6q2wYZwLV/7qdwvcu1YdDK3cq4l38crN6Vfv8Nsj5jOEi71LfKJtR6e+8R2pvUT uB5EF2sF2gch5GWplYXLYVwF7W8wBi/Nm/7+q/3DYJcIVDD+UWpgk9HiqpOTwMcG9h7H iWA148dmcjizUySrsesCUOjyaLu3W78z54l5EOEOE17G13nzK2S6SgE+su2t1Otqsdkl I7xmtE7xEu809JQt3xOrVCXh4tz+w8oFDr7rUNUEpMpGM9vCYjTJtVCwbcfVGIorr5DN xyXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7+2NOdpXS5f47BJOvy49xBFHt4Y/tu4/n0HMu0J0+tI=; b=tPw6M9hylrJd5lT9ncWmb/xLs8JnUVBs+KkFs7TAP1XCQdB8aOjhEY5NBRq1YLlLeQ bOqTBRKyB8bsqIeJJYwD/huYr+ni3p8F4fcrTgSU55oKZFoVM4+8vLFwvh++JKE9jzYQ k8JumPBOuOLdnJH4REWO1Bdhlt/pXQXCInLsesXu8mCtr9xXBk70zvUZSJBzhukWccdF bAT8H58J60GD394y4vtzOPSsi88KkTDSxUV9lfJwF1ytuZUYWc2DKeyi5dBCYRwkir5G +WA8A74CJ9t0OF4WKFF4SFEqb9P6Jt6QrCncrYb4DuZ4lHl1onhG0mDm/tUkgojh0uf3 +8cw== X-Gm-Message-State: AOAM530jeBM545lR11eO+ddhOtgyLyM5TOLZzz12P9vvdIfPWvR/+JtQ ukHEKUvPF2RkoVbI9LhzQKujhNotUTVAng== X-Received: by 2002:a63:3194:0:b0:3f6:a28:77cf with SMTP id x142-20020a633194000000b003f60a2877cfmr40260349pgx.264.1653744338287; Sat, 28 May 2022 06:25:38 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id u71-20020a63854a000000b003c14af505f6sm2952146pgd.14.2022.05.28.06.25.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:37 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: In-Reply-To: References: Date: Sat, 28 May 2022 13:25:12 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 12/23] avfilter/sbuffer: Add sbuffersrc and sbuffersink filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: VXp4kxyKUKvn From: softworkz Signed-off-by: softworkz --- configure | 2 +- libavfilter/allfilters.c | 2 ++ libavfilter/buffersink.c | 54 ++++++++++++++++++++++++++++++ libavfilter/buffersink.h | 7 ++++ libavfilter/buffersrc.c | 72 ++++++++++++++++++++++++++++++++++++++++ libavfilter/buffersrc.h | 1 + 6 files changed, 137 insertions(+), 1 deletion(-) diff --git a/configure b/configure index 5a167613a4..c26bcdebc6 100755 --- a/configure +++ b/configure @@ -7879,7 +7879,7 @@ print_enabled_components(){ fi done if [ "$name" = "filter_list" ]; then - for c in asrc_abuffer vsrc_buffer asink_abuffer vsink_buffer; do + for c in asrc_abuffer vsrc_buffer ssrc_sbuffer asink_abuffer vsink_buffer ssink_sbuffer; do printf " &ff_%s,\n" $c >> $TMPH done fi diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 2409964e53..1fe6132e0c 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -569,8 +569,10 @@ extern const AVFilter ff_avsrc_movie; * being the same while having different 'types'). */ extern const AVFilter ff_asrc_abuffer; extern const AVFilter ff_vsrc_buffer; +extern const AVFilter ff_ssrc_sbuffer; extern const AVFilter ff_asink_abuffer; extern const AVFilter ff_vsink_buffer; +extern const AVFilter ff_ssink_sbuffer; extern const AVFilter ff_af_afifo; extern const AVFilter ff_vf_fifo; diff --git a/libavfilter/buffersink.c b/libavfilter/buffersink.c index e269cf72d1..204e9bad6c 100644 --- a/libavfilter/buffersink.c +++ b/libavfilter/buffersink.c @@ -30,6 +30,8 @@ #include "libavutil/internal.h" #include "libavutil/opt.h" +#include "libavcodec/avcodec.h" + #define FF_INTERNAL_FIELDS 1 #include "framequeue.h" @@ -61,6 +63,10 @@ typedef struct BufferSinkContext { int *sample_rates; ///< list of accepted sample rates int sample_rates_size; + /* only used for subtitles */ + enum AVSubtitleType *subtitle_types; ///< list of accepted subtitle types, must be terminated with -1 + int subtitle_types_size; + AVFrame *peeked_frame; } BufferSinkContext; @@ -372,6 +378,28 @@ static int asink_query_formats(AVFilterContext *ctx) return 0; } +static int ssink_query_formats(AVFilterContext *ctx) +{ + BufferSinkContext *buf = ctx->priv; + AVFilterFormats *formats = NULL; + unsigned i; + int ret; + + CHECK_LIST_SIZE(subtitle_types) + if (buf->subtitle_types_size) { + for (i = 0; i < NB_ITEMS(buf->subtitle_types); i++) + if ((ret = ff_add_format(&formats, buf->subtitle_types[i])) < 0) + return ret; + if ((ret = ff_set_common_formats(ctx, formats)) < 0) + return ret; + } else { + if ((ret = ff_default_query_formats(ctx)) < 0) + return ret; + } + + return 0; +} + #define OFFSET(x) offsetof(BufferSinkContext, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM static const AVOption buffersink_options[] = { @@ -395,9 +423,16 @@ static const AVOption abuffersink_options[] = { { NULL }, }; #undef FLAGS +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_SUBTITLE_PARAM +static const AVOption sbuffersink_options[] = { + { "subtitle_types", "set the supported subtitle formats", OFFSET(subtitle_types), AV_OPT_TYPE_BINARY, .flags = FLAGS }, + { NULL }, +}; +#undef FLAGS AVFILTER_DEFINE_CLASS(buffersink); AVFILTER_DEFINE_CLASS(abuffersink); +AVFILTER_DEFINE_CLASS(sbuffersink); static const AVFilterPad avfilter_vsink_buffer_inputs[] = { { @@ -436,3 +471,22 @@ const AVFilter ff_asink_abuffer = { .outputs = NULL, FILTER_QUERY_FUNC(asink_query_formats), }; + +static const AVFilterPad avfilter_ssink_sbuffer_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + }, +}; + +const AVFilter ff_ssink_sbuffer = { + .name = "sbuffersink", + .description = NULL_IF_CONFIG_SMALL("Buffer subtitle frames, and make them available to the end of the filter graph."), + .priv_class = &sbuffersink_class, + .priv_size = sizeof(BufferSinkContext), + .init = common_init, + .activate = activate, + FILTER_INPUTS(avfilter_ssink_sbuffer_inputs), + .outputs = NULL, + FILTER_QUERY_FUNC(ssink_query_formats), +}; diff --git a/libavfilter/buffersink.h b/libavfilter/buffersink.h index 01e7c747d8..42c164e670 100644 --- a/libavfilter/buffersink.h +++ b/libavfilter/buffersink.h @@ -128,6 +128,13 @@ typedef struct AVABufferSinkParams { */ attribute_deprecated AVABufferSinkParams *av_abuffersink_params_alloc(void); + +/** + * Deprecated and unused struct to use for initializing an sbuffersink context. + */ +typedef struct AVSBufferSinkParams { + const int *subtitle_type; +} AVSBufferSinkParams; #endif /** diff --git a/libavfilter/buffersrc.c b/libavfilter/buffersrc.c index a3190468bb..1679c54f17 100644 --- a/libavfilter/buffersrc.c +++ b/libavfilter/buffersrc.c @@ -39,6 +39,7 @@ #include "formats.h" #include "internal.h" #include "video.h" +#include "libavcodec/avcodec.h" typedef struct BufferSourceContext { const AVClass *class; @@ -63,6 +64,9 @@ typedef struct BufferSourceContext { char *channel_layout_str; AVChannelLayout ch_layout; + /* subtitle only */ + enum AVSubtitleType subtitle_type; + int eof; } BufferSourceContext; @@ -143,6 +147,13 @@ FF_ENABLE_DEPRECATION_WARNINGS return ret; } break; + case AVMEDIA_TYPE_SUBTITLE: + s->subtitle_type = param->format; + if (param->width > 0) + s->w = param->width; + if (param->height > 0) + s->h = param->height; + break; default: return AVERROR_BUG; } @@ -224,6 +235,8 @@ FF_ENABLE_DEPRECATION_WARNINGS CHECK_AUDIO_PARAM_CHANGE(ctx, s, frame->sample_rate, frame->ch_layout, frame->format, frame->pts); break; + case AVMEDIA_TYPE_SUBTITLE: + break; default: return AVERROR(EINVAL); } @@ -296,6 +309,7 @@ unsigned av_buffersrc_get_nb_failed_requests(AVFilterContext *buffer_src) #define OFFSET(x) offsetof(BufferSourceContext, x) #define A AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_AUDIO_PARAM #define V AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM +#define S AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_SUBTITLE_PARAM static const AVOption buffer_options[] = { { "width", NULL, OFFSET(w), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, V }, @@ -325,6 +339,16 @@ static const AVOption abuffer_options[] = { AVFILTER_DEFINE_CLASS(abuffer); +static const AVOption sbuffer_options[] = { + { "time_base", NULL, OFFSET(time_base), AV_OPT_TYPE_RATIONAL, { .dbl = 0 }, 0, INT_MAX, S }, + { "subtitle_type", NULL, OFFSET(subtitle_type), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, S }, + { "width", NULL, OFFSET(w), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, V }, + { "height", NULL, OFFSET(h), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, V }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(sbuffer); + static av_cold int init_audio(AVFilterContext *ctx) { BufferSourceContext *s = ctx->priv; @@ -393,6 +417,21 @@ FF_ENABLE_DEPRECATION_WARNINGS return ret; } +static av_cold int init_subtitle(AVFilterContext *ctx) +{ + BufferSourceContext *c = ctx->priv; + + if (c->subtitle_type == AV_SUBTITLE_FMT_BITMAP) + av_log(ctx, AV_LOG_VERBOSE, "graphical subtitles - w:%d h:%d tb:%d/%d\n", + c->w, c->h, c->time_base.num, c->time_base.den); + else + av_log(ctx, AV_LOG_VERBOSE, "text subtitles - w:%d h:%d tb:%d/%d\n", + c->w, c->h, c->time_base.num, c->time_base.den); + + return 0; +} + + static av_cold void uninit(AVFilterContext *ctx) { BufferSourceContext *s = ctx->priv; @@ -426,6 +465,11 @@ static int query_formats(AVFilterContext *ctx) if ((ret = ff_set_common_channel_layouts(ctx, channel_layouts)) < 0) return ret; break; + case AVMEDIA_TYPE_SUBTITLE: + if ((ret = ff_add_format (&formats, c->subtitle_type)) < 0 || + (ret = ff_set_common_formats (ctx , formats )) < 0) + return ret; + break; default: return AVERROR(EINVAL); } @@ -456,6 +500,11 @@ static int config_props(AVFilterLink *link) return ret; } break; + case AVMEDIA_TYPE_SUBTITLE: + link->format = c->subtitle_type; + link->w = c->w; + link->h = c->h; + break; default: return AVERROR(EINVAL); } @@ -520,3 +569,26 @@ const AVFilter ff_asrc_abuffer = { FILTER_QUERY_FUNC(query_formats), .priv_class = &abuffer_class, }; + +static const AVFilterPad avfilter_ssrc_sbuffer_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .request_frame = request_frame, + .config_props = config_props, + }, +}; + +const AVFilter ff_ssrc_sbuffer = { + .name = "sbuffer", + .description = NULL_IF_CONFIG_SMALL("Buffer subtitle frames, and make them accessible to the filterchain."), + .priv_size = sizeof(BufferSourceContext), + + .init = init_subtitle, + .uninit = uninit, + + .inputs = NULL, + FILTER_OUTPUTS(avfilter_ssrc_sbuffer_outputs), + FILTER_QUERY_FUNC(query_formats), + .priv_class = &sbuffer_class, +}; diff --git a/libavfilter/buffersrc.h b/libavfilter/buffersrc.h index 3b248b37cd..45a657bbb6 100644 --- a/libavfilter/buffersrc.h +++ b/libavfilter/buffersrc.h @@ -74,6 +74,7 @@ typedef struct AVBufferSrcParameters { /** * video: the pixel format, value corresponds to enum AVPixelFormat * audio: the sample format, value corresponds to enum AVSampleFormat + * subtitles: the subtitle format, value corresponds to enum AVSubtitleType */ int format; /** From patchwork Sat May 28 13:25:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35968 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366555pzj; Sat, 28 May 2022 06:27:48 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzHCzf9k72Z7p/4TEI3jiUxNyRAbCF5FE1m7JJgxtXjSV1xOKDAT3kWUFr4KPLhm7AITCvg X-Received: by 2002:a17:907:1b24:b0:6ff:235c:2ffd with SMTP id mp36-20020a1709071b2400b006ff235c2ffdmr11806727ejc.116.1653744468246; Sat, 28 May 2022 06:27:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744468; cv=none; d=google.com; s=arc-20160816; b=lXTNa2szf1VHC3ytKThdHbpFCbh1eR6AT3ahc+jOLVoCd+LqkXS25kSCR/X1ZnKscW dTP5yZRJPm3AS3NJs7FsQvhYK6wwbApaW14Z2ok5WaLLod+itsgo3u4k8yaywa4AjVXu Y5xKWEdhL25EjHBSux6XcJCI3V6d09eIrdRpFzwRu9kz+umJq6UbqoJ8dZ3AlJyk3tMN ZGGQtKHpYG4sPa4r6jvXocwfxtv/0+p3QcIWp4wVE5G22Pw0Ftit/OhGGNBxZytaAWsY Xz/ngj2HX7Z+zNzy+GRugAOLQBjheLqyniTsKTZYPjNn5gfx2oQrKPPjjICKKHfiOPfA g4yA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=VmMoZ6k0FVmH8wD+bXz586AL8E+mtt9JUe1hjmyONgg=; b=IZo+MrjpboNLV790JvbEDabM6y/d99iLACnHRVQJX/ddQvf9yvHHDNZAaxQLZMPkl+ 4XnRdcLZ0fRFycXQFqvRe1+L0ax4JzJFImXAhzAS+VKB6tXG02KZun5++ZNyJuxnbkbK UzPu81MwlDTFVpJkdTqajF4/B42lfORrVwEb54S56n/2C5zyzYSHcce+LF37kM7c5PMR 78/gA1BETy7aRvu9niHqf7Rlnoq1uw2gbV2z+xjI9yUV/zWBYnJesoH5AMf9CTE3XdyH UVcr0U1miCS4kNCQdgLqEhUJVCsQDHSme5DY5pJSg9QZbfitKRmXitiy34u7BoWXeCpO PplQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=qrwprOxq; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ss2-20020a170907c00200b006f4cbc8d876si5286488ejc.568.2022.05.28.06.27.47; Sat, 28 May 2022 06:27:48 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=qrwprOxq; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DACC568B613; Sat, 28 May 2022 16:25:47 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CA42468B615 for ; Sat, 28 May 2022 16:25:41 +0300 (EEST) Received: by mail-pl1-f178.google.com with SMTP id c2so6465881plh.2 for ; Sat, 28 May 2022 06:25:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MdjRE04qXjd1s9sbfgBoWbnZ8285O9AkAIgrF4gmjKM=; b=qrwprOxq2ExRJe77mZQB+P72hdoVilkWOv31EO/f02v2GfT9dwWuTT288GkEoTcs0X QNyL/17HumLpQ9BNp6v8THwDwOkZmtTankU1TSsxqsCbXwnED1F7feSoZJhxNsfmPqdl 7l4LOKehL5BokT0RRg5PspcCjehTL10uZsevXAUTR2VXHjwgA2iCZkprIO+cHtoexkn/ s4Ji7pc7q9CJFwMfLesmMUHRzywVCg9EqCpSBSowYGch/n9fuvfADLNObs+C7zLr8K5F su/Bqn4VU7hchyXfpv69tSbCUcOelyWI1K7u+H0hxwXWGhrkp+YJeAtqBpp77r1Ha78x cCFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MdjRE04qXjd1s9sbfgBoWbnZ8285O9AkAIgrF4gmjKM=; b=MqnztEDTEslz4Weg7Jg4bp2dNWyHqxF/XaFU22qBd/iuo+iExJKkxX8voXaWPLKK8c 1iwRvOAI4Az8rb6AL1iKkxd4nKSn3Nms2ie7eSP7oZcfhECYZfRNI04RtMdLoCK0zfe9 /pHnd1w77w4FzMlkwXM2dreSQVNAmzVZSYKorUH+zHz8FMXYFUZ6NaML2wphxZRoMfnv AtF0gNjlOyW3LsIuzrGejb33W7bqIL0Y/VWmciKGLXutk0sHPqq/3311EJG2usEUd2Iy B6nfAg+W+BErLsrFLrwGzJkcFzd/E92G19Y9wIkd0fM9oGLznBLJPSRfk8VTe6VLsJFS Gefg== X-Gm-Message-State: AOAM533NqoXmG+LfS5vz+V1Ry11LLkE94pGGrYROyG/i+5vdkHAT0byy LsKQjVvxx4TXJRsoMTLXkxESN8OevvlByA== X-Received: by 2002:a17:902:a507:b0:162:2a77:7363 with SMTP id s7-20020a170902a50700b001622a777363mr28774535plq.115.1653744339248; Sat, 28 May 2022 06:25:39 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id e27-20020a056a0000db00b0051810d460adsm5435519pfj.114.2022.05.28.06.25.38 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:38 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: In-Reply-To: References: Date: Sat, 28 May 2022 13:25:13 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 13/23] avfilter/overlaygraphicsubs: Add overlaygraphicsubs and graphicsub2video filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: O8C7QLnlBUpy From: softworkz - overlaygraphicsubs (VS -> V) Overlay graphic subtitles onto a video stream - graphicsub2video {S -> V) Converts graphic subtitles to video frames (with alpha) Gets auto-inserted for retaining compatibility with sub2video command lines Signed-off-by: softworkz --- doc/filters.texi | 118 +++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 2 + libavfilter/vf_overlaygraphicsubs.c | 765 ++++++++++++++++++++++++++++ 4 files changed, 887 insertions(+) create mode 100644 libavfilter/vf_overlaygraphicsubs.c diff --git a/doc/filters.texi b/doc/filters.texi index 0e10946cca..875c4a9f94 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -26879,6 +26879,124 @@ tools. @c man end VIDEO SINKS +@chapter Subtitle Filters +@c man begin SUBTITLE FILTERS + +When you configure your FFmpeg build, you can disable any of the +existing filters using @code{--disable-filters}. + +Below is a description of the currently available subtitle filters. + +@section graphicsub2video + +Renders graphic subtitles as video frames. + +This filter replaces the previous "sub2video" hack which did the conversion implicitly and up-front as subtitle filtering wasn't possible at that time. +To retain compatibility with earlier sub2video command lines, this filter is being auto-inserted in those cases. + +For overlaying graphicsal subtitles it is recommended to use the 'overlaygraphicsubs' filter which is more efficient and takes less processing resources. + +This filter is still useful in cases where the overlay is done with hardware acceleration (e.g. overlay_qsv, overlay_vaapi, overlay_cuda) for preparing the overlay frames. + +Inputs: +@itemize +@item 0: Subtitles [BITMAP] +@end itemize + +Outputs: +@itemize +@item 0: Video [RGB32] +@end itemize + + +It accepts the following parameters: + +@table @option +@item size, s +Set the size of the output video frame. + +@end table + +@subsection Examples + +@itemize +@item +Overlay PGS subtitles +(not recommended - better use overlaygraphicsubs) +@example +ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:1]graphicsub2video[subs];[0:0][subs]overlay" output.mp4 +@end example + +@item +Overlay PGS subtitles implicitly +The graphicsub2video is inserted automatically for compatibility with legacy command lines. +@example +ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:0][0:1]overlay" output.mp4 +@end example +@end itemize + +@section overlaygraphicsubs + +Overlay graphic subtitles onto a video stream. + +This filter can blend graphical subtitles on a video stream directly, i.e. without creating full-size alpha images first. +The blending operation is limited to the area of the subtitle rectangles, which also means that no processing is done at times where no subtitles are to be displayed. + +Inputs: +@itemize +@item 0: Video [YUV420P, YUV422P, YUV444P, ARGB, RGBA, ABGR, BGRA, RGB24, BGR24] +@item 1: Subtitles [BITMAP] +@end itemize + +Outputs: +@itemize +@item 0: Video (same as input) +@end itemize + +It accepts the following parameters: + +@table @option +@item x +@item y +Set the expression for the x and y coordinates of the overlaid video +on the main video. Default value is "0" for both expressions. In case +the expression is invalid, it is set to a huge value (meaning that the +overlay will not be displayed within the output visible area). + +@item eof_action +See @ref{framesync}. + +@item eval +Set when the expressions for @option{x}, and @option{y} are evaluated. + +It accepts the following values: +@table @samp +@item init +only evaluate expressions once during the filter initialization or +when a command is processed + +@item frame +evaluate expressions for each incoming frame +@end table + +Default value is @samp{frame}. + +@item shortest +See @ref{framesync}. + +@end table + +@subsection Examples + +@itemize +@item +Overlay PGS subtitles +@example +ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:0][0:1]overlaygraphicsubs" output.mp4 +@end example +@end itemize +@c man end SUBTITLE FILTERS + @chapter Multimedia Filters @c man begin MULTIMEDIA FILTERS diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 842068e902..9114316f79 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -306,6 +306,7 @@ OBJS-$(CONFIG_GBLUR_FILTER) += vf_gblur.o OBJS-$(CONFIG_GBLUR_VULKAN_FILTER) += vf_gblur_vulkan.o vulkan.o vulkan_filter.o OBJS-$(CONFIG_GEQ_FILTER) += vf_geq.o OBJS-$(CONFIG_GRADFUN_FILTER) += vf_gradfun.o +OBJS-$(CONFIG_GRAPHICSUB2VIDEO_FILTER) += vf_overlaygraphicsubs.o framesync.o OBJS-$(CONFIG_GRAPHMONITOR_FILTER) += f_graphmonitor.o OBJS-$(CONFIG_GRAYWORLD_FILTER) += vf_grayworld.o OBJS-$(CONFIG_GREYEDGE_FILTER) += vf_colorconstancy.o @@ -390,6 +391,7 @@ OBJS-$(CONFIG_OVERLAY_OPENCL_FILTER) += vf_overlay_opencl.o opencl.o \ OBJS-$(CONFIG_OVERLAY_QSV_FILTER) += vf_overlay_qsv.o framesync.o OBJS-$(CONFIG_OVERLAY_VAAPI_FILTER) += vf_overlay_vaapi.o framesync.o vaapi_vpp.o OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) += vf_overlay_vulkan.o vulkan.o vulkan_filter.o +OBJS-$(CONFIG_OVERLAYGRAPHICSUBS_FILTER) += vf_overlaygraphicsubs.o framesync.o OBJS-$(CONFIG_OWDENOISE_FILTER) += vf_owdenoise.o OBJS-$(CONFIG_PAD_FILTER) += vf_pad.o OBJS-$(CONFIG_PAD_OPENCL_FILTER) += vf_pad_opencl.o opencl.o opencl/pad.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 1fe6132e0c..4fb84ff881 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -369,6 +369,7 @@ extern const AVFilter ff_vf_overlay_qsv; extern const AVFilter ff_vf_overlay_vaapi; extern const AVFilter ff_vf_overlay_vulkan; extern const AVFilter ff_vf_overlay_cuda; +extern const AVFilter ff_vf_overlaygraphicsubs; extern const AVFilter ff_vf_owdenoise; extern const AVFilter ff_vf_pad; extern const AVFilter ff_vf_pad_opencl; @@ -557,6 +558,7 @@ extern const AVFilter ff_avf_showvolume; extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; +extern const AVFilter ff_svf_graphicsub2video; /* multimedia sources */ extern const AVFilter ff_avsrc_avsynctest; diff --git a/libavfilter/vf_overlaygraphicsubs.c b/libavfilter/vf_overlaygraphicsubs.c new file mode 100644 index 0000000000..7b26d8ef37 --- /dev/null +++ b/libavfilter/vf_overlaygraphicsubs.c @@ -0,0 +1,765 @@ +/* + * Copyright (c) 2021 softworkz (derived from vf_overlay) + * Copyright (c) 2010 Stefano Sabatini + * Copyright (c) 2010 Baptiste Coudurier + * Copyright (c) 2007 Bobby Bingham + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * overlay graphical subtitles on top of a video frame + */ + +#include "libavutil/eval.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "internal.h" +#include "drawutils.h" +#include "framesync.h" + +enum var_name { + VAR_MAIN_W, VAR_MW, + VAR_MAIN_H, VAR_MH, + VAR_OVERLAY_W, VAR_OW, + VAR_OVERLAY_H, VAR_OH, + VAR_HSUB, + VAR_VSUB, + VAR_X, + VAR_Y, + VAR_N, + VAR_POS, + VAR_T, + VAR_VARS_NB +}; + +typedef struct OverlaySubsContext { + const AVClass *class; + int x, y; ///< position of overlaid picture + int w, h; + AVFrame *outpicref; + + int main_is_packed_rgb; + uint8_t main_rgba_map[4]; + int main_has_alpha; + uint8_t overlay_rgba_map[4]; + int eval_mode; ///< EvalMode + int use_caching; + AVFrame *cache_frame; + + FFFrameSync fs; + + int main_pix_step[4]; ///< steps per pixel for each plane of the main output + int hsub, vsub; ///< chroma subsampling values + const AVPixFmtDescriptor *main_desc; ///< format descriptor for main input + + double var_values[VAR_VARS_NB]; + char *x_expr, *y_expr; + + AVExpr *x_pexpr, *y_pexpr; + + int pic_counter; +} OverlaySubsContext; + +static const char *const var_names[] = { + "main_w", "W", ///< width of the main video + "main_h", "H", ///< height of the main video + "overlay_w", "w", ///< width of the overlay video + "overlay_h", "h", ///< height of the overlay video + "hsub", + "vsub", + "x", + "y", + "n", ///< number of frame + "pos", ///< position in the file + "t", ///< timestamp expressed in seconds + NULL +}; + +#define MAIN 0 +#define OVERLAY 1 + +#define R 0 +#define G 1 +#define B 2 +#define A 3 + +#define Y 0 +#define U 1 +#define V 2 + +enum EvalMode { + EVAL_MODE_INIT, + EVAL_MODE_FRAME, + EVAL_MODE_NB +}; + +static av_cold void overlay_graphicsubs_uninit(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + av_frame_free(&s->cache_frame); + ff_framesync_uninit(&s->fs); + av_expr_free(s->x_pexpr); s->x_pexpr = NULL; + av_expr_free(s->y_pexpr); s->y_pexpr = NULL; +} + +static inline int normalize_xy(double d, int chroma_sub) +{ + if (isnan(d)) + return INT_MAX; + return (int)d & ~((1 << chroma_sub) - 1); +} + +static void eval_expr(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->var_values[VAR_Y] = av_expr_eval(s->y_pexpr, s->var_values, NULL); + /* It is necessary if x is expressed from y */ + s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->x = normalize_xy(s->var_values[VAR_X], s->hsub); + s->y = normalize_xy(s->var_values[VAR_Y], s->vsub); +} + +static int set_expr(AVExpr **pexpr, const char *expr, const char *option, void *log_ctx) +{ + int ret; + AVExpr *old = NULL; + + if (*pexpr) + old = *pexpr; + ret = av_expr_parse(pexpr, expr, var_names, + NULL, NULL, NULL, NULL, 0, log_ctx); + if (ret < 0) { + av_log(log_ctx, AV_LOG_ERROR, + "Error when evaluating the expression '%s' for %s\n", + expr, option); + *pexpr = old; + return ret; + } + + av_expr_free(old); + return 0; +} + +static int overlay_graphicsubs_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink0 = ctx->inputs[0]; + AVFilterLink *inlink1 = ctx->inputs[1]; + AVFilterLink *outlink = ctx->outputs[0]; + int ret; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_BITMAP, AV_SUBTITLE_FMT_NONE }; + static const enum AVPixelFormat supported_pix_fmts[] = { + AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV444P, + AV_PIX_FMT_ARGB, AV_PIX_FMT_RGBA, + AV_PIX_FMT_ABGR, AV_PIX_FMT_BGRA, + AV_PIX_FMT_RGB24, AV_PIX_FMT_BGR24, + AV_PIX_FMT_NONE + }; + + /* set input0 video formats */ + formats = ff_make_format_list(supported_pix_fmts); + if ((ret = ff_formats_ref(formats, &inlink0->outcfg.formats)) < 0) + return ret; + + /* set output0 video formats */ + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + /* set input1 subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink1->outcfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + OverlaySubsContext *s = ctx->priv; + int ret; + + if ((ret = ff_framesync_init_dualinput(&s->fs, ctx)) < 0) + return ret; + + outlink->w = ctx->inputs[MAIN]->w; + outlink->h = ctx->inputs[MAIN]->h; + outlink->time_base = ctx->inputs[MAIN]->time_base; + outlink->frame_rate = ctx->inputs[MAIN]->frame_rate; + + return ff_framesync_configure(&s->fs); +} + +// divide by 255 and round to nearest +// apply a fast variant: (X+127)/255 = ((X+127)*257+257)>>16 = ((X+128)*257)>>16 +#define FAST_DIV255(x) ((((x) + 128) * 257) >> 16) + +// calculate the non-pre-multiplied alpha, applying the general equation: +// alpha = alpha_overlay / ( (alpha_main + alpha_overlay) - (alpha_main * alpha_overlay) ) +// (((x) << 16) - ((x) << 9) + (x)) is a faster version of: 255 * 255 * x +// ((((x) + (y)) << 8) - ((x) + (y)) - (y) * (x)) is a faster version of: 255 * (x + y) +#define UNPREMULTIPLY_ALPHA(x, y) ((((x) << 16) - ((x) << 9) + (x)) / ((((x) + (y)) << 8) - ((x) + (y)) - (y) * (x))) + +/** + * Blend image in src to destination buffer dst at position (x, y). + */ +static av_always_inline void blend_packed_rgb(const AVFilterContext *ctx, + const AVFrame *dst, const AVSubtitleArea *src, + int x, int y, + int is_straight) +{ + OverlaySubsContext *s = ctx->priv; + int i, imax, j, jmax; + const int src_w = src->w; + const int src_h = src->h; + const int dst_w = dst->width; + const int dst_h = dst->height; + uint8_t alpha; ///< the amount of overlay to blend on to main + const int dr = s->main_rgba_map[R]; + const int dg = s->main_rgba_map[G]; + const int db = s->main_rgba_map[B]; + const int da = s->main_rgba_map[A]; + const int dstep = s->main_pix_step[0]; + const int sr = s->overlay_rgba_map[R]; + const int sg = s->overlay_rgba_map[G]; + const int sb = s->overlay_rgba_map[B]; + const int sa = s->overlay_rgba_map[A]; + int slice_start, slice_end; + uint8_t *S, *sp, *d, *dp; + + i = FFMAX(-y, 0); + imax = FFMIN3(-y + dst_h, FFMIN(src_h, dst_h), y + src_h); + + slice_start = i; + slice_end = i + imax; + + sp = src->buf[0]->data + slice_start * src->linesize[0]; + dp = dst->data[0] + (slice_start + y) * dst->linesize[0]; + + for (i = slice_start; i < slice_end; i++) { + j = FFMAX(-x, 0); + S = sp + j; + d = dp + ((x + j) * dstep); + + for (jmax = FFMIN(-x + dst_w, src_w); j < jmax; j++) { + uint32_t val = src->pal[*S]; + const uint8_t *sval = (uint8_t *)&val; + alpha = sval[sa]; + + // if the main channel has an alpha channel, alpha has to be calculated + // to create an un-premultiplied (straight) alpha value + if (s->main_has_alpha && alpha != 0 && alpha != 255) { + const uint8_t alpha_d = d[da]; + alpha = UNPREMULTIPLY_ALPHA(alpha, alpha_d); + } + + switch (alpha) { + case 0: + break; + case 255: + d[dr] = sval[sr]; + d[dg] = sval[sg]; + d[db] = sval[sb]; + break; + default: + // main_value = main_value * (1 - alpha) + overlay_value * alpha + // since alpha is in the range 0-255, the result must divided by 255 + d[dr] = is_straight ? FAST_DIV255(d[dr] * (255 - alpha) + sval[sr] * alpha) : + FFMIN(FAST_DIV255(d[dr] * (255 - alpha)) + sval[sr], 255); + d[dg] = is_straight ? FAST_DIV255(d[dg] * (255 - alpha) + sval[sg] * alpha) : + FFMIN(FAST_DIV255(d[dg] * (255 - alpha)) + sval[sg], 255); + d[db] = is_straight ? FAST_DIV255(d[db] * (255 - alpha) + sval[sb] * alpha) : + FFMIN(FAST_DIV255(d[db] * (255 - alpha)) + sval[sb], 255); + } + + if (s->main_has_alpha) { + switch (alpha) { + case 0: + break; + case 255: + d[da] = sval[sa]; + break; + default: + // apply alpha compositing: main_alpha += (1-main_alpha) * overlay_alpha + d[da] += FAST_DIV255((255 - d[da]) * S[sa]); + } + } + d += dstep; + S += 1; + } + dp += dst->linesize[0]; + sp += src->linesize[0]; + } +} + +static av_always_inline void blend_plane_8_8bits(const AVFilterContext *ctx, const AVFrame *dst, const AVSubtitleArea *area, + const uint32_t *yuv_pal, int src_w, int src_h, int dst_w, int dst_h, int plane, int hsub, int vsub, + int x, int y, int dst_plane, int dst_offset, int dst_step) +{ + const int src_wp = AV_CEIL_RSHIFT(src_w, hsub); + const int src_hp = AV_CEIL_RSHIFT(src_h, vsub); + const int dst_wp = AV_CEIL_RSHIFT(dst_w, hsub); + const int dst_hp = AV_CEIL_RSHIFT(dst_h, vsub); + const int yp = y >> vsub; + const int xp = x >> hsub; + uint8_t *s, *sp, *d, *dp, *dap; + int imax, i, j, jmax; + int slice_start, slice_end; + + i = FFMAX(-yp, 0); \ + imax = FFMIN3(-yp + dst_hp, FFMIN(src_hp, dst_hp), yp + src_hp); \ + + slice_start = i; + slice_end = i + imax; + + sp = area->buf[0]->data + (slice_start << vsub) * area->linesize[0]; + dp = dst->data[dst_plane] + (yp + slice_start) * dst->linesize[dst_plane] + dst_offset; + + dap = dst->data[3] + ((yp + slice_start) << vsub) * dst->linesize[3]; + + for (i = slice_start; i < slice_end; i++) { + j = FFMAX(-xp, 0); + d = dp + (xp + j) * dst_step; + s = sp + (j << hsub); + jmax = FFMIN(-xp + dst_wp, src_wp); + + for (; j < jmax; j++) { + uint32_t val = yuv_pal[*s]; + const uint8_t *sval = (uint8_t *)&val; + const int alpha = sval[3]; + const int max = 255, mid = 128; + const int d_int = *d; + const int sval_int = sval[plane]; + + switch (alpha) { + case 0: + break; + case 255: + *d = sval[plane]; + break; + default: + if (plane > 0) + *d = av_clip(FAST_DIV255((d_int - mid) * (max - alpha) + (sval_int - mid) * alpha) , -mid, mid) + mid; + else + *d = FAST_DIV255(d_int * (max - alpha) + sval_int * alpha); + break; + } + + d += dst_step; + s += 1 << hsub; + } + dp += dst->linesize[dst_plane]; + sp += (1 << vsub) * area->linesize[0]; + dap += (1 << vsub) * dst->linesize[3]; + } +} + +#define RGB2Y(r, g, b) (uint8_t)(((66 * (r) + 129 * (g) + 25 * (b) + 128) >> 8) + 16) +#define RGB2U(r, g, b) (uint8_t)(((-38 * (r) - 74 * (g) + 112 * (b) + 128) >> 8) + 128) +#define RGB2V(r, g, b) (uint8_t)(((112 * (r) - 94 * (g) - 18 * (b) + 128) >> 8) + 128) +/* Converts R8 G8 B8 color to YUV. */ +static av_always_inline void rgb_2_yuv(uint8_t r, uint8_t g, uint8_t b, uint8_t* y, uint8_t* u, uint8_t* v) +{ + *y = RGB2Y((int)r, (int)g, (int)b); + *u = RGB2U((int)r, (int)g, (int)b); + *v = RGB2V((int)r, (int)g, (int)b); +} + + +static av_always_inline void blend_yuv_8_8bits(AVFilterContext *ctx, AVFrame *dst, const AVSubtitleArea *area, int hsub, int vsub, int x, int y) +{ + OverlaySubsContext *s = ctx->priv; + const int src_w = area->w; + const int src_h = area->h; + const int dst_w = dst->width; + const int dst_h = dst->height; + const int sr = s->overlay_rgba_map[R]; + const int sg = s->overlay_rgba_map[G]; + const int sb = s->overlay_rgba_map[B]; + const int sa = s->overlay_rgba_map[A]; + uint32_t yuvpal[256]; + + for (int i = 0; i < 256; ++i) { + const uint8_t *rgba = (const uint8_t *)&area->pal[i]; + uint8_t *yuva = (uint8_t *)&yuvpal[i]; + rgb_2_yuv(rgba[sr], rgba[sg], rgba[sb], &yuva[Y], &yuva[U], &yuva[V]); + yuva[3] = rgba[sa]; + } + + blend_plane_8_8bits(ctx, dst, area, yuvpal, src_w, src_h, dst_w, dst_h, Y, 0, 0, x, y, s->main_desc->comp[Y].plane, s->main_desc->comp[Y].offset, s->main_desc->comp[Y].step); + blend_plane_8_8bits(ctx, dst, area, yuvpal, src_w, src_h, dst_w, dst_h, U, hsub, vsub, x, y, s->main_desc->comp[U].plane, s->main_desc->comp[U].offset, s->main_desc->comp[U].step); + blend_plane_8_8bits(ctx, dst, area, yuvpal, src_w, src_h, dst_w, dst_h, V, hsub, vsub, x, y, s->main_desc->comp[V].plane, s->main_desc->comp[V].offset, s->main_desc->comp[V].step); +} + +static int config_input_main(AVFilterLink *inlink) +{ + int ret; + AVFilterContext *ctx = inlink->dst; + OverlaySubsContext *s = inlink->dst->priv; + const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(inlink->format); + + av_image_fill_max_pixsteps(s->main_pix_step, NULL, pix_desc); + ff_fill_rgba_map(s->overlay_rgba_map, AV_PIX_FMT_RGB32); // it's actually AV_PIX_FMT_PAL8); + + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + + s->main_desc = pix_desc; + + s->main_is_packed_rgb = ff_fill_rgba_map(s->main_rgba_map, inlink->format) >= 0; + s->main_has_alpha = !!(pix_desc->flags & AV_PIX_FMT_FLAG_ALPHA); + + /* Finish the configuration by evaluating the expressions + now when both inputs are configured. */ + s->var_values[VAR_MAIN_W ] = s->var_values[VAR_MW] = ctx->inputs[MAIN ]->w; + s->var_values[VAR_MAIN_H ] = s->var_values[VAR_MH] = ctx->inputs[MAIN ]->h; + s->var_values[VAR_OVERLAY_W] = s->var_values[VAR_OW] = ctx->inputs[OVERLAY]->w; + s->var_values[VAR_OVERLAY_H] = s->var_values[VAR_OH] = ctx->inputs[OVERLAY]->h; + s->var_values[VAR_HSUB] = 1<log2_chroma_w; + s->var_values[VAR_VSUB] = 1<log2_chroma_h; + s->var_values[VAR_X] = NAN; + s->var_values[VAR_Y] = NAN; + s->var_values[VAR_N] = 0; + s->var_values[VAR_T] = NAN; + s->var_values[VAR_POS] = NAN; + + if ((ret = set_expr(&s->x_pexpr, s->x_expr, "x", ctx)) < 0 || + (ret = set_expr(&s->y_pexpr, s->y_expr, "y", ctx)) < 0) + return ret; + + if (s->eval_mode == EVAL_MODE_INIT) { + eval_expr(ctx); + av_log(ctx, AV_LOG_VERBOSE, "x:%f xi:%d y:%f yi:%d\n", + s->var_values[VAR_X], s->x, + s->var_values[VAR_Y], s->y); + } + + av_log(ctx, AV_LOG_VERBOSE, + "main w:%d h:%d fmt:%s overlay w:%d h:%d fmt:%s\n", + ctx->inputs[MAIN]->w, ctx->inputs[MAIN]->h, + av_get_pix_fmt_name(ctx->inputs[MAIN]->format), + ctx->inputs[OVERLAY]->w, ctx->inputs[OVERLAY]->h, + av_get_pix_fmt_name(ctx->inputs[OVERLAY]->format)); + return 0; +} + +static int do_blend(FFFrameSync *fs) +{ + AVFilterContext *ctx = fs->parent; + AVFrame *mainpic, *second; + OverlaySubsContext *s = ctx->priv; + AVFilterLink *inlink = ctx->inputs[0]; + unsigned i; + int ret; + + ret = ff_framesync_dualinput_get_writable(fs, &mainpic, &second); + if (ret < 0) + return ret; + if (!second) + return ff_filter_frame(ctx->outputs[0], mainpic); + + if (s->eval_mode == EVAL_MODE_FRAME) { + int64_t pos = mainpic->pkt_pos; + + s->var_values[VAR_N] = (double)inlink->frame_count_out; + s->var_values[VAR_T] = mainpic->pts == AV_NOPTS_VALUE ? + NAN :(double)mainpic->pts * av_q2d(inlink->time_base); + s->var_values[VAR_POS] = pos == -1 ? NAN : (double)pos; + + s->var_values[VAR_OVERLAY_W] = s->var_values[VAR_OW] = second->width; + s->var_values[VAR_OVERLAY_H] = s->var_values[VAR_OH] = second->height; + s->var_values[VAR_MAIN_W ] = s->var_values[VAR_MW] = mainpic->width; + s->var_values[VAR_MAIN_H ] = s->var_values[VAR_MH] = mainpic->height; + + eval_expr(ctx); + av_log(ctx, AV_LOG_DEBUG, "n:%f t:%f pos:%f x:%f xi:%d y:%f yi:%d\n", + s->var_values[VAR_N], s->var_values[VAR_T], s->var_values[VAR_POS], + s->var_values[VAR_X], s->x, + s->var_values[VAR_Y], s->y); + } + + for (i = 0; i < second->num_subtitle_areas; i++) { + const AVSubtitleArea *sub_area = second->subtitle_areas[i]; + + if (sub_area->type != AV_SUBTITLE_FMT_BITMAP) { + av_log(NULL, AV_LOG_WARNING, "overlay_graphicsubs: non-bitmap subtitle\n"); + return AVERROR_INVALIDDATA; + } + + switch (inlink->format) { + case AV_PIX_FMT_YUV420P: + blend_yuv_8_8bits(ctx, mainpic, sub_area, 1, 1, sub_area->x + s->x, sub_area->y + s->y); + break; + case AV_PIX_FMT_YUV422P: + blend_yuv_8_8bits(ctx, mainpic, sub_area, 1, 0, sub_area->x + s->x, sub_area->y + s->y); + break; + case AV_PIX_FMT_YUV444P: + blend_yuv_8_8bits(ctx, mainpic, sub_area, 0, 0, sub_area->x + s->x, sub_area->y + s->y); + break; + case AV_PIX_FMT_RGB24: + case AV_PIX_FMT_BGR24: + case AV_PIX_FMT_ARGB: + case AV_PIX_FMT_RGBA: + case AV_PIX_FMT_BGRA: + case AV_PIX_FMT_ABGR: + blend_packed_rgb(ctx, mainpic, sub_area, sub_area->x + s->x, sub_area->y + s->y, 1); + break; + default: + av_log(NULL, AV_LOG_ERROR, "Unsupported input pix fmt: %d\n", inlink->format); + return AVERROR(EINVAL); + } + } + + return ff_filter_frame(ctx->outputs[0], mainpic); +} + +static av_cold int overlay_graphicsubs_init(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + s->fs.on_event = do_blend; + return 0; +} + +static int overlay_graphicsubs_activate(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + return ff_framesync_activate(&s->fs); +} + +static int graphicsub2video_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_BITMAP, AV_SUBTITLE_FMT_NONE }; + static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_RGB32, AV_PIX_FMT_NONE }; + int ret; + + /* set input subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output video formats */ + formats = ff_make_format_list(pix_fmts); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int graphicsub2video_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + OverlaySubsContext *s = ctx->priv; + + if (s->w <= 0 || s->h <= 0) { + s->w = inlink->w; + s->h = inlink->h; + } + return 0; +} + +static int graphicsub2video_config_output(AVFilterLink *outlink) +{ + const AVFilterContext *ctx = outlink->src; + OverlaySubsContext *s = ctx->priv; + const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(outlink->format); + + outlink->w = s->w; + outlink->h = s->h; + + if (outlink->w == 0 && outlink->h == 0) { + outlink->w = 1; + outlink->h = 1; + } + outlink->sample_aspect_ratio = (AVRational){1,1}; + outlink->time_base = ctx->inputs[0]->time_base; + outlink->frame_rate = ctx->inputs[0]->frame_rate; + + av_image_fill_max_pixsteps(s->main_pix_step, NULL, pix_desc); + ff_fill_rgba_map(s->overlay_rgba_map, AV_PIX_FMT_RGB32); + + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + + s->main_desc = pix_desc; + + s->main_is_packed_rgb = ff_fill_rgba_map(s->main_rgba_map, outlink->format) >= 0; + s->main_has_alpha = !!(pix_desc->flags & AV_PIX_FMT_FLAG_ALPHA); + + return 0; +} + +static int graphicsub2video_filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterLink *outlink = inlink->dst->outputs[0]; + const AVFilterContext *ctx = outlink->src; + OverlaySubsContext *s = ctx->priv; + AVFrame *out; + const unsigned num_rects = frame->num_subtitle_areas; + unsigned int i; + int ret; + + if (s->use_caching && s->cache_frame && frame->repeat_sub + && s->cache_frame->subtitle_timing.start_pts == frame->subtitle_timing.start_pts) { + out = av_frame_clone(s->cache_frame); + if (!out) + return AVERROR(ENOMEM); + + ret = av_frame_copy_props(out, frame); + if (ret < 0) + return ret; + + av_log(inlink->dst, AV_LOG_DEBUG, "graphicsub2video CACHED - size %dx%d pts: %"PRId64" areas: %d\n", frame->width, frame->height, frame->subtitle_timing.start_pts, frame->num_subtitle_areas); + av_frame_free(&frame); + return ff_filter_frame(outlink, out); + } + + + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) { + av_frame_free(&frame); + return AVERROR(ENOMEM); + } + + memset(out->data[0], 0, (size_t)out->linesize[0] * out->height); + + out->pts = out->pkt_dts = out->best_effort_timestamp = frame->pts; + out->coded_picture_number = out->display_picture_number = s->pic_counter++; + + for (i = 0; i < num_rects; i++) { + const AVSubtitleArea *sub_rect = frame->subtitle_areas[i]; + + if (sub_rect->type != AV_SUBTITLE_FMT_BITMAP) { + av_log(NULL, AV_LOG_WARNING, "graphicsub2video: non-bitmap subtitle\n"); + av_frame_free(&frame); + return AVERROR_INVALIDDATA; + } + + blend_packed_rgb(inlink->dst, out, sub_rect, sub_rect->x, sub_rect->y, 1); + } + + av_log(inlink->dst, AV_LOG_DEBUG, "graphicsub2video output - size %dx%d pts: %"PRId64" areas: %d\n", out->width, out->height, out->pts, frame->num_subtitle_areas); + + if (s->use_caching) { + av_frame_free(&s->cache_frame); + s->cache_frame = av_frame_clone(out); + } + + av_frame_free(&frame); + return ff_filter_frame(outlink, out); +} + +#define OFFSET(x) offsetof(OverlaySubsContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption overlaygraphicsubs_options[] = { + { "x", "set the x expression", OFFSET(x_expr), AV_OPT_TYPE_STRING, {.str = "0"}, 0, 0, .flags = FLAGS }, + { "y", "set the y expression", OFFSET(y_expr), AV_OPT_TYPE_STRING, {.str = "0"}, 0, 0, .flags = FLAGS }, + { "eof_action", "Action to take when encountering EOF from secondary input ", + OFFSET(fs.opt_eof_action), AV_OPT_TYPE_INT, { .i64 = EOF_ACTION_REPEAT }, + EOF_ACTION_REPEAT, EOF_ACTION_PASS, .flags = FLAGS, "eof_action" }, + { "repeat", "Repeat the previous frame.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_REPEAT }, .flags = FLAGS, "eof_action" }, + { "endall", "End both streams.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_ENDALL }, .flags = FLAGS, "eof_action" }, + { "pass", "Pass through the main input.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_PASS }, .flags = FLAGS, "eof_action" }, + { "eval", "specify when to evaluate expressions", OFFSET(eval_mode), AV_OPT_TYPE_INT, {.i64 = EVAL_MODE_FRAME}, 0, EVAL_MODE_NB-1, FLAGS, "eval" }, + { "init", "eval expressions once during initialization", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_INIT}, .flags = FLAGS, .unit = "eval" }, + { "frame", "eval expressions per-frame", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_FRAME}, .flags = FLAGS, .unit = "eval" }, + { "shortest", "force termination when the shortest input terminates", OFFSET(fs.opt_shortest), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, .flags = FLAGS }, + { "repeatlast", "repeat overlay of the last overlay frame", OFFSET(fs.opt_repeatlast), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, .flags = FLAGS }, + { NULL } +}; + +static const AVOption graphicsub2video_options[] = { + { "size", "set output frame size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, .flags = FLAGS }, + { "s", "set output frame size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, .flags = FLAGS }, + { "use_caching", "Cache output frames", OFFSET(use_caching), AV_OPT_TYPE_BOOL, { .i64 = 1}, 0, 1, .flags = FLAGS }, + { NULL } +}; + +FRAMESYNC_DEFINE_CLASS(overlaygraphicsubs, OverlaySubsContext, fs); + +static const AVFilterPad overlaygraphicsubs_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input_main, + .flags = AVFILTERPAD_FLAG_NEEDS_WRITABLE, + }, + { + .name = "overlay", + .type = AVMEDIA_TYPE_SUBTITLE, + }, +}; + +static const AVFilterPad overlaygraphicsubs_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +const AVFilter ff_vf_overlaygraphicsubs = { + .name = "overlaygraphicsubs", + .description = NULL_IF_CONFIG_SMALL("Overlay graphical subtitles on top of the input."), + .preinit = overlaygraphicsubs_framesync_preinit, + .init = overlay_graphicsubs_init, + .uninit = overlay_graphicsubs_uninit, + .priv_size = sizeof(OverlaySubsContext), + .priv_class = &overlaygraphicsubs_class, + .activate = overlay_graphicsubs_activate, + FILTER_INPUTS(overlaygraphicsubs_inputs), + FILTER_OUTPUTS(overlaygraphicsubs_outputs), + FILTER_QUERY_FUNC(overlay_graphicsubs_query_formats), +}; + +AVFILTER_DEFINE_CLASS(graphicsub2video); + +static const AVFilterPad graphicsub2video_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = graphicsub2video_filter_frame, + .config_props = graphicsub2video_config_input, + }, +}; + +static const AVFilterPad graphicsub2video_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = graphicsub2video_config_output, + }, +}; + +const AVFilter ff_svf_graphicsub2video = { + .name = "graphicsub2video", + .description = NULL_IF_CONFIG_SMALL("Convert graphical subtitles to video"), + .priv_size = sizeof(OverlaySubsContext), + .priv_class = &graphicsub2video_class, + FILTER_INPUTS(graphicsub2video_inputs), + FILTER_OUTPUTS(graphicsub2video_outputs), + FILTER_QUERY_FUNC(graphicsub2video_query_formats), +}; From patchwork Sat May 28 13:25:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35969 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366618pzj; Sat, 28 May 2022 06:27:58 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw3oJBuzH/Fc9W8uj5mwQW36ifbXOeW+ZPaDDG+kdOS+4uI3GinmIe1Bls4t3JCxCuLnVI8 X-Received: by 2002:aa7:c4d0:0:b0:42a:ce47:e9c5 with SMTP id p16-20020aa7c4d0000000b0042ace47e9c5mr7519067edr.224.1653744478606; Sat, 28 May 2022 06:27:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744478; cv=none; d=google.com; s=arc-20160816; b=ZcuXYsEcXmiL/BKJr1CZG7/u2uPhhQJcln8UFxRLXQdLXKAa9KQFFw8IIq5QWt6t/D gKq871Jb3hx37crdqlUNZjUzMuvU7JQjsQUsZB+8Wo6fxfqdHD06fKfw+pXm9DlhAtdU fzQix12QxcnktGjzN/RP2hnYIshsP7BXTYDZhldau46MShBCK/8PuIpHLC3G487CLh6o PGCfypHo8sKe8nJJtFoVWnEnYV0urMs4WJB781kpp9x2wEaYEK+b/ELPyHA/qr6cV3D8 ZiQygzSJ2FHt6vX5RadJeXtmBl2XnvUTOxDoz4DhQjyiHiqaX3MM25heltaGXqrexw9d pEfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=EJqGG9P80ttECCE5A93Ey+m8wUQY+HKK5aA2KBjeJo8=; b=oNLP85cro3lijCFtmTz34qAg8YWD6HtIB2wHtxLgSaWyTfcxd61YVCCN8yOw4cEc6b eWoqzaczZX30ttWSFA3VOeiFeJYSk7IbS48asjEUVeAWzGQ0oiL659X2VwsXxPVfTcb4 4sIkunX54HwThpddoi2MMPBkGrmnxokdgUjNivJap/QQ8o83GwJF9zxT5stdiXG4L/Iv dSAlS0QmzCkzemaUW18UAxaNNSjzE9lYt9zY3c+qenSV27DR45UKRPveZbY6oi0qEYfH 0u2WX8lIw7NhY0WQduJUSRpUFoC9ZlkxfJJMU7zyTDqwBJSu5vW9yeF4mUgSo3Ku8+UH SnPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=kE6XSKk5; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id p27-20020a1709060e9b00b006ff1420d58dsi3659377ejf.136.2022.05.28.06.27.58; Sat, 28 May 2022 06:27:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=kE6XSKk5; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D87C668B61D; Sat, 28 May 2022 16:25:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6758568B5E5 for ; Sat, 28 May 2022 16:25:42 +0300 (EEST) Received: by mail-pl1-f169.google.com with SMTP id t4so3544460pld.4 for ; Sat, 28 May 2022 06:25:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=EaCb+wC5u1l8UdbpRhItlsZC+25MxJ8vqKdzMXHUnig=; b=kE6XSKk5q986p/nBi8+oowBoWFexYxTQI/O9NkWbujmZvcauFdlTsbMG/EhZdk/TTl BN4vtEAFTl9BtbSwQ/XGRtYSCTV/7bKApWS8V+oDL9LUCVcuj4RyoLiuZoCW3i57TcKp lMAc+NvHBVkxyxtcspoUpIBBan70qQDb5ccMtaCE32H2qDTinwPmJpLsVuAJ6PQ1r7E/ jP34MwpaUfrZJWa1TjxwJ2KdqYpoSaBjRLx4/BOBNWV/bjzaj0BozJ0FSVTzVg0TorY4 xoaTj27X9GMHyRfj5MZmNjlFUzi3aHTCcP9hGiYrV75GTmx/UZQwpCxwCxxvaXIPiIu/ RZSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=EaCb+wC5u1l8UdbpRhItlsZC+25MxJ8vqKdzMXHUnig=; b=QCSCpUJWFrPtuxgbpE/3ZITOEM3Y+5JEol3+wjSar6BG12eNW+JibGtfSenVMUevXQ KVjOoKMMmW9x9ygNs35Xg2LEFyNixX1YaeONcpoe8xHbgTqMa3QqEmkVnlGKG9+5w8KJ oFZ3uDXNch0sXRr0zNwl1yF+bVAdQwPnZplx997VV7sKwUjBLpaQAdyygPNOWOXX34eW HDTKmZLH3S7wrMq6VkbWh2Jvahe8hlj0xBNVs/aOlB5ZxyhWEngpWNEtudHgJBuhSM+d kStL/ngUv22kCvqWbZ1UMT44jT8NpmTfjfOat2Kiyidg/bAyGKp3Qhj7fCcdbBJEGEml 9Y0A== X-Gm-Message-State: AOAM531sphLeqpFwqcyy54HRGv2oD0vD2CbzsHyUlEDe07hXOQH+5/ej 9IeTmWjSE+xc3a9p54niTSFmU5yAsl+Hvg== X-Received: by 2002:a17:903:2341:b0:162:17b1:cebc with SMTP id c1-20020a170903234100b0016217b1cebcmr32412875plh.106.1653744340222; Sat, 28 May 2022 06:25:40 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id q3-20020a056a0002a300b0050dc7628167sm5358347pfs.65.2022.05.28.06.25.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:39 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <4c8092357f3c2ac84d10a38104dcead07b50be88.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:14 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 14/23] avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: gK/7HchCAE72 From: softworkz - overlaytextsubs {VS -> V) Overlay text subtitles onto a video stream. - textsubs2video {S -> V) Converts text subtitles to video frames Signed-off-by: softworkz --- configure | 2 + doc/filters.texi | 113 +++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 2 + libavfilter/vf_overlaytextsubs.c | 680 +++++++++++++++++++++++++++++++ 5 files changed, 799 insertions(+) create mode 100644 libavfilter/vf_overlaytextsubs.c diff --git a/configure b/configure index c26bcdebc6..39c4e94a1f 100755 --- a/configure +++ b/configure @@ -3691,6 +3691,7 @@ overlay_qsv_filter_deps="libmfx" overlay_qsv_filter_select="qsvvpp" overlay_vaapi_filter_deps="vaapi VAProcPipelineCaps_blend_flags" overlay_vulkan_filter_deps="vulkan spirv_compiler" +overlaytextsubs_filter_deps="avcodec libass" owdenoise_filter_deps="gpl" pad_opencl_filter_deps="opencl" pan_filter_deps="swresample" @@ -3729,6 +3730,7 @@ stereo3d_filter_deps="gpl" subtitles_filter_deps="avformat avcodec libass" super2xsai_filter_deps="gpl" pixfmts_super2xsai_test_deps="super2xsai_filter" +textsub2video_filter_deps="avcodec libass" tinterlace_filter_deps="gpl" tinterlace_merge_test_deps="tinterlace_filter" tinterlace_pad_test_deps="tinterlace_filter" diff --git a/doc/filters.texi b/doc/filters.texi index 875c4a9f94..70d4c90663 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -26995,6 +26995,119 @@ Overlay PGS subtitles ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:0][0:1]overlaygraphicsubs" output.mp4 @end example @end itemize + +@section overlaytextsubs + +Overlay text subtitles onto a video stream. + +This filter supersedes the classic @ref{subtitles} filter opposed to which it does no longer require to open and access the source stream separately, which is often causing problems or doesn't even work for non-local or slow sources. + +Inputs: +@itemize +@item 0: Video [YUV420P, YUV422P, YUV444P, ARGB, RGBA, ABGR, BGRA, RGB24, BGR24] +@item 1: Subtitles [TEXT] +@end itemize + +Outputs: +@itemize +@item 0: Video (same as input) +@end itemize + +It accepts the following parameters: + +@table @option + +@item alpha +Process alpha channel, by default alpha channel is untouched. + +@item fonts_dir +Set a directory path containing fonts that can be used by the filter. +These fonts will be used in addition to whatever the font provider uses. + +@item default_font_path +Path to a font file to be used as the default font. + +@item font_size +Set the default font size. + +@item fontconfig_file +Path to ASS fontconfig configuration file. + +@item force_style +Override default style or script info parameters of the subtitles. It accepts a +string containing ASS style format @code{KEY=VALUE} couples separated by ",". + +@item margin +Set the rendering margin in pixels. + +@item render_latest_only +For rendering, alway use the latest event only, which is covering the given point in time +@end table + +@subsection Examples + +@itemize +@item +Overlay ASS subtitles with animations: +@example +ffmpeg -i "http://streams.videolan.org/samples/sub/SSA/subtitle_testing_complex.mkv" -filter_complex "[0:v]overlaytextsubs" -map 0 -y out.mkv +@end example +@end itemize + +@section textsub2video + +Converts text subtitles to video frames. + +For overlaying text subtitles onto video frames it is recommended to use the overlaytextsubs filter. +The textsub2video is useful for for creating transparent text-frames when overlay is done via hw acceleration + +Inputs: +@itemize +@item 0: Subtitles [TEXT] +@end itemize + +Outputs: +@itemize +@item 0: Video [RGB32] +@end itemize + +It accepts the following parameters: + +@table @option + +@item rate, r +Set the framerate for updating overlay frames. +Normally, overlay frames will only be updated each time when the subtitles to display are changing. +In cases where subtitles include advanced features (like animation), this parameter determines the frequency by which the overlay frames should be updated. + +@item size, s +Set the output frame size. +Allows to override the size of output video frames. + +@item fonts_dir +Set a directory path containing fonts that can be used by the filter. +These fonts will be used in addition to whatever the font provider uses. + +@item default_font_path +Path to a font file to be used as the default font. + +@item font_size +Set the default font size. + +@item fontconfig_file +Path to ASS fontconfig configuration file. + +@item force_style +Override default style or script info parameters of the subtitles. It accepts a +string containing ASS style format @code{KEY=VALUE} couples separated by ",". + +@item margin +Set the rendering margin in pixels. + +@item render_latest_only +For rendering, alway use the latest event only, which is covering the given point in time. +@end table + @c man end SUBTITLE FILTERS @chapter Multimedia Filters diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 9114316f79..84ec2fa542 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -392,6 +392,7 @@ OBJS-$(CONFIG_OVERLAY_QSV_FILTER) += vf_overlay_qsv.o framesync.o OBJS-$(CONFIG_OVERLAY_VAAPI_FILTER) += vf_overlay_vaapi.o framesync.o vaapi_vpp.o OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) += vf_overlay_vulkan.o vulkan.o vulkan_filter.o OBJS-$(CONFIG_OVERLAYGRAPHICSUBS_FILTER) += vf_overlaygraphicsubs.o framesync.o +OBJS-$(CONFIG_OVERLAYTEXTSUBS_FILTER) += vf_overlaytextsubs.o OBJS-$(CONFIG_OWDENOISE_FILTER) += vf_owdenoise.o OBJS-$(CONFIG_PAD_FILTER) += vf_pad.o OBJS-$(CONFIG_PAD_OPENCL_FILTER) += vf_pad_opencl.o opencl.o opencl/pad.o @@ -483,6 +484,7 @@ OBJS-$(CONFIG_SWAPRECT_FILTER) += vf_swaprect.o OBJS-$(CONFIG_SWAPUV_FILTER) += vf_swapuv.o OBJS-$(CONFIG_TBLEND_FILTER) += vf_blend.o framesync.o OBJS-$(CONFIG_TELECINE_FILTER) += vf_telecine.o +OBJS-$(CONFIG_TEXTSUB2VIDEO_FILTER) += vf_overlaytextsubs.o OBJS-$(CONFIG_THISTOGRAM_FILTER) += vf_histogram.o OBJS-$(CONFIG_THRESHOLD_FILTER) += vf_threshold.o framesync.o OBJS-$(CONFIG_THUMBNAIL_FILTER) += vf_thumbnail.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 4fb84ff881..a6b95785be 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -370,6 +370,7 @@ extern const AVFilter ff_vf_overlay_vaapi; extern const AVFilter ff_vf_overlay_vulkan; extern const AVFilter ff_vf_overlay_cuda; extern const AVFilter ff_vf_overlaygraphicsubs; +extern const AVFilter ff_vf_overlaytextsubs; extern const AVFilter ff_vf_owdenoise; extern const AVFilter ff_vf_pad; extern const AVFilter ff_vf_pad_opencl; @@ -559,6 +560,7 @@ extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_svf_graphicsub2video; +extern const AVFilter ff_svf_textsub2video; /* multimedia sources */ extern const AVFilter ff_avsrc_avsynctest; diff --git a/libavfilter/vf_overlaytextsubs.c b/libavfilter/vf_overlaytextsubs.c new file mode 100644 index 0000000000..1e43289bec --- /dev/null +++ b/libavfilter/vf_overlaytextsubs.c @@ -0,0 +1,680 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * overlay text subtitles on top of a video frame + */ + +#include "config_components.h" + +#include +#include "libavutil/ass_internal.h" +#include "libavutil/thread.h" + +#include "drawutils.h" +#include "filters.h" + +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" + +typedef struct TextSubsContext { + const AVClass *class; + AVMutex mutex; + int is_mutex_initialized; + + ASS_Library *library; + ASS_Renderer *renderer; + ASS_Track *track; + + char *default_font_path; + char *fonts_dir; + char *fc_file; + double font_size; + char *force_style; + char *language; + int margin; + int render_latest_only; + + int alpha; + FFDrawContext draw; + + int got_header; + int out_w, out_h; + AVRational frame_rate; + AVFrame *last_frame; + int need_frame; + int eof; +} TextSubsContext; + +/* libass supports a log level ranging from 0 to 7 */ +static const int ass_libavfilter_log_level_map[] = { + AV_LOG_QUIET, /* 0 */ + AV_LOG_PANIC, /* 1 */ + AV_LOG_FATAL, /* 2 */ + AV_LOG_ERROR, /* 3 */ + AV_LOG_WARNING, /* 4 */ + AV_LOG_INFO, /* 5 */ + AV_LOG_VERBOSE, /* 6 */ + AV_LOG_DEBUG, /* 7 */ +}; + +static void ass_log(int ass_level, const char *fmt, va_list args, void *ctx) +{ + const int ass_level_clip = av_clip(ass_level, 0, FF_ARRAY_ELEMS(ass_libavfilter_log_level_map) - 1); + const int level = ass_libavfilter_log_level_map[ass_level_clip]; + + av_vlog(ctx, level, fmt, args); + av_log(ctx, level, "\n"); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + TextSubsContext *s = ctx->priv; + + if (s->track) + ass_free_track(s->track); + if (s->renderer) + ass_renderer_done(s->renderer); + if (s->library) + ass_library_done(s->library); + + s->track = NULL; + s->renderer = NULL; + s->library = NULL; + + if (s->is_mutex_initialized) { + ff_mutex_destroy(&s->mutex); + s->is_mutex_initialized = 0; + } + + av_frame_free(&s->last_frame); +} + +static int overlay_textsubs_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink0 = ctx->inputs[0]; + AVFilterLink *inlink1 = ctx->inputs[1]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input0 and output0 video formats */ + formats = ff_draw_supported_pixel_formats(0); + if ((ret = ff_formats_ref(formats, &inlink0->outcfg.formats)) < 0) + return ret; + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + /* set input1 subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink1->outcfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + + outlink->w = ctx->inputs[0]->w; + outlink->h = ctx->inputs[0]->h; + outlink->time_base = ctx->inputs[0]->time_base; + outlink->frame_rate = ctx->inputs[0]->frame_rate; + + return 0; +} + +static int config_input_main(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = inlink->dst->priv; + int ret; + + ret = ff_draw_init(&s->draw, inlink->format, s->alpha ? FF_DRAW_PROCESS_ALPHA : 0); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize ff_draw.\n"); + return ret; + } + + ass_set_frame_size (s->renderer, inlink->w, inlink->h); + ass_set_pixel_aspect(s->renderer, av_q2d(inlink->sample_aspect_ratio)); + + av_log(ctx, AV_LOG_VERBOSE, "Subtitle screen: %dx%d\n\n\n\n", inlink->w, inlink->h); + + return 0; +} + +/* libass stores an RGBA color in the format RRGGBBTT, where TT is the transparency level */ +#define AR(c) ( (c)>>24) +#define AG(c) (((c)>>16)&0xFF) +#define AB(c) (((c)>>8) &0xFF) +#define AA(c) ((0xFF-(c)) &0xFF) + +static void overlay_ass_image(TextSubsContext *s, AVFrame *picref, + const ASS_Image *image) +{ + for (; image; image = image->next) { + uint8_t rgba_color[] = {AR(image->color), AG(image->color), AB(image->color), AA(image->color)}; + FFDrawColor color; + ff_draw_color(&s->draw, &color, rgba_color); + ff_blend_mask(&s->draw, &color, + picref->data, picref->linesize, + picref->width, picref->height, + image->bitmap, image->stride, image->w, image->h, + 3, 0, image->dst_x, image->dst_y); + } +} + +static void process_header(AVFilterContext *link, AVFrame *frame) +{ + TextSubsContext *s = link->priv; + ASS_Track *track = s->track; + ASS_Style *style; + int sid = 0; + + if (!track) + return; + + if (frame && frame->subtitle_header) { + char *subtitle_header = (char *)frame->subtitle_header->data; + ass_process_codec_private(s->track, subtitle_header, strlen(subtitle_header)); + } + else { + char* subtitle_header = avpriv_ass_get_subtitle_header_default(0); + if (!subtitle_header) + return; + + ass_process_codec_private(s->track, subtitle_header, strlen(subtitle_header)); + av_free(subtitle_header); + } + + if (s->language) + s->track->Language = av_strdup(s->language); + + if (!s->track->event_format) { + s->track->event_format = av_strdup("ReadOrder, Layer, Style, Name, MarginL, MarginR, MarginV, Effect, Text"); + } + + if (s->track->n_styles == 0) { + sid = ass_alloc_style(track); + style = &s->track->styles[sid]; + style->Name = av_strdup("Default"); + style->PrimaryColour = 0xffffff00; + style->SecondaryColour = 0x00ffff00; + style->OutlineColour = 0x00000000; + style->BackColour = 0x00000080; + style->Bold = 200; + style->ScaleX = 1.0; + style->ScaleY = 1.0; + style->Spacing = 0; + style->BorderStyle = 1; + style->Outline = 2; + style->Shadow = 3; + style->Alignment = 2; + } + else + style = &s->track->styles[sid]; + + style->FontSize = s->font_size; + style->MarginL = style->MarginR = style->MarginV = s->margin; + + track->default_style = sid; + + s->got_header = 1; +} + +static int filter_video_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + int detect_change = 0; + ASS_Image *image; + + + int64_t time_ms = (int64_t)((double)frame->pts * av_q2d(inlink->time_base) * 1000); + int64_t time_ms1 = (int64_t)((double)ctx->inputs[1]->current_pts * av_q2d(ctx->inputs[1]->time_base) * 1000); + + if (time_ms1 < time_ms + 1000) + ff_request_frame(ctx->inputs[1]); + + av_log(ctx, AV_LOG_DEBUG, "filter_video_frame - video: %"PRId64"ms sub: %"PRId64"ms rel %d\n", time_ms, time_ms1, (time_ms1 < time_ms)); + + ff_mutex_lock(&s->mutex); + image = ass_render_frame(s->renderer, s->track, time_ms, &detect_change); + ff_mutex_unlock(&s->mutex); + + if (detect_change) + av_log(ctx, AV_LOG_DEBUG, "Change happened at time ms:%"PRId64"\n", time_ms); + + overlay_ass_image(s, frame, image); + + return ff_filter_frame(ctx->outputs[0], frame); +} + +static int filter_subtitle_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + const int64_t start_time = av_rescale_q(frame->subtitle_timing.start_pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t duration = av_rescale_q(frame->subtitle_timing.duration, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t frame_time = (int64_t)((double)frame->pts * av_q2d(inlink->time_base) * 1000); + + // Postpone header processing until we receive a frame with content + if (!s->got_header && frame->num_subtitle_areas > 0) + process_header(ctx, frame); + + av_log(ctx, AV_LOG_DEBUG, "filter_subtitle_frame dur: %"PRId64"ms frame: %"PRId64"ms sub: %"PRId64"ms repeat_sub %d\n", duration, frame_time, start_time, frame->repeat_sub); + + if (frame->repeat_sub) + goto exit; + + ff_mutex_lock(&s->mutex); + + if (s->render_latest_only && s->track->n_events > 0) { + const int64_t previous_start_time = s->track->events[s->track->n_events - 1].Start; + const int64_t diff = start_time - previous_start_time; + for (int i = s->track->n_events - 1; i >= 0; i--) { + if (previous_start_time != s->track->events[i].Start) + break; + + if (s->track->events[i].Duration > diff) + s->track->events[i].Duration = diff; + } + } + + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + char *ass_line = frame->subtitle_areas[i]->ass; + if (!ass_line) + continue; + + ass_process_chunk(s->track, ass_line, strlen(ass_line), start_time, duration); + } + + ff_mutex_unlock(&s->mutex); + +exit: + av_frame_free(&frame); + return 0; +} + +static av_cold int init(AVFilterContext *ctx) +{ + int ret; + TextSubsContext *s = ctx->priv; + + s->library = ass_library_init(); + + if (!s->library) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize libass.\n"); + return AVERROR(EINVAL); + } + + ass_set_message_cb(s->library, ass_log, ctx); + + /* Initialize fonts */ + if (s->fonts_dir) + ass_set_fonts_dir(s->library, s->fonts_dir); + + ass_set_extract_fonts(s->library, 1); + + s->renderer = ass_renderer_init(s->library); + if (!s->renderer) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize libass renderer.\n"); + return AVERROR(EINVAL); + } + + s->track = ass_new_track(s->library); + if (!s->track) { + av_log(ctx, AV_LOG_ERROR, "ass_new_track() failed!\n"); + return AVERROR(EINVAL); + } + + ass_set_check_readorder(s->track, 0); + + ass_set_fonts(s->renderer, s->default_font_path, NULL, 1, s->fc_file, 1); + + if (s->force_style) { + char **list = NULL; + char *temp = NULL; + char *ptr = av_strtok(s->force_style, ",", &temp); + int i = 0; + while (ptr) { + av_dynarray_add(&list, &i, ptr); + if (!list) { + return AVERROR(ENOMEM); + } + ptr = av_strtok(NULL, ",", &temp); + } + av_dynarray_add(&list, &i, NULL); + if (!list) { + return AVERROR(ENOMEM); + } + ass_set_style_overrides(s->library, list); + av_free(list); + } + + ret = ff_mutex_init(&s->mutex, NULL); + if (ret) { + av_log(ctx, AV_LOG_ERROR, "mutex initialiuzation failed! Error code: %d\n", ret); + return AVERROR(EINVAL); + } + + s->is_mutex_initialized = 1; + + return ret; +} + +static int textsub2video_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input0 subtitle format */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output0 video format */ + formats = ff_draw_supported_pixel_formats(AV_PIX_FMT_FLAG_ALPHA); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int textsub2video_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + + if (s->out_w <= 0 || s->out_h <= 0) { + s->out_w = inlink->w; + s->out_h = inlink->h; + } + + return 0; +} + +static int textsub2video_config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + TextSubsContext *s = ctx->priv; + int ret; + + ret = ff_draw_init(&s->draw, outlink->format, FF_DRAW_PROCESS_ALPHA); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize ff_draw.\n"); + return ret; + } + + if (s->out_w <= 0 || s->out_h <= 0) { + av_log(ctx, AV_LOG_ERROR, "No output image size set.\n"); + return AVERROR(EINVAL); + } + + ass_set_frame_size (s->renderer, s->out_w, s->out_h); + + outlink->w = s->out_w; + outlink->h = s->out_h; + outlink->sample_aspect_ratio = (AVRational){1,1}; + outlink->frame_rate = s->frame_rate; + + return 0; +} + +static int textsub2video_request_frame(AVFilterLink *outlink) +{ + TextSubsContext *s = outlink->src->priv; + AVFilterLink *inlink = outlink->src->inputs[0]; + int64_t last_pts = outlink->current_pts; + int64_t next_pts, time_ms; + int i, detect_change = 0, status; + AVFrame *out; + ASS_Image *image; + + status = ff_outlink_get_status(inlink); + if (status == AVERROR_EOF) + return AVERROR_EOF; + + if (s->eof) + return AVERROR_EOF; + + if (inlink->current_pts == AV_NOPTS_VALUE) { // || outlink->current_pts > inlink->current_pts) { + int ret = ff_request_frame(inlink); + if (ret == AVERROR_EOF) { + s->eof = 1; + } + + if (ret != 0) + av_log(outlink->src, AV_LOG_DEBUG, "ff_request_frame returned: %d\n", ret); + + s->need_frame = 1; + return 0; + } + + if (last_pts == AV_NOPTS_VALUE) + next_pts = last_pts = inlink->current_pts * av_q2d(inlink->time_base) / av_q2d(outlink->time_base); + else + next_pts = last_pts + (int64_t)(1.0 / av_q2d(outlink->frame_rate) / av_q2d(outlink->time_base)); + + time_ms = (int64_t)((double)next_pts * av_q2d(outlink->time_base) * 1000); + + ff_mutex_lock(&s->mutex); + image = ass_render_frame(s->renderer, s->track, time_ms, &detect_change); + ff_mutex_unlock(&s->mutex); + + if (detect_change) + av_log(outlink->src, AV_LOG_VERBOSE, "Change happened at time ms:%"PRId64" pts:%"PRId64"\n", time_ms, next_pts); + else if (s->last_frame) { + out = av_frame_clone(s->last_frame); + if (!out) + return AVERROR(ENOMEM); + + out->pts = out->pkt_dts = out->best_effort_timestamp = next_pts; + return ff_filter_frame(outlink, out); + } + + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) + return AVERROR(ENOMEM); + + for (i = 0; i < AV_NUM_DATA_POINTERS; i++) { + if (out->buf[i] && i != 1) + memset(out->buf[i]->data, 0, out->buf[i]->size); + } + + out->pts = out->pkt_dts = out->best_effort_timestamp = next_pts; + + if (image) + overlay_ass_image(s, out, image); + + av_frame_free(&s->last_frame); + + s->last_frame = av_frame_clone(out); + + return ff_filter_frame(outlink, out); +} + +static int textsub2video_filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + const int64_t start_time = av_rescale_q(frame->subtitle_timing.start_pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t duration = av_rescale_q(frame->subtitle_timing.duration, AV_TIME_BASE_Q, av_make_q(1, 1000)); + + av_log(ctx, AV_LOG_VERBOSE, "textsub2video_filter_frame num_subtitle_rects: %d, start_time_ms: %"PRId64"\n", frame->num_subtitle_areas, start_time); + + if (!s->got_header && frame->num_subtitle_areas > 0) + process_header(ctx, frame); + + if (frame->repeat_sub) + goto exit; + + ff_mutex_lock(&s->mutex); + + if (s->render_latest_only && s->track->n_events > 0) { + const int64_t previous_start_time = s->track->events[s->track->n_events - 1].Start; + const int64_t diff = start_time - previous_start_time; + for (int i = s->track->n_events - 1; i >= 0; i--) { + if (previous_start_time != s->track->events[i].Start) + break; + + if (s->track->events[i].Duration > diff) + s->track->events[i].Duration = diff; + + } + } + + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + char *ass_line = frame->subtitle_areas[i]->ass; + if (!ass_line) + continue; + + ass_process_chunk(s->track, ass_line, strlen(ass_line), start_time, duration); + } + + + ff_mutex_unlock(&s->mutex); + +exit: + av_frame_free(&frame); + + if (s->need_frame) { + s->need_frame = 0; + return textsub2video_request_frame(ctx->outputs[0]); + } + + return 0; +} + +#define OFFSET(x) offsetof(TextSubsContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption overlaytextsubs_options[] = { + {"alpha", "enable processing of alpha channel", OFFSET(alpha), AV_OPT_TYPE_BOOL, {.i64 = 0 }, 0, 1, .flags = FLAGS}, + {"font_size", "default font size", OFFSET(font_size), AV_OPT_TYPE_DOUBLE, {.dbl = 18.0}, 0.0, 100.0, .flags = FLAGS}, + {"force_style", "force subtitle style", OFFSET(force_style), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, .flags = FLAGS}, + {"margin", "default margin", OFFSET(margin), AV_OPT_TYPE_INT, {.i64 = 20 }, 0, INT_MAX, .flags = FLAGS}, + {"default_font_path", "path to default font", OFFSET(default_font_path), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, .flags = FLAGS}, + {"fonts_dir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, .flags = FLAGS}, + {"fontsdir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, .flags = FLAGS}, + {"fontconfig_file", "fontconfig file to load", OFFSET(fc_file), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, .flags = FLAGS}, + {"language", "default language", OFFSET(language), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, .flags = FLAGS}, + {"render_latest_only", "newest sub event for each time", OFFSET(render_latest_only), AV_OPT_TYPE_BOOL, {.i64 = 0 }, 0, 1, .flags = FLAGS}, + { .name = NULL } +}; + +static const AVOption textsub2video_options[] = { + {"rate", "set frame rate", OFFSET(frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="8"}, 0, INT_MAX, .flags = FLAGS}, + {"r", "set frame rate", OFFSET(frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="8"}, 0, INT_MAX, .flags = FLAGS}, + {"size", "set video size", OFFSET(out_w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, .flags = FLAGS}, + {"s", "set video size", OFFSET(out_w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, .flags = FLAGS}, + {"font_size", "default font size", OFFSET(font_size), AV_OPT_TYPE_DOUBLE, {.dbl = 18.0}, 0.0, 100.0, .flags = FLAGS}, + {"force_style", "force subtitle style", OFFSET(force_style), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, .flags = FLAGS}, + {"margin", "default margin", OFFSET(margin), AV_OPT_TYPE_INT, {.i64 = 20 }, 0, INT_MAX, .flags = FLAGS}, + {"default_font_path", "path to default font", OFFSET(default_font_path), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, .flags = FLAGS}, + {"fonts_dir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, .flags = FLAGS}, + {"fontsdir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, .flags = FLAGS}, + {"fontconfig_file", "fontconfig file to load", OFFSET(fc_file), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, .flags = FLAGS}, + {"language", "default language", OFFSET(language), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, .flags = FLAGS}, + {"render_latest_only", "newest sub event for each time", OFFSET(render_latest_only), AV_OPT_TYPE_BOOL, {.i64 = 0 }, 0, 1, .flags = FLAGS}, + { .name = NULL } +}; + +#if CONFIG_OVERLAYTEXTSUBS_FILTER + +AVFILTER_DEFINE_CLASS(overlaytextsubs); + +static const AVFilterPad overlaytextsubs_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input_main, + .flags = AVFILTERPAD_FLAG_NEEDS_WRITABLE, + .filter_frame = filter_video_frame, + }, + { + .name = "overlay", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_subtitle_frame, + }, +}; + +static const AVFilterPad overlaytextsubs_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +const AVFilter ff_vf_overlaytextsubs = { + .name = "overlaytextsubs", + .description = NULL_IF_CONFIG_SMALL("Overlay textual subtitles on top of the input."), + .init = init, + .uninit = uninit, + .priv_size = sizeof(TextSubsContext), + .priv_class = &overlaytextsubs_class, + FILTER_INPUTS(overlaytextsubs_inputs), + FILTER_OUTPUTS(overlaytextsubs_outputs), + FILTER_QUERY_FUNC(overlay_textsubs_query_formats), +}; +#endif + +#if CONFIG_TEXTSUB2VIDEO_FILTER + +AVFILTER_DEFINE_CLASS(textsub2video); + +static const AVFilterPad textsub2video_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .config_props = textsub2video_config_input, + .filter_frame = textsub2video_filter_frame, + }, +}; + +static const AVFilterPad textsub2video_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = textsub2video_config_output, + .request_frame = textsub2video_request_frame, + }, +}; + +const AVFilter ff_svf_textsub2video = { + .name = "textsub2video", + .description = NULL_IF_CONFIG_SMALL("Convert textual subtitles to video frames"), + .init = init, + .uninit = uninit, + .priv_size = sizeof(TextSubsContext), + .priv_class = &textsub2video_class, + FILTER_INPUTS(textsub2video_inputs), + FILTER_OUTPUTS(textsub2video_outputs), + FILTER_QUERY_FUNC(textsub2video_query_formats), +}; +#endif From patchwork Sat May 28 13:25:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35970 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366669pzj; Sat, 28 May 2022 06:28:09 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyZOQu2NoxKAXoG5oWTK6R9ShRRDX6d0+xQ+i8YroqXzlxWS3GeaFAc6CM2GXwCbhkJM4w1 X-Received: by 2002:a05:6402:1147:b0:42d:b9e6:2458 with SMTP id g7-20020a056402114700b0042db9e62458mr2364031edw.48.1653744488992; Sat, 28 May 2022 06:28:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744488; cv=none; d=google.com; s=arc-20160816; b=DENsdUmP/bufZSLWrd7do5JhIB0M7npEADSmq15XC8oYmoB7TuKmgwBAo2Jrob8arp FjbQhRFrfkJFOKiXywgVzbB+p64PkoVkUhjyR/w7DsLKlNoUy1bITvqRoLcLI3wloac0 C/QCZg3DT/84wdo6lI6DLmhiHejrpLwF+Gxdfg2AFe91Z78seWD8Jf6OZba+tV0MXef1 SkZL1P67sYVtELRT7sYfMNSBs/zPhaexKGnTszsVMa9isCp/lIMSFV6pHmQ4PEnyUsjq kfbbJB3lVcc02X6NXSoK9T3nMNbpfb02Lzu1SIWI84pPBcFw+s7V7My1vFLH2EbZLVUG IVyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=/Z8a50KUYLmXRN103+UGs9orrq9Sqy57vI5dOMLvPsg=; b=OQZpWbaEZaRGdCC6xcQzbtZicXlFJeSTmpW+GNm0283PgrtIJ2T5xG88cjlNTeaoIN 6Eq7bTYY65Ha+c4+m2bgmhXfq8CYGlL3se4UmdNh/1CK9oJPdFI/6kVozMS0JndDOJ1M UYTxZa4cjyUThVOp6LWoGke0lRRYIZfgCwvbsNf2wZ4lopN+Rg4qbcHK5CqE53XIWpAf wtC+stI/8gWwhVZyuA8+JB2nb6k1Da+LOVeNlH7ChSLb9dErTs2RfOqKnVWcxGf4i3OD JNh3KfwHuata8W8zdRr7gtHMT+5Jwg0ikVlWC4qBao+Dp7W27Ik14W3ynBa51UEKZILL PZ1w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=kTbTfQjw; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id p21-20020a170906785500b006ff149c2b69si6190593ejm.329.2022.05.28.06.28.08; Sat, 28 May 2022 06:28:08 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=kTbTfQjw; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BC59668B657; Sat, 28 May 2022 16:25:50 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4F8FC68B61C for ; Sat, 28 May 2022 16:25:43 +0300 (EEST) Received: by mail-pl1-f170.google.com with SMTP id w3so6436102plp.13 for ; Sat, 28 May 2022 06:25:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=OqhbU/b0zzr3kzn9SUqIr6VqE0LJnL9y55vlOBYV5gs=; b=kTbTfQjww+sD/qZPz1jeCPXJKSwXS/Bb+jFz3mj8HgO8WBaeJofz96ZH3VU9i/nDGS jvOnLjUQBA942L0AJefCBin9KGfadFZL+ssGeRvatU9Nj2kXwFDEPVfzJFeLQoBY27MF m05NeGE2OjCDOJiGonOgFrm1hjCR/j/+JO7pD5L2HpL9EInclZoGQWi6BBKY0eDgT91u ML76L+eLnPyesDxmLNMgJVgvnkc6v9I1YKrSdAvE8cIU+tRGFW7QF3kyilG0BTLm1uni IV1lfS4K6fLSlS+dyQEi3tXbiUXNQQQTDgeHAZkegDC0WAKCnkOfZ7l6+5tbYun2ISN2 Vh/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=OqhbU/b0zzr3kzn9SUqIr6VqE0LJnL9y55vlOBYV5gs=; b=4k2/rUwfaxsmlhc8aBpcSmZCN2s7yXkdtbW5BpFIrNHEygg4FunuQgCFL3di2BeYW8 BsoxAsyoHv21laRvvrXC9hAk8OEIT13jiBmW4QhDJYmpWv9raZITvZpD3+SMapOa+Qlq X0k40p2FOe+tnxW6nexHlo/lXUm1IgOWSjKHTJ8tuVJxh4Lm8x1eCbsi1Sxs334sFAb/ njPzArToHb3z86DP7Bq5y6YDG/BpGjk/uJ4asRKzU6B6A8Bm/eT6byZff8mZ3XbcqZXP JIk4XrwSmbtwRttUFSKG7rZnwxNKRwzYbpljTJRGLcfsyhTQVqVKKpk4J8KkTXo4IGCx Vozw== X-Gm-Message-State: AOAM530dNaWQWvKUP40V5lU84P4hmUbLbt8wXcaT6qJFMvIuJM6S8cAs 8QIyBRLWKOaNsR4B0XIin4Cpc35N/Hk8YA== X-Received: by 2002:a17:902:a418:b0:161:f216:4f49 with SMTP id p24-20020a170902a41800b00161f2164f49mr41609080plq.129.1653744341217; Sat, 28 May 2022 06:25:41 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id i16-20020a170902cf1000b00161929fb1adsm5641345plg.54.2022.05.28.06.25.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:40 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <8fdbdf7c5fea1e2cd593cc90d75dd7bc885f4c82.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:15 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 15/23] avfilter/textmod: Add textmod, censor and show_speaker filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: xkmq4xrTfH3S From: softworkz - textmod {S -> S) Modify subtitle text in a number of ways - censor {S -> S) Censor subtitles using a word list - show_speaker {S -> S) Prepend speaker names from ASS subtitles to the visible text lines Signed-off-by: softworkz --- doc/filters.texi | 206 ++++++++++++ libavfilter/Makefile | 5 + libavfilter/allfilters.c | 3 + libavfilter/sf_textmod.c | 710 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 924 insertions(+) create mode 100644 libavfilter/sf_textmod.c diff --git a/doc/filters.texi b/doc/filters.texi index 70d4c90663..e0ceba80e2 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -26887,6 +26887,145 @@ existing filters using @code{--disable-filters}. Below is a description of the currently available subtitle filters. + +@section censor + +Censor selected words in text subtitles. + +Inputs: +@itemize +@item 0: Subtitles[TEXT] +@end itemize + +Outputs: +@itemize +@item 0: Subtitles[TEXT] +@end itemize + +It accepts the following parameters: + +@table @option +@item mode +The censoring mode to apply. + +Supported censoring modes are: + +@table @var +@item 0, keep_first_last +Replace all characters with the 'censor_char' except the first and the last character of a word. +For words with less than 4 characters, the last character will be replaced as well. +For words with less than 3 characters, the first character will be replaced as well. +@item 1, keep_first +Replace all characters with the 'censor_char' except the first character of a word. +For words with less than 3 characters, the first character will be replaced as well. +@item 2, all +Replace all characters with the 'censor_char'. +@end table + +@item words +A list of words to censor, separated by 'separator'. + +@item words_file +Specify a file from which to load the contents for the 'words' parameter. + +@item censor_char +Single character used as replacement for censoring. + +@item separator +Delimiter character for words. Used with replace_words and remove_words- Must be a single character. +The default is '.'. + +@end table + +@subsection Examples + +@itemize +@item +Censor a few given words with a pound character. +@example +ffmpeg -i "http://streams.videolan.org/samples/sub/SSA/subtitle_testing_complex.mkv" -filter_complex "[0:1]censor=words='diss,louder,hope,beam,word':censor_char='#'" -map 0 -y output.mkv +@end example +@end itemize + + +@section textmod + +Modify subtitle text in a number of ways. + +Inputs: +@itemize +@item 0: Subtitles[TEXT] +@end itemize + +Outputs: +@itemize +@item 0: Subtitles[TEXT] +@end itemize + +It accepts the following parameters: + +@table @option +@item mode +The kind of text modification to apply + +Supported operation modes are: + +@table @var +@item 0, leet +Convert subtitle text to 'leet speak'. It's primarily useful for testing as the modification will be visible with almost all text lines. +@item 1, to_upper +Change all text to upper case. Might improve readability. +@item 2, to_lower +Change all text to lower case. +@item 3, replace_chars +Replace one or more characters. Requires the find and replace parameters to be specified. +Both need to be equal in length. +The first char in find is replaced by the first char in replace, same for all subsequent chars. +@item 4, remove_chars +Remove certain characters. Requires the find parameter to be specified. +All chars in the find parameter string will be removed from all subtitle text. +@item 5, replace_words +Replace one or more words. Requires the find and replace parameters to be specified. Multiple words must be separated by the delimiter char specified vie the separator parameter (default: ','). +The number of words in the find and replace parameters needs to be equal. +The first word in find is replaced by the first word in replace, same for all subsequent words +@item 6, remove_words +Remove certain words. Requires the find parameter to be specified. Multiple words must be separated by the delimiter char specified vie the separator parameter (default: ','). +All words in the find parameter string will be removed from all subtitle text. +@end table + +@item find +Required for replace_chars, remove_chars, replace_words and remove_words. + +@item find_file +Specify a file from which to load the contents for the 'find' parameter. + +@item replace +Required for replace_chars and replace_words. + +@item replace_file +Specify a file from which to load the contents for the 'replace' parameter. + +@item separator +Delimiter character for words. Used with replace_words and remove_words- Must be a single character. +The default is '.'. + +@end table + +@subsection Examples + +@itemize +@item +Change all characters to upper case while keeping all styles and animations: +@example +ffmpeg -i "https://streams.videolan.org/ffmpeg/mkv_subtitles.mkv" -filter_complex "[0:s]textmod=mode=to_upper" -map 0 -y out.mkv +@end example +@item +Remove a set of symbol characters for am improved and smoother visual apperance: +@example +ffmpeg -i "https://streams.videolan.org/ffmpeg/mkv_subtitles.mkv" -filter_complex "[0:s]textmod=mode=remove_chars:find='$&#@*§'" -map 0 -y out.mkv +@end example +@end itemize + @section graphicsub2video Renders graphic subtitles as video frames. @@ -27054,6 +27193,73 @@ ffmpeg -i "http://streams.videolan.org/samples/sub/SSA/subtitle_testing_complex. @end example @end itemize +@section showspeaker + +Prepend speaker names to subtitle lines (when available). + +Subtitles in ASS/SSA format are often including the names of the persons +or character for each subtitle line. The showspeaker filter adds those names +to the actual subtitle text to make it visible on playback. + +Inputs: +@itemize +@item 0: Subtitles[TEXT] +@end itemize + +Outputs: +@itemize +@item 0: Subtitles[TEXT] +@end itemize + +It accepts the following parameters: + +@table @option +@item format +The format for prepending speaker names. Default is 'square_brackets'. + +Supported operation modes are: + +@table @var +@item 0, square_brackets +Enclose the speaker name in square brackets, followed by space ('[speaker] text'). +@item 1, round_brackets +Enclose the speaker name in round brackets, followed by space ('(speaker) text'). +@item 2, colon +Separate the speaker name with a colon and space ('speaker: text'). +@item 3, plain +Separate the speaker name with a space only ('speaker text'). +@end table + +@item line_break +Set thís parameter to insert a line break between speaker name and text instead of the space character. + +@item style +Allows to set a specific style for the speaker name text. + +This can be either a named style that exists in the ass subtitle header script (e.g. 'Default') or an ass style override code. +Example: @{\\c&HDD0000&\\be1\\i1\\bord10@} +This sets the color to blue, enables edge blurring, italic font and a border of size 10. + +The behavior is as follows: + +- When the style parameter is not provided, the filter will find the first position in the event string that is actual text. + The speaker name will be inserted at this position. This allows to have the speaker name shown in the same style like the + regular text, in case the string would start with a sequence of style codes. +- When the style parameter is provided, everything will be prepended to the original text: + Style Code or Style name >> Speaker Name >> Style Reset Code >> Original Text + +@end table + +@subsection Examples + +@itemize +@item +Prepend speaker names with blue text, smooth edges and blend/overlay that onto the video. +@example +ffmpeg -i INPUT -filter_complex "showspeaker=format=colon:style='@{\\c&HDD0000&\\be1@}',[0:v]overlaytextsubs" +@end example +@end itemize + @section textsub2video Converts text subtitles to video frames. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 84ec2fa542..6d2a75128c 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -576,6 +576,11 @@ OBJS-$(CONFIG_YUVTESTSRC_FILTER) += vsrc_testsrc.o OBJS-$(CONFIG_NULLSINK_FILTER) += vsink_nullsink.o +# subtitle filters +OBJS-$(CONFIG_CENSOR_FILTER) += sf_textmod.o +OBJS-$(CONFIG_SHOW_SPEAKER_FILTER) += sf_textmod.o +OBJS-$(CONFIG_TEXTMOD_FILTER) += sf_textmod.o + # multimedia filters OBJS-$(CONFIG_ABITSCOPE_FILTER) += avf_abitscope.o OBJS-$(CONFIG_ADRAWGRAPH_FILTER) += f_drawgraph.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index a6b95785be..960726ba08 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -559,6 +559,9 @@ extern const AVFilter ff_avf_showvolume; extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; +extern const AVFilter ff_sf_censor; +extern const AVFilter ff_sf_showspeaker; +extern const AVFilter ff_sf_textmod; extern const AVFilter ff_svf_graphicsub2video; extern const AVFilter ff_svf_textsub2video; diff --git a/libavfilter/sf_textmod.c b/libavfilter/sf_textmod.c new file mode 100644 index 0000000000..c75244cfb2 --- /dev/null +++ b/libavfilter/sf_textmod.c @@ -0,0 +1,710 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * text subtitle filter which allows to modify subtitle text in several ways + */ + +#include + +#include "libavutil/opt.h" +#include "internal.h" +#include "libavutil/ass_split_internal.h" +#include "libavutil/bprint.h" +#include "libavutil/file.h" + +static const char* leet_src = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"; +static const char* leet_dst = "abcd3f6#1jklmn0pq257uvwxyzAB(D3F6#1JKLMN0PQ257UVWXYZ"; + +enum TextModFilterType { + TM_TEXTMOD, + TM_CENSOR, + TM_SHOW_SPEAKER, +}; + +enum TextModOperation { + OP_LEET, + OP_TO_UPPER, + OP_TO_LOWER, + OP_REPLACE_CHARS, + OP_REMOVE_CHARS, + OP_REPLACE_WORDS, + OP_REMOVE_WORDS, + NB_OPS, +}; + +enum CensorMode { + CM_KEEP_FIRST_LAST, + CM_KEEP_FIRST, + CM_ALL, +}; + +enum ShowSpeakerMode { + SM_SQUARE_BRACKETS, + SM_ROUND_BRACKETS, + SM_COLON, + SM_PLAIN, +}; + +typedef struct TextModContext { + const AVClass *class; + enum AVSubtitleType format; + enum TextModFilterType filter_type; + enum TextModOperation operation; + enum CensorMode censor_mode; + enum ShowSpeakerMode speaker_mode; + char *find; + char *find_file; + char *style; + char *replace; + char *replace_file; + char *separator; + char *censor_char; + char **find_list; + int line_break; + int nb_find_list; + char **replace_list; + int nb_replace_list; +} TextModContext; + +static char **split_string(char *source, int *nb_elems, const char *delim) +{ + char **list = NULL; + char *temp = NULL; + char *ptr = av_strtok(source, delim, &temp); + + while (ptr) { + if (strlen(ptr)) { + av_dynarray_add(&list, nb_elems, ptr); + if (!list) + return NULL; + } + + ptr = av_strtok(NULL, delim, &temp); + } + + if (!list) + return NULL; + + for (int i = 0; i < *nb_elems; i++) { + list[i] = av_strdup(list[i]); + if (!list[i]) { + for (int n = 0; n < i; n++) + av_free(list[n]); + av_free(list); + return NULL; + } + } + + return list; +} + +static int load_text_from_file(AVFilterContext *ctx, const char *file_name, char **text, char separator) +{ + int err; + uint8_t *textbuf; + char *tmp; + size_t textbuf_size; + int offset = 0; + + if ((err = av_file_map(file_name, &textbuf, &textbuf_size, 0, ctx)) < 0) { + av_log(ctx, AV_LOG_ERROR, "The text file '%s' could not be read or is empty\n", file_name); + return err; + } + + if (textbuf_size > 1 && + (textbuf[0] == 0xFF && textbuf[1] == 0xFE + || textbuf[0] == 0xFE && textbuf[1] == 0xFF)) { + av_log(ctx, AV_LOG_ERROR, "UTF text files are not supported. File: %s\n", file_name); + return AVERROR(EINVAL); + } + + if (textbuf_size > 2 && textbuf[0] == 0xEF && textbuf[1] == 0xBB && textbuf[2] == 0xBF) + offset = 3; // UTF-8 + + if (textbuf_size > SIZE_MAX - 1 || !((tmp = av_strndup((char *)textbuf + offset, textbuf_size - offset)))) { + av_file_unmap(textbuf, textbuf_size); + return AVERROR(ENOMEM); + } + + av_file_unmap(textbuf, textbuf_size); + + for (size_t i = 0; i < strlen(tmp); i++) { + switch (tmp[i]) { + case '\n': + case '\r': + case '\f': + case '\v': + tmp[i] = separator; + } + } + + *text = tmp; + + return 0; +} + +static int load_files(AVFilterContext *ctx) +{ + TextModContext *s = ctx->priv; + int ret; + + if (!s->separator || strlen(s->separator) != 1) { + av_log(ctx, AV_LOG_ERROR, "A single character needs to be specified for the separator parameter.\n"); + return AVERROR(EINVAL); + } + + if (s->find_file && strlen(s->find_file)) { + ret = load_text_from_file(ctx, s->find_file, &s->find, s->separator[0]); + if (ret < 0 ) + return ret; + } + + if (s->replace_file && strlen(s->replace_file)) { + ret = load_text_from_file(ctx, s->replace_file, &s->replace, s->separator[0]); + if (ret < 0 ) + return ret; + } + + return 0; +} + +static int init_censor(AVFilterContext *ctx) +{ + TextModContext *s = ctx->priv; + int ret; + + s->filter_type = TM_CENSOR; + s->operation = OP_REPLACE_WORDS; + + ret = load_files(ctx); + if (ret < 0 ) + return ret; + + if (!s->find || !strlen(s->find)) { + av_log(ctx, AV_LOG_ERROR, "Either the 'words' or the 'words_file' parameter needs to be specified\n"); + return AVERROR(EINVAL); + } + + if (!s->censor_char || strlen(s->censor_char) != 1) { + av_log(ctx, AV_LOG_ERROR, "A single character needs to be specified for the censor_char parameter\n"); + return AVERROR(EINVAL); + } + + s->find_list = split_string(s->find, &s->nb_find_list, s->separator); + if (!s->find_list) + return AVERROR(ENOMEM); + + s->replace_list = av_calloc(s->nb_find_list, sizeof(char *)); + if (!s->replace_list) + return AVERROR(ENOMEM); + + for (int i = 0; i < s->nb_find_list; i++) { + size_t len, start = 0, end; + char *item = av_strdup(s->find_list[i]); + if (!item) + return AVERROR(ENOMEM); + + len = end = strlen(item); + + switch (s->censor_mode) { + case CM_KEEP_FIRST_LAST: + + if (len > 2) + start = 1; + if (len > 3) + end--; + + break; + case CM_KEEP_FIRST: + + if (len > 2) + start = 1; + + break; + } + + for (size_t n = start; n < end; n++) + item[n] = s->censor_char[0]; + + s->replace_list[i] = item; + } + + return 0; +} + +static int init_showspeaker(AVFilterContext *ctx) +{ + TextModContext *s = ctx->priv; + s->filter_type = TM_SHOW_SPEAKER; + + return 0; +} + +static int init(AVFilterContext *ctx) +{ + TextModContext *s = ctx->priv; + int ret; + + ret = load_files(ctx); + if (ret < 0 ) + return ret; + + switch (s->operation) { + case OP_REPLACE_CHARS: + case OP_REMOVE_CHARS: + case OP_REPLACE_WORDS: + case OP_REMOVE_WORDS: + if (!s->find || !strlen(s->find)) { + av_log(ctx, AV_LOG_ERROR, "Selected mode requires the 'find' parameter to be specified\n"); + return AVERROR(EINVAL); + } + break; + } + + switch (s->operation) { + case OP_REPLACE_CHARS: + case OP_REPLACE_WORDS: + if (!s->replace || !strlen(s->replace)) { + av_log(ctx, AV_LOG_ERROR, "Selected mode requires the 'replace' parameter to be specified\n"); + return AVERROR(EINVAL); + } + break; + } + + if (s->operation == OP_REPLACE_CHARS && strlen(s->find) != strlen(s->replace)) { + av_log(ctx, AV_LOG_ERROR, "Selected mode requires the 'find' and 'replace' parameters to have the same length\n"); + return AVERROR(EINVAL); + } + + if (s->operation == OP_REPLACE_WORDS || s->operation == OP_REMOVE_WORDS) { + if (!s->separator || strlen(s->separator) != 1) { + av_log(ctx, AV_LOG_ERROR, "Selected mode requires a single separator char to be specified\n"); + return AVERROR(EINVAL); + } + + s->find_list = split_string(s->find, &s->nb_find_list, s->separator); + if (!s->find_list) + return AVERROR(ENOMEM); + + if (s->operation == OP_REPLACE_WORDS) { + + s->replace_list = split_string(s->replace, &s->nb_replace_list, s->separator); + if (!s->replace_list) + return AVERROR(ENOMEM); + + if (s->nb_find_list != s->nb_replace_list) { + av_log(ctx, AV_LOG_ERROR, "The number of words in 'find' and 'replace' needs to be equal\n"); + return AVERROR(EINVAL); + } + } + } + + return 0; +} + +static void uninit(AVFilterContext *ctx) +{ + TextModContext *s = ctx->priv; + + for (int i = 0; i < s->nb_find_list; i++) + av_freep(&s->find_list[i]); + + s->nb_find_list = 0; + av_freep(&s->find_list); + + for (int i = 0; i < s->nb_replace_list; i++) + av_freep(&s->replace_list[i]); + + s->nb_replace_list = 0; + av_freep(&s->replace_list); +} + +static char *process_text(TextModContext *s, char *text) +{ + const char *char_src = s->find; + const char *char_dst = s->replace; + char *result = NULL; + int escape_level = 0, k = 0; + + switch (s->operation) { + case OP_LEET: + case OP_REPLACE_CHARS: + + if (s->operation == OP_LEET) { + char_src = leet_src; + char_dst = leet_dst; + } + + result = av_strdup(text); + if (!result) + return NULL; + + for (size_t n = 0; n < strlen(result); n++) { + if (result[n] == '{') + escape_level++; + + if (!escape_level) { + size_t len = strlen(char_src); + for (size_t t = 0; t < len; t++) { + if (result[n] == char_src[t]) { + result[n] = char_dst[t]; + break; + } + } + } + + if (result[n] == '}') + escape_level--; + } + + break; + case OP_TO_UPPER: + case OP_TO_LOWER: + + result = av_strdup(text); + if (!result) + return NULL; + + for (size_t n = 0; n < strlen(result); n++) { + if (result[n] == '{') + escape_level++; + if (!escape_level) + result[n] = s->operation == OP_TO_LOWER ? av_tolower(result[n]) : av_toupper(result[n]); + if (result[n] == '}') + escape_level--; + } + + break; + case OP_REMOVE_CHARS: + + result = av_strdup(text); + if (!result) + return NULL; + + for (size_t n = 0; n < strlen(result); n++) { + int skip_char = 0; + + if (result[n] == '{') + escape_level++; + + if (!escape_level) { + size_t len = strlen(char_src); + for (size_t t = 0; t < len; t++) { + if (result[n] == char_src[t]) { + skip_char = 1; + break; + } + } + } + + if (!skip_char) + result[k++] = result[n]; + + if (result[n] == '}') + escape_level--; + } + + result[k] = 0; + + break; + case OP_REPLACE_WORDS: + case OP_REMOVE_WORDS: + + result = av_strdup(text); + if (!result) + return NULL; + + for (int n = 0; n < s->nb_find_list; n++) { + char *tmp = result; + const char *replace = (s->operation == OP_REPLACE_WORDS) ? s->replace_list[n] : ""; + + result = av_strireplace(result, s->find_list[n], replace); + if (!result) + return NULL; + + av_free(tmp); + } + + break; + } + + return result; +} + +static char *process_dialog_show_speaker(TextModContext *s, char *ass_line) +{ + ASSDialog *dialog = avpriv_ass_split_dialog(NULL, ass_line); + int escape_level = 0; + unsigned pos = 0, len; + char *result, *text; + AVBPrint pbuf; + + if (!dialog) + return NULL; + + text = process_text(s, dialog->text); + if (!text) + return NULL; + + if (!dialog->name || !strlen(dialog->name) || !dialog->text || !strlen(dialog->text)) + return av_strdup(ass_line); + + // Find insertion point in case the line starts with style codes + len = (unsigned)strlen(dialog->text); + for (unsigned i = 0; i < len; i++) { + + if (dialog->text[i] == '{') + escape_level++; + + if (dialog->text[i] == '}') + escape_level--; + + if (escape_level == 0) { + pos = i; + break; + } + } + + if (s->style && strlen(s->style)) + // When a style is specified reset the insertion point + // (always add speaker plus style at the start in that case) + pos = 0; + + if (pos >= len - 1) + return av_strdup(ass_line); + + av_bprint_init(&pbuf, 1, AV_BPRINT_SIZE_UNLIMITED); + + if (pos > 0) { + av_bprint_append_data(&pbuf, dialog->text, pos); + } + + if (s->style && strlen(s->style)) { + if (s->style[0] == '{') + // Assume complete and valid style code, e.g. {\c&HFF0000&} + av_bprintf(&pbuf, "%s", s->style); + else + // Otherwise it must be a style name + av_bprintf(&pbuf, "{\\r%s}", s->style); + } + + switch (s->speaker_mode) { + case SM_SQUARE_BRACKETS: + av_bprintf(&pbuf, "[%s]", dialog->name); + break; + case SM_ROUND_BRACKETS: + av_bprintf(&pbuf, "(%s)", dialog->name); + break; + case SM_COLON: + av_bprintf(&pbuf, "%s:", dialog->name); + break; + case SM_PLAIN: + av_bprintf(&pbuf, "%s", dialog->name); + break; + } + + if (s->style && strlen(s->style)) { + // Reset line style + if (dialog->style && strlen(dialog->style) && !av_strcasecmp(dialog->style, "default")) + av_bprintf(&pbuf, "{\\r%s}", dialog->style); + else + av_bprintf(&pbuf, "{\\r}"); + } + + if (s->line_break) + av_bprintf(&pbuf, "\\N"); + else + av_bprintf(&pbuf, " "); + + av_bprint_append_data(&pbuf, dialog->text + pos, len - pos); + + av_bprint_finalize(&pbuf, &text); + + result = avpriv_ass_get_dialog_ex(dialog->readorder, dialog->layer, dialog->style, dialog->name, dialog->margin_l, dialog->margin_r, dialog->margin_v, text); + + av_free(text); + avpriv_ass_free_dialog(&dialog); + return result; +} + +static char *process_dialog(TextModContext *s, char *ass_line) +{ + ASSDialog *dialog; + char *result, *text; + + if (s->filter_type == TM_SHOW_SPEAKER) + return process_dialog_show_speaker(s, ass_line); + + dialog = avpriv_ass_split_dialog(NULL, ass_line); + if (!dialog) + return NULL; + + text = process_text(s, dialog->text); + if (!text) + return NULL; + + result = avpriv_ass_get_dialog_ex(dialog->readorder, dialog->layer, dialog->style, dialog->name, dialog->margin_l, dialog->margin_r, dialog->margin_v, text); + + av_free(text); + avpriv_ass_free_dialog(&dialog); + return result; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterLink *inlink = outlink->src->inputs[0]; + + outlink->w = inlink->w; + outlink->h = inlink->h; + outlink->time_base = inlink->time_base; + outlink->frame_rate = inlink->frame_rate; + + return 0; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + TextModContext *s = inlink->dst->priv; + AVFilterLink *outlink = inlink->dst->outputs[0]; + int ret; + + outlink->format = inlink->format; + + ret = av_frame_make_writable(frame); + if (ret < 0) + return ret; + + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + + AVSubtitleArea *area = frame->subtitle_areas[i]; + + if (area->ass) { + char *tmp = area->ass; + area->ass = process_dialog(s, area->ass); + av_free(tmp); + if (!area->ass) + return AVERROR(ENOMEM); + } + } + + return ff_filter_frame(outlink, frame); +} + +#define OFFSET(x) offsetof(TextModContext, x) +#define FLAGS (AV_OPT_FLAG_SUBTITLE_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption textmod_options[] = { + { "mode", "set operation mode", OFFSET(operation), AV_OPT_TYPE_INT, {.i64=OP_LEET}, OP_LEET, NB_OPS-1, FLAGS, "mode" }, + { "leet", "convert text to 'leet speak'", 0, AV_OPT_TYPE_CONST, {.i64=OP_LEET}, 0, 0, FLAGS, "mode" }, + { "to_upper", "change to upper case", 0, AV_OPT_TYPE_CONST, {.i64=OP_TO_UPPER}, 0, 0, FLAGS, "mode" }, + { "to_lower", "change to lower case", 0, AV_OPT_TYPE_CONST, {.i64=OP_TO_LOWER}, 0, 0, FLAGS, "mode" }, + { "replace_chars", "replace characters", 0, AV_OPT_TYPE_CONST, {.i64=OP_REPLACE_CHARS}, 0, 0, FLAGS, "mode" }, + { "remove_chars", "remove characters", 0, AV_OPT_TYPE_CONST, {.i64=OP_REMOVE_CHARS}, 0, 0, FLAGS, "mode" }, + { "replace_words", "replace words", 0, AV_OPT_TYPE_CONST, {.i64=OP_REPLACE_WORDS}, 0, 0, FLAGS, "mode" }, + { "remove_words", "remove words", 0, AV_OPT_TYPE_CONST, {.i64=OP_REMOVE_WORDS}, 0, 0, FLAGS, "mode" }, + { "find", "chars/words to find or remove", OFFSET(find), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "find_file", "load find param from file", OFFSET(find_file), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "replace", "chars/words to replace", OFFSET(replace), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "replace_file", "load replace param from file", OFFSET(replace_file), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "separator", "word separator", OFFSET(separator), AV_OPT_TYPE_STRING, {.str = ","}, 0, 0, FLAGS, NULL }, + { NULL }, +}; + + +static const AVOption censor_options[] = { + { "mode", "set censoring mode", OFFSET(censor_mode), AV_OPT_TYPE_INT, {.i64=CM_KEEP_FIRST_LAST}, 0, 2, FLAGS, "mode" }, + { "keep_first_last", "censor inner chars", 0, AV_OPT_TYPE_CONST, {.i64=CM_KEEP_FIRST_LAST}, 0, 0, FLAGS, "mode" }, + { "keep_first", "censor all but first char", 0, AV_OPT_TYPE_CONST, {.i64=CM_KEEP_FIRST}, 0, 0, FLAGS, "mode" }, + { "all", "censor all chars", 0, AV_OPT_TYPE_CONST, {.i64=CM_ALL}, 0, 0, FLAGS, "mode" }, + { "words", "list of words to censor", OFFSET(find), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "words_file", "path to word list file", OFFSET(find_file), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "separator", "word separator", OFFSET(separator), AV_OPT_TYPE_STRING, {.str = ","}, 0, 0, FLAGS, NULL }, + { "censor_char", "replacement character", OFFSET(censor_char), AV_OPT_TYPE_STRING, {.str = "*"}, 0, 0, FLAGS, NULL }, + { NULL }, +}; + +static const AVOption showspeaker_options[] = { + { "format", "speaker name formatting", OFFSET(speaker_mode), AV_OPT_TYPE_INT, {.i64=SM_SQUARE_BRACKETS}, 0, 2, FLAGS, "format" }, + { "square_brackets", "[speaker] text", 0, AV_OPT_TYPE_CONST, {.i64=SM_SQUARE_BRACKETS}, 0, 0, FLAGS, "format" }, + { "round_brackets", "(speaker) text", 0, AV_OPT_TYPE_CONST, {.i64=SM_ROUND_BRACKETS}, 0, 0, FLAGS, "format" }, + { "colon", "speaker: text", 0, AV_OPT_TYPE_CONST, {.i64=SM_COLON}, 0, 0, FLAGS, "format" }, + { "plain", "speaker text", 0, AV_OPT_TYPE_CONST, {.i64=SM_PLAIN}, 0, 0, FLAGS, "format" }, + { "line_break", "insert line break", OFFSET(line_break), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, FLAGS, NULL }, + { "style", "ass type name or style code", OFFSET(style), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(textmod); +AVFILTER_DEFINE_CLASS(censor); +AVFILTER_DEFINE_CLASS(showspeaker); + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_frame, + }, +}; + +static const AVFilterPad outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .config_props = config_output, + }, +}; + +const AVFilter ff_sf_textmod = { + .name = "textmod", + .description = NULL_IF_CONFIG_SMALL("Modify subtitle text in several ways"), + .init = init, + .uninit = uninit, + .priv_size = sizeof(TextModContext), + .priv_class = &textmod_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_SINGLE_SUBFMT(AV_SUBTITLE_FMT_ASS), +}; + +const AVFilter ff_sf_censor = { + .name = "censor", + .description = NULL_IF_CONFIG_SMALL("Censor words in subtitle text"), + .init = init_censor, + .uninit = uninit, + .priv_size = sizeof(TextModContext), + .priv_class = &censor_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_SINGLE_SUBFMT(AV_SUBTITLE_FMT_ASS), +}; + +const AVFilter ff_sf_showspeaker = { + .name = "showspeaker", + .description = NULL_IF_CONFIG_SMALL("Prepend speaker names to text subtitles (when available)"), + .init = init_showspeaker, + .uninit = uninit, + .priv_size = sizeof(TextModContext), + .priv_class = &showspeaker_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_SINGLE_SUBFMT(AV_SUBTITLE_FMT_ASS), +}; From patchwork Sat May 28 13:25:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35972 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366783pzj; Sat, 28 May 2022 06:28:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyzmsF/xZAKryA8ZjRQ2mUiPlWyx33koUWjO2BnDHXc5iKljazNqE2kjv4ZQZfPJv5TFZjY X-Received: by 2002:a17:906:49c6:b0:6fe:95bb:93cc with SMTP id w6-20020a17090649c600b006fe95bb93ccmr41473574ejv.30.1653744509459; Sat, 28 May 2022 06:28:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744509; cv=none; d=google.com; s=arc-20160816; b=SSeyHz7zn/9xvp1q7BfD1yrB4ZLY8OweyERqZdJ0dRXKiRqCwnbmeGmDOJHYpWSSIB 6Gf1KSLBjVCAjmWM+G7j6g1LtM36oI2zXwi7cMEVlrNUdkXPDIG9ZI6I6eJuqT3hRlah eaVv/6YoPSOaZ6T2MYqKvNzVOwAdRfIaBathQsc0/ggRf1/N371pfi9ZzcEA8VHY0Biu x9UvVdbiO+VjYr+RsKOTlfBCllmpTvvbUFR3BTk4KjynGkjDG/mMjKqOYCiF7G9Wxrhd MMiIlWtAtcXzcUqePaWr1w2NPb2s0l/zt9WHCwIoSZ/vsSE7ytW7krFbPA4zo7JCjbFl XfVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=FeKIMQQD1cKtUhmonENdDyP5mpGFFi4+FuWCEbT9LVY=; b=b8BZdFeE3UQaCh6hbuNZ9/aERjNh5qJsmXIWc4U6ERgZmZ4VXRTufnroka0J2l7dcG cdN0Q9r8bk7/c5QxkGRS8sOQyG+PWmTR4N9dm9mghhnU36LNuCXJr3irwJGG+l9//Z7p eOWM50aJKFaGQ6P01yvyFkAXtJvyLyJ0BDJvsdjhZZryf929pqPCwtIyvRrd8E8vp9Rj GpN9NkNAm4SooeFGkFRSlY4/f+w105leUyPKstp7tL2FraLGbWth9zZ7NLpYw8O0C3vP /V+mfF8i33Okz2LtCN9e4ukFPCOL6H6BW/Fxa530qIf1Mj7sjJ/LIppssMPAuBAfdQm5 W/dg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=FoPYPZzZ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hg3-20020a1709072cc300b006fea2fbaf6csi4912782ejc.666.2022.05.28.06.28.28; Sat, 28 May 2022 06:28:29 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=FoPYPZzZ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9849E68B649; Sat, 28 May 2022 16:25:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2A1F768B62C for ; Sat, 28 May 2022 16:25:43 +0300 (EEST) Received: by mail-pl1-f179.google.com with SMTP id t4so3544505pld.4 for ; Sat, 28 May 2022 06:25:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IDXegSdLQYPbZSzdUXvKXDHvEEy1soUnnoFR/9F1I1A=; b=FoPYPZzZhD0B3MHBsmWn3o+LYdYcax9kTj7wzVZxTswd3xDZuJtSLZlfggG9AdXbFK oM8HN3a+2DK+eiRiP4ORXCBIE2A/K1UOb+FZWgpz7pe/EEO1sh6U4S5pwbhWb1TbDeEc oL8GDd4weVHcApvtOl8clSq1Py7LEEBQBQTytaerUjMqWhyKrkiV3SaXUynYmvBSizaZ gJ8XeVfvLJpNHrRXkcwQO1AzRUbdLkqT4GZMs0mtEcMNYNlZwXU7+nSQmQcsgKoDis36 p8iWukSJnqsPF5K5WsKDW5FNjpmtPqfoGbC5S75sKYID0HW/43z+fbap6VSfiJo8J3jP kFMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IDXegSdLQYPbZSzdUXvKXDHvEEy1soUnnoFR/9F1I1A=; b=NY4/C6ZkUr4wlRqSQgcoN6QwfvrQ3kfl8JT0IgKB+Gvk1fBizu9JYHaIhvoCnn3LiN V33/kR602UQFe63HtRDwYtWrab66yP3jIfiLUy/WSyvh2oOK2tUrUm//ussji1HeQ7Le O+oJL8xnan37k82yNUIh8FZjPsktCkVambWmKG8XU9KKDNgNZCXwKrzToYgOOaDMof6a dcAo7VTiRIs8JQludIpWMOR+UFxLB+uspkuPDxhugrwbleiH1E9BQi0JJNm2G9lYd3Ic /sZMkF/ELhBCdeS30/iufE5Jyu+UrBEUnZSMlT0YsgwnfZamOLukLosZaX9X1TZidPSe Btvw== X-Gm-Message-State: AOAM531hjF6gYw1ueSmEgRxjuvR5M3XbqqVCrLx1RnthJvBwza5KmsRW o7g9FfuayeBUfg0TW6tEuw/6KM7GqsrufQ== X-Received: by 2002:a17:902:ccc2:b0:15f:4acc:f202 with SMTP id z2-20020a170902ccc200b0015f4accf202mr46787056ple.3.1653744342269; Sat, 28 May 2022 06:25:42 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id d189-20020a6368c6000000b003db8691008esm5182449pgc.12.2022.05.28.06.25.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:41 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: In-Reply-To: References: Date: Sat, 28 May 2022 13:25:16 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 16/23] avfilter/stripstyles: Add stripstyles filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: +C9g01L/vdjz From: softworkz - stripstyles {S -> S) Remove all inline styles from subtitle events Signed-off-by: softworkz --- doc/filters.texi | 37 ++++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/sf_stripstyles.c | 211 +++++++++++++++++++++++++++++++++++ 4 files changed, 250 insertions(+) create mode 100644 libavfilter/sf_stripstyles.c diff --git a/doc/filters.texi b/doc/filters.texi index e0ceba80e2..504001a4e7 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -26947,6 +26947,43 @@ ffmpeg -i "http://streams.videolan.org/samples/sub/SSA/subtitle_testing_complex. @end example @end itemize +@section stripstyles + +Remove all inline styles from subtitle events. + +Inputs: +@itemize +@item 0: Subtitles[TEXT] +@end itemize + +Outputs: +@itemize +@item 0: Subtitles[TEXT] +@end itemize + +It accepts the following parameters: + +@table @option +@item remove_animated +Also remove text which is subject to animation (default: true) +Usually, animated text elements are used used in addition to static subtitle lines for creating effects, so in most cases it is safe to remove the animation content. +If subtitle text is missing, try setting this to false. + +@item select_layer +Process only ASS subtitle events from a specific layer. This allows to filter out certain effects where an ASS author duplicates the text onto multiple layers. + +@end table + +@subsection Examples + +@itemize +@item +Remove styles and animations from ASS subtitles and output events from ass layer 0 only. Then convert asn save as SRT stream: +@example +ffmpeg -i "https://streams.videolan.org/samples/sub/SSA/subtitle_testing_complex.mkv" -filter_complex "[0:1]stripstyles=select_layer=0" -map 0 -c:s srt output.mkv +@end example +@end itemize + @section textmod diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 6d2a75128c..4290903897 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -580,6 +580,7 @@ OBJS-$(CONFIG_NULLSINK_FILTER) += vsink_nullsink.o OBJS-$(CONFIG_CENSOR_FILTER) += sf_textmod.o OBJS-$(CONFIG_SHOW_SPEAKER_FILTER) += sf_textmod.o OBJS-$(CONFIG_TEXTMOD_FILTER) += sf_textmod.o +OBJS-$(CONFIG_STRIPSTYLES_FILTER) += sf_stripstyles.o # multimedia filters OBJS-$(CONFIG_ABITSCOPE_FILTER) += avf_abitscope.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 960726ba08..94ae8c0bee 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -561,6 +561,7 @@ extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_sf_censor; extern const AVFilter ff_sf_showspeaker; +extern const AVFilter ff_sf_stripstyles; extern const AVFilter ff_sf_textmod; extern const AVFilter ff_svf_graphicsub2video; extern const AVFilter ff_svf_textsub2video; diff --git a/libavfilter/sf_stripstyles.c b/libavfilter/sf_stripstyles.c new file mode 100644 index 0000000000..bc3c5d1441 --- /dev/null +++ b/libavfilter/sf_stripstyles.c @@ -0,0 +1,211 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * text subtitle filter which removes inline-styles from subtitles + */ + +#include "libavutil/opt.h" +#include "internal.h" +#include "libavutil/ass_split_internal.h" +#include "libavutil/bprint.h" + +typedef struct StripStylesContext { + const AVClass *class; + enum AVSubtitleType format; + int remove_animated; + int select_layer; +} StripStylesContext; + +typedef struct DialogContext { + StripStylesContext* ss_ctx; + AVBPrint buffer; + int drawing_scale; + int is_animated; +} DialogContext; + +static void dialog_text_cb(void *priv, const char *text, int len) +{ + DialogContext *s = priv; + + av_log(s->ss_ctx, AV_LOG_DEBUG, "dialog_text_cb: %s\n", text); + + if (!s->drawing_scale && (!s->is_animated || !s->ss_ctx->remove_animated)) + av_bprint_append_data(&s->buffer, text, len); +} + +static void dialog_new_line_cb(void *priv, int forced) +{ + DialogContext *s = priv; + if (!s->drawing_scale && !s->is_animated) + av_bprint_append_data(&s->buffer, forced ? "\\N" : "\\n", 2); +} + +static void dialog_drawing_mode_cb(void *priv, int scale) +{ + DialogContext *s = priv; + s->drawing_scale = scale; +} + +static void dialog_animate_cb(void *priv, int t1, int t2, int accel, char *style) +{ + DialogContext *s = priv; + s->is_animated = 1; +} + +static void dialog_move_cb(void *priv, int x1, int y1, int x2, int y2, int t1, int t2) +{ + DialogContext *s = priv; + if (t1 >= 0 || t2 >= 0) + s->is_animated = 1; +} + +static const ASSCodesCallbacks dialog_callbacks = { + .text = dialog_text_cb, + .new_line = dialog_new_line_cb, + .drawing_mode = dialog_drawing_mode_cb, + .animate = dialog_animate_cb, + .move = dialog_move_cb, +}; + +static char *ass_get_line(int readorder, int layer, const char *style, + const char *speaker, const char *effect, const char *text) +{ + return av_asprintf("%d,%d,%s,%s,0,0,0,%s,%s", + readorder, layer, style ? style : "Default", + speaker ? speaker : "", effect, text); +} + +static char *process_dialog(StripStylesContext *s, const char *ass_line) +{ + DialogContext dlg_ctx = { .ss_ctx = s }; + ASSDialog *dialog = avpriv_ass_split_dialog(NULL, ass_line); + char *result = NULL; + + if (!dialog) + return NULL; + + if (s->select_layer >= 0 && dialog->layer != s->select_layer) { + avpriv_ass_free_dialog(&dialog); + return NULL; + } + + dlg_ctx.ss_ctx = s; + + av_bprint_init(&dlg_ctx.buffer, 0, AV_BPRINT_SIZE_UNLIMITED); + + avpriv_ass_split_override_codes(&dialog_callbacks, &dlg_ctx, dialog->text); + + if (av_bprint_is_complete(&dlg_ctx.buffer) + && dlg_ctx.buffer.len > 0) + result = ass_get_line(dialog->readorder, dialog->layer, dialog->style, dialog->name, dialog->effect, dlg_ctx.buffer.str); + + av_bprint_finalize(&dlg_ctx.buffer, NULL); + avpriv_ass_free_dialog(&dialog); + + return result; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterLink *inlink = outlink->src->inputs[0]; + + outlink->w = inlink->w; + outlink->h = inlink->h; + outlink->time_base = inlink->time_base; + outlink->frame_rate = inlink->frame_rate; + + return 0; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + StripStylesContext *s = inlink->dst->priv; + AVFilterLink *outlink = inlink->dst->outputs[0]; + int ret; + + outlink->format = inlink->format; + + ret = av_frame_make_writable(frame); + if (ret <0 ) { + av_frame_free(&frame); + return AVERROR(ENOMEM); + } + + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + + AVSubtitleArea *area = frame->subtitle_areas[i]; + + if (area->ass) { + char *tmp = area->ass; + area->ass = process_dialog(s, area->ass); + + if (area->ass) { + av_log(inlink->dst, AV_LOG_INFO, "original: %d %s\n", i, tmp); + av_log(inlink->dst, AV_LOG_INFO, "stripped: %d %s\n", i, area->ass); + } + else + area->ass = NULL; + + av_free(tmp); + } + } + + return ff_filter_frame(outlink, frame); +} + +#define OFFSET(x) offsetof(StripStylesContext, x) +#define FLAGS (AV_OPT_FLAG_SUBTITLE_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption stripstyles_options[] = { + { "remove_animated", "remove animated text (default: yes)", OFFSET(remove_animated), AV_OPT_TYPE_BOOL, {.i64 = 1 }, 0, 1, FLAGS, 0 }, + { "select_layer", "process a specific ass layer only", OFFSET(remove_animated), AV_OPT_TYPE_INT, {.i64 = -1 }, -1, INT_MAX, FLAGS, 0 }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(stripstyles); + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_frame, + }, +}; + +static const AVFilterPad outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .config_props = config_output, + }, +}; + +const AVFilter ff_sf_stripstyles = { + .name = "stripstyles", + .description = NULL_IF_CONFIG_SMALL("Strip subtitle inline styles"), + .priv_size = sizeof(StripStylesContext), + .priv_class = &stripstyles_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_SINGLE_SUBFMT(AV_SUBTITLE_FMT_ASS), +}; + From patchwork Sat May 28 13:25:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35971 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366736pzj; Sat, 28 May 2022 06:28:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxKtaeOrVtC9yD6RcPmYjKXCtXKwRytE6JzL0kSbkmTDz1kblJGVV/1jnRwfW8Tx3+Zsq9y X-Received: by 2002:a17:906:7d83:b0:6ce:fee:9256 with SMTP id v3-20020a1709067d8300b006ce0fee9256mr43089542ejo.647.1653744499379; Sat, 28 May 2022 06:28:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744499; cv=none; d=google.com; s=arc-20160816; b=DS7THxN/pzeqjD9mYCp8Yg4rCAync9T6fUQF8HT/DIlHBEzRy1ziW7c7FifQOlH8B+ nt8rBxqYK8xPE/umFZeC5amsa43+9zJhYrWQv35EdeZoskVh2UwPg+w7SBKBMuMIqjeF mE45jPdPGvQ9ig8bPwfEQNCW3dndE5xGwda8k0NmNagIFN01Wz9eqH+ogRjcL1b/4mwN yn3FDaTaziZ1eElFTDjlCkm0cVoC3UpUDjOpNsEkqrpLEZNhKbKfNbATd9xBwp/+QaM5 Qi/71lkpk7ByLE76gv0lydKXKfc5BA+TQGmncJPu33flniLGGHbl9eRB2UWUfDKJbAYi 7RNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=BGLb4jno9eTcv7kMzBXxkrRD2beLXcSRg5CXGzsSoFA=; b=Yv5k11BZ9UpXg8VjrY1GQ14u90+ed5b5dbjhFndKM8uQs9XBCoHAM0Ffze2DiUW0Zy b6fQS83FLi29DKVcR8Qd6FIguks22zzaD42RFawUosuUA9Vt6emFP6EU2PAa1BwaFUe+ d177ejzfaWXj0m6PLmkjZ/6/OpQWbNrbBC7pcJX5ewWqbm+ME77/cZ5YRx1pQleAxNbZ v3QuzrVis506tM0tQbGqOReOM3UoQkrPSZgovNdWiiyWcQpCE8UJb843RBrb5mUldGD5 T3FZiDC++YtXugQupurRzcs0CJwU500RTH4tx2rkLKjQc8NFTxJNKXsJJ8a+lVN3f6LI cqsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=KyFRRywb; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j24-20020aa7c418000000b0042abbcd3c60si5925498edq.362.2022.05.28.06.28.18; Sat, 28 May 2022 06:28:19 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=KyFRRywb; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B1E1A68B62E; Sat, 28 May 2022 16:25:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D479868B62C for ; Sat, 28 May 2022 16:25:43 +0300 (EEST) Received: by mail-pf1-f169.google.com with SMTP id 202so6680174pfu.0 for ; Sat, 28 May 2022 06:25:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=dfVdUYQMul27Bw5pH3GiYkFlJy7qp+b8RxyNIMRCFl4=; b=KyFRRywbrg98fl5n+Wr5NAIx1RT38PFexC5xaEQzVbCaTb8W51GrKVGBDemc2p4k7h iSuqlFJZVJIhsz5nIZgowm3RuNyNMSGO6g8SuzVxE3Ah8e5IurmXb4N9HkHP3PCSqVGh ZZew1c50Hvruo9+thnBObiXfPP6R8tmfrHIj/ks8d06KFuCorYQobjEr2CdiTX5Q67S/ 7qZVgcTKY2/UnYMZ20KDt22hLMv171bOhURlpMfuHnAqzsWAM02GI97Uzj4xjyuQXV/m 8Ztp3+nlHNEZlOiKfgGAoV7G8xBBiHWC8zs6qv5SIHa1O9BEHAzGxN3fUpk2sSgDEUsI 5HWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=dfVdUYQMul27Bw5pH3GiYkFlJy7qp+b8RxyNIMRCFl4=; b=yGISR/sI8s1ZZu9Y2IpA4lTVnYQcGcmNvQK9/R3b+PcsojL58lbLpVNo0Vklymiisk 9EYyM8oMBbNAkSnugIINmWUp45/E/RtNj083VvewdwP++cF8EFkn7GU0o2euUvAvsJOV IDXCIBv9Flv0MJvkhrdGxLTHVnLZRqt3pNwJ06U+6fdFfATaCCnLnLatbyLfg/W8i8IK YLG4p3lwSEn99DJMfnea/B56fbVrj4nvwlVzpXmwvUNscIxYU+rudIZqTub4wEECGE2M XtG+FFqvsF+nRWRjRoH1gPqqS3pe8Ia8PpI/VvJDzCe1Gl88uqy0xMQTXf/TMx18JV5w tQcg== X-Gm-Message-State: AOAM533CUS1v04rLe3FiaKp0iV5zYayhZIYu/5qECbjwvzAKpkN1+ubw oDRdSPp55lU5Soi5MTs5E6d4GXToDLTMrg== X-Received: by 2002:a63:6904:0:b0:3c6:5a3c:64bd with SMTP id e4-20020a636904000000b003c65a3c64bdmr41058375pgc.371.1653744343195; Sat, 28 May 2022 06:25:43 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id o1-20020a170902d4c100b0015e8d4eb20esm1661197plg.88.2022.05.28.06.25.42 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:42 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <28d75dc982a11e7e85224bc1cb0b2d891a06d9b2.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:17 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 17/23] avfilter/splitcc: Add splitcc filter for closed caption handling X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: wneDJtaXy9KO From: softworkz - splitcc {V -> VS) Extract closed-caption (A53) data from video frames as subtitle Frames ffmpeg -y -loglevel verbose -i "https://streams.videolan.org/streams /ts/CC/NewsStream-608-ac3.ts" -filter_complex "[0:v]splitcc[vid1], textmod=mode=remove_chars:find='@',[vid1]overlay_textsubs" output.mkv Signed-off-by: softworkz --- configure | 1 + doc/filters.texi | 63 +++++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/sf_splitcc.c | 395 +++++++++++++++++++++++++++++++++++++++ 5 files changed, 461 insertions(+) create mode 100644 libavfilter/sf_splitcc.c diff --git a/configure b/configure index 39c4e94a1f..a0c64500aa 100755 --- a/configure +++ b/configure @@ -3727,6 +3727,7 @@ spp_filter_select="fft idctdsp fdctdsp me_cmp pixblockdsp" sr_filter_deps="avformat swscale" sr_filter_select="dnn" stereo3d_filter_deps="gpl" +splitcc_filter_deps="avcodec" subtitles_filter_deps="avformat avcodec libass" super2xsai_filter_deps="gpl" pixfmts_super2xsai_test_deps="super2xsai_filter" diff --git a/doc/filters.texi b/doc/filters.texi index 504001a4e7..6662d09a82 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -27297,6 +27297,69 @@ ffmpeg -i INPUT -filter_complex "showspeaker=format=colon:style='@{\\c&HDD0000&\ @end example @end itemize + +@section splitcc + +Split-out closed-caption/A53 subtitles from video frame side data. + +This filter provides an input and an output for video frames, which are just passed through without modification. +The second out provides subtitle frames which are extracted from video frame side data. + +Inputs: +@itemize +@item 0: Video [ALL] +@end itemize + +Outputs: +@itemize +@item 0: Video (same as input) +@item 1: Subtitles [TEXT] +@end itemize + +It accepts the following parameters: + +@table @option + +@item use_cc_styles +Emit closed caption style header. +This will make closed captions appear in white font with a black rectangle background. + +@item real_time +Emit subtitle events as they are decoded for real-time display. + +@item real_time_latency_msec +Minimum elapsed time between emitting real-time subtitle events. +Only applies to real_time mode. + +@item data_field +Select data field. Possible values: + +@table @samp +@item auto +Pick first one that appears. +@item first +@item second +@end table + +@end table + +@subsection Examples + +@itemize +@item +Extract closed captions as text subtitle stream and overlay it onto the video in cc style (black bar background): +@example +ffmpeg -i "https://streams.videolan.org/streams/ts/CC/NewsStream-608-ac3.ts" -filter_complex "[0:v:0]splitcc=use_cc_styles=1[vid1][sub1];[vid1][sub1]overlaytextsubs" output.mkv +@end example + +@item +A nicer variant, using realtime output from cc_dec and rendering it with the render_latest_only parameter from overlaytextsubs to avoid ghosting by timely overlap. +@example +ffmpeg -i "https://streams.videolan.org/streams/ts/CC/NewsStream-608-ac3.ts" -filter_complex "[0:v:0]splitcc=real_time=1:real_time_latency_msec=200[vid1][sub1];[vid1][sub1]overlaytextsubs=render_latest_only=1" output.mkv +@end example +@end itemize + + @section textsub2video Converts text subtitles to video frames. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 4290903897..988c2fa692 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -580,6 +580,7 @@ OBJS-$(CONFIG_NULLSINK_FILTER) += vsink_nullsink.o OBJS-$(CONFIG_CENSOR_FILTER) += sf_textmod.o OBJS-$(CONFIG_SHOW_SPEAKER_FILTER) += sf_textmod.o OBJS-$(CONFIG_TEXTMOD_FILTER) += sf_textmod.o +OBJS-$(CONFIG_SPLITCC_FILTER) += sf_splitcc.o OBJS-$(CONFIG_STRIPSTYLES_FILTER) += sf_stripstyles.o # multimedia filters diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 94ae8c0bee..ee9d758d86 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -561,6 +561,7 @@ extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_sf_censor; extern const AVFilter ff_sf_showspeaker; +extern const AVFilter ff_sf_splitcc; extern const AVFilter ff_sf_stripstyles; extern const AVFilter ff_sf_textmod; extern const AVFilter ff_svf_graphicsub2video; diff --git a/libavfilter/sf_splitcc.c b/libavfilter/sf_splitcc.c new file mode 100644 index 0000000000..14235e822c --- /dev/null +++ b/libavfilter/sf_splitcc.c @@ -0,0 +1,395 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * subtitle filter for splitting out closed-caption/A53 subtitles from video frame side data + */ + +#include "filters.h" +#include "libavutil/opt.h" +#include "subtitles.h" +#include "libavcodec/avcodec.h" + +static const AVRational ms_tb = {1, 1000}; + +typedef struct SplitCaptionsContext { + const AVClass *class; + enum AVSubtitleType format; + AVCodecContext *cc_dec; + int eof; + AVFrame *next_sub_frame; + AVFrame *empty_sub_frame; + int new_frame; + int64_t next_repetition_pts; + int had_keyframe; + AVBufferRef *subtitle_header; + int use_cc_styles; + int real_time; + int real_time_latency_msec; + int data_field; + int scatter_realtime_output; +} SplitCaptionsContext; + +static int64_t ms_to_avtb(int64_t ms) +{ + return av_rescale_q(ms, (AVRational){ 1, 1000 }, AV_TIME_BASE_Q); +} + +static int init(AVFilterContext *ctx) +{ + SplitCaptionsContext *s = ctx->priv; + AVDictionary *options = NULL; + + int ret; + const AVCodec *codec = avcodec_find_decoder(AV_CODEC_ID_EIA_608); + if (!codec) { + av_log(ctx, AV_LOG_ERROR, "failed to find EIA-608/708 decoder\n"); + return AVERROR_DECODER_NOT_FOUND; + } + + if (!((s->cc_dec = avcodec_alloc_context3(codec)))) { + av_log(ctx, AV_LOG_ERROR, "failed to allocate EIA-608/708 decoder\n"); + return AVERROR(ENOMEM); + } + + av_dict_set_int(&options, "real_time", s->real_time, 0); + av_dict_set_int(&options, "real_time_latency_msec", s->real_time_latency_msec, 0); + av_dict_set_int(&options, "data_field", s->data_field, 0); + + if ((ret = avcodec_open2(s->cc_dec, codec, &options)) < 0) { + av_log(ctx, AV_LOG_ERROR, "failed to open EIA-608/708 decoder: %i\n", ret); + return ret; + } + + if (s->use_cc_styles && s->cc_dec->subtitle_header && s->cc_dec->subtitle_header[0] != 0) { + char* subtitle_header = av_strdup((char *)s->cc_dec->subtitle_header); + if (!subtitle_header) + return AVERROR(ENOMEM); + s->subtitle_header = av_buffer_create((uint8_t *)subtitle_header, strlen(subtitle_header) + 1, NULL, NULL, 0); + if (!s->subtitle_header) { + av_free(subtitle_header); + return AVERROR(ENOMEM); + } + } + + return 0; +} + +static void uninit(AVFilterContext *ctx) +{ + SplitCaptionsContext *s = ctx->priv; + av_frame_free(&s->next_sub_frame); + av_frame_free(&s->empty_sub_frame); + av_buffer_unref(&s->subtitle_header); +} + +static int config_input(AVFilterLink *link) +{ + const SplitCaptionsContext *context = link->dst->priv; + + if (context->cc_dec) + context->cc_dec->pkt_timebase = link->time_base; + + return 0; +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink0 = ctx->inputs[0]; + AVFilterLink *outlink0 = ctx->outputs[0]; + AVFilterLink *outlink1 = ctx->outputs[1]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input0 video formats */ + formats = ff_all_formats(AVMEDIA_TYPE_VIDEO); + if ((ret = ff_formats_ref(formats, &inlink0->outcfg.formats)) < 0) + return ret; + + /* set output0 video formats */ + if ((ret = ff_formats_ref(formats, &outlink0->incfg.formats)) < 0) + return ret; + + /* set output1 subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &outlink1->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_video_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + const AVFilterLink *inlink = outlink->src->inputs[0]; + + outlink->w = ctx->inputs[0]->w; + outlink->h = ctx->inputs[0]->h; + outlink->frame_rate = ctx->inputs[0]->frame_rate; + outlink->time_base = ctx->inputs[0]->time_base; + outlink->sample_aspect_ratio = ctx->inputs[0]->sample_aspect_ratio; + + if (inlink->hw_frames_ctx) + outlink->hw_frames_ctx = av_buffer_ref(inlink->hw_frames_ctx); + + return 0; +} + +static int config_sub_output(AVFilterLink *outlink) +{ + SplitCaptionsContext *s = outlink->src->priv; + const AVFilterLink *inlink = outlink->src->inputs[0]; + + outlink->time_base = inlink->time_base; + outlink->format = AV_SUBTITLE_FMT_ASS; + outlink->frame_rate = (AVRational){1000, s->real_time_latency_msec}; + + return 0; +} + +static int request_sub_frame(AVFilterLink *outlink) +{ + SplitCaptionsContext *s = outlink->src->priv; + int status; + int64_t pts; + + if (!s->empty_sub_frame) { + s->empty_sub_frame = ff_get_subtitles_buffer(outlink, outlink->format); + if (!s->empty_sub_frame) + return AVERROR(ENOMEM); + } + + if (!s->eof && ff_inlink_acknowledge_status(outlink->src->inputs[0], &status, &pts)) { + if (status == AVERROR_EOF) + s->eof = 1; + } + + if (s->eof) + return AVERROR_EOF; + + if (s->next_sub_frame) { + + AVFrame *out = NULL; + s->next_sub_frame->pts++; + + if (s->new_frame) + out = av_frame_clone(s->next_sub_frame); + else if (s->empty_sub_frame) { + s->empty_sub_frame->pts = s->next_sub_frame->pts; + out = av_frame_clone(s->empty_sub_frame); + av_frame_copy_props(out, s->next_sub_frame); + out->repeat_sub = 1; + } + + if (!out) + return AVERROR(ENOMEM); + + out->subtitle_timing.start_pts = av_rescale_q(s->next_sub_frame->pts, outlink->time_base, AV_TIME_BASE_Q); + s->new_frame = 0; + + return ff_filter_frame(outlink, out); + } + + return 0; +} + +static int decode(AVCodecContext *avctx, AVFrame *frame, int *got_frame, AVPacket *pkt) +{ + int ret; + + *got_frame = 0; + + if (pkt) { + ret = avcodec_send_packet(avctx, pkt); + // In particular, we don't expect AVERROR(EAGAIN), because we read all + // decoded frames with avcodec_receive_frame() until done. + if (ret < 0 && ret != AVERROR_EOF) + return ret; + } + + ret = avcodec_receive_frame(avctx, frame); + if (ret < 0 && ret != AVERROR(EAGAIN)) + return ret; + if (ret >= 0) + *got_frame = 1; + + return 0; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFrameSideData *sd; + SplitCaptionsContext *s = inlink->dst->priv; + AVFilterLink *outlink0 = inlink->dst->outputs[0]; + AVFilterLink *outlink1 = inlink->dst->outputs[1]; + AVPacket *pkt = NULL; + AVFrame *sub_out = NULL; + + int ret; + + outlink0->format = inlink->format; + + sd = av_frame_get_side_data(frame, AV_FRAME_DATA_A53_CC); + + if (sd && (s->had_keyframe || frame->key_frame)) { + int got_output = 0; + + s->had_keyframe = 1; + pkt = av_packet_alloc(); + pkt->buf = av_buffer_ref(sd->buf); + if (!pkt->buf) { + ret = AVERROR(ENOMEM); + goto fail; + } + + pkt->data = pkt->buf->data; + pkt->size = pkt->buf->size; + pkt->pts = av_rescale_q(frame->pts, inlink->time_base, AV_TIME_BASE_Q); + + sub_out = ff_get_subtitles_buffer(outlink1, AV_SUBTITLE_FMT_ASS); + if (!sub_out) { + ret = AVERROR(ENOMEM); + goto fail; + } + + if ((ret = av_buffer_replace(&sub_out->subtitle_header, s->subtitle_header)) < 0) + goto fail; + + ret = decode(s->cc_dec, sub_out, &got_output, pkt); + + ////if (got_output) { + //// av_log(inlink->dst, AV_LOG_INFO, "CC Packet PTS: %"PRId64" got_output: %d out_frame_pts: %"PRId64" out_sub_pts: %"PRId64"\n", pkt->pts, got_output, sub_out->pts, sub_out->subtitle_pts); + ////} + + av_packet_free(&pkt); + + if (ret < 0) { + av_log(inlink->dst, AV_LOG_ERROR, "Decode error: %d \n", ret); + goto fail; + } + + if (got_output) { + sub_out->pts = frame->pts; + av_frame_free(&s->next_sub_frame); + s->next_sub_frame = sub_out; + sub_out = NULL; + s->new_frame = 1; + s->next_sub_frame->pts = frame->pts; + + if ((ret = av_buffer_replace(&s->next_sub_frame->subtitle_header, s->subtitle_header)) < 0) + goto fail; + + if (s->real_time && s->scatter_realtime_output) { + if (s->next_repetition_pts) + s->next_sub_frame->pts = s->next_repetition_pts; + + s->next_sub_frame->subtitle_timing.duration = ms_to_avtb(s->real_time_latency_msec); + s->next_repetition_pts = s->next_sub_frame->pts + av_rescale_q(s->real_time_latency_msec, ms_tb, inlink->time_base); + } + + ret = request_sub_frame(outlink1); + if (ret < 0) + goto fail; + } + } + + if (s->real_time && s->scatter_realtime_output && !s->new_frame && s->next_repetition_pts > 0 && frame->pts > s->next_repetition_pts) { + s->new_frame = 1; + s->next_sub_frame->pts = s->next_repetition_pts; + s->next_repetition_pts = s->next_sub_frame->pts + av_rescale_q(s->real_time_latency_msec, ms_tb, inlink->time_base); + } + + if (!s->next_sub_frame) { + s->next_sub_frame = ff_get_subtitles_buffer(outlink1, outlink1->format); + if (!s->next_sub_frame) { + ret = AVERROR(ENOMEM); + goto fail; + } + + s->next_sub_frame->subtitle_timing.duration = ms_to_avtb(s->real_time_latency_msec); + s->next_sub_frame->pts = frame->pts; + s->new_frame = 1; + + if ((ret = av_buffer_replace(&s->next_sub_frame->subtitle_header, s->subtitle_header)) < 0) + goto fail; + } + + ret = ff_filter_frame(outlink0, frame); + +fail: + av_packet_free(&pkt); + av_frame_free(&sub_out); + return ret; +} + +#define OFFSET(x) offsetof(SplitCaptionsContext, x) +#define FLAGS (AV_OPT_FLAG_SUBTITLE_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption split_cc_options[] = { + { "use_cc_styles", "Emit closed caption style header", OFFSET(use_cc_styles), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, FLAGS, NULL }, + { "real_time", "emit subtitle events as they are decoded for real-time display", OFFSET(real_time), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, + { "real_time_latency_msec", "minimum elapsed time between emitting real-time subtitle events", OFFSET(real_time_latency_msec), AV_OPT_TYPE_INT, { .i64 = 200 }, 0, 500, FLAGS }, + { "scatter_realtime_output", "scatter output events to a duration of real_time_latency_msec", OFFSET(scatter_realtime_output), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, + { "data_field", "select data field", OFFSET(data_field), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, 1, FLAGS, "data_field" }, + { "auto", "pick first one that appears", 0, AV_OPT_TYPE_CONST, { .i64 =-1 }, 0, 0, FLAGS, "data_field" }, + { "first", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, 0, 0, FLAGS, "data_field" }, + { "second", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, "data_field" }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(split_cc); + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + .config_props = config_input, + }, +}; + +static const AVFilterPad outputs[] = { + { + .name = "video_passthrough", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_video_output, + }, + { + .name = "subtitles", + .type = AVMEDIA_TYPE_SUBTITLE, + .request_frame = request_sub_frame, + .config_props = config_sub_output, + }, +}; + +const AVFilter ff_sf_splitcc = { + .name = "splitcc", + .description = NULL_IF_CONFIG_SMALL("Extract closed-caption (A53) data from video as subtitle stream."), + .init = init, + .uninit = uninit, + .priv_size = sizeof(SplitCaptionsContext), + .priv_class = &split_cc_class, + .flags_internal = FF_FILTER_FLAG_HWFRAME_AWARE, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_QUERY_FUNC(query_formats), +}; From patchwork Sat May 28 13:25:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35973 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1366822pzj; Sat, 28 May 2022 06:28:39 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzEMFoJgkiwICJlO+rLga7y3cGNRMH/zItxGExDyEzyQJ5Q8mJFukkZFMLfdGk6E0QfPs7A X-Received: by 2002:a17:906:3a92:b0:6fe:9029:b62c with SMTP id y18-20020a1709063a9200b006fe9029b62cmr42071031ejd.569.1653744518918; Sat, 28 May 2022 06:28:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653744518; cv=none; d=google.com; s=arc-20160816; b=0Gdug7vt1Ma4hj5c+R9SsAb26c01/bLnQFqxLJ+//qq7sTqfp7BeYh9NT0+fUdUFHG lF+hXSB3KCnyeCDXyPHCCcE4or0u3qt2lkt4twF6WYTJTtAg02/Ulikhav5Jn9rVmkTD Hxphl2hJVhAufOxQnw2jRF4Yhc8zWaFCH6dAKtYCOLdnoyVqIOUuw1luavoyXEZDFtPy SQs5a+bGUwhkbj54LxbtyDahTEfatKBHQPrrMjAGFfv6GbK3SpzF+zYuBdprPaspIih5 wJByjjH480hNaNMe01NWEPmGkoP6letFTshpRo1ld3ijpBjdGfE2MLplcqng3h2AwmDL USTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=Nsy2eGS6xZgbcV0X6QBAUfvzNlMvV/kyYztFRLnHLNk=; b=i9roZDGgdUp0wUrLPtrFzgKYtp9ZoOv6XRAz478hH8IZCrlRmZfD2/3Cf8sgPf+1Sf 3tQjnqiZJxfCX5GEbUHKCyNd16JSxj4T8Pc1Rhc5Anhh1IXZP95+feDk4d7YJH1iNPS2 fM+SjSV2+JxsCoqI290TCxm3A7zMGtR8lJpVAELZ8GfFxn08/fNFRgSwbeltkikCkvlN J8MjMvC8nm+dVQmipqOeZKpB3R3EPn0CWhvgoZwMvh5rWUcQ/+frqIP7Zc1LSjHCu8gv LmsQaO3l0RgkeojJC7nFhh2zMVIVX/lG3WtUwoZd140H0fJIYllkfuoUx3j88q7GKn6b F3ZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=fUMq83qn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id dd21-20020a1709069b9500b006fee18968b4si4737001ejc.483.2022.05.28.06.28.38; Sat, 28 May 2022 06:28:38 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=fUMq83qn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6824B68B664; Sat, 28 May 2022 16:25:53 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6418B68B607 for ; Sat, 28 May 2022 16:25:46 +0300 (EEST) Received: by mail-pj1-f46.google.com with SMTP id qe5-20020a17090b4f8500b001e26126abccso4162701pjb.0 for ; Sat, 28 May 2022 06:25:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=oRf2EBhQJo3v5/ogOSHNiEb3seqpurEMRYUhZ3NvO50=; b=fUMq83qnrSxJi82zPfvh6il/w0oARFhJ5O2o/KYhqEGLpDKkz39dHuoEUiCQD4EKt+ e2UiDaiSM6vu8sPn9m4I1Nne7f9E0i4wNgit8iz5dWCzyZs5+VXsprytKlSHSLVHFpbf BJrxUwR8ShBINzbTr1XU8+5qzpGjMQsBcCxFZ6ENSF8PRFtMwqbL/RFjHKsWfYt4hEbN tXcRoVYWEovvQ+j+EDIlfv2/ze5J33aIMHknyvzQAh8QIs9+5O/ZMItXtvFOplIIFPij eqVeWnTZInnx033YlHzQy2AgMnn3FqHp8+QjYx5VVjZRqTJpvyG7RDATzK9bs/qvU9y8 u4bQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=oRf2EBhQJo3v5/ogOSHNiEb3seqpurEMRYUhZ3NvO50=; b=M6WIMgvfrU+ZBjVGLNTreYsPfVco/JvX+1lC3mG5qHcZ+fW69Hs8zUdkZqFxYyQzda 34Fvsbd24dkYLg9ycwhrgdAjWrWNix15GQOpz5JU+UAOhMq8VrTyPojUyeTBgyFscMBv iFVphPXuAakK0IgpC8tRaTEEmLpIPbgCpce/fpdIr97QLMoGy2L2NpWvhxVD5IWJI3nr I6rFNADQcs6U1wOsY0fCK4CdlFAXuJC36XlZs+QhocJfLzb50bCF8jmSlkxhNubNaSne fzk0lm55T78nvtL+6VvwqdJr8+jMyXpJLuOldNP6wHZJe1/X3+qBRyERERPZshdu9dYH nnUQ== X-Gm-Message-State: AOAM532B/OOFYlPbn+ZuMqJmC2JgcSfgfWCBJNHhonhQAvJCWznS5Zi8 XAD96fBCXrSesxe+VjTvVZkX8JzZEQIR2Q== X-Received: by 2002:a17:902:7b8f:b0:162:467:db7c with SMTP id w15-20020a1709027b8f00b001620467db7cmr35495653pll.140.1653744344185; Sat, 28 May 2022 06:25:44 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id bg16-20020a056a02011000b003c62fa02f08sm5268895pgb.43.2022.05.28.06.25.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 May 2022 06:25:43 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <42d1d1c819a6161c2a7958c4bb76b61da26ed695.1653744323.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sat, 28 May 2022 13:25:18 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 18/23] avfilter/graphicsub2text: Add new graphicsub2text filter (OCR) X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Michael Niedermayer , softworkz , Andriy Gelman , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: HuHrvIx4ShR9 From: softworkz Signed-off-by: softworkz --- configure | 1 + doc/filters.texi | 55 ++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/sf_graphicsub2text.c | 1132 ++++++++++++++++++++++++++++++ 5 files changed, 1190 insertions(+) create mode 100644 libavfilter/sf_graphicsub2text.c diff --git a/configure b/configure index a0c64500aa..e9e1fc0966 100755 --- a/configure +++ b/configure @@ -3662,6 +3662,7 @@ frei0r_filter_deps="frei0r" frei0r_src_filter_deps="frei0r" fspp_filter_deps="gpl" gblur_vulkan_filter_deps="vulkan spirv_compiler" +graphicsub2text_filter_deps="libtesseract" hflip_vulkan_filter_deps="vulkan spirv_compiler" histeq_filter_deps="gpl" hqdn3d_filter_deps="gpl" diff --git a/doc/filters.texi b/doc/filters.texi index 6662d09a82..8b823d4e12 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -27063,6 +27063,61 @@ ffmpeg -i "https://streams.videolan.org/ffmpeg/mkv_subtitles.mkv" -filter_comple @end example @end itemize +@section graphicsub2text + +Converts graphic subtitles to text subtitles by performing OCR. + +For this filter to be available, ffmpeg needs to be compiled with libtesseract (see https://github.com/tesseract-ocr/tesseract). +Language models need to be downloaded from https://github.com/tesseract-ocr/tessdata and put into as subfolder named 'tessdata' or into a folder specified via the environment variable 'TESSDATA_PREFIX'. +The path can also be specified via filter option (see below). + +Note: These models are including the data for both OCR modes. + +Inputs: +- 0: Subtitles [bitmap] + +Outputs: +- 0: Subtitles [text] + +It accepts the following parameters: + +@table @option +@item ocr_mode +The character recognition mode to use. + +Supported OCR modes are: + +@table @var +@item 0, tesseract +This is the classic libtesseract operation mode. It is fast but less accurate than LSTM. +@item 1, lstm +Newer OCR implementation based on ML models. Provides usually better results, requires more processing resources. +@item 2, both +Use a combination of both modes. +@end table + +@item tessdata_path +The path to a folder containing the language models to be used. + +@item language +The recognition language. It needs to match the first three characters of a language model file in the tessdata path. + +@end table + + +@subsection Examples + +@itemize +@item +Convert DVB graphic subtitles to ASS (text) subtitles + +Note: For this to work, you need to have the data file 'eng.traineddata' in a 'tessdata' subfolder (see above). +@example +ffmpeg -loglevel verbose -i "https://streams.videolan.org/streams/ts/video_subs_ttxt%2Bdvbsub.ts" -filter_complex "[0:13]graphicsub2text=delay_when_no_duration=1" -c:s ass -y output.mkv +@end example +@end itemize + + @section graphicsub2video Renders graphic subtitles as video frames. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 988c2fa692..66a3b73e33 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -306,6 +306,7 @@ OBJS-$(CONFIG_GBLUR_FILTER) += vf_gblur.o OBJS-$(CONFIG_GBLUR_VULKAN_FILTER) += vf_gblur_vulkan.o vulkan.o vulkan_filter.o OBJS-$(CONFIG_GEQ_FILTER) += vf_geq.o OBJS-$(CONFIG_GRADFUN_FILTER) += vf_gradfun.o +OBJS-$(CONFIG_GRAPHICSUB2TEXT_FILTER) += sf_graphicsub2text.o OBJS-$(CONFIG_GRAPHICSUB2VIDEO_FILTER) += vf_overlaygraphicsubs.o framesync.o OBJS-$(CONFIG_GRAPHMONITOR_FILTER) += f_graphmonitor.o OBJS-$(CONFIG_GRAYWORLD_FILTER) += vf_grayworld.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index ee9d758d86..46d38ef2cd 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -560,6 +560,7 @@ extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_sf_censor; +extern const AVFilter ff_sf_graphicsub2text; extern const AVFilter ff_sf_showspeaker; extern const AVFilter ff_sf_splitcc; extern const AVFilter ff_sf_stripstyles; diff --git a/libavfilter/sf_graphicsub2text.c b/libavfilter/sf_graphicsub2text.c new file mode 100644 index 0000000000..9b413d314e --- /dev/null +++ b/libavfilter/sf_graphicsub2text.c @@ -0,0 +1,1132 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * subtitle filter to convert graphical subs to text subs via OCR + */ + +#include +#include + +#include "drawutils.h" +#include "libavutil/opt.h" +#include "subtitles.h" + +#include "libavcodec/elbg.h" + +enum { + RFLAGS_NONE = 0, + RFLAGS_HALIGN = 1 << 0, + RFLAGS_VALIGN = 1 << 1, + RFLAGS_FBOLD = 1 << 2, + RFLAGS_FITALIC = 1 << 3, + RFLAGS_FUNDERLINE = 1 << 4, + RFLAGS_FONT = 1 << 5, + RFLAGS_FONTSIZE = 1 << 6, + RFLAGS_COLOR = 1 << 7, + RFLAGS_OUTLINECOLOR = 1 << 8, + RFLAGS_ALL = RFLAGS_HALIGN | RFLAGS_VALIGN | RFLAGS_FBOLD | RFLAGS_FITALIC | RFLAGS_FUNDERLINE | + RFLAGS_FONT | RFLAGS_FONTSIZE | RFLAGS_COLOR | RFLAGS_OUTLINECOLOR, +}; + +typedef struct SubOcrContext { + const AVClass *class; + int w, h; + + TessBaseAPI *tapi; + TessOcrEngineMode ocr_mode; + char *tessdata_path; + char *language; + int preprocess_images; + int dump_bitmaps; + int delay_when_no_duration; + int recognize; + double font_size_factor; + + int readorder_counter; + + AVFrame *pending_frame; + AVBufferRef *subtitle_header; + AVBPrint buffer; + + // Color Quantization Fields + struct ELBGContext *ctx; + AVLFG lfg; + int *codeword; + int *codeword_closest_codebook_idxs; + int *codebook; + int r_idx, g_idx, b_idx, a_idx; + int64_t last_subtitle_pts; +} SubOcrContext; + +typedef struct OcrImageProps { + int background_color_index; + int fill_color_index; + +} OcrImageProps; + +static int64_t ms_to_avtb(int64_t ms) +{ + return av_rescale_q(ms, (AVRational){ 1, 1000 }, AV_TIME_BASE_Q); +} + +static int create_ass_header(AVFilterContext* ctx) +{ + SubOcrContext* s = ctx->priv; + + if (!(s->w && s->h)) { + av_log(ctx, AV_LOG_WARNING, "create_ass_header: no width and height specified!\n"); + s->w = ASS_DEFAULT_PLAYRESX; + s->h = ASS_DEFAULT_PLAYRESY; + } + + char* subtitle_header_text = avpriv_ass_get_subtitle_header_full(s->w, s->h, ASS_DEFAULT_FONT, ASS_DEFAULT_FONT_SIZE, + ASS_DEFAULT_COLOR, ASS_DEFAULT_COLOR, ASS_DEFAULT_BACK_COLOR, ASS_DEFAULT_BACK_COLOR, ASS_DEFAULT_BOLD, + ASS_DEFAULT_ITALIC, ASS_DEFAULT_UNDERLINE, ASS_DEFAULT_BORDERSTYLE, ASS_DEFAULT_ALIGNMENT, 0); + + if (!subtitle_header_text) + return AVERROR(ENOMEM); + + s->subtitle_header = av_buffer_create((uint8_t*)subtitle_header_text, strlen(subtitle_header_text) + 1, NULL, NULL, 0); + + if (!s->subtitle_header) + return AVERROR(ENOMEM); + + return 0; +} + +static int init(AVFilterContext *ctx) +{ + SubOcrContext *s = ctx->priv; + const char* tver = TessVersion(); + uint8_t rgba_map[4]; + int ret; + + s->tapi = TessBaseAPICreate(); + + if (!s->tapi || !tver || !strlen(tver)) { + av_log(ctx, AV_LOG_ERROR, "Failed to access libtesseract\n"); + return AVERROR(ENOSYS); + } + + av_log(ctx, AV_LOG_VERBOSE, "Initializing libtesseract, version: %s\n", tver); + + ret = TessBaseAPIInit4(s->tapi, s->tessdata_path, s->language, s->ocr_mode, NULL, 0, NULL, NULL, 0, 1); + if (ret < 0 ) { + av_log(ctx, AV_LOG_ERROR, "Failed to initialize libtesseract. Error: %d\n", ret); + return AVERROR(ENOSYS); + } + + ret = TessBaseAPISetVariable(s->tapi, "tessedit_char_blacklist", "|"); + if (ret < 0 ) { + av_log(ctx, AV_LOG_ERROR, "Failed to set 'tessedit_char_blacklist'. Error: %d\n", ret); + return AVERROR(EINVAL); + } + + av_bprint_init(&s->buffer, 0, AV_BPRINT_SIZE_UNLIMITED); + + ff_fill_rgba_map(&rgba_map[0], AV_PIX_FMT_RGB32); + + s->r_idx = rgba_map[0]; // R + s->g_idx = rgba_map[1]; // G + s->b_idx = rgba_map[2]; // B + s->a_idx = rgba_map[3]; // A + + av_lfg_init(&s->lfg, 123456789); + + return 0; +} + +static void uninit(AVFilterContext *ctx) +{ + SubOcrContext *s = ctx->priv; + + av_buffer_unref(&s->subtitle_header); + av_bprint_finalize(&s->buffer, NULL); + + if (s->tapi) { + TessBaseAPIEnd(s->tapi); + TessBaseAPIDelete(s->tapi); + } + + avpriv_elbg_free(&s->ctx); +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats, *formats2; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType in_fmts[] = { AV_SUBTITLE_FMT_BITMAP, AV_SUBTITLE_FMT_NONE }; + static const enum AVSubtitleType out_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input format */ + formats = ff_make_format_list(in_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output format */ + formats2 = ff_make_format_list(out_fmts); + if ((ret = ff_formats_ref(formats2, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + SubOcrContext *s = ctx->priv; + + if (s->w <= 0 || s->h <= 0) { + s->w = inlink->w; + s->h = inlink->h; + } + + return create_ass_header(ctx); +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterLink *inlink = outlink->src->inputs[0]; + const AVFilterContext *ctx = outlink->src; + SubOcrContext *s = ctx->priv; + + outlink->format = AV_SUBTITLE_FMT_ASS; + outlink->w = s->w; + outlink->h = s->h; + outlink->time_base = inlink->time_base; + outlink->frame_rate = inlink->frame_rate; + + return 0; +} + +static void free_subtitle_area(AVSubtitleArea *area) +{ + for (unsigned n = 0; n < FF_ARRAY_ELEMS(area->buf); n++) + av_buffer_unref(&area->buf[n]); + + av_freep(&area->text); + av_freep(&area->ass); + av_free(area); + +} + +static AVSubtitleArea *copy_subtitle_area(const AVSubtitleArea *src) +{ + AVSubtitleArea *dst = av_mallocz(sizeof(AVSubtitleArea)); + + if (!dst) + return NULL; + + dst->x = src->x; + dst->y = src->y; + dst->w = src->w; + dst->h = src->h; + dst->nb_colors = src->nb_colors; + dst->type = src->type; + dst->flags = src->flags; + + for (unsigned i = 0; i < AV_NUM_BUFFER_POINTERS; i++) { + if (src->h > 0 && src->w > 0 && src->buf[i]) { + dst->buf[0] = av_buffer_ref(src->buf[i]); + if (!dst->buf[i]) + return NULL; + + const int ret = av_buffer_make_writable(&dst->buf[i]); + if (ret < 0) + return NULL; + + dst->linesize[i] = src->linesize[i]; + } + } + + memcpy(&dst->pal[0], &src->pal[0], sizeof(src->pal[0]) * 256); + + + return dst; +} + +static int quantize_image_colors(SubOcrContext *const s, AVSubtitleArea *subtitle_area) +{ + const int num_quantized_colors = 3; + int k, ret; + const int codeword_length = subtitle_area->w * subtitle_area->h; + uint8_t *src_data = subtitle_area->buf[0]->data; + + if (subtitle_area->nb_colors <= num_quantized_colors) { + av_log(s, AV_LOG_DEBUG, "No need to quantize colors. Color count: %d\n", subtitle_area->nb_colors); + return 0; + } + + // Convert palette to grayscale + for (int i = 0; i < subtitle_area->nb_colors; i++) { + uint8_t *color = (uint8_t *)&subtitle_area->pal[i]; + const uint8_t average = (uint8_t)(((int)color[s->r_idx] + color[s->g_idx] + color[s->b_idx]) / 3); + color[s->b_idx] = average; + color[s->g_idx] = average; + color[s->r_idx] = average; + } + + /* Re-Initialize */ + s->codeword = av_realloc_f(s->codeword, codeword_length, 4 * sizeof(*s->codeword)); + if (!s->codeword) + return AVERROR(ENOMEM); + + s->codeword_closest_codebook_idxs = av_realloc_f(s->codeword_closest_codebook_idxs, + codeword_length, sizeof(*s->codeword_closest_codebook_idxs)); + if (!s->codeword_closest_codebook_idxs) + return AVERROR(ENOMEM); + + s->codebook = av_realloc_f(s->codebook, num_quantized_colors, 4 * sizeof(*s->codebook)); + if (!s->codebook) + return AVERROR(ENOMEM); + + /* build the codeword */ + k = 0; + for (int i = 0; i < subtitle_area->h; i++) { + uint8_t *p = src_data; + for (int j = 0; j < subtitle_area->w; j++) { + const uint8_t *color = (uint8_t *)&subtitle_area->pal[*p]; + s->codeword[k++] = color[s->b_idx]; + s->codeword[k++] = color[s->g_idx]; + s->codeword[k++] = color[s->r_idx]; + s->codeword[k++] = color[s->a_idx]; + p++; + } + src_data += subtitle_area->linesize[0]; + } + + /* compute the codebook */ + ret = avpriv_elbg_do(&s->ctx, s->codeword, 4, codeword_length, s->codebook, + num_quantized_colors, 1, s->codeword_closest_codebook_idxs, &s->lfg, 0); + if (ret < 0) + return ret; + + /* Write Palette */ + for (int i = 0; i < num_quantized_colors; i++) { + subtitle_area->pal[i] = s->codebook[i*4+3] << 24 | + (s->codebook[i*4+2] << 16) | + (s->codebook[i*4+1] << 8) | + (s->codebook[i*4 ] << 0); + } + + + av_log(s, AV_LOG_DEBUG, "Quantized colors from %d to %d\n", subtitle_area->nb_colors, num_quantized_colors); + + subtitle_area->nb_colors = num_quantized_colors; + src_data = subtitle_area->buf[0]->data; + + /* Write Image */ + k = 0; + for (int i = 0; i < subtitle_area->h; i++) { + uint8_t *p = src_data; + for (int j = 0; j < subtitle_area->w; j++, p++) { + p[0] = (uint8_t)s->codeword_closest_codebook_idxs[k++]; + } + + src_data += subtitle_area->linesize[0]; + } + + return ret; +} + +#define MEASURE_LINE_COUNT 6 + +static uint8_t get_background_color_index(SubOcrContext *const s, const AVSubtitleArea *subtitle_area) +{ + const int linesize = subtitle_area->linesize[0]; + int index_counts[256] = {0}; + const unsigned int line_offsets[MEASURE_LINE_COUNT] = { + 0, + linesize, + 2 * linesize, + (subtitle_area->h - 3) * linesize, + (subtitle_area->h - 2) * linesize, + (subtitle_area->h - 1) * linesize + }; + + const uint8_t *src_data = subtitle_area->buf[0]->data; + const uint8_t tl = src_data[0]; + const uint8_t tr = src_data[subtitle_area->w - 1]; + const uint8_t bl = src_data[(subtitle_area->h - 1) * linesize + 0]; + const uint8_t br = src_data[(subtitle_area->h - 1) * linesize + subtitle_area->w - 1]; + uint8_t max_index = 0; + int max_count; + + // When all corner pixels are equal, assume that as background color + if (tl == tr == bl == br || subtitle_area->h < 6) + return tl; + + for (unsigned int i = 0; i < MEASURE_LINE_COUNT; i++) { + uint8_t *p = subtitle_area->buf[0]->data + line_offsets[i]; + for (int k = 0; k < subtitle_area->w; k++) + index_counts[p[k]]++; + } + + max_count = index_counts[0]; + + for (uint8_t i = 1; i < subtitle_area->nb_colors; i++) { + if (index_counts[i] > max_count) { + max_count = index_counts[i]; + max_index = i; + } + } + + return max_index; +} + +static uint8_t get_text_color_index(SubOcrContext *const s, const AVSubtitleArea *subtitle_area, const uint8_t bg_color_index, uint8_t *outline_color_index) +{ + const int linesize = subtitle_area->linesize[0]; + int index_counts[256] = {0}; + uint8_t last_index = bg_color_index; + int max_count, min_req_count; + uint8_t max_index = 0; + + for (int i = 3; i < subtitle_area->h - 3; i += 5) { + const uint8_t *p = subtitle_area->buf[0]->data + ((ptrdiff_t)linesize * i); + for (int k = 0; k < subtitle_area->w; k++) { + const uint8_t cur_index = p[k]; + + // When color hasn't changed, continue + if (cur_index == last_index) + continue; + + if (cur_index != bg_color_index) + index_counts[cur_index]++; + + last_index = cur_index; + } + } + + max_count = index_counts[0]; + + for (uint8_t i = 1; i < subtitle_area->nb_colors; i++) { + if (index_counts[i] > max_count) { + max_count = index_counts[i]; + max_index = i; + } + } + + min_req_count = max_count / 3; + + for (uint8_t i = 1; i < subtitle_area->nb_colors; i++) { + if (index_counts[i] < min_req_count) + index_counts[i] = 0; + } + + *outline_color_index = max_index; + + index_counts[max_index] = 0; + max_count = 0; + + for (uint8_t i = 0; i < subtitle_area->nb_colors; i++) { + if (index_counts[i] > max_count) { + max_count = index_counts[i]; + max_index = i; + } + } + + if (*outline_color_index == max_index) + *outline_color_index = 255; + + return max_index; +} + +static void make_image_binary(SubOcrContext *const s, AVSubtitleArea *subtitle_area, const uint8_t text_color_index) +{ + for (int i = 0; i < subtitle_area->nb_colors; i++) { + + if (i != text_color_index) + subtitle_area->pal[i] = 0xffffffff; + else + subtitle_area->pal[i] = 0xff000000; + } +} + +static int get_crop_region(SubOcrContext *const s, const AVSubtitleArea *subtitle_area, uint8_t text_color_index, int *x, int *y, int *w, int *h) +{ + const int linesize = subtitle_area->linesize[0]; + int max_y = 0, max_x = 0; + int min_y = subtitle_area->h - 1, min_x = subtitle_area->w - 1; + + for (int i = 0; i < subtitle_area->h; i += 3) { + const uint8_t *p = subtitle_area->buf[0]->data + ((ptrdiff_t)linesize * i); + for (int k = 0; k < subtitle_area->w; k += 2) { + if (p[k] == text_color_index) { + min_y = FFMIN(min_y, i); + min_x = FFMIN(min_x, k); + max_y = FFMAX(max_y, i); + max_x = FFMAX(max_x, k); + } + } + } + + if (max_y <= min_y || max_x <= min_x) { + av_log(s, AV_LOG_WARNING, "Unable to detect crop region\n"); + *x = 0; + *y = 0; + *w = subtitle_area->w; + *h = subtitle_area->h; + } else { + *x = FFMAX(min_x - 10, 0); + *y = FFMAX(min_y - 10, 0); + *w = FFMIN(max_x + 10 - *x, (subtitle_area->w - *x)); + *h = FFMIN(max_y + 10 - *y, (subtitle_area->h - *y)); + } + + return 0; +} + +static int crop_area_bitmap(SubOcrContext *const s, AVSubtitleArea *subtitle_area, int x, int y, int w, int h) +{ + const int linesize = subtitle_area->linesize[0]; + AVBufferRef *dst = av_buffer_allocz(h * w); + uint8_t *d; + + if (!dst) + return AVERROR(ENOMEM); + + d = dst->data; + + for (int i = y; i < y + h; i++) { + const uint8_t *p = subtitle_area->buf[0]->data + ((ptrdiff_t)linesize * i); + for (int k = x; k < x + w; k++) { + *d = p[k]; + d++; + } + } + + subtitle_area->w = w; + subtitle_area->h = h; + subtitle_area->x += x; + subtitle_area->y += y; + subtitle_area->linesize[0] = w; + av_buffer_replace(&subtitle_area->buf[0], dst); + + av_buffer_unref(&dst); + return 0; +} + +#define R 0 +#define G 1 +#define B 2 +#define A 3 + +static int print_code(AVBPrint *buf, int in_code, const char *fmt, ...) +{ + va_list vl; + + if (!in_code) + av_bprint_chars(buf, '{', 1); + + va_start(vl, fmt); + av_vbprintf(buf, fmt, vl); + va_end(vl); + + return 1; +} + +static int end_code(AVBPrint *buf, int in_code) +{ + if (in_code) + av_bprint_chars(buf, '}', 1); + return 0; +} + +static uint8_t* create_grayscale_image(AVFilterContext *ctx, AVSubtitleArea *area, int invert) +{ + uint8_t gray_pal[256]; + const size_t img_size = area->buf[0]->size; + const uint8_t* img = area->buf[0]->data; + uint8_t* gs_img = av_malloc(img_size); + + if (!gs_img) + return NULL; + + for (unsigned i = 0; i < 256; i++) { + const uint8_t *col = (uint8_t*)&area->pal[i]; + const int val = (int)col[3] * FFMAX3(col[0], col[1], col[2]); + gray_pal[i] = (uint8_t)(val >> 8); + } + + if (invert) + for (unsigned i = 0; i < img_size; i++) + gs_img[i] = 255 - gray_pal[img[i]]; + else + for (unsigned i = 0; i < img_size; i++) + gs_img[i] = gray_pal[img[i]]; + + return gs_img; +} + +static uint8_t* create_bitmap_image(AVFilterContext *ctx, AVSubtitleArea *area, const uint8_t text_color_index) +{ + const size_t img_size = area->buf[0]->size; + const uint8_t* img = area->buf[0]->data; + uint8_t* gs_img = av_malloc(img_size); + + if (!gs_img) + return NULL; + + for (unsigned i = 0; i < img_size; i++) { + if (img[i] == text_color_index) + gs_img[i] = 0; + else + gs_img[i] = 255; + } + + return gs_img; +} + +static void png_save(AVFilterContext *ctx, const char *filename, AVSubtitleArea *area) +{ + int x, y; + int v; + FILE *f; + char fname[40]; + const uint8_t *data = area->buf[0]->data; + + snprintf(fname, sizeof(fname), "%s.ppm", filename); + + f = fopen(fname, "wb"); + if (!f) { + perror(fname); + return; + } + fprintf(f, "P6\n" + "%d %d\n" + "%d\n", + area->w, area->h, 255); + for(y = 0; y < area->h; y++) { + for(x = 0; x < area->w; x++) { + const uint8_t index = data[y * area->linesize[0] + x]; + v = (int)area->pal[index]; + putc(v >> 16 & 0xff, f); + putc(v >> 8 & 0xff, f); + putc(v >> 0 & 0xff, f); + } + } + + fclose(f); +} + +static int get_max_index(int score[256]) +{ + int max_val = 0, max_index = 0; + + for (int i = 0; i < 256; i++) { + if (score[i] > max_val) { + max_val = score[i]; + max_index = i; + } + } + + return max_index; +} + +static int get_word_colors(AVFilterContext *ctx, TessResultIterator* ri, const AVSubtitleArea* area, const AVSubtitleArea* original_area, + uint8_t bg_color_index, uint8_t text_color_index, uint8_t outline_color_index, + uint32_t* bg_color, uint32_t* text_color, uint32_t* outline_color) +{ + int left = 0, top = 0, right = 0, bottom = 0, ret; + int bg_score[256] = {0}, text_score[256] = {0}, outline_score[256] = {0}; + int max_index; + + ret = TessPageIteratorBoundingBox((TessPageIterator*)ri, RIL_WORD, &left, &top, &right, &bottom); + if (ret < 0) { + av_log(ctx, AV_LOG_WARNING, "get_word_colors: IteratorBoundingBox failed: %d\n", ret); + return ret; + } + + if (left >= area->w || right >= area->w || top >= area->h || bottom >= area->h) { + av_log(ctx, AV_LOG_WARNING, "get_word_colors: word bounding box (l: %d, t: %d r: %d, b: %d) out of image bounds (%dx%d)\n", left,top, right, bottom, area->w, area->h); + return AVERROR(EINVAL); + } + + for (int y = top; y < bottom; y += 3) { + uint8_t *p = area->buf[0]->data + (y * area->linesize[0]) + left; + uint8_t *porig = original_area->buf[0]->data + (y * original_area->linesize[0]) + left; + uint8_t current_index = 255; + + for (int x = left; x < right; x++, p++, porig++) { + + if (*p == current_index) { + if (*p == bg_color_index) + bg_score[*porig]++; + if (*p == text_color_index) + text_score[*porig]++; + if (*p == outline_color_index) + outline_score[*porig]++; + } + + current_index = *p; + } + } + + max_index = get_max_index(bg_score); + if (bg_score[max_index] > 0) + *bg_color = original_area->pal[max_index]; + + max_index = get_max_index(text_score); + if (text_score[max_index] > 0) + *text_color = original_area->pal[max_index]; + + max_index = get_max_index(outline_score); + if (outline_score[max_index] > 0) + *outline_color = original_area->pal[max_index]; + + return 0; +} + +static int convert_area(AVFilterContext *ctx, AVSubtitleArea *area, const AVFrame *frame, const unsigned area_index, int *margin_v) +{ + SubOcrContext *s = ctx->priv; + char *ocr_text = NULL; + int ret = 0; + uint8_t *gs_img; + uint8_t bg_color_index; + uint8_t text_color_index = 255; + uint8_t outline_color_index = 255; + char filename[32]; + AVSubtitleArea *original_area = copy_subtitle_area(area); + + if (!original_area) + return AVERROR(ENOMEM); + + if (area->w < 6 || area->h < 6) { + area->ass = NULL; + goto exit; + } + + if (s->dump_bitmaps) { + snprintf(filename, sizeof(filename), "graphicsub2text_%"PRId64"_%d_original", frame->subtitle_timing.start_pts, area_index); + png_save(ctx, filename, area); + } + + if (s->preprocess_images) { + ret = quantize_image_colors(s, area); + if (ret < 0) + goto exit; + if (s->dump_bitmaps && original_area->nb_colors != area->nb_colors) { + snprintf(filename, sizeof(filename), "graphicsub2text_%"PRId64"_%d_quantized", frame->subtitle_timing.start_pts, area_index); + png_save(ctx, filename, area); + } + } + + bg_color_index = get_background_color_index(s, area); + + if (s->preprocess_images) { + int x, y, w, h; + + for (int i = 0; i < area->nb_colors; ++i) { + av_log(s, AV_LOG_DEBUG, "Color #%d: %0.8X\n", i, area->pal[i]); + } + + text_color_index = get_text_color_index(s, area, bg_color_index, &outline_color_index); + + get_crop_region(s, area, text_color_index, &x, &y, &w, &h); + + if ((ret = crop_area_bitmap(s, area, x, y, w, h) < 0)) + goto exit; + + if ((ret = crop_area_bitmap(s, original_area, x, y, w, h) < 0)) + goto exit; + + make_image_binary(s, area, text_color_index); + + if (s->dump_bitmaps) { + snprintf(filename, sizeof(filename), "graphicsub2text_%"PRId64"_%d_preprocessed", frame->subtitle_timing.start_pts, area_index); + png_save(ctx, filename, area); + } + + gs_img = create_bitmap_image(ctx, area, text_color_index); + } else + gs_img = create_grayscale_image(ctx, area, 1); + + if (!gs_img) { + ret = AVERROR(ENOMEM); + goto exit; + } + + area->type = AV_SUBTITLE_FMT_ASS; + TessBaseAPISetImage(s->tapi, gs_img, area->w, area->h, 1, area->linesize[0]); + + TessBaseAPISetSourceResolution(s->tapi, 72); + + ret = TessBaseAPIRecognize(s->tapi, NULL); + if (ret == 0) + ocr_text = TessBaseAPIGetUTF8Text(s->tapi); + + if (!ocr_text || !strlen(ocr_text)) { + av_log(ctx, AV_LOG_WARNING, "OCR didn't return a text. ret=%d\n", ret); + area->ass = NULL; + + goto exit; + } + + const size_t len = strlen(ocr_text); + if (len > 0 && ocr_text[len - 1] == '\n') + ocr_text[len - 1] = 0; + + av_log(ctx, AV_LOG_VERBOSE, "OCR Result: %s\n", ocr_text); + + area->ass = av_strdup(ocr_text); + TessDeleteText(ocr_text); + + // End of simple recognition + + if (s->recognize != RFLAGS_NONE) { + TessResultIterator* ri = 0; + const TessPageIteratorLevel level = RIL_WORD; + int cur_is_bold = 0, cur_is_italic = 0, cur_is_underlined = 0, cur_pointsize = 0; + uint32_t cur_text_color = 0, cur_bg_color = 0, cur_outline_color = 0; + + char *cur_font_name = NULL; + int valign = 0; // 0: bottom, 4: top, 8 middle + int halign = 2; // 1: left, 2: center, 3: right + int in_code = 0; + double font_factor = (0.000666 * (s->h - 480) + 1) * s->font_size_factor; + + av_freep(&area->ass); + av_bprint_clear(&s->buffer); + + ri = TessBaseAPIGetIterator(s->tapi); + + // Horizontal Alignment + if (s->w && s->recognize & RFLAGS_HALIGN) { + int left_margin = area->x; + int right_margin = s->w - area->x - area->w; + double relative_diff = ((double)left_margin - right_margin) / s->w; + + if (FFABS(relative_diff) < 0.1) + halign = 2; // center + else if (relative_diff > 0) + halign = 3; // right + else + halign = 1; // left + } + + // Vertical Alignment + if (s->h && frame->height && s->recognize & RFLAGS_VALIGN) { + int left = 0, top = 0, right = 0, bottom = 0; + + TessPageIteratorBoundingBox((TessPageIterator*)ri, RIL_TEXTLINE, &left, &top, &right, &bottom); + av_log(s, AV_LOG_DEBUG, "RIL_TEXTLINE - TOP: %d BOTTOM: %d HEIGHT: %d\n", top, bottom, bottom - top); + + TessPageIteratorBoundingBox((TessPageIterator*)ri, RIL_BLOCK, &left, &top, &right, &bottom); + + const int vertical_pos = area->y + area->h / 2; + if (vertical_pos < s->h / 3) { + *margin_v = area->y + top; + valign = 4; + } + else if (vertical_pos < s->h / 3 * 2) { + *margin_v = 0; + valign = 8; + } else { + *margin_v = frame->height - area->y - area->h; + valign = 0; + } + } + + if (*margin_v < 0) + *margin_v = 0; + + // Set alignment when not default (2) + if ((valign | halign) != 2) + in_code = print_code(&s->buffer, in_code, "\\a%d", valign | halign); + + do { + int is_bold, is_italic, is_underlined, is_monospace, is_serif, is_smallcaps, pointsize, font_id; + char* word; + const char *font_name = TessResultIteratorWordFontAttributes(ri, &is_bold, &is_italic, &is_underlined, &is_monospace, &is_serif, &is_smallcaps, &pointsize, &font_id); + uint32_t text_color = 0, bg_color = 0, outline_color = 0; + + if (cur_is_underlined && !is_underlined && s->recognize & RFLAGS_FUNDERLINE) + in_code = print_code(&s->buffer, in_code, "\\u0"); + + if (cur_is_bold && !is_bold && s->recognize & RFLAGS_FBOLD) + in_code = print_code(&s->buffer, in_code, "\\b0"); + + if (cur_is_italic && !is_italic && s->recognize & RFLAGS_FITALIC) + in_code = print_code(&s->buffer, in_code, "\\i0"); + + + if (TessPageIteratorIsAtBeginningOf((TessPageIterator*)ri, RIL_TEXTLINE) && !TessPageIteratorIsAtBeginningOf((TessPageIterator*)ri, RIL_BLOCK)) { + in_code = end_code(&s->buffer, in_code); + av_bprintf(&s->buffer, "\\N"); + } + + if (get_word_colors(ctx, ri, area, original_area, bg_color_index, text_color_index, outline_color_index, &bg_color, &text_color, &outline_color) == 0) { + + if (text_color > 0 && cur_text_color != text_color && s->recognize & RFLAGS_COLOR) { + const uint8_t* tval = (uint8_t*)&text_color; + const int color = (int)tval[R] << 16 | (int)tval[G] << 8 | tval[B]; + + in_code = print_code(&s->buffer, in_code, "\\1c&H%0.6X&", color); + if (tval[A] != 255) + in_code = print_code(&s->buffer, in_code, "\\1a&H%0.2X&", 255 - tval[A]); + } + + if (outline_color > 0 && cur_outline_color != outline_color && s->recognize & RFLAGS_OUTLINECOLOR) { + const uint8_t* tval = (uint8_t*)&outline_color; + const int color = (int)tval[R] << 16 | (int)tval[G] << 8 | tval[B]; + + in_code = print_code(&s->buffer, in_code, "\\3c&H%0.6X&\\bord2", color); + in_code = print_code(&s->buffer, in_code, "\\3a&H%0.2X&", FFMIN(255 - tval[A], 30)); + } + + cur_text_color = text_color; + cur_outline_color = outline_color; + } + + if (font_name && strlen(font_name) && s->recognize & RFLAGS_FONT) { + if (!cur_font_name || !strlen(cur_font_name) || strcmp(cur_font_name, font_name) != 0) { + char *sanitized_font_name = av_strireplace(font_name, "_", " "); + if (!sanitized_font_name) { + ret = AVERROR(ENOMEM); + goto exit; + } + + in_code = print_code(&s->buffer, in_code, "\\fn%s", sanitized_font_name); + av_freep(&sanitized_font_name); + + if (cur_font_name) + av_freep(&cur_font_name); + cur_font_name = av_strdup(font_name); + if (!cur_font_name) { + ret = AVERROR(ENOMEM); + goto exit; + } + } + } + + if (pointsize != cur_pointsize && s->recognize & RFLAGS_FONTSIZE) { + av_log(s, AV_LOG_DEBUG, "pointsize - pointsize: %d\n", pointsize); + in_code = print_code(&s->buffer, in_code, "\\fs%d", (int)(pointsize * font_factor)); + cur_pointsize = pointsize; + } + + if (is_italic && !cur_is_italic && s->recognize & RFLAGS_FITALIC) + in_code = print_code(&s->buffer, in_code, "\\i1"); + + if (is_bold && !cur_is_bold && s->recognize & RFLAGS_FBOLD) + in_code = print_code(&s->buffer, in_code, "\\b1"); + + if (is_underlined && !cur_is_underlined && s->recognize & RFLAGS_FUNDERLINE) + in_code = print_code(&s->buffer, in_code, "\\u1"); + + in_code = end_code(&s->buffer, in_code); + + cur_is_underlined = is_underlined; + cur_is_bold = is_bold; + cur_is_italic = is_italic; + + if (!TessPageIteratorIsAtBeginningOf((TessPageIterator*)ri, RIL_TEXTLINE)) + av_bprint_chars(&s->buffer, ' ', 1); + + word = TessResultIteratorGetUTF8Text(ri, level); + av_bprint_append_data(&s->buffer, word, strlen(word)); + TessDeleteText(word); + + } while (TessResultIteratorNext(ri, level)); + + if (!av_bprint_is_complete(&s->buffer)) + ret = AVERROR(ENOMEM); + else { + av_log(ctx, AV_LOG_VERBOSE, "ASS Result: %s\n", s->buffer.str); + area->ass = av_strdup(s->buffer.str); + } + + TessResultIteratorDelete(ri); + av_freep(&cur_font_name); + } + +exit: + free_subtitle_area(original_area); + av_freep(&gs_img); + av_buffer_unref(&area->buf[0]); + area->type = AV_SUBTITLE_FMT_ASS; + + return ret; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + SubOcrContext *s = ctx->priv; + AVFilterLink *outlink = inlink->dst->outputs[0]; + int ret, frame_sent = 0; + + if (s->pending_frame && !frame->repeat_sub) { + const int64_t pts_diff = frame->subtitle_timing.start_pts - s->pending_frame->subtitle_timing.start_pts; + + if (pts_diff == 0) { + // This is just a repetition of the previous frame, ignore it + av_frame_free(&frame); + return 0; + } + + s->pending_frame->subtitle_timing.duration = pts_diff; + + if ((ret = av_buffer_replace(&s->pending_frame->subtitle_header, s->subtitle_header)) < 0) + return ret; + + ret = ff_filter_frame(outlink, s->pending_frame); + s->pending_frame = NULL; + if (ret < 0) + return ret; + + frame_sent = 1; + s->last_subtitle_pts = frame->subtitle_timing.start_pts; + } + + if (frame->repeat_sub) { + // Ignore repeated frame + av_frame_free(&frame); + return 0; + } + + s->last_subtitle_pts = frame->subtitle_timing.start_pts; + + ret = av_frame_make_writable(frame); + + if (ret < 0) { + av_frame_free(&frame); + return ret; + } + + frame->format = AV_SUBTITLE_FMT_ASS; + + av_log(ctx, AV_LOG_VERBOSE, "filter_frame sub_pts: %"PRIu64", duration: %"PRIu64", num_areas: %d\n", + frame->subtitle_timing.start_pts, frame->subtitle_timing.duration, frame->num_subtitle_areas); + + if (frame->num_subtitle_areas > 1 && + frame->subtitle_areas[0]->y > frame->subtitle_areas[frame->num_subtitle_areas - 1]->y) { + + for (unsigned i = 0; i < frame->num_subtitle_areas / 2; i++) + FFSWAP(AVSubtitleArea*, frame->subtitle_areas[i], frame->subtitle_areas[frame->num_subtitle_areas - i - 1]); + } + + for (int i = 0; i < frame->num_subtitle_areas; i++) { + AVSubtitleArea *area = frame->subtitle_areas[i]; + int margin_v = 0; + + ret = convert_area(ctx, area, frame, i, &margin_v); + if (ret < 0) + return ret; + + if (area->ass && area->ass[0] != '\0') { + + const int layer = s->recognize ? i : 0; + char *tmp = area->ass; + area->ass = avpriv_ass_get_dialog_ex(s->readorder_counter++, layer, "Default", NULL, 0, 0, margin_v, tmp); + av_free(tmp); + } + } + + // When decoders can't determine the end time, they are setting it either to UINT32_NAX + // or 30s (dvbsub). + if (s->delay_when_no_duration && frame->subtitle_timing.duration >= ms_to_avtb(29000)) { + // Can't send it without end time, wait for the next frame to determine the end_display time + s->pending_frame = frame; + + if (frame_sent) + return 0; + + // To keep all going, send an empty frame instead + frame = ff_get_subtitles_buffer(outlink, AV_SUBTITLE_FMT_ASS); + if (!frame) + return AVERROR(ENOMEM); + + av_frame_copy_props(frame, s->pending_frame); + frame->subtitle_timing.start_pts = 0; + frame->subtitle_timing.duration = 1; + frame->repeat_sub = 1; + } + + if ((ret = av_buffer_replace(&frame->subtitle_header, s->subtitle_header)) < 0) + return ret; + + return ff_filter_frame(outlink, frame); +} + +#define OFFSET(x) offsetof(SubOcrContext, x) +#define FLAGS (AV_OPT_FLAG_SUBTITLE_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption graphicsub2text_options[] = { + { "delay_when_no_duration", "delay output when duration is unknown", OFFSET(delay_when_no_duration), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS, NULL }, + { "dump_bitmaps", "save processed bitmaps as .ppm", OFFSET(dump_bitmaps), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS, NULL }, + { "font_size_factor", "font size adjustment factor", OFFSET(font_size_factor), AV_OPT_TYPE_DOUBLE, { .dbl = 1.0 }, 0.2, 5, FLAGS, NULL }, + { "language", "ocr language", OFFSET(language), AV_OPT_TYPE_STRING, { .str = "eng" }, 0, 0, FLAGS, NULL }, + { "ocr_mode", "set ocr mode", OFFSET(ocr_mode), AV_OPT_TYPE_INT, { .i64=OEM_TESSERACT_ONLY }, OEM_TESSERACT_ONLY, 2, FLAGS, "ocr_mode" }, + { "tesseract", "classic tesseract ocr", 0, AV_OPT_TYPE_CONST, { .i64=OEM_TESSERACT_ONLY }, 0, 0, FLAGS, "ocr_mode" }, + { "lstm", "lstm (ML based)", 0, AV_OPT_TYPE_CONST, { .i64=OEM_LSTM_ONLY}, 0, 0, FLAGS, "ocr_mode" }, + { "both", "use both models combined", 0, AV_OPT_TYPE_CONST, { .i64=OEM_TESSERACT_LSTM_COMBINED }, 0, 0, FLAGS, "ocr_mode" }, + { "preprocess_images", "reduce colors, remove outlines", OFFSET(preprocess_images), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, FLAGS, NULL }, + { "recognize", "detect fonts, styles and colors", OFFSET(recognize), AV_OPT_TYPE_FLAGS, { .i64 = RFLAGS_ALL}, 0, INT_MAX, FLAGS, "reco_flags" }, + { "none", "no format detection", 0, AV_OPT_TYPE_CONST, { .i64 = RFLAGS_NONE }, 0, 0, FLAGS, "reco_flags" }, + { "halign", "horizontal alignment", 0, AV_OPT_TYPE_CONST, { .i64 = RFLAGS_HALIGN }, 0, 0, FLAGS, "reco_flags" }, + { "valign", "vertical alignment", 0, AV_OPT_TYPE_CONST, { .i64 = RFLAGS_VALIGN }, 0, 0, FLAGS, "reco_flags" }, + { "bold", "font bold", 0, AV_OPT_TYPE_CONST, { .i64 = RFLAGS_FBOLD }, 0, 0, FLAGS, "reco_flags" }, + { "italic", "font italic", 0, AV_OPT_TYPE_CONST, { .i64 = RFLAGS_FITALIC }, 0, 0, FLAGS, "reco_flags" }, + { "underline", "font underline", 0, AV_OPT_TYPE_CONST, { .i64 = RFLAGS_FUNDERLINE }, 0, 0, FLAGS, "reco_flags" }, + { "font", "font name", 0, AV_OPT_TYPE_CONST, { .i64 = RFLAGS_FONT }, 0, 0, FLAGS, "reco_flags" }, + { "fontsize", "font size", 0, AV_OPT_TYPE_CONST, { .i64 = RFLAGS_FONTSIZE }, 0, 0, FLAGS, "reco_flags" }, + { "color", "font color", 0, AV_OPT_TYPE_CONST, { .i64 = RFLAGS_COLOR }, 0, 0, FLAGS, "reco_flags" }, + { "outlinecolor", "outline color", 0, AV_OPT_TYPE_CONST, { .i64 = RFLAGS_OUTLINECOLOR }, 0, 0, FLAGS, "reco_flags" }, + { "tessdata_path", "path to tesseract data", OFFSET(tessdata_path), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS, NULL }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(graphicsub2text); + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_frame, + .config_props = config_input, + }, +}; + +static const AVFilterPad outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .config_props = config_output, + }, +}; + +const AVFilter ff_sf_graphicsub2text = { + .name = "graphicsub2text", + .description = NULL_IF_CONFIG_SMALL("Convert graphical subtitles to text subtitles via OCR"), + .init = init, + .uninit = uninit, + .priv_size = sizeof(SubOcrContext), + .priv_class = &graphicsub2text_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_QUERY_FUNC(query_formats), +};