From patchwork Mon Aug 30 08:16:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29868 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp3737261iov; Mon, 30 Aug 2021 01:16:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwj7OfXfhtbO2HOlDLiAYlWQGzdNP9Th6Mhm/qMhY11E9tZnY7Cfnh0hEULVI0KkZvuXjYi X-Received: by 2002:a17:906:5e4b:: with SMTP id b11mr4958617eju.472.1630311387258; Mon, 30 Aug 2021 01:16:27 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id v20si14545892ejv.515.2021.08.30.01.16.26; Mon, 30 Aug 2021 01:16:27 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=BCUptc5A; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 03BA368A17B; Mon, 30 Aug 2021 11:16:24 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10olkn2083.outbound.protection.outlook.com [40.92.40.83]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 98AC9687F49 for ; Mon, 30 Aug 2021 11:16:17 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LjXVotLn/tT/eX8Gx9q0FczXqXIZnGco+zUvujT4zLDO0gA1n93ik557pemQt63LhBsLwAUuiRGolYUaOchMv8Syt8Vi+k5kq0QmPBs4Zlub5/pIe8tlljKL+urIumOru4khZH+ouKATvLmZ8TnOtML4gA0K0wnz7H6aGu+/dIgvJwqfvE/WFhiPR++giuW+9vbHVxg5GgFCh9oRml2670RsjjAv18Yo6OJLnjAGIaS0Zjqr5B8r8Bt9JwtJOvrAVvEf+5AvE7TiyHvaK2RQ3bbDimoIbapjhrcFiiEe2Bp6iHX0978elBwJgBs77HanJpcjZzVjgyNatUqtxzjczQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eePRXanIq62EyxIaiHemR82cVZuXiQRZl+tcVSyCtEk=; b=VkLgrcab9cN5+9jGh+FvTXtJz8lBvI7jkkwo4INuRLOkERfLOtMTZnzYW5jbZ/UU8J65D/dUK7ptoreSMZBOT4uJ+IpIo2k7IM8mdFKm0/JhcyuHLqkpCd9nnh8ZZja/sdf1uiYTbYhyKcDXXP1GkAvzf50BQfTz0uRUqgg6ZiQHKwe4t6h+8l+hHAElYlDEWml+HyjkGZG5H9FX1BSiM8eXVzJ9KLvehWVIlQLup1QSmnB+RZM2+BqMsHf2I7PjnYds+SA7kUYxkPABr5toCmuo00g4QykPfWtMPi2MaMoj3jaquPejOTQLQ8TCncnwbYpzfJpEcBph+GUHiXq+fw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eePRXanIq62EyxIaiHemR82cVZuXiQRZl+tcVSyCtEk=; b=BCUptc5Adq1+TnJ6ap+6X14RHTnKv5lkOggoSqTjiNGhXu2jeWkjbvHVf4nVk+Usm1QvgoySpGGO1kthk/GKZdEo/4oJ9rGBjgxzXYZxEG0IJoQi+IyVPyxZZq81DonloWZvkF9qnWD3XbPT97aCA2a9G8RQcaSfqQaYbTEYhvKxa6Crmr/47x52fKqioqF6Bmbq121OS+R8OxrmNyKGIF6wq4Jb/SbIjFU6WbP81ijhvDWdRtDGnJ6ejy1ozMspYDIBNz5DxVDN1zaY04u9sJkS6/KBjwA3oGpACMzduN61eAG3Bwd4PJJCXE8sn+QTfJmJYifsFxPGevy3afpn7Q== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB6191.namprd04.prod.outlook.com (2603:10b6:208:de::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.23; Mon, 30 Aug 2021 08:16:15 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 08:16:14 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v2 1/8] lavu/frame: avframe add type property Thread-Index: AdeUhtWMaTdAZp5sQQiYaJJEL0FffQ== Date: Mon, 30 Aug 2021 08:16:14 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [WDA0SPo5HCtlcamJdSpYcAjCMcKRX/uL] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 26e6fe13-dbf0-490c-a015-08d96b8e6f7c x-ms-traffictypediagnostic: MN2PR04MB6191: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: JwPkQa2bg6rM/21nHY8sqZVR7imTRD0W3/GQSxYpD6kQL+dcx0At1/qDOTvcYA0b+ZnRDfM1MAeDgTg+IuL+wAby1/usYKRaSRiOyNo5m8LMoCNvOY7lW+rY+4EVY25mu4TW9y/WpeK9pzUufMZD92QC+1FgtpvSHhPoc/oBTZPfE0M2FoWv2KygjOWoEALZs6xdRAqaUbdi0EFoOmOgF436EfVSs8Q2Kb1D97HLfoGTuH8Mmz+UK7MFMoiZNIYZ4+LslUSDCA0DxUdDqORKHMdOTQVz49ma7PML3mFLiUa6zdQMX9zemjf71/iLZrCENiBFJrE8uhCWU8tY0HsagQByDB3Q90PXCOsKN6KGj0l9h02DtOK5Ioc98jszdo4vyD9H2k0lyrfR3/56z5z/xFL6F3iNK+k3YaC96C2gtr56cyK3V4rIxn9CykkN83Ux x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: PdfI8y66iIB2YwLJ3d0cO+zSE7hmMtxLMgoCuE3VJXNkTipCm0VGIVX6wHTqrAW4JLxBrj5JLYz/mXhdFUFKh13WNNs0IB3ZaPOu54VvkvGoZcrsbdP74e04vI2j1OgkFzLPdN0bmHTJTY1rMYzXbA== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 26e6fe13-dbf0-490c-a015-08d96b8e6f7c X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2021 08:16:14.8442 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB6191 Subject: [FFmpeg-devel] [PATCH v2 1/8] lavu/frame: avframe add type property X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: QUT8D1CAcLx6 Signed-off-by: softworkz --- v2 Update: - Implemented Andreas' suggestions - overlay_subs filter: - removed duplicated code - implemented direct (no pre-conversion) blending of graphical subtitle rects - Supported input formats: - all packed RGB formats (with and without alpha) - yuv420p, yuv422p, yuv444p libavutil/frame.c | 74 ++++++++++++++++++++++++++++++++++++--------- libavutil/frame.h | 39 ++++++++++++++++++++++-- libavutil/version.h | 2 +- 3 files changed, 97 insertions(+), 18 deletions(-) diff --git a/libavutil/frame.c b/libavutil/frame.c index b0ceaf7145..7d95849cef 100644 --- a/libavutil/frame.c +++ b/libavutil/frame.c @@ -244,22 +244,39 @@ static int get_audio_buffer(AVFrame *frame, int align) } int av_frame_get_buffer(AVFrame *frame, int align) +{ + if (frame->width > 0 && frame->height > 0) + return av_frame_get_buffer2(frame, AVMEDIA_TYPE_VIDEO, align); + else if (frame->nb_samples > 0 && (frame->channel_layout || frame->channels > 0)) + return av_frame_get_buffer2(frame, AVMEDIA_TYPE_AUDIO, align); + + return AVERROR(EINVAL); +} + +int av_frame_get_buffer2(AVFrame *frame, enum AVMediaType type, int align) { if (frame->format < 0) return AVERROR(EINVAL); - if (frame->width > 0 && frame->height > 0) + frame->type = type; + + switch(frame->type) { + case AVMEDIA_TYPE_VIDEO: return get_video_buffer(frame, align); - else if (frame->nb_samples > 0 && (frame->channel_layout || frame->channels > 0)) + case AVMEDIA_TYPE_AUDIO: return get_audio_buffer(frame, align); - - return AVERROR(EINVAL); + case AVMEDIA_TYPE_SUBTITLE: + return 0; + default: + return AVERROR(EINVAL); + } } static int frame_copy_props(AVFrame *dst, const AVFrame *src, int force_copy) { int ret, i; + dst->type = src->type; dst->key_frame = src->key_frame; dst->pict_type = src->pict_type; dst->sample_aspect_ratio = src->sample_aspect_ratio; @@ -331,6 +348,7 @@ int av_frame_ref(AVFrame *dst, const AVFrame *src) av_assert1(dst->width == 0 && dst->height == 0); av_assert1(dst->channels == 0); + dst->type = src->type; dst->format = src->format; dst->width = src->width; dst->height = src->height; @@ -499,6 +517,7 @@ int av_frame_make_writable(AVFrame *frame) return 0; memset(&tmp, 0, sizeof(tmp)); + tmp.type = frame->type; tmp.format = frame->format; tmp.width = frame->width; tmp.height = frame->height; @@ -544,14 +563,22 @@ AVBufferRef *av_frame_get_plane_buffer(AVFrame *frame, int plane) uint8_t *data; int planes, i; - if (frame->nb_samples) { - int channels = frame->channels; - if (!channels) - return NULL; - CHECK_CHANNELS_CONSISTENCY(frame); - planes = av_sample_fmt_is_planar(frame->format) ? channels : 1; - } else + switch(frame->type) { + case AVMEDIA_TYPE_VIDEO: planes = 4; + break; + case AVMEDIA_TYPE_AUDIO: + { + int channels = frame->channels; + if (!channels) + return NULL; + CHECK_CHANNELS_CONSISTENCY(frame); + planes = av_sample_fmt_is_planar(frame->format) ? channels : 1; + break; + } + default: + return NULL; + } if (plane < 0 || plane >= planes || !frame->extended_data[plane]) return NULL; @@ -675,17 +702,34 @@ static int frame_copy_audio(AVFrame *dst, const AVFrame *src) return 0; } +static int frame_copy_subtitles(AVFrame *dst, const AVFrame *src) +{ + dst->type = AVMEDIA_TYPE_SUBTITLE; + dst->format = src->format; + + if (src->buf[0]) { + dst->buf[0] = av_buffer_ref(src->buf[0]); + dst->data[0] = src->data[0]; + } + + return 0; +} + int av_frame_copy(AVFrame *dst, const AVFrame *src) { if (dst->format != src->format || dst->format < 0) return AVERROR(EINVAL); - if (dst->width > 0 && dst->height > 0) + switch(dst->type) { + case AVMEDIA_TYPE_VIDEO: return frame_copy_video(dst, src); - else if (dst->nb_samples > 0 && dst->channels > 0) + case AVMEDIA_TYPE_AUDIO: return frame_copy_audio(dst, src); - - return AVERROR(EINVAL); + case AVMEDIA_TYPE_SUBTITLE: + return frame_copy_subtitles(dst, src); + default: + return AVERROR(EINVAL); + } } void av_frame_remove_side_data(AVFrame *frame, enum AVFrameSideDataType type) diff --git a/libavutil/frame.h b/libavutil/frame.h index ff2540a20f..c104815df9 100644 --- a/libavutil/frame.h +++ b/libavutil/frame.h @@ -271,7 +271,7 @@ typedef struct AVRegionOfInterest { } AVRegionOfInterest; /** - * This structure describes decoded (raw) audio or video data. + * This structure describes decoded (raw) audio, video or subtitle data. * * AVFrame must be allocated using av_frame_alloc(). Note that this only * allocates the AVFrame itself, the buffers for the data must be managed @@ -302,6 +302,13 @@ typedef struct AVRegionOfInterest { */ typedef struct AVFrame { #define AV_NUM_DATA_POINTERS 8 + /** + * Media type of the frame (audio, video, subtitles..) + * + * See AVMEDIA_TYPE_xxx + */ + enum AVMediaType type; + /** * pointer to the picture/channel planes. * This might be different from the first allocated byte @@ -371,7 +378,7 @@ typedef struct AVFrame { /** * format of the frame, -1 if unknown or unset * Values correspond to enum AVPixelFormat for video frames, - * enum AVSampleFormat for audio) + * enum AVSampleFormat for audio, AVSubtitleType for subtitles) */ int format; @@ -721,6 +728,8 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src); /** * Allocate new buffer(s) for audio or video data. * + * Note: For subtitle data, use av_frame_get_buffer2 + * * The following fields must be set on frame before calling this function: * - format (pixel format for video, sample format for audio) * - width and height for video @@ -743,6 +752,32 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src); */ int av_frame_get_buffer(AVFrame *frame, int align); +/** + * Allocate new buffer(s) for audio, video or subtitle data. + * + * The following fields must be set on frame before calling this function: + * - format (pixel format for video, sample format for audio) + * - width and height for video + * - nb_samples and channel_layout for audio + * + * This function will fill AVFrame.data and AVFrame.buf arrays and, if + * necessary, allocate and fill AVFrame.extended_data and AVFrame.extended_buf. + * For planar formats, one buffer will be allocated for each plane. + * + * @warning: if frame already has been allocated, calling this function will + * leak memory. In addition, undefined behavior can occur in certain + * cases. + * + * @param frame frame in which to store the new buffers. + * @param type media type to set for the frame. + * @param align Required buffer size alignment. If equal to 0, alignment will be + * chosen automatically for the current CPU. It is highly + * recommended to pass 0 here unless you know what you are doing. + * + * @return 0 on success, a negative AVERROR on error. + */ +int av_frame_get_buffer2(AVFrame *frame, enum AVMediaType type, int align); + /** * Check if the frame data is writable. * diff --git a/libavutil/version.h b/libavutil/version.h index 4b77387b08..1e6a80f86e 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -79,7 +79,7 @@ */ #define LIBAVUTIL_VERSION_MAJOR 57 -#define LIBAVUTIL_VERSION_MINOR 4 +#define LIBAVUTIL_VERSION_MINOR 5 #define LIBAVUTIL_VERSION_MICRO 101 #define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \ From patchwork Mon Aug 30 08:16:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29867 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp3737482iov; Mon, 30 Aug 2021 01:16:40 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxBe8YdG5z7f6nN5MrymVOFLc7qji9duKIpZR/K5D7zE4ngCG0z1rtcDzQCa+tzfikZYnNL X-Received: by 2002:a05:6402:358e:: with SMTP id y14mr15800180edc.296.1630311400270; Mon, 30 Aug 2021 01:16:40 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j4si3181485ejj.357.2021.08.30.01.16.38; Mon, 30 Aug 2021 01:16:40 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=FWQ5u4TU; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2664268A1F0; Mon, 30 Aug 2021 11:16:35 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10olkn2052.outbound.protection.outlook.com [40.92.40.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 57625688181 for ; Mon, 30 Aug 2021 11:16:29 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aYss6YOZGUOMGi5XIStYds7at/Hpd6uTYOLqJPd4VxXjJ5zo3qDLEwb3rp8ZmpmyEXB2Q+Ij/EzhIb9Et+f551GckoLSCfOdNr5sJWzaAuKXF0XiF3lmCdH5HkuWxW2K0K+iXZKtnD4WtdsSmlLeYw5Kry9T4CIsq5EAHnVG1r2tXSMbS2hvntfaoBULnTP5ySa2L2zE0wdSMHg0BioqQsImPZcpDW7i2DD8fRNWsZswiUFK5BBMqkwH8CoODZT8afuAd53b3bPli/HteJui9tDrblP3gHCF5jU4yORUsY3OeLFVEgDliUdXw/S6dd1bgjBCVQ2OXv2eY1Xwyo9SOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J2pPSyLvneqeWTRFVt31F2VRPFXYUFLyIqwyG2Mn9K0=; b=IhppVlyeAvvwkccCjJVLJBtqBFgzpAyWVQNooyPIXiqQlagiLTjmL/E5K3q0Ere3zUojce1E0rTjTdVECsakGWs7InirtYEQnq1zlUv2AaIHvoNPSmPr3E52MdYaogzYtUt8xr2lGUr3uTlKcoInXYHUGFcX5GXSmYsVNVgXrRmysv14QubhYRU8rmQ780IwD1VtzGQn5dAgT1z+4IfdKCdW394Q0V82SuuDqxRmC9Wpm3QwlaDv2cwjvVipZ87QBZJ8HMLrKZ0Y5Gjy2D0bkqZ4pHV0NuBvqF4FgrmdkgppJeYx0HqVAUVBQCjyElxDNv91dAPXN4yJHuelWjDdWA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J2pPSyLvneqeWTRFVt31F2VRPFXYUFLyIqwyG2Mn9K0=; b=FWQ5u4TUCt9r2cuWb1gJvXluBVE7Caq2V7SKqNj8rmM+ewe4w3Kw0gCRpVopEBPJ8gnZH22YKEmQfIW4+di1eqvPsu6hUMtNnofUoMDZj//0ejA8qhI3n2GAe1kjgKGRqlknosCDRXsWc3C94EMsx8fdCZkrNXB/rTaHCvI1BfoY3hTmqjzCRP6f0Fz6RzRCibnm1Sdo8dluW/CpombXdHqvgGkWuGDzGaz4SIzRqL+Vhdl+4qjfF/joYXRLL/1nYBzXjJ7C/b3BttYkYvRVBZu9jLNWE46yL41WIZvVYRHE5Gd+BCmkcVL25345tcNuuMl9HR7FO+6Veru9dY6Eeg== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB6191.namprd04.prod.outlook.com (2603:10b6:208:de::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.23; Mon, 30 Aug 2021 08:16:27 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 08:16:27 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v2 2/8] avfilter/subtitles: Add subtitles.c Thread-Index: AdedcjHqoNWdqt5YTeuSX/XxFkJE7A== Date: Mon, 30 Aug 2021 08:16:27 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [s64LF3HAQS9yRJirhKS5gdPk3d/SwYV9] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 457f235d-8e2e-4351-b41f-08d96b8e7716 x-ms-traffictypediagnostic: MN2PR04MB6191: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: GbJ4k86kQVU++QLkY60Apb7D7QuD3b2c+6D9Pg+lXYS7Ql8iBH6+AC+Mm6zzrw+SrRL0N/cGsZ7Dh5XuXaOyPsGMWIKz5yTS5rnBoXazR8nuQxsn6QCf0HUtTcg0ltLBMtUMZufQ2NO46cg5t+91XqanEpEPhQTGGYa3GwgFwsaIaQAv4UVjxnvNlQPI3b2BiwtjG0W9JRYqAw1PN2UZCCTYhmff4smZvJXA3zROBLnwEXEQPQoULgJ/JmcfREbHxkmzxuUyucz5f1CqTblNO0KPTzTJfSMRUiXoecaxUWUvFn+O5DFu/k8U3mpJHPPOve0vpuxbxbLJRjRkVBHOn3wT+UlFw3hbCNGmUaVGPp3QDDj8y91a/aXjJ8YbWkSrXxEnzUBTjVkHV/Ptq5aEC5hvbhknC1Y7OeV1CoSc9d0VbrIYB6yEhDPVPWeffGCe x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: Nv+HqZVo5f+KTVZn7riT3iziB1vMXUXkAMpCpOJ2l/Tg/YyiaCK6sLDbqLpY6LXcpUHpgHMiQfL2BmXegTFT6nj6edXq1tYYQCZOc7+nNQvOIks/TTULz6+uTh+/db0XwDTyeiz+7JjPddmhP5ipag== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 457f235d-8e2e-4351-b41f-08d96b8e7716 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2021 08:16:27.6600 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB6191 Subject: [FFmpeg-devel] [PATCH v2 2/8] avfilter/subtitles: Add subtitles.c X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: qCMFtbuNAqNO Signed-off-by: softworkz --- v2 Update: - Implemented Andreas' suggestions - overlay_subs filter: - removed duplicated code - implemented direct (no pre-conversion) blending of graphical subtitle rects - Supported input formats: - all packed RGB formats (with and without alpha) - yuv420p, yuv422p, yuv444p libavfilter/Makefile | 1 + libavfilter/f_interleave.c | 3 ++ libavfilter/internal.h | 1 + libavfilter/subtitles.c | 61 ++++++++++++++++++++++++++++++++++++++ libavfilter/subtitles.h | 44 +++++++++++++++++++++++++++ 5 files changed, 110 insertions(+) create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 102ce7beff..82a7394adb 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -19,6 +19,7 @@ OBJS = allfilters.o \ framequeue.o \ graphdump.o \ graphparser.o \ + subtitles.o \ video.o \ OBJS-$(HAVE_THREADS) += pthread.o diff --git a/libavfilter/f_interleave.c b/libavfilter/f_interleave.c index 2845d72b79..aefa9a11cb 100644 --- a/libavfilter/f_interleave.c +++ b/libavfilter/f_interleave.c @@ -32,6 +32,7 @@ #include "filters.h" #include "internal.h" #include "audio.h" +#include "subtitles.h" #include "video.h" typedef struct InterleaveContext { @@ -170,6 +171,8 @@ static av_cold int init(AVFilterContext *ctx) inpad.get_buffer.video = ff_null_get_video_buffer; break; case AVMEDIA_TYPE_AUDIO: inpad.get_buffer.audio = ff_null_get_audio_buffer; break; + case AVMEDIA_TYPE_SUBTITLE: + inpad.get_buffer.subtitle = ff_null_get_subtitles_buffer; break; default: av_assert0(0); } diff --git a/libavfilter/internal.h b/libavfilter/internal.h index a0aa32af4d..85519a1076 100644 --- a/libavfilter/internal.h +++ b/libavfilter/internal.h @@ -85,6 +85,7 @@ struct AVFilterPad { union { AVFrame *(*video)(AVFilterLink *link, int w, int h); AVFrame *(*audio)(AVFilterLink *link, int nb_samples); + AVFrame *(*subtitle)(AVFilterLink *link, int format); } get_buffer; /** diff --git a/libavfilter/subtitles.c b/libavfilter/subtitles.c new file mode 100644 index 0000000000..90ec479e51 --- /dev/null +++ b/libavfilter/subtitles.c @@ -0,0 +1,61 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/common.h" + +#include "subtitles.h" +#include "avfilter.h" +#include "internal.h" + + +AVFrame *ff_null_get_subtitles_buffer(AVFilterLink *link, int format) +{ + return ff_get_subtitles_buffer(link->dst->outputs[0], format); +} + +AVFrame *ff_default_get_subtitles_buffer(AVFilterLink *link, int format) +{ + AVFrame *frame = NULL; + + // TODO: + //frame = ff_frame_pool_get(link->frame_pool); + + frame = av_frame_alloc(); + if (!frame) + return NULL; + + frame->format = format; + frame->type = AVMEDIA_TYPE_SUBTITLE; + + return frame; +} + +AVFrame *ff_get_subtitles_buffer(AVFilterLink *link, int format) +{ + AVFrame *ret = NULL; + + if (link->dstpad->get_buffer.subtitle) + ret = link->dstpad->get_buffer.subtitle(link, format); + + if (!ret) + ret = ff_default_get_subtitles_buffer(link, format); + + return ret; +} diff --git a/libavfilter/subtitles.h b/libavfilter/subtitles.h new file mode 100644 index 0000000000..d3d5491652 --- /dev/null +++ b/libavfilter/subtitles.h @@ -0,0 +1,44 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVFILTER_SUBTITLES_H +#define AVFILTER_SUBTITLES_H + +#include "avfilter.h" +#include "internal.h" + +/** default handler for get_subtitles_buffer() for subtitle inputs */ +AVFrame *ff_default_get_subtitles_buffer(AVFilterLink *link, int format); + +/** get_subtitles_buffer() handler for filters which simply pass subtitles along */ +AVFrame *ff_null_get_subtitles_buffer(AVFilterLink *link, int format); + +/** + * Request a subtitles buffer with a specific set of permissions. + * + * @param link the output link to the filter from which the buffer will + * be requested + * @param format The subtitles format. + * @return A reference to the subtitles. This must be unreferenced with + * avfilter_unref_buffer when you are finished with it. +*/ +AVFrame *ff_get_subtitles_buffer(AVFilterLink *link, int format); + +#endif /* AVFILTER_SUBTITLES_H */ From patchwork Mon Aug 30 08:16:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29875 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp3737609iov; Mon, 30 Aug 2021 01:16:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzALX9MvSPBc5jtbYzuaLkSH6XttmZRqtTl2/9o/3p1nfSHYlP82WHzzoYCv+3R0P+9ea/N X-Received: by 2002:a05:6402:b65:: with SMTP id cb5mr7232281edb.343.1630311410578; Mon, 30 Aug 2021 01:16:50 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 18si17378436ejc.508.2021.08.30.01.16.50; Mon, 30 Aug 2021 01:16:50 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=a50AadJm; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7D4E968A1CC; Mon, 30 Aug 2021 11:16:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10olkn2033.outbound.protection.outlook.com [40.92.40.33]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8EAFD688181 for ; Mon, 30 Aug 2021 11:16:38 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GxdXCF38H/0rC311u2svbYIb49/mXTb2o1IyL5nBMSVjFLKFnvYgkTI9Y8P9DubiYyb+K3lnV3ljbH2LeMJ4ziTsZEP5YZfuxjmMcESBZOZBPYc/MiLMzYj889SyR1VJbP+8ovxA3Z5l+uPVfIkgU9dNmkbuzM0NMUirIANaTlR6Kdyn9ZcS4GKDG1JiFnzZE8TlEpJJCqyhhR2TH0XnP6Vgsqsgcqm+cdYq3xwNDpyaSVsdnsirltOasvYbzWBAwzG20c/hEfk6q1FDv3uaHgECG0Mk7DFLR61IYfzP1bDkyY5nm234An2U3YKTcAMeW0aniBU3GfqVPZL9VAbUdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6mQ1JRbpXaVnG+svbScYk7m3sMvQAZvDan9X40VibaM=; b=VhIWAUj9HeBQfVZeuC1MBUczcH4KezFNn8Limbb/UBFZYsVlt2PXxbgFGmSrORrUyZAOhEDb8W13k9NytCRYvBjAS9O8iVfBxIFSfNZFxh3JoUTQuClH5QpXfawQ7TNbJo0XKIQxpZvImoO6LTPFtkX73dLi3YX8W2c4zJrMC98TgRnONVX/ZNVRRfTJwrJkVikXTojECOwXHa5vZKa6eKvsxgYb0Be67DyRAfvHYKk0cZd/yrAtShAJlRuzFtq+soADp6W9xE3Wca2txa1MIDE4S8S3YLWyESwEH0xZnXKK6/1/XcUHJrRr3I0CdpeXdwUJnfK3xBPHP78oWCo8Gg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6mQ1JRbpXaVnG+svbScYk7m3sMvQAZvDan9X40VibaM=; b=a50AadJmJtzgziio3TSu1OPvHIJKuFwUxkxgIuj6qEwAJGReFUADaaNNuP09G/V6iIA+QtLwemN9P2VHN92P8l7luwqCjhLGiTSJPD35BNwa/jODE2IyRKQWpM8/na9OB1sB6Tc9TlKzPcY9B9Ara2p+9BVwOiJCQsM6fBmV2i27IQvjcKJCMZ3Ku1BhMYdhLRF+LmXfxDUbOERhamPVjwOCnLVMpvUN/808VPdehtXyO8aits0jAu5uktc9mVWrNLdz3ak5Q4QCypLvn0N1W4E580x6IG4dUs0YaVkX1Si0Y/MlHJK9cRD0Dl5XIjMndORGjw2IWpFTzC6/RYVesg== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB6191.namprd04.prod.outlook.com (2603:10b6:208:de::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.23; Mon, 30 Aug 2021 08:16:36 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 08:16:36 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v2 3/8] avfilter/avfilter: Handle subtitle frames Thread-Index: AdeUdjgpG4tB/pZ/T5+8zhj48gT42A== Date: Mon, 30 Aug 2021 08:16:36 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [FJfNb0DvR84/shzsLeDSEX+34ijy9qYN] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: bbed67de-4ea1-4819-0e60-08d96b8e7c63 x-ms-traffictypediagnostic: MN2PR04MB6191: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: dnhcHXeSF89vNEWheFlPLeyvPfneNW52ZAO/iwjywTlcwEeNlczRtU8tIYtwq3RTLsKNiJAD7HA0VpQwP6qBrpnMwpwtQpIOaqqFcQg+2bWlcKbnsD/hQ5LTSfY5hhI9/6pYL86FWms/XwlgKGOqVbEW7FB7oDUPMqQ93OiUzrDAT86kvjBeyyqNux804ZTotWiXyPqVXFldaSJXXqC38wIWknSE3BSThu4C+pHeKa+Y2+RjVo3dPeO+7g5bPF3hoNVdw8YUIcQAkaqx+9usCkazx6ZU5p0x/NtThPCfgFMJIOxBdLfLiNmzJt+txIAv7m3O81kb+oaQS6p3AJrJBwW2f2xFGfcNSY0q/ztVUFgb4soHbzWK9j+I6Rg6FqnuPLzVeorS/Tbgr6iVhklqs7GT6vFMnBPjiWopTKuwwXa+PJuIjRZCIK7xfPtAkHJ1 x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: H4k0pZ7Tt1dYh9oDXwHHbTsb1eXdS8/in2G9rpnNNnXwP3FQ3+8SyEIAtKeYXlGke4mxT+9f3peK44UMjPLCcbRg5qa+cxvtVvRRrVnRFkhaRqna2Toyx9vgGi/cmH5HdzD6uVFogicaCY2mXdT6uQ== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: bbed67de-4ea1-4819-0e60-08d96b8e7c63 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2021 08:16:36.5120 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB6191 Subject: [FFmpeg-devel] [PATCH v2 3/8] avfilter/avfilter: Handle subtitle frames X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: PH6at/+dAPfV Signed-off-by: softworkz --- v2 Update: - Implemented Andreas' suggestions - overlay_subs filter: - removed duplicated code - implemented direct (no pre-conversion) blending of graphical subtitle rects - Supported input formats: - all packed RGB formats (with and without alpha) - yuv420p, yuv422p, yuv444p libavfilter/avfilter.c | 8 +++++--- libavfilter/avfiltergraph.c | 5 +++++ libavfilter/formats.c | 11 +++++++++++ 3 files changed, 21 insertions(+), 3 deletions(-) diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index ea22b247de..5505c34678 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -56,7 +56,8 @@ void ff_tlog_ref(void *ctx, AVFrame *ref, int end) ref->linesize[0], ref->linesize[1], ref->linesize[2], ref->linesize[3], ref->pts, ref->pkt_pos); - if (ref->width) { + switch(ref->type) { + case AVMEDIA_TYPE_VIDEO: ff_tlog(ctx, " a:%d/%d s:%dx%d i:%c iskey:%d type:%c", ref->sample_aspect_ratio.num, ref->sample_aspect_ratio.den, ref->width, ref->height, @@ -64,12 +65,13 @@ void ff_tlog_ref(void *ctx, AVFrame *ref, int end) ref->top_field_first ? 'T' : 'B', /* Top / Bottom */ ref->key_frame, av_get_picture_type_char(ref->pict_type)); - } - if (ref->nb_samples) { + break; + case AVMEDIA_TYPE_AUDIO: ff_tlog(ctx, " cl:%"PRId64"d n:%d r:%d", ref->channel_layout, ref->nb_samples, ref->sample_rate); + break; } ff_tlog(ctx, "]%s", end ? "\n" : ""); diff --git a/libavfilter/avfiltergraph.c b/libavfilter/avfiltergraph.c index 41a91a9bda..4f581bc7a6 100644 --- a/libavfilter/avfiltergraph.c +++ b/libavfilter/avfiltergraph.c @@ -328,6 +328,8 @@ static int filter_link_check_formats(void *log, AVFilterLink *link, AVFilterForm return ret; break; + case AVMEDIA_TYPE_SUBTITLE: + return 0; default: av_assert0(!"reached"); } @@ -463,6 +465,9 @@ static int query_formats(AVFilterGraph *graph, AVClass *log_ctx) if (!link) continue; + if (link->type == AVMEDIA_TYPE_SUBTITLE) + continue; + neg = ff_filter_get_negotiation(link); av_assert0(neg); for (neg_step = 1; neg_step < neg->nb; neg_step++) { diff --git a/libavfilter/formats.c b/libavfilter/formats.c index 9e39d65a3c..26d8e45263 100644 --- a/libavfilter/formats.c +++ b/libavfilter/formats.c @@ -19,6 +19,7 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include "libavcodec/avcodec.h" #include "libavutil/avassert.h" #include "libavutil/channel_layout.h" #include "libavutil/common.h" @@ -430,6 +431,12 @@ int ff_add_channel_layout(AVFilterChannelLayouts **l, uint64_t channel_layout) return 0; } +static int ff_add_subtitle_type(AVFilterFormats **avff, int64_t fmt) +{ + ADD_FORMAT(avff, fmt, ff_formats_unref, int, formats, nb_formats); + return 0; +} + AVFilterFormats *ff_all_formats(enum AVMediaType type) { AVFilterFormats *ret = NULL; @@ -447,6 +454,10 @@ AVFilterFormats *ff_all_formats(enum AVMediaType type) return NULL; fmt++; } + } else if (type == AVMEDIA_TYPE_SUBTITLE) { + ff_add_subtitle_type(&ret, SUBTITLE_BITMAP); + ff_add_subtitle_type(&ret, SUBTITLE_ASS); + ff_add_subtitle_type(&ret, SUBTITLE_TEXT); } return ret; From patchwork Mon Aug 30 08:16:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29872 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp3737735iov; Mon, 30 Aug 2021 01:17:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwPf6+7gApnHcfT7BvFmWuU/5OnuZsifNXv+0nRWgkJmnIPXLlDWJa0GfvqNz3+07ZQ2O7v X-Received: by 2002:a05:6402:48e:: with SMTP id k14mr23008819edv.212.1630311422059; Mon, 30 Aug 2021 01:17:02 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id eg43si13355567edb.107.2021.08.30.01.17.01; Mon, 30 Aug 2021 01:17:02 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=cAiTB2P3; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 852EA68A2B2; Mon, 30 Aug 2021 11:16:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08olkn2070.outbound.protection.outlook.com [40.92.46.70]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1C012689AA0 for ; Mon, 30 Aug 2021 11:16:49 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jxyKLYjrmdLnLIIhFcT8aaqAzgkSPJXlzr3hK95mGe0M2hAPOasy7S7ji1L9T7SwhGeT0iHt+Ud09HByHWBPJkI8OcShnOC2TkpqlFhFVZxGWTBLKV/uEg0m51MYrjuRLz01T1fKv82WawK2CbLCeNvi0vY0cY4RhhddZ8TMjaotNgzd2/zn0U58vDQ9IKJtsuYUi1y9f5u1dWv3hqiNLKub0CrVfXSi+BZjB8Use4kO4L3LJwjNcVY1Y2Vq/7+SqSRVnKbiS8Rcp4gEPRTd490CjY07o5mmTy2NiWZViGjPEECOPn2zl/mUCr73N0BYSbWKCHS9Qma6+StaqUbs2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=U9I9rV67HbyyNQhZdTt3JJw9rEQPtVHUaBEflOw1MXM=; b=lgJzeiE5yjSiNmN6ippqqI7ZhJ4/oyvtPmjHrMK3/NGrgzkRRZOehcA4C+0VFyDBjOM43oCDhkQD6jplR5qmrX784R4b4Mf077Yrshs5ZLdYX/p8WmbuZSoC7GJDOc+zQdNkVBHoRrxuzLCR3tiSXKtXjUTrblUyY3si1Mvek26vfgoyfOO08L97z34tMQ9fTW6W1XzrUiu5n4082Vwyb/EnMzB0bKzo7AaY7RtFbBnU6MQhULmSBSIQsOfkI9JC4aoJMiDLN1PkuSSvW2HTKKwzhhh+CJ6jDVZumziQ34N03CtTC+LJ+c+rWrsCNJYyQO3fWFPM5Yr4EaID557oEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=U9I9rV67HbyyNQhZdTt3JJw9rEQPtVHUaBEflOw1MXM=; b=cAiTB2P31pQ/W0GBprofVd6HKjpmmKiZmar9XHod3CPe+jf2aBq3egAeYgtjVVj+Xljb47PyRIrqEAvHrzg/rqOpHMJwPN4qzrpv3d34UfsSN8+q8I8ywDXm5ot8ZbkMD5mZIM/25iCTcSyuBtGJJMvmQCS3YaVGbdKA4CHlkfWO6kiwtFQSyJ2hHJsNdUvYUIhOrW4Y6Ve0hLJnyOp6QDHhmb8xGkOSoVTY03+q9oDEK3HxThoVasaO/1vkHDqIzi/ky4Pc1hWZmHPyR1xstjClT558/CxSAN9yv3YeYHFAH7DbvW8wD33//u+Js//GDLqq6d7yRyO0TJJBfru7qQ== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5853.namprd04.prod.outlook.com (2603:10b6:208:a6::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.19; Mon, 30 Aug 2021 08:16:46 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 08:16:46 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v2 4/8] avfilter/overlay_subs: Add overlay_subs filter Thread-Index: AdeddT2LfqP+ZxC3Rn6qBCTJTShc+A== Date: Mon, 30 Aug 2021 08:16:45 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [5SqYNlDejJ4XofNqMpQxmgU0JABcW72x] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 153d1013-dd3d-45bb-3c16-08d96b8e8200 x-ms-traffictypediagnostic: MN2PR04MB5853: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: E9/C9WX/SYIrK0/i9fColwzDBiltyQHJ4VKb98KXw7vmOMyK6tcdJEpJtlGfPm+482F3/dUMt8wuid4IoLdDJL0STDBMUp3wMgFwGc6PscU7PLCGMhkAwUH0hjW8M3B5++Vt4cjICAvZa3hIFMASXVqchFYpl//DWuxyGn0GdqUfEpOPt2MY0YghgvSGGi3f9yK8+zH0w4KHr4nzYYJ2ThonvC5aaqNPsbTFu0/mrfkp8xphXzm9E/FQ6pWqwTUspMbKnmRlRIq4OdHjYGiaV1jrIfMrWCKFWwbaLikJZYyYh/QfMdNiksjd3DtikCcB0hLEYwzw/kHzUSgtlhCH7wQQQVEF+R7lyn2iyiEbtT3LuksDDrho+QmozyHS0J/swj7glNyPiRY9DBn8yLkr2AzT+JFWwMmdRCQClfgcFwoFb/maygjdm5iKud8yq/nk x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: Qj5/uonQVb7q1hT2+ehIOJbvWOEuu11MUtmSO0q1w3EY8mhmY4w1KBFMZ/u5lMWnySDBp0pJ7lpQ1W51eGs+44f54ANdMELTvIWTSh6U9fs0sXaNH2wJKiaqTnIpQhQcmRcfkhQlTnl3g4/Yds2k4Q== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 153d1013-dd3d-45bb-3c16-08d96b8e8200 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2021 08:16:45.8778 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5853 Subject: [FFmpeg-devel] [PATCH v2 4/8] avfilter/overlay_subs: Add overlay_subs filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: cn6vo/5A8M9u Signed-off-by: softworkz --- v2 Update: - Implemented Andreas' suggestions - overlay_subs filter: - removed duplicated code - implemented direct (no pre-conversion) blending of graphical subtitle rects - Supported input formats: - all packed RGB formats (with and without alpha) - yuv420p, yuv422p, yuv444p libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_overlay_subs.c | 549 ++++++++++++++++++++++++++++++++++ libavfilter/vf_overlay_subs.h | 65 ++++ 4 files changed, 616 insertions(+) create mode 100644 libavfilter/vf_overlay_subs.c create mode 100644 libavfilter/vf_overlay_subs.h diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 82a7394adb..8130f7b46c 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -357,6 +357,7 @@ OBJS-$(CONFIG_OVERLAY_CUDA_FILTER) += vf_overlay_cuda.o framesync.o vf OBJS-$(CONFIG_OVERLAY_OPENCL_FILTER) += vf_overlay_opencl.o opencl.o \ opencl/overlay.o framesync.o OBJS-$(CONFIG_OVERLAY_QSV_FILTER) += vf_overlay_qsv.o framesync.o +OBJS-$(CONFIG_OVERLAY_SUBS_FILTER) += vf_overlay_subs.o framesync.o OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) += vf_overlay_vulkan.o vulkan.o OBJS-$(CONFIG_OWDENOISE_FILTER) += vf_owdenoise.o OBJS-$(CONFIG_PAD_FILTER) += vf_pad.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 73040d2824..abd0a47750 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -339,6 +339,7 @@ extern const AVFilter ff_vf_oscilloscope; extern const AVFilter ff_vf_overlay; extern const AVFilter ff_vf_overlay_opencl; extern const AVFilter ff_vf_overlay_qsv; +extern const AVFilter ff_vf_overlay_subs; extern const AVFilter ff_vf_overlay_vulkan; extern const AVFilter ff_vf_overlay_cuda; extern const AVFilter ff_vf_owdenoise; diff --git a/libavfilter/vf_overlay_subs.c b/libavfilter/vf_overlay_subs.c new file mode 100644 index 0000000000..177f9b1cc9 --- /dev/null +++ b/libavfilter/vf_overlay_subs.c @@ -0,0 +1,549 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * overlay graphical subtitles on top of a video frame + */ + +#include "avfilter.h" +#include "formats.h" +#include "libavutil/common.h" +#include "libavutil/eval.h" +#include "libavutil/avstring.h" +#include "libavutil/pixdesc.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "internal.h" +#include "drawutils.h" +#include "framesync.h" +#include "vf_overlay_subs.h" + +#include "libavcodec/avcodec.h" + +static const char *const var_names[] = { + "main_w", "W", ///< width of the main video + "main_h", "H", ///< height of the main video + "overlay_w", "w", ///< width of the overlay video + "overlay_h", "h", ///< height of the overlay video + "hsub", + "vsub", + "x", + "y", + "n", ///< number of frame + "pos", ///< position in the file + "t", ///< timestamp expressed in seconds + NULL +}; + +#define MAIN 0 +#define OVERLAY 1 + +#define R 0 +#define G 1 +#define B 2 +#define A 3 + +#define Y 0 +#define U 1 +#define V 2 + +enum EvalMode { + EVAL_MODE_INIT, + EVAL_MODE_FRAME, + EVAL_MODE_NB +}; + +static av_cold void uninit(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + ff_framesync_uninit(&s->fs); + av_expr_free(s->x_pexpr); s->x_pexpr = NULL; + av_expr_free(s->y_pexpr); s->y_pexpr = NULL; +} + +static inline int normalize_xy(double d, int chroma_sub) +{ + if (isnan(d)) + return INT_MAX; + return (int)d & ~((1 << chroma_sub) - 1); +} + +static void eval_expr(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->var_values[VAR_Y] = av_expr_eval(s->y_pexpr, s->var_values, NULL); + /* It is necessary if x is expressed from y */ + s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->x = normalize_xy(s->var_values[VAR_X], s->hsub); + s->y = normalize_xy(s->var_values[VAR_Y], s->vsub); +} + +static int set_expr(AVExpr **pexpr, const char *expr, const char *option, void *log_ctx) +{ + int ret; + AVExpr *old = NULL; + + if (*pexpr) + old = *pexpr; + ret = av_expr_parse(pexpr, expr, var_names, + NULL, NULL, NULL, NULL, 0, log_ctx); + if (ret < 0) { + av_log(log_ctx, AV_LOG_ERROR, + "Error when evaluating the expression '%s' for %s\n", + expr, option); + *pexpr = old; + return ret; + } + + av_expr_free(old); + return 0; +} + +static const enum AVPixelFormat alpha_pix_fmts[] = { + AV_PIX_FMT_YUVA420P, AV_PIX_FMT_YUVA422P, AV_PIX_FMT_YUVA444P, + AV_PIX_FMT_YUVA420P10, AV_PIX_FMT_YUVA422P10, + AV_PIX_FMT_ARGB, AV_PIX_FMT_ABGR, AV_PIX_FMT_RGBA, + AV_PIX_FMT_BGRA, AV_PIX_FMT_NONE +}; + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat supported_pix_fmts[] = { + AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV444P, + AV_PIX_FMT_ARGB, AV_PIX_FMT_RGBA, + AV_PIX_FMT_ABGR, AV_PIX_FMT_BGRA, + AV_PIX_FMT_RGB24, AV_PIX_FMT_BGR24, + AV_PIX_FMT_NONE + }; + + return ff_set_common_formats(ctx, ff_make_format_list(supported_pix_fmts)); +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + OverlaySubsContext *s = ctx->priv; + int ret; + + if ((ret = ff_framesync_init_dualinput(&s->fs, ctx)) < 0) + return ret; + + outlink->w = ctx->inputs[MAIN]->w; + outlink->h = ctx->inputs[MAIN]->h; + outlink->time_base = ctx->inputs[MAIN]->time_base; + + return ff_framesync_configure(&s->fs); +} + +// divide by 255 and round to nearest +// apply a fast variant: (X+127)/255 = ((X+127)*257+257)>>16 = ((X+128)*257)>>16 +#define FAST_DIV255(x) ((((x) + 128) * 257) >> 16) + +// calculate the non-pre-multiplied alpha, applying the general equation: +// alpha = alpha_overlay / ( (alpha_main + alpha_overlay) - (alpha_main * alpha_overlay) ) +// (((x) << 16) - ((x) << 9) + (x)) is a faster version of: 255 * 255 * x +// ((((x) + (y)) << 8) - ((x) + (y)) - (y) * (x)) is a faster version of: 255 * (x + y) +#define UNPREMULTIPLY_ALPHA(x, y) ((((x) << 16) - ((x) << 9) + (x)) / ((((x) + (y)) << 8) - ((x) + (y)) - (y) * (x))) + +/** + * Blend image in src to destination buffer dst at position (x, y). + */ +static av_always_inline void blend_packed_rgb(const AVFilterContext *ctx, + const AVFrame *dst, const AVSubtitleRect *src, + int main_has_alpha, int x, int y, + int is_straight) +{ + OverlaySubsContext *s = ctx->priv; + int i, imax, j, jmax; + const int src_w = src->w; + const int src_h = src->h; + const int dst_w = dst->width; + const int dst_h = dst->height; + uint8_t alpha; ///< the amount of overlay to blend on to main + const int dr = s->main_rgba_map[R]; + const int dg = s->main_rgba_map[G]; + const int db = s->main_rgba_map[B]; + const int da = s->main_rgba_map[A]; + const int dstep = s->main_pix_step[0]; + const int sr = s->overlay_rgba_map[R]; + const int sg = s->overlay_rgba_map[G]; + const int sb = s->overlay_rgba_map[B]; + const int sa = s->overlay_rgba_map[A]; + const int sstep = s->overlay_pix_step[0]; + uint32_t *pal = (uint32_t *)src->data[1]; + int slice_start, slice_end; + uint8_t *S, *sp, *d, *dp; + + i = FFMAX(-y, 0); + imax = FFMIN3(-y + dst_h, FFMIN(src_h, dst_h), y + src_h); + + slice_start = i; + slice_end = i + imax; + + sp = src->data[0] + slice_start * src->linesize[0]; + dp = dst->data[0] + (slice_start + y) * dst->linesize[0]; + + for (i = slice_start; i < slice_end; i++) { + j = FFMAX(-x, 0); + S = sp + (j * sstep); + d = dp + ((x + j) * dstep); + + for (jmax = FFMIN(-x + dst_w, src_w); j < jmax; j++) { + uint32_t val = pal[*S]; + const uint8_t *sval = (uint8_t *)&val; + alpha = sval[sa]; + + // if the main channel has an alpha channel, alpha has to be calculated + // to create an un-premultiplied (straight) alpha value + if (main_has_alpha && alpha != 0 && alpha != 255) { + const uint8_t alpha_d = d[da]; + alpha = UNPREMULTIPLY_ALPHA(alpha, alpha_d); + } + + switch (alpha) { + case 0: + break; + case 255: + d[dr] = sval[sr]; + d[dg] = sval[sg]; + d[db] = sval[sb]; + break; + default: + // main_value = main_value * (1 - alpha) + overlay_value * alpha + // since alpha is in the range 0-255, the result must divided by 255 + d[dr] = is_straight ? FAST_DIV255(d[dr] * (255 - alpha) + sval[sr] * alpha) : + FFMIN(FAST_DIV255(d[dr] * (255 - alpha)) + sval[sr], 255); + d[dg] = is_straight ? FAST_DIV255(d[dg] * (255 - alpha) + sval[sg] * alpha) : + FFMIN(FAST_DIV255(d[dg] * (255 - alpha)) + sval[sg], 255); + d[db] = is_straight ? FAST_DIV255(d[db] * (255 - alpha) + sval[sb] * alpha) : + FFMIN(FAST_DIV255(d[db] * (255 - alpha)) + sval[sb], 255); + } + + if (main_has_alpha) { + switch (alpha) { + case 0: + break; + case 255: + d[da] = sval[sa]; + break; + default: + // apply alpha compositing: main_alpha += (1-main_alpha) * overlay_alpha + d[da] += FAST_DIV255((255 - d[da]) * S[sa]); + } + } + d += dstep; + S += sstep; + } + dp += dst->linesize[0]; + sp += src->linesize[0]; + } +} + +static av_always_inline void blend_plane_8_8bits(const AVFilterContext *ctx, const AVFrame *dst, const AVSubtitleRect *src, + const uint32_t *yuv_pal, int src_w, int src_h, int dst_w, int dst_h, int plane, int hsub, int vsub, + int x, int y, int dst_plane, int dst_offset, int dst_step) +{ + OverlaySubsContext *octx = ctx->priv; + int src_wp = AV_CEIL_RSHIFT(src_w, hsub); + int src_hp = AV_CEIL_RSHIFT(src_h, vsub); + int dst_wp = AV_CEIL_RSHIFT(dst_w, hsub); + int dst_hp = AV_CEIL_RSHIFT(dst_h, vsub); + int yp = y >> vsub; + int xp = x >> hsub; + uint8_t *s, *sp, *d, *dp, *dap; + int imax, i, j, jmax; + const int sstep = octx->overlay_pix_step[0]; + int slice_start, slice_end; + + i = FFMAX(-yp, 0); \ + imax = FFMIN3(-yp + dst_hp, FFMIN(src_hp, dst_hp), yp + src_hp); \ + + slice_start = i; + slice_end = i + imax; + + sp = src->data[0] + slice_start * src->linesize[0]; + dp = dst->data[dst_plane] + (yp + slice_start) * dst->linesize[dst_plane] + dst_offset; + + dap = dst->data[3] + ((yp + slice_start) << vsub) * dst->linesize[3]; + + for (i = slice_start; i < slice_end; i++) { + j = FFMAX(-xp, 0); + d = dp + (xp + j) * dst_step; + s = sp + j * sstep; + jmax = FFMIN(-xp + dst_wp, src_wp); + + for (; j < jmax; j++) { + uint32_t val = yuv_pal[*s]; + const uint8_t *sval = (uint8_t *)&val; + const int alpha = sval[3]; + + *d = (*d * (255 - alpha) + sval[plane] * alpha + 128) * 257 >> 16; + d += dst_step; + s += (1 << hsub); + } + dp += dst->linesize[dst_plane]; + sp += src->linesize[plane]; + dap += (1 << vsub) * dst->linesize[3]; + } +} + +#define RGB2Y(r, g, b) (uint8_t)(((66 * (r) + 129 * (g) + 25 * (b) + 128) >> 8) + 16) +#define RGB2U(r, g, b) (uint8_t)(((-38 * (r) - 74 * (g) + 112 * (b) + 128) >> 8) + 128) +#define RGB2V(r, g, b) (uint8_t)(((112 * (r) - 94 * (g) - 18 * (b) + 128) >> 8) + 128) +/* Converts R8 G8 B8 color to YUV. */ +static av_always_inline void rgb_2_yuv(uint8_t r, uint8_t g, uint8_t b, uint8_t* y, uint8_t* u, uint8_t* v) +{ + *y = RGB2Y((int)r, (int)g, (int)b); + *u = RGB2U((int)r, (int)g, (int)b); + *v = RGB2V((int)r, (int)g, (int)b); +} + +static av_always_inline void blend_yuv_8_8bits(AVFilterContext *ctx, AVFrame *dst, const AVSubtitleRect *src, int hsub, int vsub, int x, int y) +{ + OverlaySubsContext *s = ctx->priv; + const int src_w = src->w; + const int src_h = src->h; + const int dst_w = dst->width; + const int dst_h = dst->height; + const int sr = s->overlay_rgba_map[R]; + const int sg = s->overlay_rgba_map[G]; + const int sb = s->overlay_rgba_map[B]; + const int sa = s->overlay_rgba_map[A]; + uint32_t *pal = (uint32_t *)src->data[1]; + uint32_t yuvpal[256]; + + for (int i = 0; i < 256; ++i) { + const uint8_t *rgba = (uint8_t *)&pal[i]; + uint8_t *yuva = (uint8_t *)&yuvpal[i]; + rgb_2_yuv(rgba[sr], rgba[sg], rgba[sb], &yuva[Y], &yuva[U], &yuva[V]); + yuva[3] = rgba[sa]; + } + + blend_plane_8_8bits(ctx, dst, src, yuvpal, src_w, src_h, dst_w, dst_h, Y, 0, 0, x, y, s->main_desc->comp[Y].plane, s->main_desc->comp[Y].offset, s->main_desc->comp[Y].step); + blend_plane_8_8bits(ctx, dst, src, yuvpal, src_w, src_h, dst_w, dst_h, U, hsub, vsub, x, y, s->main_desc->comp[U].plane, s->main_desc->comp[U].offset, s->main_desc->comp[U].step); + blend_plane_8_8bits(ctx, dst, src, yuvpal, src_w, src_h, dst_w, dst_h, V, hsub, vsub, x, y, s->main_desc->comp[V].plane, s->main_desc->comp[V].offset, s->main_desc->comp[V].step); +} + +static int config_input_main(AVFilterLink *inlink) +{ + int ret; + AVFilterContext *ctx = inlink->dst; + OverlaySubsContext *s = inlink->dst->priv; + const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(inlink->format); + const AVPixFmtDescriptor *pix_desc_overlay = av_pix_fmt_desc_get(AV_PIX_FMT_PAL8); + + av_image_fill_max_pixsteps(s->main_pix_step, NULL, pix_desc); + av_image_fill_max_pixsteps(s->overlay_pix_step, NULL, pix_desc_overlay); + ff_fill_rgba_map(s->overlay_rgba_map, AV_PIX_FMT_RGB32); + + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + + s->main_desc = pix_desc; + + s->main_is_packed_rgb = ff_fill_rgba_map(s->main_rgba_map, inlink->format) >= 0; + s->main_has_alpha = ff_fmt_is_in(inlink->format, alpha_pix_fmts); + + /* Finish the configuration by evaluating the expressions + now when both inputs are configured. */ + s->var_values[VAR_MAIN_W ] = s->var_values[VAR_MW] = ctx->inputs[MAIN ]->w; + s->var_values[VAR_MAIN_H ] = s->var_values[VAR_MH] = ctx->inputs[MAIN ]->h; + s->var_values[VAR_OVERLAY_W] = s->var_values[VAR_OW] = ctx->inputs[OVERLAY]->w; + s->var_values[VAR_OVERLAY_H] = s->var_values[VAR_OH] = ctx->inputs[OVERLAY]->h; + s->var_values[VAR_HSUB] = 1<log2_chroma_w; + s->var_values[VAR_VSUB] = 1<log2_chroma_h; + s->var_values[VAR_X] = NAN; + s->var_values[VAR_Y] = NAN; + s->var_values[VAR_N] = 0; + s->var_values[VAR_T] = NAN; + s->var_values[VAR_POS] = NAN; + + if ((ret = set_expr(&s->x_pexpr, s->x_expr, "x", ctx)) < 0 || + (ret = set_expr(&s->y_pexpr, s->y_expr, "y", ctx)) < 0) + return ret; + + if (s->eval_mode == EVAL_MODE_INIT) { + eval_expr(ctx); + av_log(ctx, AV_LOG_VERBOSE, "x:%f xi:%d y:%f yi:%d\n", + s->var_values[VAR_X], s->x, + s->var_values[VAR_Y], s->y); + } + + av_log(ctx, AV_LOG_VERBOSE, + "main w:%d h:%d fmt:%s overlay w:%d h:%d fmt:%s\n", + ctx->inputs[MAIN]->w, ctx->inputs[MAIN]->h, + av_get_pix_fmt_name(ctx->inputs[MAIN]->format), + ctx->inputs[OVERLAY]->w, ctx->inputs[OVERLAY]->h, + av_get_pix_fmt_name(ctx->inputs[OVERLAY]->format)); + return 0; +} + +static int do_blend(FFFrameSync *fs) +{ + AVFilterContext *ctx = fs->parent; + AVFrame *mainpic, *second; + OverlaySubsContext *s = ctx->priv; + AVFilterLink *inlink = ctx->inputs[0]; + int ret; + + ret = ff_framesync_dualinput_get_writable(fs, &mainpic, &second); + if (ret < 0) + return ret; + if (!second) + return ff_filter_frame(ctx->outputs[0], mainpic); + + if (s->eval_mode == EVAL_MODE_FRAME) { + int64_t pos = mainpic->pkt_pos; + + s->var_values[VAR_N] = inlink->frame_count_out; + s->var_values[VAR_T] = mainpic->pts == AV_NOPTS_VALUE ? + NAN : mainpic->pts * av_q2d(inlink->time_base); + s->var_values[VAR_POS] = pos == -1 ? NAN : pos; + + s->var_values[VAR_OVERLAY_W] = s->var_values[VAR_OW] = second->width; + s->var_values[VAR_OVERLAY_H] = s->var_values[VAR_OH] = second->height; + s->var_values[VAR_MAIN_W ] = s->var_values[VAR_MW] = mainpic->width; + s->var_values[VAR_MAIN_H ] = s->var_values[VAR_MH] = mainpic->height; + + eval_expr(ctx); + av_log(ctx, AV_LOG_DEBUG, "n:%f t:%f pos:%f x:%f xi:%d y:%f yi:%d\n", + s->var_values[VAR_N], s->var_values[VAR_T], s->var_values[VAR_POS], + s->var_values[VAR_X], s->x, + s->var_values[VAR_Y], s->y); + } + + if (second->buf[0] && second->buf[0]->data) { + const AVSubtitle *sub = (AVSubtitle *)second->buf[0]->data; + unsigned num_rects, i; + + num_rects = sub->num_rects; + + for (i = 0; i < num_rects; i++) { + const AVSubtitleRect *sub_rect = sub->rects[i]; + + if (sub_rect->type != SUBTITLE_BITMAP) { + av_log(NULL, AV_LOG_WARNING, "sub2video: non-bitmap subtitle\n"); + return AVERROR_INVALIDDATA; + } + + switch (inlink->format) { + case AV_PIX_FMT_YUV420P: + blend_yuv_8_8bits(ctx, mainpic, sub_rect, 1, 1, sub_rect->x + s->x, sub_rect->y + s->y); + break; + case AV_PIX_FMT_YUV422P: + blend_yuv_8_8bits(ctx, mainpic, sub_rect, 1, 0, sub_rect->x + s->x, sub_rect->y + s->y); + break; + case AV_PIX_FMT_YUV444P: + blend_yuv_8_8bits(ctx, mainpic, sub_rect, 0, 0, sub_rect->x + s->x, sub_rect->y + s->y); + break; + case AV_PIX_FMT_RGB24: + case AV_PIX_FMT_BGR24: + case AV_PIX_FMT_ARGB: + case AV_PIX_FMT_RGBA: + case AV_PIX_FMT_BGRA: + case AV_PIX_FMT_ABGR: + blend_packed_rgb(ctx, mainpic, sub_rect, s->main_has_alpha, sub_rect->x + s->x, sub_rect->y + s->y, 1); + break; + default: + av_assert0(0); + break; + } + } + } + + return ff_filter_frame(ctx->outputs[0], mainpic); +} + +static av_cold int init(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + s->fs.on_event = do_blend; + return 0; +} + +static int activate(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + return ff_framesync_activate(&s->fs); +} + +#define OFFSET(x) offsetof(OverlaySubsContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_FILTERING_PARAM + +static const AVOption overlay_subs_options[] = { + { "x", "set the x expression", OFFSET(x_expr), AV_OPT_TYPE_STRING, {.str = "0"}, 0, 0, FLAGS }, + { "y", "set the y expression", OFFSET(y_expr), AV_OPT_TYPE_STRING, {.str = "0"}, 0, 0, FLAGS }, + { "eof_action", "Action to take when encountering EOF from secondary input ", + OFFSET(fs.opt_eof_action), AV_OPT_TYPE_INT, { .i64 = EOF_ACTION_REPEAT }, + EOF_ACTION_REPEAT, EOF_ACTION_PASS, .flags = FLAGS, "eof_action" }, + { "repeat", "Repeat the previous frame.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_REPEAT }, .flags = FLAGS, "eof_action" }, + { "endall", "End both streams.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_ENDALL }, .flags = FLAGS, "eof_action" }, + { "pass", "Pass through the main input.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_PASS }, .flags = FLAGS, "eof_action" }, + { "eval", "specify when to evaluate expressions", OFFSET(eval_mode), AV_OPT_TYPE_INT, {.i64 = EVAL_MODE_FRAME}, 0, EVAL_MODE_NB-1, FLAGS, "eval" }, + { "init", "eval expressions once during initialization", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_INIT}, .flags = FLAGS, .unit = "eval" }, + { "frame", "eval expressions per-frame", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_FRAME}, .flags = FLAGS, .unit = "eval" }, + { "shortest", "force termination when the shortest input terminates", OFFSET(fs.opt_shortest), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, + { "repeatlast", "repeat overlay of the last overlay frame", OFFSET(fs.opt_repeatlast), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, FLAGS }, + { NULL } +}; + +FRAMESYNC_DEFINE_CLASS(overlay_subs, OverlaySubsContext, fs); + +static const AVFilterPad overlay_subs_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input_main, + .flags = AVFILTERPAD_FLAG_NEEDS_WRITABLE, + }, + { + .name = "overlay", + .type = AVMEDIA_TYPE_SUBTITLE, + }, + { NULL } +}; + +static const AVFilterPad overlay_subs_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, + { NULL } +}; + +const AVFilter ff_vf_overlay_subs = { + .name = "overlay_subs", + .description = NULL_IF_CONFIG_SMALL("Overlay graphical subtitles on top of the input."), + .preinit = overlay_subs_framesync_preinit, + .init = init, + .uninit = uninit, + .priv_size = sizeof(OverlaySubsContext), + .priv_class = &overlay_subs_class, + .query_formats = query_formats, + .activate = activate, + .inputs = overlay_subs_inputs, + .outputs = overlay_subs_outputs, +}; diff --git a/libavfilter/vf_overlay_subs.h b/libavfilter/vf_overlay_subs.h new file mode 100644 index 0000000000..08e8637bff --- /dev/null +++ b/libavfilter/vf_overlay_subs.h @@ -0,0 +1,65 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVFILTER_OVERLAY_SUBS_H +#define AVFILTER_OVERLAY_SUBS_H + +#include "libavutil/eval.h" +#include "libavutil/pixdesc.h" +#include "framesync.h" +#include "avfilter.h" + +enum var_name { + VAR_MAIN_W, VAR_MW, + VAR_MAIN_H, VAR_MH, + VAR_OVERLAY_W, VAR_OW, + VAR_OVERLAY_H, VAR_OH, + VAR_HSUB, + VAR_VSUB, + VAR_X, + VAR_Y, + VAR_N, + VAR_POS, + VAR_T, + VAR_VARS_NB +}; + +typedef struct OverlaySubsContext { + const AVClass *class; + int x, y; ///< position of overlaid picture + + uint8_t main_is_packed_rgb; + uint8_t main_rgba_map[4]; + uint8_t main_has_alpha; + uint8_t overlay_rgba_map[4]; + int eval_mode; ///< EvalMode + + FFFrameSync fs; + + int main_pix_step[4]; ///< steps per pixel for each plane of the main output + int overlay_pix_step[4]; ///< steps per pixel for each plane of the overlay + int hsub, vsub; ///< chroma subsampling values + const AVPixFmtDescriptor *main_desc; ///< format descriptor for main input + + double var_values[VAR_VARS_NB]; + char *x_expr, *y_expr; + + AVExpr *x_pexpr, *y_pexpr; +} OverlaySubsContext; + +#endif /* AVFILTER_OVERLAY_SUBS_H */ From patchwork Mon Aug 30 08:16:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29873 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp3737864iov; Mon, 30 Aug 2021 01:17:16 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwnse/Yv3xCrc8QUQ5mVCLv62u0v/oYRn7ckK/MEYOnuSahduVwJlPZSBeSoNWDcFhZtmEg X-Received: by 2002:a17:907:1b06:: with SMTP id mp6mr24073467ejc.188.1630311435933; Mon, 30 Aug 2021 01:17:15 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id o20si8388289eds.216.2021.08.30.01.17.15; Mon, 30 Aug 2021 01:17:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=VNOl63be; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9E40668A2CE; Mon, 30 Aug 2021 11:16:56 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08olkn2027.outbound.protection.outlook.com [40.92.46.27]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AF231689F94 for ; Mon, 30 Aug 2021 11:16:54 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VdtjbQxDs/IcE9A4fqaWfJ3z2OjCq9WSB7KuuyEdtdN8Gn9djWUjCujUJ1PBYF+cRY806d174n2Ilgtt7AMFnT4NzxFt8ckLoFKAAEt66C2lMHS4yU/8SoE1y8tOtDcHnxxVfyC4UnFXIjYWXCMs74grzBaSqr6OHmoIzqQcM0yP7KDAP8F7BnAURkueEczJORWnGhAe8vAAc5FeA9ILtqNxBnFzIIarMlnA086MBN32UOO9vkbdJvA18+HLr5q3Is8I4ZsylzTNeL4DEoV2dQEpr96LLpYPA7klUGBVMUumePut/NUgJm0ukfJL3uuM119rUm375o6VGe7rM2N4+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1l33jcmVP1veB9cu3OhyuqCY30g9BBAqlRDbDLgbTf4=; b=KRV/D7lJ1cQsGn8SZwFJ8sauqWLYe8COHJdmVHSCp6mPIXY5M2C8Y2db6ECjcpVQKirvvVvaqij/3GxTx64fUsqk1vLdTtaq7k3wW7eJgeQ/gpiVCVkq1/suQi2jyw8rZ7VyMHPw3MPQvc3Pq/olv+UZIqR3EtRCcvAqS8FQJXQAmYj91oKWe58nQ+BS/ni79ZYLzLB8WW4XRysRA59rbGigZGDMxvActI71ij8aa92BOcK0CWreo81HVKEMOwSM02DKnNQFSxKOF0HfiOQQg1K86XwBiFVlm9r6q/epyqGlFQ1iaITsVR85q4f25pkjl/BhtLMg9dUwXw6D3fct8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1l33jcmVP1veB9cu3OhyuqCY30g9BBAqlRDbDLgbTf4=; b=VNOl63beXJynzKKUAKFxPe1XPa0/ER9dmr3xxEtScCcguudp1EhSN0vLGALkX47PdK8UWz3VTPriDPMtIRu5+qvkGVoRv7KuciGnezmiC3QGTM7Nh9sFVnbtgS6ai0Loxoe/87ki7DUkNimDX822vvtQuxbAscGYa4oqJWpCkM39akqSyqj3W54L+bh1im74BPF7YevqTDQLiuto4UIS2p2MuOLKXqnf3Ttzo5XyFsVX6XNRLd642djoPLAPPOgL16ueF6IkF5qxtl4cIZkse9+FCDXmeYfNxGS9+wjdrIQNJzD4DS5EJSim9PAe/SVdjz7Y1wE3QsSE9sTL8p6XUA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5853.namprd04.prod.outlook.com (2603:10b6:208:a6::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.19; Mon, 30 Aug 2021 08:16:51 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 08:16:51 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v2 5/8] avfilter/sub2video: Add sub2video filter Thread-Index: AdeUd97C+XQyjRkXSyOJSLj3QMpn5g== Date: Mon, 30 Aug 2021 08:16:51 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [kDkD0Lz4NZhx8LUCyB8J7wTGTvNY1HlP] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: ad313395-7f4b-453d-7ee4-08d96b8e8565 x-ms-traffictypediagnostic: MN2PR04MB5853: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 5hg5dOdNcbe0DuFVt5q/5rviTqs3ZAx9zTnfIHxGIWhXEWUuzV/r9DQ7NmkZYm29kFuQF+euvt1qRQdRoqpJgxOyiLBRqRGhDOwtX55sRqTXY4ShWgkI1QiqHZ850vcZhLPPNor9ePLIUUImi9G+jMd2NBlQtMcZxsUoyaI15jiQQ8kKfhFmB9cUPenMTWSQt7L3Tntifjr4Lz70mfxtus7D6k1OM7IRYh/lKUysgcqOrGH3im0BDS9LdXHKDekoGkXabmZ0R0GcnAfhOZrLnn5YcpJ/XKICgNG3JxnR/sppcbvQaiurNqXs9MlH0C4LILm6YxHhQQ0/Ez9T6NISUrxnbZ/rz7zrZUGru4fkECgLHlGdmZaflxitrBuEp3U5gOIj6FBRZ03WPRafvCwq1RcVDpOHWBVnx2y4hpnrvA4vtvSqyoUKZFjObQOTNLjg x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: Opyre+skEh3O09yn4Z6+91SMxA4RdZW9+rMpM4ZJZwLx5ZKM0S27kfUM7qshLaNNnMYF03fg8KjlgKY2+gZhj3Q27dL5oonA9Lz2VRTH4ki1BNcmzpwsEqf5RZcKgTgFKk5oIgcFR15Rr9B/CRLEaQ== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: ad313395-7f4b-453d-7ee4-08d96b8e8565 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2021 08:16:51.6295 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5853 Subject: [FFmpeg-devel] [PATCH v2 5/8] avfilter/sub2video: Add sub2video filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: nIFL220BUUxt Signed-off-by: softworkz --- v2 Update: - Implemented Andreas' suggestions - overlay_subs filter: - removed duplicated code - implemented direct (no pre-conversion) blending of graphical subtitle rects - Supported input formats: - all packed RGB formats (with and without alpha) - yuv420p, yuv422p, yuv444p libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/svf_sub2video.c | 260 ++++++++++++++++++++++++++++++++++++ 3 files changed, 262 insertions(+) create mode 100644 libavfilter/svf_sub2video.c diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 8130f7b46c..e38c6b6f6d 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -542,6 +542,7 @@ OBJS-$(CONFIG_SHOWSPECTRUMPIC_FILTER) += avf_showspectrum.o OBJS-$(CONFIG_SHOWVOLUME_FILTER) += avf_showvolume.o OBJS-$(CONFIG_SHOWWAVES_FILTER) += avf_showwaves.o OBJS-$(CONFIG_SHOWWAVESPIC_FILTER) += avf_showwaves.o +OBJS-$(CONFIG_SUB2VIDEO_FILTER) += svf_sub2video.o OBJS-$(CONFIG_SPECTRUMSYNTH_FILTER) += vaf_spectrumsynth.o # multimedia sources diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index abd0a47750..5b631b3617 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -518,6 +518,7 @@ extern const AVFilter ff_avf_showvolume; extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; +extern const AVFilter ff_svf_sub2video; /* multimedia sources */ extern const AVFilter ff_avsrc_amovie; diff --git a/libavfilter/svf_sub2video.c b/libavfilter/svf_sub2video.c new file mode 100644 index 0000000000..689c2a565c --- /dev/null +++ b/libavfilter/svf_sub2video.c @@ -0,0 +1,260 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * graphical subtitles to video conversion, based on previous sub2video + * implementation. + */ + +#include + +#include "libavutil/audio_fifo.h" +#include "libavutil/avassert.h" +#include "libavutil/avstring.h" +#include "libavutil/channel_layout.h" +#include "libavutil/opt.h" +#include "libavutil/parseutils.h" +#include "libavutil/xga_font_data.h" +#include "avfilter.h" +#include "blend.h" +#include "filters.h" +#include "internal.h" +#include "libavcodec/avcodec.h" +typedef struct Sub2VideoContext { + const AVClass *class; + int w, h; + AVFrame *outpicref; + int pixstep; +} Sub2VideoContext; + +#define OFFSET(x) offsetof(Sub2VideoContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM + +////static int alloc_out_frame(Sub2VideoContext *s2v_ctx, const int16_t *p, +//// const AVFilterLink *inlink, AVFilterLink *outlink, +//// const AVFrame *in) +////{ +//// if (!s2v_ctx->outpicref) { +//// int j; +//// AVFrame *out = s2v_ctx->outpicref = +//// ff_get_video_buffer(outlink, outlink->w, outlink->h); +//// if (!out) +//// return AVERROR(ENOMEM); +//// out->width = outlink->w; +//// out->height = outlink->h; +//// out->pts = in->pts + av_rescale_q((p - (int16_t *)in->data[0]) / inlink->channels, +//// av_make_q(1, inlink->sample_rate), +//// outlink->time_base); +//// for (j = 0; j < outlink->h; j++) +//// memset(out->data[0] + j*out->linesize[0], 0, outlink->w * s2v_ctx->pixstep); +//// } +//// return 0; +////} + + +static const AVOption sub2video_options[] = { + { "size", "set video size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = "640x512"}, 0, 0, FLAGS }, + { "s", "set video size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = "640x512"}, 0, 0, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(sub2video); + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats = NULL; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { SUBTITLE_BITMAP, SUBTITLE_NONE }; + static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_RGB32, AV_PIX_FMT_NONE }; + int ret; + + /* set input subtitle format */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output video format */ + formats = ff_make_format_list(pix_fmts); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + Sub2VideoContext *s = ctx->priv; + + s->w = 1920; // inlink->w; + s->h = 1080; // inlink->h; + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + Sub2VideoContext *s = outlink->src->priv; + float overlap; + + outlink->w = s->w; + outlink->h = s->h; + outlink->sample_aspect_ratio = (AVRational){1,1}; + + return 0; +} + +static void sub2video_copy_rect(uint8_t *dst, int dst_linesize, int w, int h, + AVSubtitleRect *r) +{ + uint32_t *pal, *dst2; + uint8_t *src, *src2; + int x, y; + + if (r->type != SUBTITLE_BITMAP) { + av_log(NULL, AV_LOG_WARNING, "sub2video: non-bitmap subtitle\n"); + return; + } + if (r->x < 0 || r->x + r->w > w || r->y < 0 || r->y + r->h > h) { + av_log(NULL, AV_LOG_WARNING, "sub2video: rectangle (%d %d %d %d) overflowing %d %d\n", + r->x, r->y, r->w, r->h, w, h + ); + return; + } + + dst += r->y * dst_linesize + r->x * 4; + src = r->data[0]; + pal = (uint32_t *)r->data[1]; + for (y = 0; y < r->h; y++) { + dst2 = (uint32_t *)dst; + src2 = src; + for (x = 0; x < r->w; x++) + *(dst2++) = pal[*(src2++)]; + dst += dst_linesize; + src += r->linesize[0]; + } +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *src_frame) +{ + Sub2VideoContext *s = inlink->dst->priv; + AVFilterLink *outlink = inlink->dst->outputs[0]; + AVSubtitle *sub; + int ret; + int dst_linesize; + AVFrame *out; + unsigned int num_rects, i; + uint8_t *dst; + + outlink->w = 1920; + outlink->h = 1080; + + out = av_frame_alloc(); + if (!out) { + av_frame_free(&src_frame); + return AVERROR(ENOMEM); + } + + out->width = outlink->w; + out->height = outlink->h; + out->format = AV_PIX_FMT_RGB32; + + if ((ret = av_frame_get_buffer(out, 0)) < 0) + return ret; + + memset(out->data[0], 0, out->height * out->linesize[0]); + + ////out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + ////if (!out) { + //// av_frame_free(&src_frame); + //// return AVERROR(ENOMEM); + ////} + ////memset(out->data[0], 0, out->linesize[0] * out->height ); + + out->pts = src_frame->pts; + out->repeat_pict = src_frame->repeat_pict; + out->pkt_dts = src_frame->pkt_dts; + out->pkt_pos = src_frame->pkt_pos; + out->pkt_size = src_frame->pkt_size; + out->pkt_duration = src_frame->pkt_duration; + out->reordered_opaque = src_frame->reordered_opaque; + out->best_effort_timestamp = src_frame->best_effort_timestamp; + out->flags = src_frame->flags; + + sub = (AVSubtitle *)src_frame->data[0]; + + if (sub) { + num_rects = sub->num_rects; + dst = out->data [0]; + dst_linesize = out->linesize[0]; + for (i = 0; i < num_rects; i++) + sub2video_copy_rect(dst, dst_linesize, out->width, out->height, sub->rects[i]); + } + + av_frame_free(&src_frame); + return ff_filter_frame(outlink, out); +} + +////static int activate(AVFilterContext *ctx) +////{ +//// AVFilterLink *inlink = ctx->inputs[0]; +//// AVFilterLink *outlink = ctx->outputs[0]; +//// Sub2VideoContext *s = ctx->priv; +//// +//// do { +//// int ret = ff_outlink_get_status(outlink); +//// if (ret) { +//// ff_inlink_set_status(inlink, ret); +//// return 0; +//// } +//// } while (0); +//// +//// return FFERROR_NOT_READY; +////} + +static const AVFilterPad sub2video_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_frame, + .config_props = config_input, + }, + { NULL } +}; + +static const AVFilterPad sub2video_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, + { NULL } +}; + +AVFilter ff_svf_sub2video = { + .name = "sub2video", + .description = NULL_IF_CONFIG_SMALL("Convert graphical subtitles to video"), + .query_formats = query_formats, + .priv_size = sizeof(Sub2VideoContext), + .priv_class = &sub2video_class, + .inputs = sub2video_inputs, + .outputs = sub2video_outputs, +}; From patchwork Mon Aug 30 08:16:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29871 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp3737985iov; Mon, 30 Aug 2021 01:17:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzxztN8Z5KCQpWjG9CG4n7OC/Ap1e53fEeS7O+Du7lOCBlb83FMVa67uDp2ZuwVhc+7lwCs X-Received: by 2002:a17:906:774f:: with SMTP id o15mr24150643ejn.200.1630311448376; Mon, 30 Aug 2021 01:17:28 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l7si9070424edb.31.2021.08.30.01.17.26; Mon, 30 Aug 2021 01:17:28 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=jJ+fpam7; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 859CE68A360; Mon, 30 Aug 2021 11:17:02 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08olkn2108.outbound.protection.outlook.com [40.92.46.108]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 422AA68A1D4 for ; Mon, 30 Aug 2021 11:17:00 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DmSAgbPP+oz3ZNnqJ3/S7x4Icf6DXhSJUQSNYCZJVoaZYKyj2ePyJ5lVh0G8nnZ027WuihV3hvkJNroJKxV3GNHMCL9jJBRcQULMbNCZnByvnVlqWO0ML5qjrsnlJzEeWg9bJeGgoDp540zLO34F8b2XT326D9eM00D3Sh4XnSp3FyXPCzT3QjEujN9TR3g15KD6qvM2z0fJE/xmLHE6jxt+sbBYQp08JCGEf230MYwjETq3E+MBFgzv6b7MZ+d4Xi1BZcTo1NKjuEtiTNhRK+jMSv/UOYsaMaMiLrzqAGyZeEFf9cbfSrzHLtxfDSrMeFFaw73bY1qhMz/RjZ4LVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=M++LZzAXzIr4CWwE7htwpzgAhSBf8XdZsImyZ4r1NKg=; b=SG49cXloHVU8V0g47iERx4seCEvCQf69yb7fS1KbzCHup7qj/YMrOwt93Jt7w/+aiAORNDWNXKBFuRSQQeoDmdV6FmtKs23GWU44w4556NPatup8qQ9o5U28la1AHvJ/zMACfQzPyXSkzoRMXmxesHtvi7k/XRo1gDoBhRm3/e9CHOGqNBzXTVfIDaKYbTrhzM+q7j5UudPYjOVl8qybGF321ATMAKwpBuA0Ipkp1+PBdefBtwRdZI5fMsurp9QUusH71fZS4ArhE/ptlIhKD+AhztbA8f+GWY5QCHJRg/j08JAERn6qnUbQW4Vgr9WyesTIi++AIQ201eVH3r4Hjg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=M++LZzAXzIr4CWwE7htwpzgAhSBf8XdZsImyZ4r1NKg=; b=jJ+fpam7iz2/uoTerQWWbluF8RAiN7GAHz5UCmjjshD17IQpPrckJ4M9aN086inqjW79O4OhXK8iqY2YeEmEDJQ5iV/RbRmkQUX1MsP/UCoq+XY4Dme6J/HPMHY5R2mfmc0NYxYu4xUWR++nFM3KFMuZswcThHqTR49wJ2A0VW2SFrdxwjHpg0uOMXUngWRb1gOtSw9RhdM6T5M8s2pFIYU+QPGcgAj+ezUek5Kl4g1uICMppUokxjjRQR640Vx3tKHpiX9RVsOW7ZsYBgVWgwVn5buhuBoHt7iHsbu5kRNO3tI98PDhBoPyM5Dnvxl0upvzsRcRjNGK2ZHaxKdR+w== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5853.namprd04.prod.outlook.com (2603:10b6:208:a6::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.19; Mon, 30 Aug 2021 08:16:58 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 08:16:58 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v2 6/8] avfilter/sbuffer: Add sbuffersrv and sbuffersink filters Thread-Index: AdeUpHY9YnLYcJe+Q/y5MTrjWNM3hg== Date: Mon, 30 Aug 2021 08:16:58 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [F1tKAGptf4TWyXAqapS78/GVcZpLj+aS] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 76d7b9c1-c296-42af-38a4-08d96b8e8970 x-ms-traffictypediagnostic: MN2PR04MB5853: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: sw6omXRbGHXloXHxAZAzMbvmI/dhrg4ChFpPw4cPpGc7S+5MWKSHUq5KrUZGG7xq+bRlgXclC+iKpmzqhBu6R5S4US7j5MOXHiIR6CB7De9NtyMosAXrAcFwUf/G/hZily2etAME0rQ6SRPkhaOXIIjiMHC9wA2LxglTO2kK0RTNlWxqEpD2r3meCtxJr5cNtpL72KQZIbnTnIkui587OpJW6zXomb5y0BjBpDp3nFrHgm3eK7BGDg9HapHpQ033rZReh7z3cPbzbB6UDT1o1dyiKf7EyQEoyN39CJylznCxrG32KpxuY4oXNXSXYlN5fB0O6WTgOvNyUii5BIM3ZvV1HzpYiFO63EGB05crErT/uoUUraiytlYJ4bBUaVkXKXv6QeFmBqYxfedOoiK+hGiDkFM+DNRRQdiROEiiUmBx+pQwwM+jbhepWhtyt7hs x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: FguNZGWTCbo2yRCk/UxrvM3OKQlIe/trGVyWmke+5jAhs3sMUBQE7NHKGQdPqOxP+/Ia2OwuEwthngTsgP3j3uvgNXQV20dXcKDPRqle92lFFGvPlQxzaG1RjYldX41B7xC+etw+EUvQgTDCF87fiw== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 76d7b9c1-c296-42af-38a4-08d96b8e8970 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2021 08:16:58.4657 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5853 Subject: [FFmpeg-devel] [PATCH v2 6/8] avfilter/sbuffer: Add sbuffersrv and sbuffersink filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: NdiutS8IQYaM Signed-off-by: softworkz --- v2 Update: - Implemented Andreas' suggestions - overlay_subs filter: - removed duplicated code - implemented direct (no pre-conversion) blending of graphical subtitle rects - Supported input formats: - all packed RGB formats (with and without alpha) - yuv420p, yuv422p, yuv444p configure | 2 +- libavfilter/allfilters.c | 10 ++++--- libavfilter/buffersink.c | 65 ++++++++++++++++++++++++++++++++++++++++ libavfilter/buffersink.h | 15 ++++++++++ libavfilter/buffersrc.c | 65 ++++++++++++++++++++++++++++++++++++++++ libavfilter/buffersrc.h | 1 + 6 files changed, 153 insertions(+), 5 deletions(-) diff --git a/configure b/configure index 9249254b70..fdb3bf1714 100755 --- a/configure +++ b/configure @@ -7715,7 +7715,7 @@ print_enabled_components(){ fi done if [ "$name" = "filter_list" ]; then - for c in asrc_abuffer vsrc_buffer asink_abuffer vsink_buffer; do + for c in asrc_abuffer vsrc_buffer ssrc_sbuffer asink_abuffer vsink_buffer ssink_sbuffer; do printf " &ff_%s,\n" $c >> $TMPH done fi diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 5b631b3617..5bd54db2c8 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -528,10 +528,12 @@ extern const AVFilter ff_avsrc_movie; * they are formatted to not be found by the grep * as they are manually added again (due to their 'names' * being the same while having different 'types'). */ -extern const AVFilter ff_asrc_abuffer; -extern const AVFilter ff_vsrc_buffer; -extern const AVFilter ff_asink_abuffer; -extern const AVFilter ff_vsink_buffer; +extern const AVFilter ff_asrc_abuffer; +extern const AVFilter ff_vsrc_buffer; +extern const AVFilter ff_ssrc_sbuffer; +extern const AVFilter ff_asink_abuffer; +extern const AVFilter ff_vsink_buffer; +extern const AVFilter ff_ssink_sbuffer; extern const AVFilter ff_af_afifo; extern const AVFilter ff_vf_fifo; diff --git a/libavfilter/buffersink.c b/libavfilter/buffersink.c index 07c4812f29..20845eb10f 100644 --- a/libavfilter/buffersink.c +++ b/libavfilter/buffersink.c @@ -57,6 +57,10 @@ typedef struct BufferSinkContext { int *sample_rates; ///< list of accepted sample rates, terminated by -1 int sample_rates_size; + /* only used for subtitles */ + enum AVSubtitleType *subtitle_types; ///< list of accepted subtitle types, must be terminated with -1 + int subtitle_types_size; + AVFrame *peeked_frame; } BufferSinkContext; @@ -168,6 +172,15 @@ AVABufferSinkParams *av_abuffersink_params_alloc(void) return NULL; return params; } + +AVSBufferSinkParams *av_sbuffersink_params_alloc(void) +{ + AVSBufferSinkParams *params = av_mallocz(sizeof(AVSBufferSinkParams)); + + if (!params) + return NULL; + return params; +} #endif static av_cold int common_init(AVFilterContext *ctx) @@ -305,6 +318,31 @@ static int asink_query_formats(AVFilterContext *ctx) return 0; } +static int ssink_query_formats(AVFilterContext *ctx) +{ + BufferSinkContext *buf = ctx->priv; + AVFilterFormats *formats = NULL; + unsigned i; + int ret; + + if ((ret = ff_default_query_formats(ctx)) < 0) + return ret; + + ////CHECK_LIST_SIZE(pixel_fmts) + ////if (buf->pixel_fmts_size) { + //// for (i = 0; i < NB_ITEMS(buf->pixel_fmts); i++) + //// if ((ret = ff_add_format(&formats, buf->pixel_fmts[i])) < 0) + //// return ret; + //// if ((ret = ff_set_common_formats(ctx, formats)) < 0) + //// return ret; + ////} else { + //// if ((ret = ff_default_query_formats(ctx)) < 0) + //// return ret; + ////} + + return 0; +} + #define OFFSET(x) offsetof(BufferSinkContext, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM static const AVOption buffersink_options[] = { @@ -322,9 +360,16 @@ static const AVOption abuffersink_options[] = { { NULL }, }; #undef FLAGS +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_SUBTITLE_PARAM +static const AVOption sbuffersink_options[] = { + { "subtitle_types", "set the supported subtitle formats", OFFSET(subtitle_types), AV_OPT_TYPE_BINARY, .flags = FLAGS }, + { NULL }, +}; +#undef FLAGS AVFILTER_DEFINE_CLASS(buffersink); AVFILTER_DEFINE_CLASS(abuffersink); +AVFILTER_DEFINE_CLASS(sbuffersink); static const AVFilterPad avfilter_vsink_buffer_inputs[] = { { @@ -365,3 +410,23 @@ const AVFilter ff_asink_abuffer = { .inputs = avfilter_asink_abuffer_inputs, .outputs = NULL, }; + +static const AVFilterPad avfilter_ssink_sbuffer_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + }, + { NULL } +}; + +AVFilter ff_ssink_sbuffer = { + .name = "sbuffersink", + .description = NULL_IF_CONFIG_SMALL("Buffer subtitle frames, and make them available to the end of the filter graph."), + .priv_class = &sbuffersink_class, + .priv_size = sizeof(BufferSinkContext), + .init = common_init, + .query_formats = ssink_query_formats, + .activate = activate, + .inputs = avfilter_ssink_sbuffer_inputs, + .outputs = NULL, +}; diff --git a/libavfilter/buffersink.h b/libavfilter/buffersink.h index 69ed0f29a8..b439b586c5 100644 --- a/libavfilter/buffersink.h +++ b/libavfilter/buffersink.h @@ -129,6 +129,21 @@ typedef struct AVABufferSinkParams { */ attribute_deprecated AVABufferSinkParams *av_abuffersink_params_alloc(void); + +/** + * Deprecated and unused struct to use for initializing an sbuffersink context. + */ +typedef struct AVSBufferSinkParams { + const int *subtitle_type; +} AVSBufferSinkParams; + +/** + * Create an AVSBufferSinkParams structure. + * + * Must be freed with av_free(). + */ +attribute_deprecated +AVSBufferSinkParams *av_sbuffersink_params_alloc(void); #endif /** diff --git a/libavfilter/buffersrc.c b/libavfilter/buffersrc.c index 4d0bb4f91d..eeef3e2390 100644 --- a/libavfilter/buffersrc.c +++ b/libavfilter/buffersrc.c @@ -39,6 +39,7 @@ #include "formats.h" #include "internal.h" #include "video.h" +#include "libavcodec/avcodec.h" typedef struct BufferSourceContext { const AVClass *class; @@ -63,6 +64,9 @@ typedef struct BufferSourceContext { uint64_t channel_layout; char *channel_layout_str; + /* subtitle only */ + enum AVSubtitleType subtitle_type; + int eof; } BufferSourceContext; @@ -130,6 +134,9 @@ int av_buffersrc_parameters_set(AVFilterContext *ctx, AVBufferSrcParameters *par if (param->channel_layout) s->channel_layout = param->channel_layout; break; + case AVMEDIA_TYPE_SUBTITLE: + s->subtitle_type = param->format; + break; default: return AVERROR_BUG; } @@ -197,6 +204,8 @@ int attribute_align_arg av_buffersrc_add_frame_flags(AVFilterContext *ctx, AVFra CHECK_AUDIO_PARAM_CHANGE(ctx, s, frame->sample_rate, frame->channel_layout, frame->channels, frame->format, frame->pts); break; + case AVMEDIA_TYPE_SUBTITLE: + break; default: return AVERROR(EINVAL); } @@ -269,6 +278,7 @@ unsigned av_buffersrc_get_nb_failed_requests(AVFilterContext *buffer_src) #define OFFSET(x) offsetof(BufferSourceContext, x) #define A AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_AUDIO_PARAM #define V AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM +#define S AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_SUBTITLE_PARAM static const AVOption buffer_options[] = { { "width", NULL, OFFSET(w), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, V }, @@ -298,6 +308,14 @@ static const AVOption abuffer_options[] = { AVFILTER_DEFINE_CLASS(abuffer); +static const AVOption sbuffer_options[] = { + { "time_base", NULL, OFFSET(time_base), AV_OPT_TYPE_RATIONAL, { .dbl = 0 }, 0, INT_MAX, S }, + { "subtitle_type", NULL, OFFSET(subtitle_type), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, S }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(sbuffer); + static av_cold int init_audio(AVFilterContext *ctx) { BufferSourceContext *s = ctx->priv; @@ -347,6 +365,21 @@ static av_cold int init_audio(AVFilterContext *ctx) return ret; } +static av_cold int init_subtitle(AVFilterContext *ctx) +{ + BufferSourceContext *c = ctx->priv; + + if (c->subtitle_type == SUBTITLE_BITMAP) + av_log(ctx, AV_LOG_VERBOSE, "graphical subtitles - w:%d h:%d tb:%d/%d\n", + c->w, c->h, c->time_base.num, c->time_base.den); + else + av_log(ctx, AV_LOG_VERBOSE, "text subtitles - tb:%d/%d\n", + c->time_base.num, c->time_base.den); + + return 0; +} + + static av_cold void uninit(AVFilterContext *ctx) { BufferSourceContext *s = ctx->priv; @@ -381,6 +414,11 @@ static int query_formats(AVFilterContext *ctx) if ((ret = ff_set_common_channel_layouts(ctx, channel_layouts)) < 0) return ret; break; + case AVMEDIA_TYPE_SUBTITLE: + if ((ret = ff_add_format (&formats, c->subtitle_type)) < 0 || + (ret = ff_set_common_formats (ctx , formats )) < 0) + return ret; + break; default: return AVERROR(EINVAL); } @@ -408,6 +446,9 @@ static int config_props(AVFilterLink *link) if (!c->channel_layout) c->channel_layout = link->channel_layout; break; + case AVMEDIA_TYPE_SUBTITLE: + link->format = c->subtitle_type; + break; default: return AVERROR(EINVAL); } @@ -474,3 +515,27 @@ const AVFilter ff_asrc_abuffer = { .outputs = avfilter_asrc_abuffer_outputs, .priv_class = &abuffer_class, }; + +static const AVFilterPad avfilter_ssrc_sbuffer_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .request_frame = request_frame, + .config_props = config_props, + }, + { NULL } +}; + +const AVFilter ff_ssrc_sbuffer = { + .name = "sbuffer", + .description = NULL_IF_CONFIG_SMALL("Buffer subtitle frames, and make them accessible to the filterchain."), + .priv_size = sizeof(BufferSourceContext), + .query_formats = query_formats, + + .init = init_subtitle, + .uninit = uninit, + + .inputs = NULL, + .outputs = avfilter_ssrc_sbuffer_outputs, + .priv_class = &sbuffer_class, +}; diff --git a/libavfilter/buffersrc.h b/libavfilter/buffersrc.h index 08fbd18a47..929a2fa249 100644 --- a/libavfilter/buffersrc.h +++ b/libavfilter/buffersrc.h @@ -74,6 +74,7 @@ typedef struct AVBufferSrcParameters { /** * video: the pixel format, value corresponds to enum AVPixelFormat * audio: the sample format, value corresponds to enum AVSampleFormat + * subtitles: the subtitle format, value corresponds to enum AVSubtitleType */ int format; /** From patchwork Mon Aug 30 08:17:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29870 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp3738099iov; Mon, 30 Aug 2021 01:17:39 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw6AKkkuhrKTIUjgqLMBDhaSooPyOKDAR7LIZt6BRn3lg69e4JdYsdHBWhIHq8dl2dcKrIu X-Received: by 2002:a50:ff19:: with SMTP id a25mr21948832edu.311.1630311458856; Mon, 30 Aug 2021 01:17:38 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hv22si8573765ejc.188.2021.08.30.01.17.38; Mon, 30 Aug 2021 01:17:38 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=ZR3fGpjP; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 745EB68A2D5; Mon, 30 Aug 2021 11:17:15 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08olkn2061.outbound.protection.outlook.com [40.92.46.61]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7933F68A10A for ; Mon, 30 Aug 2021 11:17:08 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WX2BeKnMrYfi4evm9CXSUjBSzS5Kf4+X0hD2LBY/jji6H3ADTCS/Q7Jlsv81Vuqu2CWqZ60y/8TeWic20BiFQvEoYhaxiZCXL3G+O1G2oTP6/Jm0RRpMcjsP/4HmRAZd3Tm2g9Wja/mtFTIlRb2QvM/c59cwu3Y1lN13LvvlT615Ng6REyaE89ASmvGxq7KN5dCP1/haiNciWVbrT7lwYD/gTZfue0SO4h9Ka3IYxY556Ikm3dK88Phjb/jRx7St41+wpgzGnHhUE7VZ7cwXl427ewvjNsj4aGEjFQn8b9TVzkDJbeDNSFHmK9wRNg1v/H/EtMAU8US/Zale6FDM6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cliA7OcqsRBJmhkPDtin53W9YR2aJY6xcHqIcJpgkVU=; b=mDx8X4u3ITAHuoqUlolRBBnaKmE1cQVjfuuxuvcVuTCkb/ANN0vWHT6Q2ItPt0X998d0VT+zVxvLYp7yY7j5+TaMJ5ED9fbdn4/bvmx7+SntUhCFByiZ+KBRrNmBosbmQgCGUXSmgojFDmud4B5ze85aHpmH0zi8KNm8TeCU+Elyc3IsFiZnmRm1Hue18sY+1ioKFowYpajW4y070b1WicdTWa8krgr8wU7j3oVn15rOPRhC7PzU9qgPFQNHWCnMkjGtHF38hfUyzH4FD6M07R4PiHBJTsgktx7jaRCCk2Y2AORAgxeGqyOLIZQjMm86asflSAHLnwvECQp78Nhiww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cliA7OcqsRBJmhkPDtin53W9YR2aJY6xcHqIcJpgkVU=; b=ZR3fGpjPdUE8F+gg72A3QfnmWwj8rviGfIGcOnglOMPoKMWTvYICVkTRaAsoqZ0fRWNhtaqklZjVqTOxAG+FwTbzW7AmcoJVbiN47bup2mRquHmotuwy8+fE/w25SGgWG1p+A4k5ZFtsBcO7eFaL2K5n6HUP+QeQDVqwOgBuVSviD8FbvbrofJdGQFj5aF2UpKc2FeMHiQCswkK2iQBlLgmL37Ciw+bUY1mJw6lHuPXDg3f/VOth///62bxr1fwJXvRtGA7jNF93JLPbfDGkB7POaHQze63MH/vCOxZ/Oh/2vPkLqcpt/QEnKhEFWDBC/k6qfNNyptQ/+tUCnlhaUQ== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5853.namprd04.prod.outlook.com (2603:10b6:208:a6::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.19; Mon, 30 Aug 2021 08:17:06 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 08:17:06 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v2 7/8] avfilter/sleet: Add sleet filter Thread-Index: AdeUeEEb3Th5Ze/STdKI6w1tUaGrBg== Date: Mon, 30 Aug 2021 08:17:06 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [bxOEGm50zY0X12N7KA3lTfmDgvD+x9B2] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 947d9c87-6892-40f6-d87e-08d96b8e8df7 x-ms-traffictypediagnostic: MN2PR04MB5853: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: keEg206+p7+zB1I/HlcpEgeUHk+WFuP5BOpd6Cw/ZbmU21hScWvxFjVYNLVRVZo1HS0vcYeF7cYtEAJ9nP0+P9DPiEw/WwmlEQyDU84sJno7L42pKXqh38P1FJCWw1WEQXsRY+XemOEaYva8iEIGpcLyZRRS1pK7CxZQxHRHsOGsCS4jr1iAf6tqe9psHMdJmVJWdYnboMoZ+7T1/xx1cNlZb+h+2+oFSYIFNcYfK6uuDPSAXpHxDhbqzDJvHDg4sfI3Jz5mcpHHoH/ejFym9NjvRckQ4AlIzd5q+EWiDR5/F5IgwaHH7k1bCXbuGohZQnH50bSpdBk9VhnmTyQpdHXbOl1fe5lzocqLtM1/s5W/sgeAIHVd8GvPSwSSzWzIfMBSN0e0rDQB6Rc/54Qe66gpoTuEypmeIkPby2WRFsb6Z3Fq9mB+OSQcpD2UXeeT x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: JXpCHef3g+Eb1gEKPFIUehLL07m+8RM3RTfO1OVvxXmhfECtK2eYPBQzKniuq9KCT1vAsVOordDo7SPs4c90yy4Hp7VQgxBOK5i2SAH3yxDBBsihdTXF/YWHnhNYGIbXPrJLgV80FzH32lGFW7RQpg== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 947d9c87-6892-40f6-d87e-08d96b8e8df7 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2021 08:17:06.0704 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5853 Subject: [FFmpeg-devel] [PATCH v2 7/8] avfilter/sleet: Add sleet filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 4tms4CXTpQDz Signed-off-by: softworkz --- v2 Update: - Implemented Andreas' suggestions - overlay_subs filter: - removed duplicated code - implemented direct (no pre-conversion) blending of graphical subtitle rects - Supported input formats: - all packed RGB formats (with and without alpha) - yuv420p, yuv422p, yuv444p libavfilter/Makefile | 3 + libavfilter/allfilters.c | 1 + libavfilter/sf_sleet.c | 209 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 213 insertions(+) create mode 100644 libavfilter/sf_sleet.c diff --git a/libavfilter/Makefile b/libavfilter/Makefile index e38c6b6f6d..25dd1276de 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -526,6 +526,9 @@ OBJS-$(CONFIG_YUVTESTSRC_FILTER) += vsrc_testsrc.o OBJS-$(CONFIG_NULLSINK_FILTER) += vsink_nullsink.o +# subtitle filters +OBJS-$(CONFIG_SLEET_FILTER) += sf_sleet.o + # multimedia filters OBJS-$(CONFIG_ABITSCOPE_FILTER) += avf_abitscope.o OBJS-$(CONFIG_ADRAWGRAPH_FILTER) += f_drawgraph.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 5bd54db2c8..efe16b8e1b 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -519,6 +519,7 @@ extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_svf_sub2video; +extern const AVFilter ff_sf_sleet; /* multimedia sources */ extern const AVFilter ff_avsrc_amovie; diff --git a/libavfilter/sf_sleet.c b/libavfilter/sf_sleet.c new file mode 100644 index 0000000000..cf7701c01f --- /dev/null +++ b/libavfilter/sf_sleet.c @@ -0,0 +1,209 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * text subtitle filter which translates to 'leet speak' + */ + +#include "libavutil/avassert.h" +#include "libavutil/avstring.h" +#include "libavutil/opt.h" +#include "avfilter.h" +#include "internal.h" +#include "libavcodec/avcodec.h" + +static const char* alphabet_src = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"; +static const char* alphabet_dst = "abcd3f6#1jklmn0pq257uvwxyzAB(D3F6#1JKLMN0PQ257UVWXYZ"; + + +typedef struct LeetContext { + const AVClass *class; + enum AVSubtitleType format; +} LeetContext; + +static const AVOption sleet_options[] = { + { NULL } +}; + +AVFILTER_DEFINE_CLASS(sleet); + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats = NULL; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { SUBTITLE_ASS, SUBTITLE_NONE }; + static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_RGB32, AV_PIX_FMT_NONE }; + int ret; + + /* set input subtitle format */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output video format */ + formats = ff_make_format_list(pix_fmts); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + LeetContext *s = ctx->priv; + + s->format = inlink->format; + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + LeetContext *s = outlink->src->priv; + + outlink->format = s->format; + + return 0; +} + +static void avsubtitle_free_ref(void *opaque, uint8_t *data) +{ + avsubtitle_free((AVSubtitle *)data); +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *src_frame) +{ + LeetContext *s = inlink->dst->priv; + AVFilterLink *outlink = inlink->dst->outputs[0]; + AVSubtitle *sub; + int ret; + AVFrame *out; + unsigned int num_rects; + uint8_t *dst; + + outlink->format = inlink->format; + + out = av_frame_alloc(); + if (!out) { + av_frame_free(&src_frame); + return AVERROR(ENOMEM); + } + + out->format = outlink->format; + + if ((ret = av_frame_get_buffer2(out, AVMEDIA_TYPE_SUBTITLE, 0)) < 0) + return ret; + + out->pts = src_frame->pts; + out->repeat_pict = src_frame->repeat_pict; + out->pkt_dts = src_frame->pkt_dts; + out->pkt_pos = src_frame->pkt_pos; + out->pkt_size = src_frame->pkt_size; + out->pkt_duration = src_frame->pkt_duration; + out->reordered_opaque = src_frame->reordered_opaque; + out->best_effort_timestamp = src_frame->best_effort_timestamp; + out->flags = src_frame->flags; + + sub = (AVSubtitle *)src_frame->data[0]; + + if (sub) { + AVSubtitle *out_sub = av_memdup(sub, sizeof(*out_sub)); + if (!out_sub) + return AVERROR(ENOMEM); + + out->buf[0] = av_buffer_create((uint8_t*)out_sub, sizeof(*out_sub), avsubtitle_free_ref, NULL, AV_BUFFER_FLAG_READONLY); + out->data[0] = (uint8_t*)out_sub; + + if (sub->num_rects) { + out_sub->rects = av_malloc_array(sub->num_rects, sizeof(AVSubtitleRect *)); + } + + for (unsigned i = 0; i < sub->num_rects; i++) { + + AVSubtitleRect *src_rect = sub->rects[i]; + AVSubtitleRect *dst_rect = av_memdup(src_rect, sizeof(*dst_rect)); + out_sub->rects[i] = dst_rect; + + if (src_rect->text) { + dst_rect->text = av_strdup(src_rect->text); + if (!dst_rect->text) + return AVERROR(ENOMEM); + + for (size_t n = 0; n < strlen(dst_rect->text); n++) { + for (size_t t = 0; t < FF_ARRAY_ELEMS(alphabet_src); t++) { + if (dst_rect->text[n] == alphabet_src[t]) { + dst_rect->text[n] = alphabet_dst[t]; + break; + } + } + } + } + + if (src_rect->ass) { + dst_rect->ass = av_strdup(src_rect->ass); + if (!dst_rect->ass) + return AVERROR(ENOMEM); + + for (size_t n = 0; n < strlen(dst_rect->ass); n++) { + for (size_t t = 0; t < FF_ARRAY_ELEMS(alphabet_src); t++) { + if (dst_rect->ass[n] == alphabet_src[t]) { + dst_rect->ass[n] = alphabet_dst[t]; + break; + } + } + } + } + } + } + + av_frame_free(&src_frame); + return ff_filter_frame(outlink, out); +} + +static const AVFilterPad sleet_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_frame, + .config_props = config_input, + }, + { NULL } +}; + +static const AVFilterPad sleet_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .config_props = config_output, + }, + { NULL } +}; + +const AVFilter ff_sf_sleet = { + .name = "sleet", + .description = NULL_IF_CONFIG_SMALL("Translate text subtitles to 'leet speak'"), + .query_formats = query_formats, + .priv_size = sizeof(LeetContext), + .priv_class = &sleet_class, + .inputs = sleet_inputs, + .outputs = sleet_outputs, +}; From patchwork Mon Aug 30 08:17:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 29869 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp3738197iov; Mon, 30 Aug 2021 01:17:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxQ6tpSU41w2lOkXw99vNanH/IAYxf8NQyBZGTAyWS8dB+rmP1vnvpUFYUFi6pAo2rdlpYv X-Received: by 2002:aa7:c9cd:: with SMTP id i13mr4992176edt.178.1630311469507; Mon, 30 Aug 2021 01:17:49 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id mp24si14192718ejc.731.2021.08.30.01.17.49; Mon, 30 Aug 2021 01:17:49 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=hmlCubvE; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 93F0168A412; Mon, 30 Aug 2021 11:17:18 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08olkn2042.outbound.protection.outlook.com [40.92.46.42]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A7A8E68A3F4 for ; Mon, 30 Aug 2021 11:17:16 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m9undjrfwk+PLVRxHLwthTUK21IhiQST70HRpMRLXuoHq628K3hS69jiLe4nYe91o4KxZKYxaT0svnCVpDWJ1r3ZAsRn08JwsMm/0PDdXlO3P+RnI+ni51w0A5Y6hyXv8YZE9O6bsZckB/EV7g9kcz4aUHFu7xRnx2cVTN+ndzbsKG4JIy42qo519N/IwYiLKTx56n8F57il6ts56EEygmwCQiqDMxaB7pCzoaQm4KPEciAjVZU4QxcGMdRlRN489lrM+IVwBrk3YaO6q8esslSmXCtReC3twN27M57CbGCQ84LXWl+h28xl9/sqI6CX7zP5f/Rm4XG5fTYlf5PK2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=S/yMOTuF6jEbk4VRcGrL3DpVvlJdFR+z2B34kO2/3lE=; b=eNt+sFn5GW/Ach8OHThn7lO+q5o/yuP5cWY377ttCz7GZ6pMX+OoxN8VLuEN++ClvT6rge+k7XQrS4Q6oWzRHkCSQDOuPK3CXpgE+GKcymoyazgbd62nTKmu0m8SyJV3CNwS+3iTfbOIqN/DkKW/pEoA0qUlXwIXAXNydKCPgqZMziI0yOYQBMCaF95DWlT6hgsI9nyY73CC2UFojYB2NItGSFoBuNojshKAl98pB4h6hkSqaMZlfNr68y4+Im0P3bF/UbmqYsWLZC/Byelcd1jmwZgIT5pqop1/FnUdh44do8G3EdqD3CVeNe64joqve5Eswkq0vCm9KKx+vuyy+Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=S/yMOTuF6jEbk4VRcGrL3DpVvlJdFR+z2B34kO2/3lE=; b=hmlCubvEQ1dpqmBRk2TVNAcMgTNKqhFX2DpeA54dHXSJAaaI5+NHpLr9hYCShLe5zUM5Uq4SbZ1HKqESXCJ/A3cx1TqmomaZec1VTsPRWzw0IUvoJ1HYIc2RTweYSNMON90Y7nCcs7pqtDo9lH9lOexBkNBOruPMspY/JYWfqo7lkyFeJuBlS5S0ituYcBGi7Re8wsAl9vzp3gapMv0HTPqG/u7wWE7cXz74UZFzazv9KvuVClGKg01/99JL57r9NOwz1+nthK6S/y8Cl6O09TRSWrcZr85DEHKmdaFjJDx8+pj7lGg31SXAe58MLvTcqXNU0MPYYr1ILjOt2jEpRA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5853.namprd04.prod.outlook.com (2603:10b6:208:a6::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.19; Mon, 30 Aug 2021 08:17:13 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 08:17:13 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v2 8/8] fftools/ffmpeg: Replace sub2video with subtitle frame filtering Thread-Index: AdeUsAnIVebg7m2nSyS7vkXjpkE5Yw== Date: Mon, 30 Aug 2021 08:17:13 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [4UqpqLc38+vk5nXRsubfEA7MWivGgZbp] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f5dcae16-7b89-4d69-a343-08d96b8e9286 x-ms-traffictypediagnostic: MN2PR04MB5853: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Vz63B9izUgCqFP8nO8G0j1YdGHKnCpxvJtKSv5DAwbwxxjGcHkuOfH4z/ipjHWFgTiHqUBYdqnCiNK1IU+JUqaxcUhrYwkTSpC9JkFU7yqhPthZ/E7QRJvzRgdX1Uu5r8uiAFNIVZxsloggfiHzhf5l+y6fH7TZB2gGxURB7bfMKaL6cmT1e3b8MVKHrBgEXEx1x6/ItkvHOBcZvqvEvso8K0RvSXaptSy2Zo7Y/RqgHK3HqvepPevX+QCl6W8RWPEVIYjQWH8gs43zH5aGVaIN9oFiIf86C7SbDLRJ6c2uIDTwC8ndIWeaDHaqH/P6b3I2SrWaqdcl0sh9HeBCTBwe8mjfHfr31deKBLwyL7/fW+AFfHH1SESsZcqrwowybV62pfDyL0jHPRldBSANR1Gfp+E+YziDR2ZYRjMz74cuzRU/Md9Skxp5ysJrEL29y x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: 4bNXhDCG4W48Msn/Y+rpvGmaV5xColEuoMnoq3F8aXn4GuXuNiOnWBMzg5fJa2lmGhU8PkI+nEe1PcpySvxAB8nkA3wtpK+bwNr8tRuku7HUFQAWwDm6SfCvIJp99ap1OK6NMC0vawvGRXzLLNFNjw== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: f5dcae16-7b89-4d69-a343-08d96b8e9286 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2021 08:17:13.6551 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5853 Subject: [FFmpeg-devel] [PATCH v2 8/8] fftools/ffmpeg: Replace sub2video with subtitle frame filtering X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: HIQnrxcaA82n Signed-off-by: softworkz --- v2 Update: - Implemented Andreas' suggestions - overlay_subs filter: - removed duplicated code - implemented direct (no pre-conversion) blending of graphical subtitle rects - Supported input formats: - all packed RGB formats (with and without alpha) - yuv420p, yuv422p, yuv444p fftools/ffmpeg.c | 324 +++++++++++++++++++++++----------------- fftools/ffmpeg.h | 7 +- fftools/ffmpeg_filter.c | 198 +++++++++++++++++------- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- 5 files changed, 330 insertions(+), 204 deletions(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index b0ce7c7c32..aec8422111 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -174,114 +174,99 @@ static void free_input_threads(void); This is a temporary solution until libavfilter gets real subtitles support. */ -static int sub2video_get_blank_frame(InputStream *ist) +static int send_frame_to_filters(InputStream *ist, AVFrame *decoded_frame); + + +static int get_subtitle_format_from_codecdesc(const AVCodecDescriptor *codec_descriptor) { - int ret; - AVFrame *frame = ist->sub2video.frame; + int format; - av_frame_unref(frame); - ist->sub2video.frame->width = ist->dec_ctx->width ? ist->dec_ctx->width : ist->sub2video.w; - ist->sub2video.frame->height = ist->dec_ctx->height ? ist->dec_ctx->height : ist->sub2video.h; - ist->sub2video.frame->format = AV_PIX_FMT_RGB32; - if ((ret = av_frame_get_buffer(frame, 0)) < 0) - return ret; - memset(frame->data[0], 0, frame->height * frame->linesize[0]); - return 0; + if(codec_descriptor->props & AV_CODEC_PROP_BITMAP_SUB) + format = SUBTITLE_BITMAP; + else if(codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB) + format = SUBTITLE_ASS; + else + format = SUBTITLE_TEXT; + + return format; } -static void sub2video_copy_rect(uint8_t *dst, int dst_linesize, int w, int h, - AVSubtitleRect *r) +static void avsubtitle_free_ref(void *opaque, uint8_t *data) { - uint32_t *pal, *dst2; - uint8_t *src, *src2; - int x, y; + avsubtitle_free((AVSubtitle *)data); +} - if (r->type != SUBTITLE_BITMAP) { - av_log(NULL, AV_LOG_WARNING, "sub2video: non-bitmap subtitle\n"); +static void sub2video_resend_current(InputStream *ist, int64_t heartbeat_pts) +{ + AVFrame *frame; + AVSubtitle *current_sub; + int8_t *dst; + int num_rects, i, ret; + int64_t pts, end_pts, pts_sub; + int format = get_subtitle_format_from_codecdesc(ist->dec_ctx->codec_descriptor); + + /* If we are initializing the system, utilize current heartbeat + PTS as the start time, and show until the following subpicture + is received. Otherwise, utilize the previous subpicture's end time + as the fall-back value. */ + pts = ist->sub2video.end_pts <= 0 ? + heartbeat_pts : ist->sub2video.end_pts; + end_pts = INT64_MAX; + + ////av_log(ist->dec_ctx, AV_LOG_ERROR, "sub2video_resend_current1: heartbeat_pts: %lld ist->sub2video.end_pts: %lld\n", heartbeat_pts, ist->sub2video.end_pts); + + pts = av_rescale_q(pts * 1000LL, + AV_TIME_BASE_Q, ist->st->time_base); + + frame = av_frame_alloc(); + if (!frame) { + av_log(ist->dec_ctx, AV_LOG_ERROR, "Unable to alloc frame (out of memory).\n"); return; } - if (r->x < 0 || r->x + r->w > w || r->y < 0 || r->y + r->h > h) { - av_log(NULL, AV_LOG_WARNING, "sub2video: rectangle (%d %d %d %d) overflowing %d %d\n", - r->x, r->y, r->w, r->h, w, h - ); + + frame->format = get_subtitle_format_from_codecdesc(ist->dec_ctx->codec_descriptor); + + if ((ret = av_frame_get_buffer2(frame, AVMEDIA_TYPE_SUBTITLE, 0)) < 0) { + av_log(ist->dec_ctx, AV_LOG_ERROR, "Error (av_frame_get_buffer): %d.\n", ret); return; } - dst += r->y * dst_linesize + r->x * 4; - src = r->data[0]; - pal = (uint32_t *)r->data[1]; - for (y = 0; y < r->h; y++) { - dst2 = (uint32_t *)dst; - src2 = src; - for (x = 0; x < r->w; x++) - *(dst2++) = pal[*(src2++)]; - dst += dst_linesize; - src += r->linesize[0]; + frame->width = ist->sub2video.w; + frame->height = ist->sub2video.h; + + if (ist->sub2video.current_subtitle) { + frame->buf[0] = av_buffer_ref(ist->sub2video.current_subtitle); + frame->data[0] = ist->sub2video.current_subtitle->data; + } + else { + AVBufferRef *empty_sub_buffer; + AVSubtitle *empty_sub = av_mallocz(sizeof(*empty_sub)); + empty_sub->format = format; + empty_sub->num_rects = 0; + empty_sub->pts = av_rescale_q(pts, ist->st->time_base, AV_TIME_BASE_Q); + empty_sub->end_display_time = 1000; + empty_sub_buffer = av_buffer_create((uint8_t*)empty_sub, sizeof(*empty_sub), avsubtitle_free_ref, NULL, AV_BUFFER_FLAG_READONLY); + frame->buf[0] = empty_sub_buffer; + frame->data[0] = empty_sub_buffer->data; } -} -static void sub2video_push_ref(InputStream *ist, int64_t pts) -{ - AVFrame *frame = ist->sub2video.frame; - int i; - int ret; + current_sub = (AVSubtitle *)frame->data[0]; - av_assert1(frame->data[0]); - ist->sub2video.last_pts = frame->pts = pts; - for (i = 0; i < ist->nb_filters; i++) { - ret = av_buffersrc_add_frame_flags(ist->filters[i]->filter, frame, - AV_BUFFERSRC_FLAG_KEEP_REF | - AV_BUFFERSRC_FLAG_PUSH); - if (ret != AVERROR_EOF && ret < 0) - av_log(NULL, AV_LOG_WARNING, "Error while add the frame to buffer source(%s).\n", - av_err2str(ret)); - } -} + frame->pts = pts; + ist->sub2video.last_pts = pts; -void sub2video_update(InputStream *ist, int64_t heartbeat_pts, AVSubtitle *sub) -{ - AVFrame *frame = ist->sub2video.frame; - int8_t *dst; - int dst_linesize; - int num_rects, i; - int64_t pts, end_pts; + ////av_log(ist->dec_ctx, AV_LOG_ERROR, ": frame->pts: %lld current_sub: %lld\n", frame->pts, current_sub->pts); + + send_frame_to_filters(ist, frame); - if (!frame) - return; - if (sub) { - pts = av_rescale_q(sub->pts + sub->start_display_time * 1000LL, - AV_TIME_BASE_Q, ist->st->time_base); - end_pts = av_rescale_q(sub->pts + sub->end_display_time * 1000LL, - AV_TIME_BASE_Q, ist->st->time_base); - num_rects = sub->num_rects; - } else { - /* If we are initializing the system, utilize current heartbeat - PTS as the start time, and show until the following subpicture - is received. Otherwise, utilize the previous subpicture's end time - as the fall-back value. */ - pts = ist->sub2video.initialize ? - heartbeat_pts : ist->sub2video.end_pts; - end_pts = INT64_MAX; - num_rects = 0; - } - if (sub2video_get_blank_frame(ist) < 0) { - av_log(ist->dec_ctx, AV_LOG_ERROR, - "Impossible to get a blank canvas.\n"); - return; - } - dst = frame->data [0]; - dst_linesize = frame->linesize[0]; - for (i = 0; i < num_rects; i++) - sub2video_copy_rect(dst, dst_linesize, frame->width, frame->height, sub->rects[i]); - sub2video_push_ref(ist, pts); ist->sub2video.end_pts = end_pts; - ist->sub2video.initialize = 0; } static void sub2video_heartbeat(InputStream *ist, int64_t pts) { - InputFile *infile = input_files[ist->file_index]; - int i, j, nb_reqs; + const InputFile *infile = input_files[ist->file_index]; + int i, j; + unsigned nb_reqs; int64_t pts2; /* When a frame is read from a file, examine all sub2video streams in @@ -290,7 +275,7 @@ static void sub2video_heartbeat(InputStream *ist, int64_t pts) (possibly overlay) is desperately waiting for a subtitle frame. */ for (i = 0; i < infile->nb_streams; i++) { InputStream *ist2 = input_streams[infile->ist_index + i]; - if (!ist2->sub2video.frame) + if (!ist2->sub2video.is_active) continue; /* subtitles seem to be usually muxed ahead of other streams; if not, subtracting a larger time here is necessary */ @@ -298,15 +283,18 @@ static void sub2video_heartbeat(InputStream *ist, int64_t pts) /* do not send the heartbeat frame if the subtitle is already ahead */ if (pts2 <= ist2->sub2video.last_pts) continue; - if (pts2 >= ist2->sub2video.end_pts || ist2->sub2video.initialize) + if (pts2 >= ist2->sub2video.end_pts) { /* if we have hit the end of the current displayed subpicture, or if we need to initialize the system, update the overlayed subpicture and its start/end times */ - sub2video_update(ist2, pts2 + 1, NULL); - for (j = 0, nb_reqs = 0; j < ist2->nb_filters; j++) - nb_reqs += av_buffersrc_get_nb_failed_requests(ist2->filters[j]->filter); - if (nb_reqs) - sub2video_push_ref(ist2, pts2); + av_buffer_unref(&ist2->sub2video.current_subtitle); + ist2->sub2video.current_subtitle = NULL; + sub2video_resend_current(ist2, pts2 + 1); + } + //for (j = 0, nb_reqs = 0; j < ist2->nb_filters; j++) + // nb_reqs += av_buffersrc_get_nb_failed_requests(ist2->filters[j]->filter); + //if (nb_reqs) + // sub2video_resend_current(ist2, pts2); } } @@ -316,7 +304,7 @@ static void sub2video_flush(InputStream *ist) int ret; if (ist->sub2video.end_pts < INT64_MAX) - sub2video_update(ist, INT64_MAX, NULL); + sub2video_resend_current(ist, INT64_MAX); for (i = 0; i < ist->nb_filters; i++) { ret = av_buffersrc_add_frame(ist->filters[i]->filter, NULL); if (ret != AVERROR_EOF && ret < 0) @@ -535,15 +523,6 @@ static void ffmpeg_cleanup(int ret) av_frame_free(&frame); } av_fifo_freep(&ifilter->frame_queue); - if (ist->sub2video.sub_queue) { - while (av_fifo_size(ist->sub2video.sub_queue)) { - AVSubtitle sub; - av_fifo_generic_read(ist->sub2video.sub_queue, - &sub, sizeof(sub), NULL); - avsubtitle_free(&sub); - } - av_fifo_freep(&ist->sub2video.sub_queue); - } av_buffer_unref(&ifilter->hw_frames_ctx); av_freep(&ifilter->name); av_freep(&fg->inputs[j]); @@ -636,7 +615,6 @@ static void ffmpeg_cleanup(int ret) av_packet_free(&ist->pkt); av_dict_free(&ist->decoder_opts); avsubtitle_free(&ist->prev_sub.subtitle); - av_frame_free(&ist->sub2video.frame); av_freep(&ist->filters); av_freep(&ist->hwaccel_device); av_freep(&ist->dts_buffer); @@ -1061,13 +1039,19 @@ error: static void do_subtitle_out(OutputFile *of, OutputStream *ost, - AVSubtitle *sub) + AVFrame *frame) { int subtitle_out_max_size = 1024 * 1024; int subtitle_out_size, nb, i; AVCodecContext *enc; AVPacket *pkt = ost->pkt; int64_t pts; + AVSubtitle *sub = (AVSubtitle *)frame->data[0]; + + if (!sub) + return; + + ////av_log(NULL, AV_LOG_VERBOSE, "do_subtitle_out: sub->pts: %lld frame->pts: %lld\n", sub->pts, frame->pts); if (sub->pts == AV_NOPTS_VALUE) { av_log(NULL, AV_LOG_ERROR, "Subtitle packets must have a pts\n"); @@ -1576,8 +1560,11 @@ static int reap_filters(int flush) } do_audio_out(of, ost, filtered_frame); break; + case AVMEDIA_TYPE_SUBTITLE: + + do_subtitle_out(of, ost, filtered_frame); + break; default: - // TODO support subtitle filters av_assert0(0); } @@ -2173,7 +2160,8 @@ static int ifilter_has_all_input_formats(FilterGraph *fg) int i; for (i = 0; i < fg->nb_inputs; i++) { if (fg->inputs[i]->format < 0 && (fg->inputs[i]->type == AVMEDIA_TYPE_AUDIO || - fg->inputs[i]->type == AVMEDIA_TYPE_VIDEO)) + fg->inputs[i]->type == AVMEDIA_TYPE_VIDEO || + fg->inputs[i]->type == AVMEDIA_TYPE_SUBTITLE)) return 0; } return 1; @@ -2270,7 +2258,7 @@ static int ifilter_send_eof(InputFilter *ifilter, int64_t pts) // the filtergraph was never configured if (ifilter->format < 0) ifilter_parameters_from_codecpar(ifilter, ifilter->ist->st->codecpar); - if (ifilter->format < 0 && (ifilter->type == AVMEDIA_TYPE_AUDIO || ifilter->type == AVMEDIA_TYPE_VIDEO)) { + if (ifilter->format < 0 && (ifilter->type == AVMEDIA_TYPE_AUDIO || ifilter->type == AVMEDIA_TYPE_VIDEO || ifilter->type == AVMEDIA_TYPE_SUBTITLE)) { av_log(NULL, AV_LOG_ERROR, "Cannot determine format of input stream %d:%d after EOF\n", ifilter->ist->file_index, ifilter->ist->st->index); return AVERROR_INVALIDDATA; } @@ -2528,12 +2516,24 @@ fail: static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, int *decode_failed) { - AVSubtitle subtitle; - int free_sub = 1; - int i, ret = avcodec_decode_subtitle2(ist->dec_ctx, - &subtitle, got_output, pkt); + AVFrame *decoded_frame; + AVCodecContext *avctx = ist->dec_ctx; + int i = 0, ret = 0, err = 0; + int64_t pts, end_pts; + AVSubtitle *subtitle = av_mallocz(sizeof(*subtitle)); + if (!subtitle) + return NULL; + + if (!ist->decoded_frame && !(ist->decoded_frame = av_frame_alloc())) + return AVERROR(ENOMEM); + if (!ist->filter_frame && !(ist->filter_frame = av_frame_alloc())) + return AVERROR(ENOMEM); + decoded_frame = ist->decoded_frame; + + ret = avcodec_decode_subtitle2(avctx, subtitle, got_output, pkt); - check_decode_result(NULL, got_output, ret); + if (ret != AVERROR_EOF) + check_decode_result(NULL, got_output, ret); if (ret < 0 || !*got_output) { *decode_failed = 1; @@ -2545,10 +2545,10 @@ static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, if (ist->fix_sub_duration) { int end = 1; if (ist->prev_sub.got_output) { - end = av_rescale(subtitle.pts - ist->prev_sub.subtitle.pts, + end = av_rescale(subtitle->pts - ist->prev_sub.subtitle.pts, 1000, AV_TIME_BASE); if (end < ist->prev_sub.subtitle.end_display_time) { - av_log(ist->dec_ctx, AV_LOG_DEBUG, + av_log(avctx, AV_LOG_DEBUG, "Subtitle duration reduced from %"PRId32" to %d%s\n", ist->prev_sub.subtitle.end_display_time, end, end <= 0 ? ", dropping it" : ""); @@ -2557,35 +2557,79 @@ static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, } FFSWAP(int, *got_output, ist->prev_sub.got_output); FFSWAP(int, ret, ist->prev_sub.ret); - FFSWAP(AVSubtitle, subtitle, ist->prev_sub.subtitle); + FFSWAP(AVSubtitle, *subtitle, ist->prev_sub.subtitle); if (end <= 0) - goto out; + return end; } if (!*got_output) return ret; - if (ist->sub2video.frame) { - sub2video_update(ist, INT64_MIN, &subtitle); - } else if (ist->nb_filters) { - if (!ist->sub2video.sub_queue) - ist->sub2video.sub_queue = av_fifo_alloc(8 * sizeof(AVSubtitle)); - if (!ist->sub2video.sub_queue) + ////decoded_frame = av_frame_alloc(); + ////if (!decoded_frame) + //// return AVERROR(ENOMEM); + + decoded_frame->format = get_subtitle_format_from_codecdesc(avctx->codec_descriptor); + + if ((ret = av_frame_get_buffer2(decoded_frame, AVMEDIA_TYPE_SUBTITLE, 0)) < 0) + return ret; + + av_buffer_unref(&ist->sub2video.current_subtitle); + ist->sub2video.current_subtitle = av_buffer_create((uint8_t*)subtitle, sizeof(*subtitle), avsubtitle_free_ref, NULL, AV_BUFFER_FLAG_READONLY); + + decoded_frame->buf[0] = av_buffer_ref(ist->sub2video.current_subtitle); + decoded_frame->data[0] = ist->sub2video.current_subtitle->data; + + pts = av_rescale_q(subtitle->pts + subtitle->start_display_time * 1000LL, + AV_TIME_BASE_Q, ist->st->time_base); + end_pts = av_rescale_q(subtitle->pts + subtitle->end_display_time * 1000LL, + AV_TIME_BASE_Q, ist->st->time_base); + + ist->sub2video.last_pts = decoded_frame->pts = pts; + ist->sub2video.end_pts = end_pts; + + ////av_log(ist->dec_ctx, AV_LOG_ERROR, "frame->pts: %lld subtitle->pts: %lld\n", decoded_frame->pts, subtitle->pts); + + for (i = 0; i < nb_output_streams; i++) { + OutputStream *ost = output_streams[i]; + + if (!ost->pkt && !(ost->pkt = av_packet_alloc())) exit_program(1); - if (!av_fifo_space(ist->sub2video.sub_queue)) { - ret = av_fifo_realloc2(ist->sub2video.sub_queue, 2 * av_fifo_size(ist->sub2video.sub_queue)); - if (ret < 0) - exit_program(1); - } - av_fifo_generic_write(ist->sub2video.sub_queue, &subtitle, sizeof(subtitle), NULL); - free_sub = 0; + if (!check_output_constraints(ist, ost) || !ost->encoding_needed + || ost->enc->type != AVMEDIA_TYPE_SUBTITLE) + continue; + + do_subtitle_out(output_files[ost->file_index], ost, decoded_frame); } - if (!subtitle.num_rects) - goto out; + err = send_frame_to_filters(ist, decoded_frame); - ist->frames_decoded++; + av_frame_unref(ist->filter_frame); + av_frame_unref(decoded_frame); + return err < 0 ? err : ret; +//// if (ist->sub2video.frame) { +//// sub2video_update(ist, INT64_MIN, subtitle); +//// free_sub = 0; +//// } else if (ist->nb_filters) { +//// if (!ist->sub2video.sub_queue) +//// ist->sub2video.sub_queue = av_fifo_alloc(8 * sizeof(AVSubtitle)); +//// if (!ist->sub2video.sub_queue) +//// exit_program(1); +//// if (!av_fifo_space(ist->sub2video.sub_queue)) { +//// ret = av_fifo_realloc2(ist->sub2video.sub_queue, 2 * av_fifo_size(ist->sub2video.sub_queue)); +//// if (ret < 0) +//// exit_program(1); +//// } +//// av_fifo_generic_write(ist->sub2video.sub_queue, &subtitle, sizeof(subtitle), NULL); +//// free_sub = 0; +//// } +//// +//// if (!subtitle->num_rects) +//// goto out; +//// +//// ist->frames_decoded++; +//// for (i = 0; i < nb_output_streams; i++) { OutputStream *ost = output_streams[i]; @@ -2595,13 +2639,13 @@ static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, || ost->enc->type != AVMEDIA_TYPE_SUBTITLE) continue; - do_subtitle_out(output_files[ost->file_index], ost, &subtitle); + do_subtitle_out(output_files[ost->file_index], ost, subtitle); } - -out: - if (free_sub) - avsubtitle_free(&subtitle); - return ret; +//// +////out: +//// if (free_sub) +//// avsubtitle_free(subtitle); +//// return ret; } static int send_filter_eof(InputStream *ist) diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index d2dd7ca092..59e9dd775c 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -352,12 +352,11 @@ typedef struct InputStream { } prev_sub; struct sub2video { + int is_active; int64_t last_pts; int64_t end_pts; - AVFifoBuffer *sub_queue; ///< queue of AVSubtitle* before filter init - AVFrame *frame; + AVBufferRef *current_subtitle; int w, h; - unsigned int initialize; ///< marks if sub2video_update should force an initialization } sub2video; int dr1; @@ -664,8 +663,6 @@ int filtergraph_is_simple(FilterGraph *fg); int init_simple_filtergraph(InputStream *ist, OutputStream *ost); int init_complex_filtergraph(FilterGraph *fg); -void sub2video_update(InputStream *ist, int64_t heartbeat_pts, AVSubtitle *sub); - int ifilter_parameters_from_frame(InputFilter *ifilter, const AVFrame *frame); int ffmpeg_parse_options(int argc, char **argv); diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c index 49076f13ee..139843402c 100644 --- a/fftools/ffmpeg_filter.c +++ b/fftools/ffmpeg_filter.c @@ -19,6 +19,7 @@ */ #include +#include #include "ffmpeg.h" @@ -222,8 +223,8 @@ static void init_input_filter(FilterGraph *fg, AVFilterInOut *in) int i; // TODO: support other filter types - if (type != AVMEDIA_TYPE_VIDEO && type != AVMEDIA_TYPE_AUDIO) { - av_log(NULL, AV_LOG_FATAL, "Only video and audio filters supported " + if (type != AVMEDIA_TYPE_VIDEO && type != AVMEDIA_TYPE_AUDIO && type != AVMEDIA_TYPE_SUBTITLE) { + av_log(NULL, AV_LOG_FATAL, "Only video, audio and subtitle filters supported " "currently.\n"); exit_program(1); } @@ -243,10 +244,6 @@ static void init_input_filter(FilterGraph *fg, AVFilterInOut *in) for (i = 0; i < s->nb_streams; i++) { enum AVMediaType stream_type = s->streams[i]->codecpar->codec_type; - if (stream_type != type && - !(stream_type == AVMEDIA_TYPE_SUBTITLE && - type == AVMEDIA_TYPE_VIDEO /* sub2video hack */)) - continue; if (check_stream_specifier(s, s->streams[i], *p == ':' ? p + 1 : p) == 1) { st = s->streams[i]; break; @@ -416,6 +413,39 @@ static int insert_filter(AVFilterContext **last_filter, int *pad_idx, return 0; } +static int configure_output_subtitle_filter(FilterGraph *fg, OutputFilter *ofilter, AVFilterInOut *out) +{ + char *pix_fmts; + OutputStream *ost = ofilter->ost; + OutputFile *of = output_files[ost->file_index]; + AVFilterContext *last_filter = out->filter_ctx; + int pad_idx = out->pad_idx; + int ret; + char name[255]; + + snprintf(name, sizeof(name), "out_%d_%d", ost->file_index, ost->index); + ret = avfilter_graph_create_filter(&ofilter->filter, + avfilter_get_by_name("sbuffersink"), + name, NULL, NULL, fg->graph); + + if (ret < 0) + return ret; + + ////snprintf(name, sizeof(name), "trim_out_%d_%d", + //// ost->file_index, ost->index); + ////ret = insert_trim(of->start_time, of->recording_time, + //// &last_filter, &pad_idx, name); + ////if (ret < 0) + //// return ret; + + ////ost->st->codecpar->codec_tag = MKTAG('a', 's', 's', 's'); + + if ((ret = avfilter_link(last_filter, pad_idx, ofilter->filter, 0)) < 0) + return ret; + + return 0; +} + static int configure_output_video_filter(FilterGraph *fg, OutputFilter *ofilter, AVFilterInOut *out) { char *pix_fmts; @@ -594,7 +624,8 @@ static int configure_output_audio_filter(FilterGraph *fg, OutputFilter *ofilter, int i; for (i=0; ictx->nb_streams; i++) - if (of->ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) + if (of->ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO || + of->ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) break; if (ictx->nb_streams) { @@ -628,6 +659,7 @@ static int configure_output_filter(FilterGraph *fg, OutputFilter *ofilter, switch (avfilter_pad_get_type(out->filter_ctx->output_pads, out->pad_idx)) { case AVMEDIA_TYPE_VIDEO: return configure_output_video_filter(fg, ofilter, out); case AVMEDIA_TYPE_AUDIO: return configure_output_audio_filter(fg, ofilter, out); + case AVMEDIA_TYPE_SUBTITLE: return configure_output_subtitle_filter(fg, ofilter, out); default: av_assert0(0); return 0; } } @@ -652,46 +684,110 @@ static int sub2video_prepare(InputStream *ist, InputFilter *ifilter) AVFormatContext *avf = input_files[ist->file_index]->ctx; int i, w, h; - /* Compute the size of the canvas for the subtitles stream. - If the subtitles codecpar has set a size, use it. Otherwise use the - maximum dimensions of the video streams in the same file. */ + ist->sub2video.is_active = 1; + w = ifilter->width; h = ifilter->height; if (!(w && h)) { - for (i = 0; i < avf->nb_streams; i++) { - if (avf->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) { - w = FFMAX(w, avf->streams[i]->codecpar->width); - h = FFMAX(h, avf->streams[i]->codecpar->height); - } - } - if (!(w && h)) { - w = FFMAX(w, 720); - h = FFMAX(h, 576); - } - av_log(avf, AV_LOG_INFO, "sub2video: using %dx%d canvas\n", w, h); + w = ist->dec_ctx->width; + h = ist->dec_ctx->height; } - ist->sub2video.w = ifilter->width = w; - ist->sub2video.h = ifilter->height = h; - ifilter->width = ist->dec_ctx->width ? ist->dec_ctx->width : ist->sub2video.w; - ifilter->height = ist->dec_ctx->height ? ist->dec_ctx->height : ist->sub2video.h; + ist->sub2video.w = w; + ist->sub2video.h = h; + av_log(avf, AV_LOG_INFO, "sub2video: decoding size %dx%d\n", ist->sub2video.w, ist->sub2video.h); - /* rectangles are AV_PIX_FMT_PAL8, but we have no guarantee that the - palettes for all rectangles are identical or compatible */ - ifilter->format = AV_PIX_FMT_RGB32; + ifilter->width = w; + ifilter->height = h; - ist->sub2video.frame = av_frame_alloc(); - if (!ist->sub2video.frame) - return AVERROR(ENOMEM); ist->sub2video.last_pts = INT64_MIN; - ist->sub2video.end_pts = INT64_MIN; + ist->sub2video.end_pts = 0; + + return 0; +} + +static int configure_input_subtitle_filter(FilterGraph *fg, InputFilter *ifilter, + AVFilterInOut *in) +{ + AVFilterContext *last_filter; + const AVFilter *buffer_filt = avfilter_get_by_name("sbuffer"); + InputStream *ist = ifilter->ist; + InputFile *f = input_files[ist->file_index]; + AVBPrint args; + char name[255]; + int ret, pad_idx = 0; + int64_t tsoffset = 0; + AVBufferSrcParameters *par = av_buffersrc_parameters_alloc(); + + if (!par) + return AVERROR(ENOMEM); + memset(par, 0, sizeof(*par)); + par->format = AV_PIX_FMT_NONE; + + if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_AUDIO) { + av_log(NULL, AV_LOG_ERROR, "Cannot connect subtitle filter to audio input\n"); + ret = AVERROR(EINVAL); + goto fail; + } + + if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_VIDEO) { + av_log(NULL, AV_LOG_ERROR, "Cannot connect subtitle filter to video input\n"); + ret = AVERROR(EINVAL); + goto fail; + } + + ret = sub2video_prepare(ist, ifilter); + if (ret < 0) + goto fail; + + snprintf(name, sizeof(name), "graph %d subtitle input from stream %d:%d", fg->index, + ist->file_index, ist->st->index); + - /* sub2video structure has been (re-)initialized. - Mark it as such so that the system will be - initialized with the first received heartbeat. */ - ist->sub2video.initialize = 1; + av_bprint_init(&args, 0, AV_BPRINT_SIZE_AUTOMATIC); + av_bprintf(&args, + "subtitle_type=%d:time_base=%d/%d:", + ifilter->format, + ist->st->time_base.num, ist->st->time_base.den); + if ((ret = avfilter_graph_create_filter(&ifilter->filter, buffer_filt, name, + args.str, NULL, fg->graph)) < 0) + goto fail; + + par->hw_frames_ctx = ifilter->hw_frames_ctx; + par->format = ifilter->format; + + ret = av_buffersrc_parameters_set(ifilter->filter, par); + if (ret < 0) + goto fail; + av_freep(&par); + last_filter = ifilter->filter; + if (in->filter_ctx->input_pads[in->pad_idx].type == AVMEDIA_TYPE_VIDEO) { + ret = insert_filter(&last_filter, &pad_idx, "sub2video", NULL); + if (ret < 0) + return ret; + } + + ////snprintf(name, sizeof(name), "trim_in_%d_%d", + //// ist->file_index, ist->st->index); + ////if (copy_ts) { + //// tsoffset = f->start_time == AV_NOPTS_VALUE ? 0 : f->start_time; + //// if (!start_at_zero && f->ctx->start_time != AV_NOPTS_VALUE) + //// tsoffset += f->ctx->start_time; + ////} + ////ret = insert_trim(((f->start_time == AV_NOPTS_VALUE) || !f->accurate_seek) ? + //// AV_NOPTS_VALUE : tsoffset, f->recording_time, + //// &last_filter, &pad_idx, name); + ////if (ret < 0) + //// return ret; + + if ((ret = avfilter_link(last_filter, 0, in->filter_ctx, in->pad_idx)) < 0) + return ret; return 0; +fail: + av_freep(&par); + + return ret; } static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, @@ -709,8 +805,13 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, char name[255]; int ret, pad_idx = 0; int64_t tsoffset = 0; - AVBufferSrcParameters *par = av_buffersrc_parameters_alloc(); + AVBufferSrcParameters *par; + if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_SUBTITLE) { + return configure_input_subtitle_filter(fg, ifilter, in); + } + + par = av_buffersrc_parameters_alloc(); if (!par) return AVERROR(ENOMEM); memset(par, 0, sizeof(*par)); @@ -725,12 +826,6 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, if (!fr.num) fr = av_guess_frame_rate(input_files[ist->file_index]->ctx, ist->st, NULL); - if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_SUBTITLE) { - ret = sub2video_prepare(ist, ifilter); - if (ret < 0) - goto fail; - } - sar = ifilter->sample_aspect_ratio; if(!sar.den) sar = (AVRational){0,1}; @@ -742,7 +837,7 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, tb.num, tb.den, sar.num, sar.den); if (fr.num && fr.den) av_bprintf(&args, ":frame_rate=%d/%d", fr.num, fr.den); - snprintf(name, sizeof(name), "graph %d input from stream %d:%d", fg->index, + snprintf(name, sizeof(name), "graph %d video input from stream %d:%d", fg->index, ist->file_index, ist->st->index); @@ -938,6 +1033,7 @@ static int configure_input_filter(FilterGraph *fg, InputFilter *ifilter, switch (avfilter_pad_get_type(in->filter_ctx->input_pads, in->pad_idx)) { case AVMEDIA_TYPE_VIDEO: return configure_input_video_filter(fg, ifilter, in); case AVMEDIA_TYPE_AUDIO: return configure_input_audio_filter(fg, ifilter, in); + case AVMEDIA_TYPE_SUBTITLE: return configure_input_subtitle_filter(fg, ifilter, in); default: av_assert0(0); return 0; } } @@ -1110,19 +1206,6 @@ int configure_filtergraph(FilterGraph *fg) } } - /* process queued up subtitle packets */ - for (i = 0; i < fg->nb_inputs; i++) { - InputStream *ist = fg->inputs[i]->ist; - if (ist->sub2video.sub_queue && ist->sub2video.frame) { - while (av_fifo_size(ist->sub2video.sub_queue)) { - AVSubtitle tmp; - av_fifo_generic_read(ist->sub2video.sub_queue, &tmp, sizeof(tmp), NULL); - sub2video_update(ist, INT64_MIN, &tmp); - avsubtitle_free(&tmp); - } - } - } - return 0; fail: @@ -1143,6 +1226,7 @@ int ifilter_parameters_from_frame(InputFilter *ifilter, const AVFrame *frame) ifilter->sample_rate = frame->sample_rate; ifilter->channels = frame->channels; ifilter->channel_layout = frame->channel_layout; + ifilter->type = frame->type; if (frame->hw_frames_ctx) { ifilter->hw_frames_ctx = av_buffer_ref(frame->hw_frames_ctx); diff --git a/fftools/ffmpeg_hw.c b/fftools/ffmpeg_hw.c index 41aaf776d7..f8b49fd5ac 100644 --- a/fftools/ffmpeg_hw.c +++ b/fftools/ffmpeg_hw.c @@ -449,7 +449,7 @@ int hw_device_setup_for_encode(OutputStream *ost) AVBufferRef *frames_ref = NULL; int i; - if (ost->filter) { + if (ost->filter && ost->filter->filter) { frames_ref = av_buffersink_get_hw_frames_ctx(ost->filter->filter); if (frames_ref && ((AVHWFramesContext*)frames_ref->data)->format == diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index 428934a3d8..9776455cca 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -2144,8 +2144,9 @@ static void init_output_filter(OutputFilter *ofilter, OptionsContext *o, switch (ofilter->type) { case AVMEDIA_TYPE_VIDEO: ost = new_video_stream(o, oc, -1); break; case AVMEDIA_TYPE_AUDIO: ost = new_audio_stream(o, oc, -1); break; + case AVMEDIA_TYPE_SUBTITLE: ost = new_subtitle_stream(o, oc, -1); break; default: - av_log(NULL, AV_LOG_FATAL, "Only video and audio filters are supported " + av_log(NULL, AV_LOG_FATAL, "Only video, audio and subtitle filters are supported " "currently.\n"); exit_program(1); }