From patchwork Sat Sep 11 06:02:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30111 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp2167301iov; Fri, 10 Sep 2021 23:03:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzxSD9azCKJuV0x71ozOh57onlyz0D8lmCgKyBLepk3o97aN/y5xyEUExx0qE38kGBRIL0J X-Received: by 2002:aa7:c945:: with SMTP id h5mr1737380edt.350.1631340212254; Fri, 10 Sep 2021 23:03:32 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b16si875288edy.546.2021.09.10.23.03.31; Fri, 10 Sep 2021 23:03:32 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=QddQuWuW; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B5A0768A545; Sat, 11 Sep 2021 09:02:58 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12olkn2093.outbound.protection.outlook.com [40.92.23.93]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id ADDAA68A3AF for ; Sat, 11 Sep 2021 09:02:50 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mqzNsml1RgTrkasHG2h4xrEv07FNrSerD6AT9BIh1JzuEfPHc4n1hPtekDfbqWmjEbB3zoz8NTgaZcZ0FXmlYGIeV4Ns1M1x5QS5G9ZtC79JRBFcCEti5uisDOud8yc9jEKxSqb1kQKBZjgMvlw5KX2kGzr4vDyDCUyaMAYHrMJW2FnUzIXgoGW+2072QZlC6zn+ZHMCsDeKTsOcCT37eGNXnH11OHK2JCBxZGy7D/oPQ7NijEIehUcFRXj9Viyh0MmjqUCjQzLvzRR0VkqFuE+eEhHj3JXQouV6SyPUm1yMg69tmd6H4EnDMGGtgXojnOWxFUn7fIWpIB7+xD3wEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=L+0hREfX4eUh5Q7rzQiUyM+QqSYqC3DUo5fqX+5qvlY=; b=lMtQy09thdQSml4aeAxqgtJsYnsXWvZ9ZEOjDpWydPwo61l9XIGuBD3XHiEBuSmbmB+cYmmgU+3z74tzgaklTnI3jS6TU44kQEibZ20LFkKSOowGaaaeHLXHb34IW9R8P/OcPtCCx8ZTR3Xk5xnjB3bXWsnGItB0OXozF/qlfO+BtPczGyW2Cp7qvROLuaB1bFTg2uZw5txhoiPk+yVbdb8hwKryCZSLpNpZuMjcvBjfJemGExtIcfdcmx1+MxywrxlW2s+nEqQR5GjnecYSOIwpclRdh3OzPcjepqOsnUanhyNa8XtG+dMnmqlBGA7c6gswQxhojBom18aHBu4N9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=L+0hREfX4eUh5Q7rzQiUyM+QqSYqC3DUo5fqX+5qvlY=; b=QddQuWuWPGA9ahK9TAcLfiifIPY/xRSgjocJWU4JIaD1Fn5TF8DFryxd110Y2v5DJu9zGHG/jgvci4/0yBsuWngS2GBwGjLGtnFIsYhT/3SG7iDx8/reRQUAm5KYjtkgNwHnymK4av1vyfcNyEozToTkrED6ksCGJZVTj/P/FC+ocU0BAGJKmem75KRbQQ32Ql9DzkR8ZAemURABex2roZ+BOlj1bvaKitmLfn9cVqINxjpz/Q54wi8qL7F1hqUw317E/Sw/sTrTgjZsEnIATSu5s3DBV5uU928Nu8p6ZSonUnjTTR8Cn87TQqKMrDjeDZvcVnUMjo97CrfwYloNaA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5629.namprd04.prod.outlook.com (2603:10b6:208:a6::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.16; Sat, 11 Sep 2021 06:02:40 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4500.018; Sat, 11 Sep 2021 06:02:40 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v3 02/18] avutil/subfmt: Move AVSubtitleType, AVSubtitle and AVSubtitleRect to avutil Thread-Index: AdemwRmnf+l4c62oQFOSxFEx/uauNQ== Date: Sat, 11 Sep 2021 06:02:39 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [/QVw2uMhWhkvxjjRKb2eLD947ggSeVDaLXyrhzYVENA=] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 512421e8-7ea9-4469-0eed-08d974e9c329 x-ms-traffictypediagnostic: MN2PR04MB5629: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: C1WgjPPCtxZmnYLxTX+/hN/BjCZ8dDEDV7Drzo9mLE7NXuyVa7fTC3jDgsehK8364CdwfSFguop8ZImqjP0rKBtffS2E36aEKSnRWkgqkQ/svMQXnMeombM8IRLRvn3gn1Bu63kA7gr0VjDCqtY1lEQd6UkWMpcpOlvn1NtFkmz46UeldKcSZdfeLMAdVVIRngcK0rde82XBDvgyBF8/4kPXHLwRc7UmZwfNNNnxPovzbJDq2X+MRUeD9CZN0kASS3qdv4uOv0dbQB9SSXxr25oe9MYzV21KJt4baIb6DinTR0eg36/AUffEZmImgbdmd6YHpyHzb23im+X5+KndYrJXcnEJNpPWv27d0NEi4m7FLaPT74v0aLQmuv9YE8DSXgLo1HFDABOYf5ch0NMCxkxwwEW4RhZHlbVKft4CapHXxAeGPhrrVEAbLuV8FEaY x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: dO7Uzic6PCNgsEHr/aEhJfrlpkgzYx7l+IH69wwa7caGwVE2BlbjanST06MMgUEKrbt6PxzLH5I+AJYgOEy0EYooyBiZYfGS0SQynaO+H4MpTHXfbONgPrcnWF6uDIsPcuTQzawivmZiTpC3AxBHcA== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 512421e8-7ea9-4469-0eed-08d974e9c329 X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2021 06:02:39.9606 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5629 Subject: [FFmpeg-devel] [PATCH v3 02/18] avutil/subfmt: Move AVSubtitleType, AVSubtitle and AVSubtitleRect to avutil X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: NUCBaEekzoRf Moving these type to avutil - analog to e.g. samplefmt, pixfmt Signed-off-by: softworkz --- libavutil/Makefile | 2 + libavutil/subfmt.c | 78 ++++++++++++++++++++++++++++ libavutil/subfmt.h | 125 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 205 insertions(+) create mode 100644 libavutil/subfmt.c create mode 100644 libavutil/subfmt.h diff --git a/libavutil/Makefile b/libavutil/Makefile index 410ac636f7..04e1101bf3 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -74,6 +74,7 @@ HEADERS = adler32.h \ sha512.h \ spherical.h \ stereo3d.h \ + subfmt.h \ threadmessage.h \ time.h \ timecode.h \ @@ -159,6 +160,7 @@ OBJS = adler32.o \ slicethread.o \ spherical.o \ stereo3d.o \ + subfmt.o \ threadmessage.o \ time.o \ timecode.o \ diff --git a/libavutil/subfmt.c b/libavutil/subfmt.c new file mode 100644 index 0000000000..8476c1a15d --- /dev/null +++ b/libavutil/subfmt.c @@ -0,0 +1,78 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include +#include "common.h" +#include "subfmt.h" + +typedef struct SubtitleFmtInfo { + enum AVSubtitleType fmt; + char* name; +} SubtitleFmtInfo; + +static const SubtitleFmtInfo sub_fmt_info[AV_SUBTITLE_FMT_NB] = { + {.fmt = AV_SUBTITLE_FMT_UNKNOWN, .name = "Unknown subtitle format", }, + {.fmt = AV_SUBTITLE_FMT_BITMAP, .name = "Graphical subtitles", }, + {.fmt = AV_SUBTITLE_FMT_TEXT, .name = "Text subtitles (plain)", }, + {.fmt = AV_SUBTITLE_FMT_ASS, .name = "Text subtitles (ass)", }, +}; + +const char *av_get_subtitle_fmt_name(enum AVSubtitleType sub_fmt) +{ + if (sub_fmt < 0 || sub_fmt >= AV_SUBTITLE_FMT_NB) + return NULL; + return sub_fmt_info[sub_fmt].name; +} + +enum AVSubtitleType av_get_subtitle_fmt(const char *name) +{ + int i; + + for (i = 0; i < AV_SUBTITLE_FMT_NB; i++) + if (!strcmp(sub_fmt_info[i].name, name)) + return i; + return AV_SUBTITLE_FMT_NONE; +} + + +void avsubtitle_free(AVSubtitle *sub) +{ + unsigned i; + + for (i = 0; i < sub->num_rects; i++) { + av_freep(&sub->rects[i]->data[0]); + av_freep(&sub->rects[i]->data[1]); + av_freep(&sub->rects[i]->data[2]); + av_freep(&sub->rects[i]->data[3]); + av_freep(&sub->rects[i]->text); + av_freep(&sub->rects[i]->ass); + av_freep(&sub->rects[i]); + } + + av_freep(&sub->rects); + + memset(sub, 0, sizeof(*sub)); +} + +void avsubtitle_free_ref(void *opaque, uint8_t *data) +{ + avsubtitle_free((AVSubtitle *)data); +} + diff --git a/libavutil/subfmt.h b/libavutil/subfmt.h new file mode 100644 index 0000000000..c6544433d0 --- /dev/null +++ b/libavutil/subfmt.h @@ -0,0 +1,125 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_SUBFMT_H +#define AVUTIL_SUBFMT_H + +#include + +#include "attributes.h" + +enum AVSubtitleType { + + /** + * Subtitle format unknown. + */ + AV_SUBTITLE_FMT_NONE = -1, + + /** + * Subtitle format unknown. + */ + AV_SUBTITLE_FMT_UNKNOWN = 0, + + /** + * Bitmap area in AVSubtitleRect.data, pixfmt AV_PIX_FMT_PAL8. + */ + AV_SUBTITLE_FMT_BITMAP = 1, + + /** + * Plain text in AVSubtitleRect.text. + */ + AV_SUBTITLE_FMT_TEXT = 2, + + /** + * Text Formatted as per ASS specification, contained AVSubtitleRect.ass. + */ + AV_SUBTITLE_FMT_ASS = 3, + + AV_SUBTITLE_FMT_NB, +}; + +typedef struct AVSubtitleRect { + int x; ///< top left corner of pict, undefined when pict is not set + int y; ///< top left corner of pict, undefined when pict is not set + int w; ///< width of pict, undefined when pict is not set + int h; ///< height of pict, undefined when pict is not set + int nb_colors; ///< number of colors in pict, undefined when pict is not set + + /** + * data+linesize for the bitmap of this subtitle. + */ + uint8_t *data[4]; + int linesize[4]; + + enum AVSubtitleType type; + + char *text; ///< 0 terminated plain UTF-8 text + + /** + * 0-terminated ASS/SSA compatible event line. + */ + char *ass; + + int flags; +} AVSubtitleRect; + +typedef struct AVSubtitle { + enum AVSubtitleType format; + uint32_t start_display_time; /* relative to packet pts, in ms */ + uint32_t end_display_time; /* relative to packet pts, in ms */ + unsigned num_rects; + AVSubtitleRect **rects; + int64_t pts; ///< Same as packet pts, in AV_TIME_BASE + + /** + * Header containing style information for text subtitles. + */ + char *header; +} AVSubtitle; + +/** + * Return the name of sub_fmt, or NULL if sub_fmt is not + * recognized. + */ +const char *av_get_subtitle_fmt_name(enum AVSubtitleType sub_fmt); + +/** + * Return a subtitle format corresponding to name, or AV_SUBTITLE_FMT_NONE + * on error. + */ +enum AVSubtitleType av_get_subtitle_fmt(const char *name); + +/** + * Free all allocated data in the given subtitle struct. + * + * @param sub AVSubtitle to free. + */ +void avsubtitle_free(AVSubtitle *sub); + +/** + * Free subtitle (for use with av_buffer) + * + * @param opaque unused + * @param data AVSubtitle to free. + */ +void avsubtitle_free_ref(void *opaque, uint8_t *data); + + +#endif /* AVUTIL_SUBFMT_H */ From patchwork Sat Sep 11 06:02:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30112 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp2167978iov; Fri, 10 Sep 2021 23:04:43 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwnk3xjOu9jLQHymfvN2d4YlZO1Nm/aPOwXQQenrZotKJyguBrFQQu1QJUuBFEKgswI3oXA X-Received: by 2002:a17:906:d04b:: with SMTP id bo11mr1334409ejb.513.1631340283107; Fri, 10 Sep 2021 23:04:43 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id js19si1193685ejc.737.2021.09.10.23.04.42; Fri, 10 Sep 2021 23:04:43 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=rYgq3pwl; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8209468A5D1; Sat, 11 Sep 2021 09:03:08 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12olkn2093.outbound.protection.outlook.com [40.92.23.93]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CA85868A5B3 for ; Sat, 11 Sep 2021 09:03:02 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NUlQpuV+dQqI06zPtty6RY2U7w7KA88emxMfdrogOhxPPWWBt7C81++Eh6rG42WdVBQbNmdLdKD6KbjT3+OV707d34iu6L4XXnd9AQIjH/1ex+ynThUmHDKm4/Spzpqy+4WF6TOVxMI81RmF4uD3v0E1juGwQtghxsMmT4ZBNUFh5Ngr3GwvCYPmqj2jCd1kqIsNDsQ5wcZ+DEkiuNQWYAHAZlFC/wCfUByP66ZuFiBL9sVFrZAoRQsoK4x45UNyX5uYO2PnwAO/oBrqvWQTvRKzxTPxQylUwsHd+1rO0e7Qt4m3uSfArSvstovY2tNHBv+EQofchIzQNkbgVSLtfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=mdnP4vNiSOIF14sCXlqB6GPZNiEbEet1qK1hSWjkmgQ=; b=YNTgLLGNBKUUb3WAyud0n5L+3ZNsnkurbLE7xQYFQWYQxOBnLdh/aNEu+e5qjIvBabZO4fK5leKr9LGm5WZivXR7v8DJM/z8Z0eeUssi2Zp16j4bar3ggAKN9MV7vFtNJdHYsSweGGn4EU2FLSQLBfeSxAzsaP+VNrgiV9Bd3p9kU8/sx7v8GRf5gQqCDiuGWRKVSKLTWel3UE5ZS31+fNKaID5TPiyQxsw8gwmT8ObV8ude9r4MbSbzkDezlSWf4vLuw0xVeeKDKM+t6dLuAAao8Ryw69cW8fCeXeIJSc4lH1wAa9qkYrrchDj2ffQUt2dYowZMCIsZv/i0WOF3WA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mdnP4vNiSOIF14sCXlqB6GPZNiEbEet1qK1hSWjkmgQ=; b=rYgq3pwldrZTIWnFOAUKchzwRO/iWcQAfwr8upTXpQwjfnL/FMuaRsaKwNSZhhkoDTkFgSSkcmtQzTgMb2L8m5ym50JfvF19is9ePB/NTkMNfhUVwQEnyAOMYJ6rHHciDunyRHQrahiiHQPZ6dDQb8E1HemEcgZfN0+DaKU90f8iYQe4kR2y/G4p6RzR6hJ1whGq8DnpP1+LGiGlHuIBrjIOpBhhUXiHEuCSoRfvArbR1Ku0u1hDi8DVtsq4Qo4MgdOzetp8m1xJ/JL7LQQhitovTbvz7+8KJkpd9Z7vXRZ5uJs0JpIYcRdLQFncRD16UPVbISAkPeST+mfrcK8Rtw== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5629.namprd04.prod.outlook.com (2603:10b6:208:a6::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.16; Sat, 11 Sep 2021 06:02:50 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4500.018; Sat, 11 Sep 2021 06:02:50 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v3 07/18] fftools/play, probe: Adjust for subtitle format type change Thread-Index: AdemwSKXqpERtxGbS0KgpcR2ktb1Fg== Date: Sat, 11 Sep 2021 06:02:50 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [+NfgUQzAtVHN0J+ombL54JvVCNx462ygtAFJ4dKKEJw=] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0bb4def9-abe1-417b-1f79-08d974e9c9a4 x-ms-traffictypediagnostic: MN2PR04MB5629: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: npCc9B23DmwAexY9JkSq2K3ObeaAf4wdN/WQN5mmL+TF3k+fYD5/bm8pPOqpgjn7chBo60ZyJKTUmyMw+7fsmfKbabQr4WOW7e9X5m4En0zB/vcAb4UK+io6RUwk4MA4j6EbJ5ogsws/P2lNL132rkjvTSJvBR3VZYn/p1s1NOCjBDSPeJd6pl0P3WKf7PbBtKIuaukVLbf261gJh97sojyjDwtnn2XOcWCTLCIzILiFeE+7DroiMa8CS3UdDZd2gph2SbgkLQAHOrnYiRzvD3Z5Cd7oo0DLrTdqvc4Wika1LNRngy/xbUL1tyan3ZeBPTLYYkwCk9UwSEdRLBc68CLH7Rth/cycWGj28Ymvf6oiUG60TvGLW7HJaBVeOb+SADvkU/qIfJiPKI6gBNKBI5AYiQ85AffmWQC2TTR4+FYyV4IMA57ZA1kD7qRKEP4W x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: G0YsS6lFm/bnQua9CTaoz4F45vvis6rH0fr8YCMgklE4zT7f8gBuAh+5Ewd8WJo0T/1OgiVfotAcuP/9yuWhXUY5mzvubVMyW3U/pmigEh3bszUVJsRlFXlG+v0mtGRTmV3J3vXQhLqufUsyWPt6nw== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 0bb4def9-abe1-417b-1f79-08d974e9c9a4 X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2021 06:02:50.7934 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5629 Subject: [FFmpeg-devel] [PATCH v3 07/18] fftools/play, probe: Adjust for subtitle format type change X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: MbSnDLcIE1qb Signed-off-by: softworkz --- fftools/ffplay.c | 2 +- fftools/ffprobe.c | 23 ++++++++++++++++++++++- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/fftools/ffplay.c b/fftools/ffplay.c index 46758b9f55..f6a4d242c3 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -2250,7 +2250,7 @@ static int subtitle_thread(void *arg) pts = 0; - if (got_subtitle && sp->sub.format == 0) { + if (got_subtitle && sp->sub.format == AV_SUBTITLE_FMT_BITMAP) { if (sp->sub.pts != AV_NOPTS_VALUE) pts = sp->sub.pts / (double)AV_TIME_BASE; sp->pts = pts; diff --git a/fftools/ffprobe.c b/fftools/ffprobe.c index acfec09656..fb55f3b292 100644 --- a/fftools/ffprobe.c +++ b/fftools/ffprobe.c @@ -2212,6 +2212,7 @@ static void show_subtitle(WriterContext *w, AVSubtitle *sub, AVStream *stream, AVFormatContext *fmt_ctx) { AVBPrint pbuf; + const char *s; av_bprint_init(&pbuf, 1, AV_BPRINT_SIZE_UNLIMITED); @@ -2220,7 +2221,27 @@ static void show_subtitle(WriterContext *w, AVSubtitle *sub, AVStream *stream, print_str ("media_type", "subtitle"); print_ts ("pts", sub->pts); print_time("pts_time", sub->pts, &AV_TIME_BASE_Q); - print_int ("format", sub->format); + + // Remain compatible with previous outputs? + switch (sub->format) { + case AV_SUBTITLE_FMT_BITMAP: + print_int ("format", 0); + break; + case AV_SUBTITLE_FMT_TEXT: + print_int ("format", 1); + break; + case AV_SUBTITLE_FMT_ASS: + print_int ("format", 1); + break; + default: + print_int ("format", -1); + break; + } + + s = av_get_subtitle_fmt_name(sub->format); + if (s) print_str ("format_str", s); + else print_str_opt("format_str", "unknown"); + print_int ("start_display_time", sub->start_display_time); print_int ("end_display_time", sub->end_display_time); print_int ("num_rects", sub->num_rects); From patchwork Sat Sep 11 06:02:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30109 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp2167412iov; Fri, 10 Sep 2021 23:03:44 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwlBsxoDKQndScA/yIAk7Qc48IuA9gH9BCMWUcQ6Fqc7xdqzRen3E3RDkYHnAAd61skAdHy X-Received: by 2002:a17:907:7694:: with SMTP id jv20mr212310ejc.356.1631340224059; Fri, 10 Sep 2021 23:03:44 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b3si884128ejp.504.2021.09.10.23.03.43; Fri, 10 Sep 2021 23:03:44 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=YKoeS3k+; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id F202F6804A0; Sat, 11 Sep 2021 09:03:02 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12olkn2097.outbound.protection.outlook.com [40.92.21.97]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6916E68A43A for ; Sat, 11 Sep 2021 09:02:55 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aOvCV0kB8wepObJl90JAZ4h5nVCl8ZxR6VfzeWG1tgiZ4TAqtTwoS+2lVwY83EmrxZFOFxbZmUzYlSjr4M6iJuOKg926etnk0wWxFmkfyvM8g9DT9mJw0s02NJBVzEp+3GD/Gc3JIwaNSjDy0XbZmU5CiCMTuJTkXDFc9dZhpFa05RHIyV4c+6iuKI4heIrsEAKrxeQewaEWg5irs/8aWUDWkAY/Y/GJ0Rp06sCTibAvY2lYLhH7pDAewfpnOJh/r8xs3/3WLcaBLFCpOG6k/pOIPuf4DC0r6LbeqeQeIomufngrCyiQTGQTLE4+SeLYZzIc60fSafQpWKEuQocyBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=lfaQsTwq+uspqY+VEcudFzTzLi4gs8P9wkFvuHzYBR0=; b=EqZYn1aVfEQZSERCLMlZ0JBJODvOFnXXkhWXgtXLd+XGFPth8Ep7nvgMZYfx5s9L7Z0sCUFaoyD6gtEOPqoFuM5vQN+ktXz07WNkKWKCwbHyLskOieK4MmN1nFFH4SL9njlNVvP7kR2t3HbwokqzgR4J1QFk1z2a5aKA6mZCum9IIlr1dY7XJ9WdGjTnANktjJa0PhVFDIydQ3IhYrItdTqNKHfIWOzEhf8LCesq7IwQWtZkXfxb+BatK0jT40a2VqgQTjWPTUXSbC1qz+RK6RSXIksPEvYaAK1nSkKz0FNVS3Z1dhGlhjaWgJvc2YekJtO/oIQrDsKPDl70/6HVWA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lfaQsTwq+uspqY+VEcudFzTzLi4gs8P9wkFvuHzYBR0=; b=YKoeS3k+hd1IE28g4Bo/ZRO1h1PGpGzEHNCuXx6QSr91JKIVM32RcKRILhvcrb8s/RF9phmswmEoFwGAMLUXLjpbxOC+eKSTCKMmLGB79mIxf352sPR9RVy14HL24anZDeGnko5YqoAScOh7oZNK/3OAZ9mHtzakqv5U4BCbCBBmUyEcYAx3yrJaOgQkCS3PsM1dCGkTnTNFmBCbAGnKAKdqGOk+i6GF5Dzvz13D6JH7LZpXn6HSRMDkZXZJVqwRgKeomM9N/eGiU5hhLR2KIWqMqhTrOR9fA4Q/hoJknbB70jERSSDVeCFrZBWY8R48+UZ4QLSdxQSfopasWgJCog== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5824.namprd04.prod.outlook.com (2603:10b6:208:3d::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.14; Sat, 11 Sep 2021 06:02:53 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4500.018; Sat, 11 Sep 2021 06:02:53 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v3 08/18] fftools/ffmpeg: Replace sub2video with subtitle frame filtering Thread-Index: AdemwSrwlc2HdUEQQDKBr5MWy3LBYw== Date: Sat, 11 Sep 2021 06:02:53 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [GJ78IaQxcJvr0YDNZo3Bb7Ot4MhbR3pgg8FNzOCisSc=] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 28dc678b-dd59-44fc-3d46-08d974e9cb28 x-ms-traffictypediagnostic: MN2PR04MB5824: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: k+iDjO8fETOQNp/gUBnua400EHUOoc1xy6yOUab1SSVDE+/IUUSvyOHAF/pvt5YiUDX0MgSsmW3Uuwu1wH4KLxGH4Y5N99w7zi+4DaSnzV3MSK1wobZWmtce2Nx4ABYvDx4HWMrZArdc26HyhfG18AjRuRBPIPjpekSnOH2RfUiQihdoVjw5+89bQ3QOXwhjkm32v6akiA/Fj8jfXhEadcO6ZrZUhZyCP9jsp0kUAahNJoKS1/Y0pcaSLHSpguxokTdBOzCYahCodt43pX3nP2kg4gsogrHYtmQwN1RKg1bB/955LGON2HSSoeHGFkZevqkpeDYDihV/A1fMCK/Wite4xpps/xAL/0+R6rbscyO7TMGUJXWFpJ5s4uvLqIAgknHLWhXTCYou77r48Dyt695rWwsVgcZNbo67u8ySVpJpvXLAYBFljQkXNdpT236h x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: 7JkQf7C5BZt352bgm2oa8iZuwAfwIVGpa4eKtKzUqYpe1OqYXz30eKDyYB4GBTvOxzlooXTEKMfaTKLa5sycxsA9sfW9x+vNhlW3VbhNnPcmD5Kta0nENuTUQ1DZ3G7P4DLIOrvHwmSCjhGgcBDSiA== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 28dc678b-dd59-44fc-3d46-08d974e9cb28 X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2021 06:02:53.2710 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5824 Subject: [FFmpeg-devel] [PATCH v3 08/18] fftools/ffmpeg: Replace sub2video with subtitle frame filtering X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: h9wL2EPtiscK Signed-off-by: softworkz --- fftools/ffmpeg.c | 408 +++++++++++++++++----------------------- fftools/ffmpeg.h | 12 +- fftools/ffmpeg_filter.c | 198 +++++++++++++------ fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- 5 files changed, 323 insertions(+), 300 deletions(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index a9bb9d964d..36fba765de 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -169,163 +169,6 @@ static int restore_tty; static void free_input_threads(void); #endif -/* sub2video hack: - Convert subtitles to video with alpha to insert them in filter graphs. - This is a temporary solution until libavfilter gets real subtitles support. - */ - -static int sub2video_get_blank_frame(InputStream *ist) -{ - int ret; - AVFrame *frame = ist->sub2video.frame; - - av_frame_unref(frame); - ist->sub2video.frame->width = ist->dec_ctx->width ? ist->dec_ctx->width : ist->sub2video.w; - ist->sub2video.frame->height = ist->dec_ctx->height ? ist->dec_ctx->height : ist->sub2video.h; - ist->sub2video.frame->format = AV_PIX_FMT_RGB32; - if ((ret = av_frame_get_buffer(frame, 0)) < 0) - return ret; - memset(frame->data[0], 0, frame->height * frame->linesize[0]); - return 0; -} - -static void sub2video_copy_rect(uint8_t *dst, int dst_linesize, int w, int h, - AVSubtitleRect *r) -{ - uint32_t *pal, *dst2; - uint8_t *src, *src2; - int x, y; - - if (r->type != SUBTITLE_BITMAP) { - av_log(NULL, AV_LOG_WARNING, "sub2video: non-bitmap subtitle\n"); - return; - } - if (r->x < 0 || r->x + r->w > w || r->y < 0 || r->y + r->h > h) { - av_log(NULL, AV_LOG_WARNING, "sub2video: rectangle (%d %d %d %d) overflowing %d %d\n", - r->x, r->y, r->w, r->h, w, h - ); - return; - } - - dst += r->y * dst_linesize + r->x * 4; - src = r->data[0]; - pal = (uint32_t *)r->data[1]; - for (y = 0; y < r->h; y++) { - dst2 = (uint32_t *)dst; - src2 = src; - for (x = 0; x < r->w; x++) - *(dst2++) = pal[*(src2++)]; - dst += dst_linesize; - src += r->linesize[0]; - } -} - -static void sub2video_push_ref(InputStream *ist, int64_t pts) -{ - AVFrame *frame = ist->sub2video.frame; - int i; - int ret; - - av_assert1(frame->data[0]); - ist->sub2video.last_pts = frame->pts = pts; - for (i = 0; i < ist->nb_filters; i++) { - ret = av_buffersrc_add_frame_flags(ist->filters[i]->filter, frame, - AV_BUFFERSRC_FLAG_KEEP_REF | - AV_BUFFERSRC_FLAG_PUSH); - if (ret != AVERROR_EOF && ret < 0) - av_log(NULL, AV_LOG_WARNING, "Error while add the frame to buffer source(%s).\n", - av_err2str(ret)); - } -} - -void sub2video_update(InputStream *ist, int64_t heartbeat_pts, AVSubtitle *sub) -{ - AVFrame *frame = ist->sub2video.frame; - int8_t *dst; - int dst_linesize; - int num_rects, i; - int64_t pts, end_pts; - - if (!frame) - return; - if (sub) { - pts = av_rescale_q(sub->pts + sub->start_display_time * 1000LL, - AV_TIME_BASE_Q, ist->st->time_base); - end_pts = av_rescale_q(sub->pts + sub->end_display_time * 1000LL, - AV_TIME_BASE_Q, ist->st->time_base); - num_rects = sub->num_rects; - } else { - /* If we are initializing the system, utilize current heartbeat - PTS as the start time, and show until the following subpicture - is received. Otherwise, utilize the previous subpicture's end time - as the fall-back value. */ - pts = ist->sub2video.initialize ? - heartbeat_pts : ist->sub2video.end_pts; - end_pts = INT64_MAX; - num_rects = 0; - } - if (sub2video_get_blank_frame(ist) < 0) { - av_log(ist->dec_ctx, AV_LOG_ERROR, - "Impossible to get a blank canvas.\n"); - return; - } - dst = frame->data [0]; - dst_linesize = frame->linesize[0]; - for (i = 0; i < num_rects; i++) - sub2video_copy_rect(dst, dst_linesize, frame->width, frame->height, sub->rects[i]); - sub2video_push_ref(ist, pts); - ist->sub2video.end_pts = end_pts; - ist->sub2video.initialize = 0; -} - -static void sub2video_heartbeat(InputStream *ist, int64_t pts) -{ - InputFile *infile = input_files[ist->file_index]; - int i, j, nb_reqs; - int64_t pts2; - - /* When a frame is read from a file, examine all sub2video streams in - the same file and send the sub2video frame again. Otherwise, decoded - video frames could be accumulating in the filter graph while a filter - (possibly overlay) is desperately waiting for a subtitle frame. */ - for (i = 0; i < infile->nb_streams; i++) { - InputStream *ist2 = input_streams[infile->ist_index + i]; - if (!ist2->sub2video.frame) - continue; - /* subtitles seem to be usually muxed ahead of other streams; - if not, subtracting a larger time here is necessary */ - pts2 = av_rescale_q(pts, ist->st->time_base, ist2->st->time_base) - 1; - /* do not send the heartbeat frame if the subtitle is already ahead */ - if (pts2 <= ist2->sub2video.last_pts) - continue; - if (pts2 >= ist2->sub2video.end_pts || ist2->sub2video.initialize) - /* if we have hit the end of the current displayed subpicture, - or if we need to initialize the system, update the - overlayed subpicture and its start/end times */ - sub2video_update(ist2, pts2 + 1, NULL); - for (j = 0, nb_reqs = 0; j < ist2->nb_filters; j++) - nb_reqs += av_buffersrc_get_nb_failed_requests(ist2->filters[j]->filter); - if (nb_reqs) - sub2video_push_ref(ist2, pts2); - } -} - -static void sub2video_flush(InputStream *ist) -{ - int i; - int ret; - - if (ist->sub2video.end_pts < INT64_MAX) - sub2video_update(ist, INT64_MAX, NULL); - for (i = 0; i < ist->nb_filters; i++) { - ret = av_buffersrc_add_frame(ist->filters[i]->filter, NULL); - if (ret != AVERROR_EOF && ret < 0) - av_log(NULL, AV_LOG_WARNING, "Flush the frame error.\n"); - } -} - -/* end of sub2video hack */ - static void term_exit_sigsafe(void) { #if HAVE_TERMIOS_H @@ -526,7 +369,6 @@ static void ffmpeg_cleanup(int ret) avfilter_graph_free(&fg->graph); for (j = 0; j < fg->nb_inputs; j++) { InputFilter *ifilter = fg->inputs[j]; - struct InputStream *ist = ifilter->ist; while (av_fifo_size(ifilter->frame_queue)) { AVFrame *frame; @@ -535,15 +377,6 @@ static void ffmpeg_cleanup(int ret) av_frame_free(&frame); } av_fifo_freep(&ifilter->frame_queue); - if (ist->sub2video.sub_queue) { - while (av_fifo_size(ist->sub2video.sub_queue)) { - AVSubtitle sub; - av_fifo_generic_read(ist->sub2video.sub_queue, - &sub, sizeof(sub), NULL); - avsubtitle_free(&sub); - } - av_fifo_freep(&ist->sub2video.sub_queue); - } av_buffer_unref(&ifilter->hw_frames_ctx); av_freep(&ifilter->name); av_freep(&fg->inputs[j]); @@ -636,11 +469,13 @@ static void ffmpeg_cleanup(int ret) av_packet_free(&ist->pkt); av_dict_free(&ist->decoder_opts); avsubtitle_free(&ist->prev_sub.subtitle); - av_frame_free(&ist->sub2video.frame); av_freep(&ist->filters); av_freep(&ist->hwaccel_device); av_freep(&ist->dts_buffer); + av_buffer_unref(&ist->subtitle_heartbeat.recent_sub); + av_freep(&ist->subtitle_heartbeat.header); + avcodec_free_context(&ist->dec_ctx); av_freep(&input_streams[i]); @@ -1062,13 +897,19 @@ error: static void do_subtitle_out(OutputFile *of, OutputStream *ost, - AVSubtitle *sub) + AVFrame *frame) { int subtitle_out_max_size = 1024 * 1024; int subtitle_out_size, nb, i; AVCodecContext *enc; AVPacket *pkt = ost->pkt; int64_t pts; + AVSubtitle *sub = (AVSubtitle *)frame->data[0]; + + if (!sub) + return; + + av_log(NULL, AV_LOG_TRACE, "do_subtitle_out: sub->pts: %"PRId64" frame->pts: %"PRId64"\n", sub->pts, frame->pts); if (sub->pts == AV_NOPTS_VALUE) { av_log(NULL, AV_LOG_ERROR, "Subtitle packets must have a pts\n"); @@ -1577,8 +1418,11 @@ static int reap_filters(int flush) } do_audio_out(of, ost, filtered_frame); break; + case AVMEDIA_TYPE_SUBTITLE: + + do_subtitle_out(of, ost, filtered_frame); + break; default: - // TODO support subtitle filters av_assert0(0); } @@ -2174,7 +2018,8 @@ static int ifilter_has_all_input_formats(FilterGraph *fg) int i; for (i = 0; i < fg->nb_inputs; i++) { if (fg->inputs[i]->format < 0 && (fg->inputs[i]->type == AVMEDIA_TYPE_AUDIO || - fg->inputs[i]->type == AVMEDIA_TYPE_VIDEO)) + fg->inputs[i]->type == AVMEDIA_TYPE_VIDEO || + fg->inputs[i]->type == AVMEDIA_TYPE_SUBTITLE)) return 0; } return 1; @@ -2271,7 +2116,7 @@ static int ifilter_send_eof(InputFilter *ifilter, int64_t pts) // the filtergraph was never configured if (ifilter->format < 0) ifilter_parameters_from_codecpar(ifilter, ifilter->ist->st->codecpar); - if (ifilter->format < 0 && (ifilter->type == AVMEDIA_TYPE_AUDIO || ifilter->type == AVMEDIA_TYPE_VIDEO)) { + if (ifilter->format < 0 && (ifilter->type == AVMEDIA_TYPE_AUDIO || ifilter->type == AVMEDIA_TYPE_VIDEO || ifilter->type == AVMEDIA_TYPE_SUBTITLE)) { av_log(NULL, AV_LOG_ERROR, "Cannot determine format of input stream %d:%d after EOF\n", ifilter->ist->file_index, ifilter->ist->st->index); return AVERROR_INVALIDDATA; } @@ -2526,30 +2371,136 @@ fail: return err < 0 ? err : ret; } -static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, +static void subtitle_resend_current(InputStream *ist, int64_t heartbeat_pts) +{ + AVFrame *frame; + AVBufferRef *sub_buffer; + int64_t pts, end_pts; + + frame = av_frame_alloc(); + if (!frame) { + av_log(ist->dec_ctx, AV_LOG_ERROR, "Unable to alloc frame (out of memory).\n"); + return; + } + + frame->format = av_get_subtitle_format_from_codecdesc(ist->dec_ctx->codec_descriptor); + + frame->width = ist->subtitle_heartbeat.w; + frame->height = ist->subtitle_heartbeat.h; + + if (ist->subtitle_heartbeat.recent_sub) { + AVSubtitle *sub = (AVSubtitle *)ist->subtitle_heartbeat.recent_sub->data; + sub_buffer = av_buffer_ref(ist->subtitle_heartbeat.recent_sub); + + pts = av_rescale_q(sub->pts + sub->start_display_time * 1000LL, AV_TIME_BASE_Q, ist->st->time_base); + end_pts = av_rescale_q(sub->pts + sub->end_display_time * 1000LL, AV_TIME_BASE_Q, ist->st->time_base); + } + else { + AVSubtitle *empty_sub = av_mallocz(sizeof(*empty_sub)); + + pts = ist->subtitle_heartbeat.end_pts <= 0 ? heartbeat_pts : ist->subtitle_heartbeat.end_pts; + end_pts = INT64_MAX; + + empty_sub->format = frame->format; + empty_sub->pts = av_rescale_q(pts, ist->st->time_base, AV_TIME_BASE_Q); + empty_sub->end_display_time = 1000; + sub_buffer = av_buffer_create((uint8_t*)empty_sub, sizeof(*empty_sub), avsubtitle_free_ref, NULL, AV_BUFFER_FLAG_READONLY); + } + + frame->buf[0] = sub_buffer; + frame->data[0] = sub_buffer->data; + frame->pts = pts; + ist->subtitle_heartbeat.last_pts = pts; + ist->subtitle_heartbeat.end_pts = end_pts; + + send_frame_to_filters(ist, frame); +} + +static void subtitle_heartbeat(InputStream *ist, int64_t pts) +{ + const InputFile *infile = input_files[ist->file_index]; + int i; + int64_t pts2; + + /* When a frame is read from a file, examine all subtitle streams in + the same file and send the subtitle frame again. Otherwise, decoded + video frames could be accumulating in the filter graph while a filter + (possibly overlay) is desperately waiting for a subtitle frame. */ + for (i = 0; i < infile->nb_streams; i++) { + InputStream *ist2 = input_streams[infile->ist_index + i]; + if (!ist2->subtitle_heartbeat.is_active) + continue; + /* subtitles seem to be usually muxed ahead of other streams; + if not, subtracting a larger time here is necessary */ + pts2 = av_rescale_q(pts, ist->st->time_base, ist2->st->time_base) - 1; + /* do not send the heartbeat frame if the subtitle is already ahead */ + if (pts2 <= ist2->subtitle_heartbeat.last_pts) + continue; + if (pts2 >= ist2->subtitle_heartbeat.end_pts) { + /* if we have hit the end of the current displayed subpicture, + or if we need to initialize the system, update the + overlayed subpicture and its start/end times */ + if (ist2->subtitle_heartbeat.recent_sub) { + av_buffer_unref(&ist2->subtitle_heartbeat.recent_sub); + ist2->subtitle_heartbeat.recent_sub = NULL; + } + subtitle_resend_current(ist2, pts2 + 1); + } + //for (j = 0, nb_reqs = 0; j < ist2->nb_filters; j++) + // nb_reqs += av_buffersrc_get_nb_failed_requests(ist2->filters[j]->filter); + //if (nb_reqs) + // subtitle_heartbeat_resend_current(ist2, pts2); + } +} + +static int decode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, int *decode_failed) { - AVSubtitle subtitle; - int free_sub = 1; - int i, ret = avcodec_decode_subtitle2(ist->dec_ctx, - &subtitle, got_output, pkt); + AVFrame *decoded_frame; + AVCodecContext *avctx = ist->dec_ctx; + int i = 0, ret = 0, err = 0; + int64_t pts, end_pts; + AVSubtitle *subtitle = av_mallocz(sizeof(*subtitle)); - check_decode_result(NULL, got_output, ret); + if (!subtitle) + return AVERROR(ENOMEM); + + if (!ist->decoded_frame && !(ist->decoded_frame = av_frame_alloc())) + return AVERROR(ENOMEM); + if (!ist->filter_frame && !(ist->filter_frame = av_frame_alloc())) + return AVERROR(ENOMEM); + + decoded_frame = ist->decoded_frame; + + if (!ist->subtitle_heartbeat.header && avctx->subtitle_header) + ist->subtitle_heartbeat.header = av_strdup((char *)avctx->subtitle_header); + + ret = avcodec_decode_subtitle2(avctx, subtitle, got_output, pkt); + subtitle->header = ist->subtitle_heartbeat.header; + + if (ret != AVERROR_EOF) + check_decode_result(NULL, got_output, ret); if (ret < 0 || !*got_output) { *decode_failed = 1; - if (!pkt->size) - sub2video_flush(ist); + if (!pkt->size) { + // Flush + for (i = 0; i < ist->nb_filters; i++) { + ret = av_buffersrc_add_frame(ist->filters[i]->filter, NULL); + if (ret != AVERROR_EOF && ret < 0) + av_log(NULL, AV_LOG_WARNING, "Flush the frame error.\n"); + } + } return ret; } if (ist->fix_sub_duration) { int end = 1; if (ist->prev_sub.got_output) { - end = av_rescale(subtitle.pts - ist->prev_sub.subtitle.pts, + end = av_rescale(subtitle->pts - ist->prev_sub.subtitle.pts, 1000, AV_TIME_BASE); if (end < ist->prev_sub.subtitle.end_display_time) { - av_log(ist->dec_ctx, AV_LOG_DEBUG, + av_log(avctx, AV_LOG_DEBUG, "Subtitle duration reduced from %"PRId32" to %d%s\n", ist->prev_sub.subtitle.end_display_time, end, end <= 0 ? ", dropping it" : ""); @@ -2558,51 +2509,52 @@ static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, } FFSWAP(int, *got_output, ist->prev_sub.got_output); FFSWAP(int, ret, ist->prev_sub.ret); - FFSWAP(AVSubtitle, subtitle, ist->prev_sub.subtitle); + FFSWAP(AVSubtitle, *subtitle, ist->prev_sub.subtitle); if (end <= 0) - goto out; + return end; } if (!*got_output) return ret; - if (ist->sub2video.frame) { - sub2video_update(ist, INT64_MIN, &subtitle); - } else if (ist->nb_filters) { - if (!ist->sub2video.sub_queue) - ist->sub2video.sub_queue = av_fifo_alloc(8 * sizeof(AVSubtitle)); - if (!ist->sub2video.sub_queue) - exit_program(1); - if (!av_fifo_space(ist->sub2video.sub_queue)) { - ret = av_fifo_realloc2(ist->sub2video.sub_queue, 2 * av_fifo_size(ist->sub2video.sub_queue)); - if (ret < 0) - exit_program(1); - } - av_fifo_generic_write(ist->sub2video.sub_queue, &subtitle, sizeof(subtitle), NULL); - free_sub = 0; - } + decoded_frame->format = av_get_subtitle_format_from_codecdesc(avctx->codec_descriptor); - if (!subtitle.num_rects) - goto out; + if ((ret = av_frame_get_buffer2(decoded_frame, AVMEDIA_TYPE_SUBTITLE, 0)) < 0) + return ret; - ist->frames_decoded++; + av_buffer_unref(&ist->subtitle_heartbeat.recent_sub); + ist->subtitle_heartbeat.recent_sub = av_buffer_create((uint8_t*)subtitle, sizeof(*subtitle), avsubtitle_free_ref, NULL, AV_BUFFER_FLAG_READONLY); - for (i = 0; i < nb_output_streams; i++) { - OutputStream *ost = output_streams[i]; + decoded_frame->buf[0] = av_buffer_ref(ist->subtitle_heartbeat.recent_sub); + decoded_frame->data[0] = ist->subtitle_heartbeat.recent_sub->data; - if (!ost->pkt && !(ost->pkt = av_packet_alloc())) - exit_program(1); - if (!check_output_constraints(ist, ost) || !ost->encoding_needed - || ost->enc->type != AVMEDIA_TYPE_SUBTITLE) - continue; + pts = av_rescale_q(subtitle->pts + subtitle->start_display_time * 1000LL, + AV_TIME_BASE_Q, ist->st->time_base); + end_pts = av_rescale_q(subtitle->pts + subtitle->end_display_time * 1000LL, + AV_TIME_BASE_Q, ist->st->time_base); - do_subtitle_out(output_files[ost->file_index], ost, &subtitle); + ist->subtitle_heartbeat.last_pts = decoded_frame->pts = pts; + ist->subtitle_heartbeat.end_pts = end_pts; + + if (ist->nb_filters == 0) { + for (i = 0; i < nb_output_streams; i++) { + OutputStream *ost = output_streams[i]; + + if (!ost->pkt && !(ost->pkt = av_packet_alloc())) + exit_program(1); + if (!check_output_constraints(ist, ost) || !ost->encoding_needed + || ost->enc->type != AVMEDIA_TYPE_SUBTITLE) + continue; + + do_subtitle_out(output_files[ost->file_index], ost, decoded_frame); + } } + else + err = send_frame_to_filters(ist, decoded_frame); -out: - if (free_sub) - avsubtitle_free(&subtitle); - return ret; + av_frame_unref(ist->filter_frame); + av_frame_unref(decoded_frame); + return err < 0 ? err : ret; } static int send_filter_eof(InputStream *ist) @@ -2710,7 +2662,7 @@ static int process_input_packet(InputStream *ist, const AVPacket *pkt, int no_eo case AVMEDIA_TYPE_SUBTITLE: if (repeating) break; - ret = transcode_subtitles(ist, avpkt, &got_output, &decode_failed); + ret = decode_subtitles(ist, avpkt, &got_output, &decode_failed); if (!pkt && ret >= 0) ret = AVERROR_EOF; av_packet_unref(avpkt); @@ -2983,8 +2935,6 @@ FF_ENABLE_DEPRECATION_WARNINGS av_log(NULL, AV_LOG_WARNING, "Warning using DVB subtitles for filtering and output at the same time is not fully supported, also see -compute_edt [0|1]\n"); } - av_dict_set(&ist->decoder_opts, "sub_text_format", "ass", AV_DICT_DONT_OVERWRITE); - /* Useful for subtitles retiming by lavf (FIXME), skipping samples in * audio, and video decoders such as cuvid or mediacodec */ ist->dec_ctx->pkt_timebase = ist->st->time_base; @@ -3582,19 +3532,14 @@ static int init_output_stream(OutputStream *ost, AVFrame *frame, } if (ist && ist->dec->type == AVMEDIA_TYPE_SUBTITLE && ost->enc->type == AVMEDIA_TYPE_SUBTITLE) { - int input_props = 0, output_props = 0; - AVCodecDescriptor const *input_descriptor = - avcodec_descriptor_get(dec->codec_id); - AVCodecDescriptor const *output_descriptor = - avcodec_descriptor_get(ost->enc_ctx->codec_id); - if (input_descriptor) - input_props = input_descriptor->props & (AV_CODEC_PROP_TEXT_SUB | AV_CODEC_PROP_BITMAP_SUB); - if (output_descriptor) - output_props = output_descriptor->props & (AV_CODEC_PROP_TEXT_SUB | AV_CODEC_PROP_BITMAP_SUB); - if (input_props && output_props && input_props != output_props) { - snprintf(error, error_len, - "Subtitle encoding currently only possible from text to text " - "or bitmap to bitmap"); + AVCodecDescriptor const *input_descriptor = avcodec_descriptor_get(dec->codec_id); + AVCodecDescriptor const *output_descriptor = avcodec_descriptor_get(ost->enc_ctx->codec_id); + const enum AVSubtitleType in_subtitle_format = output_descriptor ? av_get_subtitle_format_from_codecdesc(input_descriptor) : AV_SUBTITLE_FMT_UNKNOWN; + const enum AVSubtitleType out_subtitle_format = output_descriptor ? av_get_subtitle_format_from_codecdesc(output_descriptor) : AV_SUBTITLE_FMT_UNKNOWN; + + if (in_subtitle_format != AV_SUBTITLE_FMT_UNKNOWN && out_subtitle_format != AV_SUBTITLE_FMT_UNKNOWN + && in_subtitle_format != out_subtitle_format) { + snprintf(error, error_len, "Subtitle encoding is only possible from text to text or bitmap to bitmap"); return AVERROR_INVALIDDATA; } } @@ -4627,7 +4572,7 @@ static int process_input(int file_index) av_ts2timestr(input_files[ist->file_index]->ts_offset, &AV_TIME_BASE_Q)); } - sub2video_heartbeat(ist, pkt->pts); + subtitle_heartbeat(ist, pkt->pts); process_input_packet(ist, pkt, 0); @@ -4839,6 +4784,7 @@ static int transcode(void) /* at the end of stream, we must flush the decoder buffers */ for (i = 0; i < nb_input_streams; i++) { ist = input_streams[i]; + ist->subtitle_heartbeat.is_active = 0; if (!input_files[ist->file_index]->eof_reached) { process_input_packet(ist, NULL, 0); } diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index 1194bb0cae..d1c6ae9769 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -351,14 +351,14 @@ typedef struct InputStream { AVSubtitle subtitle; } prev_sub; - struct sub2video { + struct subtitle_heartbeat { + int is_active; int64_t last_pts; int64_t end_pts; - AVFifoBuffer *sub_queue; ///< queue of AVSubtitle* before filter init - AVFrame *frame; + AVBufferRef *recent_sub; int w, h; - unsigned int initialize; ///< marks if sub2video_update should force an initialization - } sub2video; + char *header; + } subtitle_heartbeat; /* decoded data from this stream goes into all those filters * currently video and audio only */ @@ -660,8 +660,6 @@ int filtergraph_is_simple(FilterGraph *fg); int init_simple_filtergraph(InputStream *ist, OutputStream *ost); int init_complex_filtergraph(FilterGraph *fg); -void sub2video_update(InputStream *ist, int64_t heartbeat_pts, AVSubtitle *sub); - int ifilter_parameters_from_frame(InputFilter *ifilter, const AVFrame *frame); int ffmpeg_parse_options(int argc, char **argv); diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c index 6b7b6ca1b3..983b49cb8b 100644 --- a/fftools/ffmpeg_filter.c +++ b/fftools/ffmpeg_filter.c @@ -22,6 +22,9 @@ #include "ffmpeg.h" +#include +#include + #include "libavfilter/avfilter.h" #include "libavfilter/buffersink.h" #include "libavfilter/buffersrc.h" @@ -221,9 +224,8 @@ static void init_input_filter(FilterGraph *fg, AVFilterInOut *in) enum AVMediaType type = avfilter_pad_get_type(in->filter_ctx->input_pads, in->pad_idx); int i; - // TODO: support other filter types - if (type != AVMEDIA_TYPE_VIDEO && type != AVMEDIA_TYPE_AUDIO) { - av_log(NULL, AV_LOG_FATAL, "Only video and audio filters supported " + if (type != AVMEDIA_TYPE_VIDEO && type != AVMEDIA_TYPE_AUDIO && type != AVMEDIA_TYPE_SUBTITLE) { + av_log(NULL, AV_LOG_FATAL, "Only video, audio and subtitle filters supported " "currently.\n"); exit_program(1); } @@ -244,8 +246,9 @@ static void init_input_filter(FilterGraph *fg, AVFilterInOut *in) for (i = 0; i < s->nb_streams; i++) { enum AVMediaType stream_type = s->streams[i]->codecpar->codec_type; if (stream_type != type && - !(stream_type == AVMEDIA_TYPE_SUBTITLE && - type == AVMEDIA_TYPE_VIDEO /* sub2video hack */)) + // in the followng case we auto-insert the graphicsub2video conversion filter + // for retaining compatibility with the previous sub2video hack + !(stream_type == AVMEDIA_TYPE_SUBTITLE && type == AVMEDIA_TYPE_VIDEO)) continue; if (check_stream_specifier(s, s->streams[i], *p == ':' ? p + 1 : p) == 1) { st = s->streams[i]; @@ -416,6 +419,37 @@ static int insert_filter(AVFilterContext **last_filter, int *pad_idx, return 0; } +static int configure_output_subtitle_filter(FilterGraph *fg, OutputFilter *ofilter, AVFilterInOut *out) +{ + OutputStream *ost = ofilter->ost; + AVFilterContext *last_filter = out->filter_ctx; + int pad_idx = out->pad_idx; + int ret; + char name[255]; + + snprintf(name, sizeof(name), "out_%d_%d", ost->file_index, ost->index); + ret = avfilter_graph_create_filter(&ofilter->filter, + avfilter_get_by_name("sbuffersink"), + name, NULL, NULL, fg->graph); + + if (ret < 0) + return ret; + + ////snprintf(name, sizeof(name), "trim_out_%d_%d", + //// ost->file_index, ost->index); + ////ret = insert_trim(of->start_time, of->recording_time, + //// &last_filter, &pad_idx, name); + ////if (ret < 0) + //// return ret; + + ////ost->st->codecpar->codec_tag = MKTAG('a', 's', 's', 's'); + + if ((ret = avfilter_link(last_filter, pad_idx, ofilter->filter, 0)) < 0) + return ret; + + return 0; +} + static int configure_output_video_filter(FilterGraph *fg, OutputFilter *ofilter, AVFilterInOut *out) { char *pix_fmts; @@ -594,7 +628,8 @@ static int configure_output_audio_filter(FilterGraph *fg, OutputFilter *ofilter, int i; for (i=0; ictx->nb_streams; i++) - if (of->ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) + if (of->ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO || + of->ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) break; if (ictx->nb_streams) { @@ -628,6 +663,7 @@ static int configure_output_filter(FilterGraph *fg, OutputFilter *ofilter, switch (avfilter_pad_get_type(out->filter_ctx->output_pads, out->pad_idx)) { case AVMEDIA_TYPE_VIDEO: return configure_output_video_filter(fg, ofilter, out); case AVMEDIA_TYPE_AUDIO: return configure_output_audio_filter(fg, ofilter, out); + case AVMEDIA_TYPE_SUBTITLE: return configure_output_subtitle_filter(fg, ofilter, out); default: av_assert0(0); return 0; } } @@ -647,51 +683,103 @@ void check_filter_outputs(void) } } -static int sub2video_prepare(InputStream *ist, InputFilter *ifilter) +static int configure_input_subtitle_filter(FilterGraph *fg, InputFilter *ifilter, + AVFilterInOut *in) { - AVFormatContext *avf = input_files[ist->file_index]->ctx; - int i, w, h; + AVFilterContext *last_filter; + const AVFilter *buffer_filt = avfilter_get_by_name("sbuffer"); + InputStream *ist = ifilter->ist; + AVBPrint args; + char name[255]; + int ret, pad_idx = 0; + int w, h; + AVBufferSrcParameters *par = av_buffersrc_parameters_alloc(); + + if (!par) + return AVERROR(ENOMEM); + memset(par, 0, sizeof(*par)); + par->format = AV_PIX_FMT_NONE; + + if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_AUDIO) { + av_log(NULL, AV_LOG_ERROR, "Cannot connect subtitle filter to audio input\n"); + ret = AVERROR(EINVAL); + goto fail; + } + + if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_VIDEO) { + av_log(NULL, AV_LOG_ERROR, "Cannot connect subtitle filter to video input\n"); + ret = AVERROR(EINVAL); + goto fail; + } + + if (!ist->subtitle_heartbeat.header && ist->dec_ctx->subtitle_header) + ist->subtitle_heartbeat.header = av_strdup((char *)ist->dec_ctx->subtitle_header); + + ist->subtitle_heartbeat.is_active = 1; - /* Compute the size of the canvas for the subtitles stream. - If the subtitles codecpar has set a size, use it. Otherwise use the - maximum dimensions of the video streams in the same file. */ w = ifilter->width; h = ifilter->height; + if (!(w && h)) { - for (i = 0; i < avf->nb_streams; i++) { - if (avf->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) { - w = FFMAX(w, avf->streams[i]->codecpar->width); - h = FFMAX(h, avf->streams[i]->codecpar->height); - } - } - if (!(w && h)) { - w = FFMAX(w, 720); - h = FFMAX(h, 576); - } - av_log(avf, AV_LOG_INFO, "sub2video: using %dx%d canvas\n", w, h); + w = ist->dec_ctx->width; + h = ist->dec_ctx->height; } - ist->sub2video.w = ifilter->width = w; - ist->sub2video.h = ifilter->height = h; - ifilter->width = ist->dec_ctx->width ? ist->dec_ctx->width : ist->sub2video.w; - ifilter->height = ist->dec_ctx->height ? ist->dec_ctx->height : ist->sub2video.h; + if (!(w && h) && ist->dec_ctx->subtitle_header) { + ASSSplitContext *ass_ctx = ff_ass_split((char *)ist->dec_ctx->subtitle_header); + ASS *ass = (ASS *)ass_ctx; + w = ass->script_info.play_res_x; + h = ass->script_info.play_res_y; + ff_ass_split_free(ass_ctx); + } - /* rectangles are AV_PIX_FMT_PAL8, but we have no guarantee that the - palettes for all rectangles are identical or compatible */ - ifilter->format = AV_PIX_FMT_RGB32; + ist->subtitle_heartbeat.w = w; + ist->subtitle_heartbeat.h = h; + av_log(ifilter, AV_LOG_INFO, "subtitle input filter: decoding size %dx%d\n", ist->subtitle_heartbeat.w, ist->subtitle_heartbeat.h); - ist->sub2video.frame = av_frame_alloc(); - if (!ist->sub2video.frame) - return AVERROR(ENOMEM); - ist->sub2video.last_pts = INT64_MIN; - ist->sub2video.end_pts = INT64_MIN; + ifilter->width = w; + ifilter->height = h; + + ist->subtitle_heartbeat.last_pts = INT64_MIN; + + snprintf(name, sizeof(name), "graph %d subtitle input from stream %d:%d", fg->index, + ist->file_index, ist->st->index); + + + av_bprint_init(&args, 0, AV_BPRINT_SIZE_AUTOMATIC); + av_bprintf(&args, + "subtitle_type=%d:time_base=%d/%d:", + ifilter->format, + ist->st->time_base.num, ist->st->time_base.den); + if ((ret = avfilter_graph_create_filter(&ifilter->filter, buffer_filt, name, + args.str, NULL, fg->graph)) < 0) + goto fail; - /* sub2video structure has been (re-)initialized. - Mark it as such so that the system will be - initialized with the first received heartbeat. */ - ist->sub2video.initialize = 1; + par->hw_frames_ctx = ifilter->hw_frames_ctx; + par->format = ifilter->format; + par->width = ifilter->width; + par->height = ifilter->height; + + ret = av_buffersrc_parameters_set(ifilter->filter, par); + if (ret < 0) + goto fail; + av_freep(&par); + last_filter = ifilter->filter; + + if (in->filter_ctx->input_pads[in->pad_idx].type == AVMEDIA_TYPE_VIDEO) { + ret = insert_filter(&last_filter, &pad_idx, "graphicsub2video", NULL); + if (ret < 0) + return ret; + } + + if ((ret = avfilter_link(last_filter, 0, in->filter_ctx, in->pad_idx)) < 0) + return ret; return 0; +fail: + av_freep(&par); + + return ret; } static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, @@ -709,8 +797,15 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, char name[255]; int ret, pad_idx = 0; int64_t tsoffset = 0; - AVBufferSrcParameters *par = av_buffersrc_parameters_alloc(); + AVBufferSrcParameters *par; + if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_SUBTITLE) { + // Automatically insert conversion filter to retain compatibility + // with sub2video command lines + return configure_input_subtitle_filter(fg, ifilter, in); + } + + par = av_buffersrc_parameters_alloc(); if (!par) return AVERROR(ENOMEM); memset(par, 0, sizeof(*par)); @@ -725,12 +820,6 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, if (!fr.num) fr = av_guess_frame_rate(input_files[ist->file_index]->ctx, ist->st, NULL); - if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_SUBTITLE) { - ret = sub2video_prepare(ist, ifilter); - if (ret < 0) - goto fail; - } - sar = ifilter->sample_aspect_ratio; if(!sar.den) sar = (AVRational){0,1}; @@ -742,7 +831,7 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, tb.num, tb.den, sar.num, sar.den); if (fr.num && fr.den) av_bprintf(&args, ":frame_rate=%d/%d", fr.num, fr.den); - snprintf(name, sizeof(name), "graph %d input from stream %d:%d", fg->index, + snprintf(name, sizeof(name), "graph %d video input from stream %d:%d", fg->index, ist->file_index, ist->st->index); @@ -938,6 +1027,7 @@ static int configure_input_filter(FilterGraph *fg, InputFilter *ifilter, switch (avfilter_pad_get_type(in->filter_ctx->input_pads, in->pad_idx)) { case AVMEDIA_TYPE_VIDEO: return configure_input_video_filter(fg, ifilter, in); case AVMEDIA_TYPE_AUDIO: return configure_input_audio_filter(fg, ifilter, in); + case AVMEDIA_TYPE_SUBTITLE: return configure_input_subtitle_filter(fg, ifilter, in); default: av_assert0(0); return 0; } } @@ -1105,19 +1195,6 @@ int configure_filtergraph(FilterGraph *fg) } } - /* process queued up subtitle packets */ - for (i = 0; i < fg->nb_inputs; i++) { - InputStream *ist = fg->inputs[i]->ist; - if (ist->sub2video.sub_queue && ist->sub2video.frame) { - while (av_fifo_size(ist->sub2video.sub_queue)) { - AVSubtitle tmp; - av_fifo_generic_read(ist->sub2video.sub_queue, &tmp, sizeof(tmp), NULL); - sub2video_update(ist, INT64_MIN, &tmp); - avsubtitle_free(&tmp); - } - } - } - return 0; fail: @@ -1138,6 +1215,7 @@ int ifilter_parameters_from_frame(InputFilter *ifilter, const AVFrame *frame) ifilter->sample_rate = frame->sample_rate; ifilter->channels = frame->channels; ifilter->channel_layout = frame->channel_layout; + ifilter->type = frame->type; if (frame->hw_frames_ctx) { ifilter->hw_frames_ctx = av_buffer_ref(frame->hw_frames_ctx); diff --git a/fftools/ffmpeg_hw.c b/fftools/ffmpeg_hw.c index 41aaf776d7..f8b49fd5ac 100644 --- a/fftools/ffmpeg_hw.c +++ b/fftools/ffmpeg_hw.c @@ -449,7 +449,7 @@ int hw_device_setup_for_encode(OutputStream *ost) AVBufferRef *frames_ref = NULL; int i; - if (ost->filter) { + if (ost->filter && ost->filter->filter) { frames_ref = av_buffersink_get_hw_frames_ctx(ost->filter->filter); if (frames_ref && ((AVHWFramesContext*)frames_ref->data)->format == diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index 6ca28cf974..21b5228bb2 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -2149,8 +2149,9 @@ static void init_output_filter(OutputFilter *ofilter, OptionsContext *o, switch (ofilter->type) { case AVMEDIA_TYPE_VIDEO: ost = new_video_stream(o, oc, -1); break; case AVMEDIA_TYPE_AUDIO: ost = new_audio_stream(o, oc, -1); break; + case AVMEDIA_TYPE_SUBTITLE: ost = new_subtitle_stream(o, oc, -1); break; default: - av_log(NULL, AV_LOG_FATAL, "Only video and audio filters are supported " + av_log(NULL, AV_LOG_FATAL, "Only video, audio and subtitle filters are supported " "currently.\n"); exit_program(1); } From patchwork Sat Sep 11 06:02:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30113 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp2168100iov; Fri, 10 Sep 2021 23:04:56 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy9xK13OfXNHd/gKRfd0LIptmWaqdelvXEkfLRvqqI4DaWYZr5lWKNFnjjom+CU/jhF8Znd X-Received: by 2002:a17:906:7714:: with SMTP id q20mr1393422ejm.551.1631340296757; Fri, 10 Sep 2021 23:04:56 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id n15si1179886ejd.124.2021.09.10.23.04.56; Fri, 10 Sep 2021 23:04:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=NaDD2qFS; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8A2F168A733; Sat, 11 Sep 2021 09:03:09 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11olkn2078.outbound.protection.outlook.com [40.92.19.78]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 13E9468A5E7 for ; Sat, 11 Sep 2021 09:03:03 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Lz1Akrh6tb8qEcmzyJvc4Gsz6aR6zakKD0GmFPOdEzyV8a3cfhV551n4pEK44v9DLqi9h2IvTziAhSJ4qxnAZYUwVftmp+hiEGc6BnliexoWCpTebUfuyrkf/9rfw49G4Fjbhp4e3bRC9aAjRtvGo3+wN45OKzNKbpANHKQK+S7U2XF0qHS4nQUYXpanhZo7Js3ACpsmEVv5hXujbOU1FxJ5Ei7mDlX/jw+B0cPdTsFKdVJdICn3NxpOepox8McWmovsfxkf/hqJOiB5rTLfcpzRLCYD79d4NW3ce2HJCFyUyttwkfhkiPKqGi4gAECNjVYVJxDrCIBwA/+YoL8qEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=F7KiL0JgeBQiLvU1j+IFUBxLV16yq73XsRMrhvmcWuA=; b=cwNUTg9oRNBayjQa5urHlyHbe6Bssh/zKVyAxWB7QHhT8XimktMXEEVefYodgJEJa8eHDy5dMcrZqIO8fqTcUPO8QMOx/q1r+rxBkYsshtQGo8HesoxiLxcoOdGUmOmpTzWK8dpZXvs2gYz1orVy1H1N6tPKtKgi8SKT+YjR/j+dm5lbF8BJM6Kp+2cGeRlZlrZ97MaDmuZKsVXJjf6MeFC3Ey2sGarMoqLjwAK/ODKDRaIA2/qaoVO70nU6MgZ2aPXWAXAb0gnDTgBO1nPUgHYdLF513ehfy+yffem23Wdt+Q1fXkWjU2yfiibssvPzOPxG/tmtRRSW4V4jQ/J+mQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F7KiL0JgeBQiLvU1j+IFUBxLV16yq73XsRMrhvmcWuA=; b=NaDD2qFSGbnHnw8FPDhQzRYXK4a9dijQ5I4XKmpbuUHxRjhxG3GUCP3YoXZLsH01sOyFjBKITLoTHDjyhWj+Xlu6Pk+FF06edj8LNM4K+9QpeyCCeYc8x2pqamGQnnbIriW2FCp1zjPPbmlGxXVXvb3d1MbHZfEsywCBGL71dL4281Ha4ZYrriiG4mi2XKHjS9RZL6JJNRp5M1kZAxKjBRxKpK40uGulkNwtfshIvmJp+kIYnkzFqW7ig5PCJuy2iit/Lg7HQa0jVI+ikohTHr1IjY8QbyRxephLLbXRKoc8zOghloRbCjM4bSace3g+yt1vFzdZ3LwRGmM6mjZltA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5776.namprd04.prod.outlook.com (2603:10b6:208:a2::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.16; Sat, 11 Sep 2021 06:02:56 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4500.018; Sat, 11 Sep 2021 06:02:56 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v3 10/18] avfilter/avfilter: Handle subtitle frames Thread-Index: Ademwa4RTEBxTn8FSGa0mXu3u8AkwQ== Date: Sat, 11 Sep 2021 06:02:56 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [GbCMza5yCTy6O+KR/HiQ5pGCVHbG46FsrBjhAQKoXR8=] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f0f702ca-8540-4399-9adc-08d974e9cd3e x-ms-traffictypediagnostic: MN2PR04MB5776: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: imsFPtxwfT+gjSYdmfIEzoArAmr9n5aseL0QxKuxCrDryUoAXtJqzNZZBA2SjWsFzp5wE4W2B0me1VV+j/BcSw7gUmjiwkdSAA3VY3ELCD6nBdV6Sh87iYy3QAhaymVRU4B88KeVU0h2Y5XE3/7+Xoh5yf+jXDB6JabOp3FrP6c6Sy2CxTRaDGzzc1YeAPNa7DvKjGRyyIMwVF1eb0VDI4NVtU0T2JmppnfNJMzIvOcH7qBxOcHq79hDVCsI1rw/pJ03qkg1fp+18esseqkb+6Y5t5UXnHKxTgTgLNr9U4f/e4hlJpyNc/xeDqpa9Nfa1x0xNuZUPwMLm6PXM5S8beEJtRWXJisc5OYQNTzuAx3lTEj+PguGKT++YEzWBJEgBMHDqSdf3oGEB9wtqXFznvTnHG2OnuGvn7eejRoW24rh0Pu5lDGWjAm+LpsBtJeU x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: hNv+hPiES063Dghag8+MuJyehhvYxq+cDLFh67ZEtB3fUvP1uK+YJNFbyxN9IZ0hUHMhM908afesj2CjHhLBIkzi1jDqRrfKPoQGKNXGo4lt4wNysDwSgpsnzp87tIqNg4RR27OM7bI28AI8TMYMrw== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: f0f702ca-8540-4399-9adc-08d974e9cd3e X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2021 06:02:56.8940 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5776 Subject: [FFmpeg-devel] [PATCH v3 10/18] avfilter/avfilter: Handle subtitle frames X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: QZbAriM3VKfu Signed-off-by: softworkz --- libavfilter/avfilter.c | 8 +++++--- libavfilter/avfiltergraph.c | 5 +++++ libavfilter/formats.c | 14 ++++++++++++++ libavfilter/formats.h | 3 +++ 4 files changed, 27 insertions(+), 3 deletions(-) diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index a04f8ed62f..961e55e7b4 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -56,7 +56,8 @@ void ff_tlog_ref(void *ctx, AVFrame *ref, int end) ref->linesize[0], ref->linesize[1], ref->linesize[2], ref->linesize[3], ref->pts, ref->pkt_pos); - if (ref->width) { + switch(ref->type) { + case AVMEDIA_TYPE_VIDEO: ff_tlog(ctx, " a:%d/%d s:%dx%d i:%c iskey:%d type:%c", ref->sample_aspect_ratio.num, ref->sample_aspect_ratio.den, ref->width, ref->height, @@ -64,12 +65,13 @@ void ff_tlog_ref(void *ctx, AVFrame *ref, int end) ref->top_field_first ? 'T' : 'B', /* Top / Bottom */ ref->key_frame, av_get_picture_type_char(ref->pict_type)); - } - if (ref->nb_samples) { + break; + case AVMEDIA_TYPE_AUDIO: ff_tlog(ctx, " cl:%"PRId64"d n:%d r:%d", ref->channel_layout, ref->nb_samples, ref->sample_rate); + break; } ff_tlog(ctx, "]%s", end ? "\n" : ""); diff --git a/libavfilter/avfiltergraph.c b/libavfilter/avfiltergraph.c index 45b028cd9c..7a5a4ea419 100644 --- a/libavfilter/avfiltergraph.c +++ b/libavfilter/avfiltergraph.c @@ -314,6 +314,8 @@ static int filter_link_check_formats(void *log, AVFilterLink *link, AVFilterForm return ret; break; + case AVMEDIA_TYPE_SUBTITLE: + return 0; default: av_assert0(!"reached"); } @@ -444,6 +446,9 @@ static int query_formats(AVFilterGraph *graph, void *log_ctx) if (!link) continue; + if (link->type == AVMEDIA_TYPE_SUBTITLE) + continue; + neg = ff_filter_get_negotiation(link); av_assert0(neg); for (neg_step = 1; neg_step < neg->nb_mergers; neg_step++) { diff --git a/libavfilter/formats.c b/libavfilter/formats.c index 1bf7d36195..704774d763 100644 --- a/libavfilter/formats.c +++ b/libavfilter/formats.c @@ -19,6 +19,7 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include "libavcodec/avcodec.h" #include "libavutil/avassert.h" #include "libavutil/channel_layout.h" #include "libavutil/common.h" @@ -430,6 +431,12 @@ int ff_add_channel_layout(AVFilterChannelLayouts **l, uint64_t channel_layout) return 0; } +int ff_add_subtitle_type(AVFilterFormats **avff, int64_t fmt) +{ + ADD_FORMAT(avff, fmt, ff_formats_unref, int, formats, nb_formats); + return 0; +} + AVFilterFormats *ff_all_formats(enum AVMediaType type) { AVFilterFormats *ret = NULL; @@ -447,6 +454,13 @@ AVFilterFormats *ff_all_formats(enum AVMediaType type) return NULL; fmt++; } + } else if (type == AVMEDIA_TYPE_SUBTITLE) { + if (ff_add_subtitle_type(&ret, AV_SUBTITLE_FMT_BITMAP) < 0) + return NULL; + if (ff_add_subtitle_type(&ret, AV_SUBTITLE_FMT_ASS) < 0) + return NULL; + if (ff_add_subtitle_type(&ret, AV_SUBTITLE_FMT_TEXT) < 0) + return NULL; } return ret; diff --git a/libavfilter/formats.h b/libavfilter/formats.h index 471cb42bc4..25ae511fc1 100644 --- a/libavfilter/formats.h +++ b/libavfilter/formats.h @@ -180,6 +180,9 @@ int ff_set_common_formats_from_list(AVFilterContext *ctx, const int *fmts); av_warn_unused_result int ff_add_channel_layout(AVFilterChannelLayouts **l, uint64_t channel_layout); +av_warn_unused_result +int ff_add_subtitle_type(AVFilterFormats **avff, int64_t fmt); + /** * Add *ref as a new reference to f. */ From patchwork Sat Sep 11 06:03:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30110 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp2168487iov; Fri, 10 Sep 2021 23:05:35 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwC2sjc7rr4hu8iWaqlPm6sf5fwR5e1w6kitMzOwwFsGgRjSmDpjzzhaVaaOgQJ+IUZP7xG X-Received: by 2002:aa7:dd48:: with SMTP id o8mr1767886edw.44.1631340334997; Fri, 10 Sep 2021 23:05:34 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id v21si1146687ejg.118.2021.09.10.23.05.34; Fri, 10 Sep 2021 23:05:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=RKzZ+X23; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 88E8768A773; Sat, 11 Sep 2021 09:03:12 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11olkn2048.outbound.protection.outlook.com [40.92.19.48]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id DF3A868A5D1 for ; Sat, 11 Sep 2021 09:03:07 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YCjD6RoOkA/fBkkWlAl6fDszv7yh1+xts6YAjXI3+5QOxpDtbQM5duPISCugACQ63/GXIIwmPUQ6qipP5or+aCL+ymDIIS74bfMbT3fuJlrBHzK9EyNYjHl+XgkFqOx+pRsdF1RopwdPXxbs2f7aleh4s/u6q9hbnXNBP2XywjY7M4CW9unQjbW4GMqVSSMcsN8JtPatPcOjpBc8lGgu/3HSddClYXJtL9IP3V+fMC0JheLqwpwptDS1Gc1LUDvBBWuNYVNFStcCHYPsMIX3Z1ff2t0RBvMagZGnhOnRHnwcDm2wetzlbejLEjS2Vqs25l5VP7YLIlrZbeMcwTjlhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=4zHD60lP+uhDH5A1DkA3paa/Tv/pa3ic0kwAnIm6RCw=; b=nLUr2DB8Ger5aI0wOoHcHVUkP3bDd9dktffEAQPsqct6t2JTBRT+khh6t+hKOAxhOYxzoRD3oNhuKgI1GYPl79V6ZZFhiCDBleDouEpvjCkkAc0ilDbxOVwg0z2DqyeYSQfs+PxgFlZ3Vr3kyixJbUEz01luRcIeLiMKHbTdyCh//JePoaN23YRi4VgDNKgZ7Ts4CJp44KUVwOKM+XQG++y7J90VgEwOadojcrIhQE9g6hKt12GPAJthKCFqt1niwV4i8h02vj+YGB+HCfuKV66GW7ajnAI4DsJptIz7tC3ko5K/QJ8hT87VOiKEYVNKUE2Im1DjbifzMyUEuCQHUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4zHD60lP+uhDH5A1DkA3paa/Tv/pa3ic0kwAnIm6RCw=; b=RKzZ+X23tiKCSa9k9Ng1I+shl+y1yYjVkooht7EUIu8A1Mp1izW4Ep3M8FnG3HvnF1+8z/JSNPcryia5EdRSGYRNBkqr0gjWZteIZVxbo5mzNOq4ws+y953NDaHvase2w//n/fjH//e/VLPzBsvYOsatshPhZoGhdD2Kp0opgl0Qz++0U+PAOrv/X27mgP2nQYLp+G5Wb2IZUBQ5sYWk1m+qlP7hoickJRrPNNIqvuYEnx4V90Dxp8juWvg+/41G5HH4/XuS3sbZfGbIWi9g7AMF8z/gnaoOBinpLJ5odlH9QFfxJRKHekneZobR3fOzlkIsTvXXTsnMnTC2yWKDMw== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5776.namprd04.prod.outlook.com (2603:10b6:208:a2::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.16; Sat, 11 Sep 2021 06:03:04 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4500.018; Sat, 11 Sep 2021 06:03:04 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v3 13/18] avfilter/overlay_graphicsubs: Add overlay_graphicsubs and graphicsub2video filters Thread-Index: AdemwZINXsFjv2HgRG2Wt8AamH7ODg== Date: Sat, 11 Sep 2021 06:03:04 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [FZr/O71Sj3qK6MxUv+vN93cqPV4kU0Bo79rCgjD2O1s=] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 612bd72a-20ba-4558-e8e5-08d974e9d1f2 x-ms-traffictypediagnostic: MN2PR04MB5776: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 3nTrtXYJPcxNrCF7yl37tXCmeo3MU53J3PYgDfCQ2BvjrQxZ4n2CcpQVDefsV9apvcI7o8UAxRU4DIpd19+cNLVZ8ppxMtRVcQh5NnN11uq3mPGYHWiLVz34D+tpfl1b0SQm8+EVkXFwc8fc7O1pHJIoBuIt7V/u/e+Avt8dkY3t5nAxVrUZ3Fbu34UmVRCjJgnNCJLXtKebrMQviiuRveSjeFsQgcvtH6g68654bJBO+OcEU9H+GW8o/thAMSKjyFNF7Ov9KwMI3nDK1e0RNf0WjJDNRwiSsIyYBIizLYUJrKhagZSA84C8llleIeXx2jKMKTNlR4yvNfUJNVF2hAdvXN5YGtCRO4wrwXdnVmbw8mw30pLp2pmjto7S7fmkTXlDMwSsHmfhmswe+CdcfuadcbFCH1fc8ZgCtT3oSCHXmq65VuNT6EFJBx+Wd301 x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: sI+OSdqKuBGx9pSsJRBZdPBzGw/c+rimekCbz8l8rUYTw6TkRYaJ7hlOlZ4BkMLCFmAR7DPPcTAJivVSwz1Dpjb/EeEhHYKtI0sJfzO5Eocw2cflmMvwmP2ty8IPViCunTAlESTD/a8JChszK7seww== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 612bd72a-20ba-4558-e8e5-08d974e9d1f2 X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2021 06:03:04.7535 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5776 Subject: [FFmpeg-devel] [PATCH v3 13/18] avfilter/overlay_graphicsubs: Add overlay_graphicsubs and graphicsub2video filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: aeS6WHEuhnly Signed-off-by: softworkz --- libavfilter/Makefile | 2 + libavfilter/allfilters.c | 2 + libavfilter/vf_overlay_graphicsubs.c | 740 +++++++++++++++++++++++++++ 3 files changed, 744 insertions(+) create mode 100644 libavfilter/vf_overlay_graphicsubs.c diff --git a/libavfilter/Makefile b/libavfilter/Makefile index a510dd12b4..f9b33d4ead 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -289,6 +289,7 @@ OBJS-$(CONFIG_FSPP_FILTER) += vf_fspp.o qp_table.o OBJS-$(CONFIG_GBLUR_FILTER) += vf_gblur.o OBJS-$(CONFIG_GEQ_FILTER) += vf_geq.o OBJS-$(CONFIG_GRADFUN_FILTER) += vf_gradfun.o +OBJS-$(CONFIG_GRAPHICSUB2VIDEO_FILTER) += vf_overlay_graphicsubs.o framesync.o OBJS-$(CONFIG_GRAPHMONITOR_FILTER) += f_graphmonitor.o OBJS-$(CONFIG_GRAYWORLD_FILTER) += vf_grayworld.o OBJS-$(CONFIG_GREYEDGE_FILTER) += vf_colorconstancy.o @@ -362,6 +363,7 @@ OBJS-$(CONFIG_OVERLAY_CUDA_FILTER) += vf_overlay_cuda.o framesync.o vf OBJS-$(CONFIG_OVERLAY_OPENCL_FILTER) += vf_overlay_opencl.o opencl.o \ opencl/overlay.o framesync.o OBJS-$(CONFIG_OVERLAY_QSV_FILTER) += vf_overlay_qsv.o framesync.o +OBJS-$(CONFIG_OVERLAY_GRAPHICSUBS_FILTER) += vf_overlay_graphicsubs.o framesync.o OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) += vf_overlay_vulkan.o vulkan.o OBJS-$(CONFIG_OWDENOISE_FILTER) += vf_owdenoise.o OBJS-$(CONFIG_PAD_FILTER) += vf_pad.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index b27ef6f027..ca983db5b4 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -344,6 +344,7 @@ extern const AVFilter ff_vf_oscilloscope; extern const AVFilter ff_vf_overlay; extern const AVFilter ff_vf_overlay_opencl; extern const AVFilter ff_vf_overlay_qsv; +extern const AVFilter ff_vf_overlay_graphicsubs; extern const AVFilter ff_vf_overlay_vulkan; extern const AVFilter ff_vf_overlay_cuda; extern const AVFilter ff_vf_owdenoise; @@ -522,6 +523,7 @@ extern const AVFilter ff_avf_showvolume; extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; +extern const AVFilter ff_svf_graphicsub2video; /* multimedia sources */ extern const AVFilter ff_avsrc_amovie; diff --git a/libavfilter/vf_overlay_graphicsubs.c b/libavfilter/vf_overlay_graphicsubs.c new file mode 100644 index 0000000000..85af58a9d4 --- /dev/null +++ b/libavfilter/vf_overlay_graphicsubs.c @@ -0,0 +1,740 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * overlay graphical subtitles on top of a video frame + */ + +#include "avfilter.h" +#include "formats.h" +#include "libavutil/common.h" +#include "libavutil/eval.h" +#include "libavutil/avstring.h" +#include "libavutil/pixdesc.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "internal.h" +#include "drawutils.h" +#include "framesync.h" + +#include "libavcodec/avcodec.h" + +enum var_name { + VAR_MAIN_W, VAR_MW, + VAR_MAIN_H, VAR_MH, + VAR_OVERLAY_W, VAR_OW, + VAR_OVERLAY_H, VAR_OH, + VAR_HSUB, + VAR_VSUB, + VAR_X, + VAR_Y, + VAR_N, + VAR_POS, + VAR_T, + VAR_VARS_NB +}; + +typedef struct OverlaySubsContext { + const AVClass *class; + int x, y; ///< position of overlaid picture + int w, h; + AVFrame *outpicref; + + int main_is_packed_rgb; + uint8_t main_rgba_map[4]; + int main_has_alpha; + uint8_t overlay_rgba_map[4]; + int eval_mode; ///< EvalMode + + FFFrameSync fs; + + int main_pix_step[4]; ///< steps per pixel for each plane of the main output + int hsub, vsub; ///< chroma subsampling values + const AVPixFmtDescriptor *main_desc; ///< format descriptor for main input + + double var_values[VAR_VARS_NB]; + char *x_expr, *y_expr; + + AVExpr *x_pexpr, *y_pexpr; +} OverlaySubsContext; + +static const char *const var_names[] = { + "main_w", "W", ///< width of the main video + "main_h", "H", ///< height of the main video + "overlay_w", "w", ///< width of the overlay video + "overlay_h", "h", ///< height of the overlay video + "hsub", + "vsub", + "x", + "y", + "n", ///< number of frame + "pos", ///< position in the file + "t", ///< timestamp expressed in seconds + NULL +}; + +#define MAIN 0 +#define OVERLAY 1 + +#define R 0 +#define G 1 +#define B 2 +#define A 3 + +#define Y 0 +#define U 1 +#define V 2 + +enum EvalMode { + EVAL_MODE_INIT, + EVAL_MODE_FRAME, + EVAL_MODE_NB +}; + +static av_cold void overlay_graphicsubs_uninit(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + ff_framesync_uninit(&s->fs); + av_expr_free(s->x_pexpr); s->x_pexpr = NULL; + av_expr_free(s->y_pexpr); s->y_pexpr = NULL; +} + +static inline int normalize_xy(double d, int chroma_sub) +{ + if (isnan(d)) + return INT_MAX; + return (int)d & ~((1 << chroma_sub) - 1); +} + +static void eval_expr(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->var_values[VAR_Y] = av_expr_eval(s->y_pexpr, s->var_values, NULL); + /* It is necessary if x is expressed from y */ + s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->x = normalize_xy(s->var_values[VAR_X], s->hsub); + s->y = normalize_xy(s->var_values[VAR_Y], s->vsub); +} + +static int set_expr(AVExpr **pexpr, const char *expr, const char *option, void *log_ctx) +{ + int ret; + AVExpr *old = NULL; + + if (*pexpr) + old = *pexpr; + ret = av_expr_parse(pexpr, expr, var_names, + NULL, NULL, NULL, NULL, 0, log_ctx); + if (ret < 0) { + av_log(log_ctx, AV_LOG_ERROR, + "Error when evaluating the expression '%s' for %s\n", + expr, option); + *pexpr = old; + return ret; + } + + av_expr_free(old); + return 0; +} + +static int overlay_graphicsubs_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink0 = ctx->inputs[0]; + AVFilterLink *inlink1 = ctx->inputs[1]; + AVFilterLink *outlink = ctx->outputs[0]; + int ret; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_BITMAP, AV_SUBTITLE_FMT_NONE }; + static const enum AVPixelFormat supported_pix_fmts[] = { + AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV444P, + AV_PIX_FMT_ARGB, AV_PIX_FMT_RGBA, + AV_PIX_FMT_ABGR, AV_PIX_FMT_BGRA, + AV_PIX_FMT_RGB24, AV_PIX_FMT_BGR24, + AV_PIX_FMT_NONE + }; + + /* set input0 video formats */ + formats = ff_make_format_list(supported_pix_fmts); + if ((ret = ff_formats_ref(formats, &inlink0->outcfg.formats)) < 0) + return ret; + + /* set input1 subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink1->outcfg.formats)) < 0) + return ret; + + /* set output0 video formats */ + formats = ff_make_format_list(supported_pix_fmts); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + OverlaySubsContext *s = ctx->priv; + int ret; + + if ((ret = ff_framesync_init_dualinput(&s->fs, ctx)) < 0) + return ret; + + outlink->w = ctx->inputs[MAIN]->w; + outlink->h = ctx->inputs[MAIN]->h; + outlink->time_base = ctx->inputs[MAIN]->time_base; + + return ff_framesync_configure(&s->fs); +} + +// divide by 255 and round to nearest +// apply a fast variant: (X+127)/255 = ((X+127)*257+257)>>16 = ((X+128)*257)>>16 +#define FAST_DIV255(x) ((((x) + 128) * 257) >> 16) + +// calculate the non-pre-multiplied alpha, applying the general equation: +// alpha = alpha_overlay / ( (alpha_main + alpha_overlay) - (alpha_main * alpha_overlay) ) +// (((x) << 16) - ((x) << 9) + (x)) is a faster version of: 255 * 255 * x +// ((((x) + (y)) << 8) - ((x) + (y)) - (y) * (x)) is a faster version of: 255 * (x + y) +#define UNPREMULTIPLY_ALPHA(x, y) ((((x) << 16) - ((x) << 9) + (x)) / ((((x) + (y)) << 8) - ((x) + (y)) - (y) * (x))) + +/** + * Blend image in src to destination buffer dst at position (x, y). + */ +static av_always_inline void blend_packed_rgb(const AVFilterContext *ctx, + const AVFrame *dst, const AVSubtitleRect *src, + int x, int y, + int is_straight) +{ + OverlaySubsContext *s = ctx->priv; + int i, imax, j, jmax; + const int src_w = src->w; + const int src_h = src->h; + const int dst_w = dst->width; + const int dst_h = dst->height; + uint8_t alpha; ///< the amount of overlay to blend on to main + const int dr = s->main_rgba_map[R]; + const int dg = s->main_rgba_map[G]; + const int db = s->main_rgba_map[B]; + const int da = s->main_rgba_map[A]; + const int dstep = s->main_pix_step[0]; + const int sr = s->overlay_rgba_map[R]; + const int sg = s->overlay_rgba_map[G]; + const int sb = s->overlay_rgba_map[B]; + const int sa = s->overlay_rgba_map[A]; + uint32_t *pal = (uint32_t *)src->data[1]; + int slice_start, slice_end; + uint8_t *S, *sp, *d, *dp; + + i = FFMAX(-y, 0); + imax = FFMIN3(-y + dst_h, FFMIN(src_h, dst_h), y + src_h); + + slice_start = i; + slice_end = i + imax; + + sp = src->data[0] + slice_start * src->linesize[0]; + dp = dst->data[0] + (slice_start + y) * dst->linesize[0]; + + for (i = slice_start; i < slice_end; i++) { + j = FFMAX(-x, 0); + S = sp + j; + d = dp + ((x + j) * dstep); + + for (jmax = FFMIN(-x + dst_w, src_w); j < jmax; j++) { + uint32_t val = pal[*S]; + const uint8_t *sval = (uint8_t *)&val; + alpha = sval[sa]; + + // if the main channel has an alpha channel, alpha has to be calculated + // to create an un-premultiplied (straight) alpha value + if (s->main_has_alpha && alpha != 0 && alpha != 255) { + const uint8_t alpha_d = d[da]; + alpha = UNPREMULTIPLY_ALPHA(alpha, alpha_d); + } + + switch (alpha) { + case 0: + break; + case 255: + d[dr] = sval[sr]; + d[dg] = sval[sg]; + d[db] = sval[sb]; + break; + default: + // main_value = main_value * (1 - alpha) + overlay_value * alpha + // since alpha is in the range 0-255, the result must divided by 255 + d[dr] = is_straight ? FAST_DIV255(d[dr] * (255 - alpha) + sval[sr] * alpha) : + FFMIN(FAST_DIV255(d[dr] * (255 - alpha)) + sval[sr], 255); + d[dg] = is_straight ? FAST_DIV255(d[dg] * (255 - alpha) + sval[sg] * alpha) : + FFMIN(FAST_DIV255(d[dg] * (255 - alpha)) + sval[sg], 255); + d[db] = is_straight ? FAST_DIV255(d[db] * (255 - alpha) + sval[sb] * alpha) : + FFMIN(FAST_DIV255(d[db] * (255 - alpha)) + sval[sb], 255); + } + + if (s->main_has_alpha) { + switch (alpha) { + case 0: + break; + case 255: + d[da] = sval[sa]; + break; + default: + // apply alpha compositing: main_alpha += (1-main_alpha) * overlay_alpha + d[da] += FAST_DIV255((255 - d[da]) * S[sa]); + } + } + d += dstep; + S += 1; + } + dp += dst->linesize[0]; + sp += src->linesize[0]; + } +} + +static av_always_inline void blend_plane_8_8bits(const AVFilterContext *ctx, const AVFrame *dst, const AVSubtitleRect *src, + const uint32_t *yuv_pal, int src_w, int src_h, int dst_w, int dst_h, int plane, int hsub, int vsub, + int x, int y, int dst_plane, int dst_offset, int dst_step) +{ + const int src_wp = AV_CEIL_RSHIFT(src_w, hsub); + const int src_hp = AV_CEIL_RSHIFT(src_h, vsub); + const int dst_wp = AV_CEIL_RSHIFT(dst_w, hsub); + const int dst_hp = AV_CEIL_RSHIFT(dst_h, vsub); + const int yp = y >> vsub; + const int xp = x >> hsub; + uint8_t *s, *sp, *d, *dp, *dap; + int imax, i, j, jmax; + int slice_start, slice_end; + + i = FFMAX(-yp, 0); \ + imax = FFMIN3(-yp + dst_hp, FFMIN(src_hp, dst_hp), yp + src_hp); \ + + slice_start = i; + slice_end = i + imax; + + sp = src->data[0] + (slice_start << vsub) * src->linesize[0]; + dp = dst->data[dst_plane] + (yp + slice_start) * dst->linesize[dst_plane] + dst_offset; + + dap = dst->data[3] + ((yp + slice_start) << vsub) * dst->linesize[3]; + + for (i = slice_start; i < slice_end; i++) { + j = FFMAX(-xp, 0); + d = dp + (xp + j) * dst_step; + s = sp + (j << hsub); + jmax = FFMIN(-xp + dst_wp, src_wp); + + for (; j < jmax; j++) { + uint32_t val = yuv_pal[*s]; + const uint8_t *sval = (uint8_t *)&val; + const int alpha = sval[3]; + const int max = 255, mid = 128; + const int d_int = *d; + const int sval_int = sval[plane]; + + switch (alpha) { + case 0: + break; + case 255: + *d = sval[plane]; + break; + default: + if (plane > 0) + *d = av_clip(FAST_DIV255((d_int - mid) * (max - alpha) + (sval_int - mid) * alpha) , -mid, mid) + mid; + else + *d = FAST_DIV255(d_int * (max - alpha) + sval_int * alpha); + break; + } + + d += dst_step; + s += 1 << hsub; + } + dp += dst->linesize[dst_plane]; + sp += (1 << vsub) * src->linesize[0]; + dap += (1 << vsub) * dst->linesize[3]; + } +} + +#define RGB2Y(r, g, b) (uint8_t)(((66 * (r) + 129 * (g) + 25 * (b) + 128) >> 8) + 16) +#define RGB2U(r, g, b) (uint8_t)(((-38 * (r) - 74 * (g) + 112 * (b) + 128) >> 8) + 128) +#define RGB2V(r, g, b) (uint8_t)(((112 * (r) - 94 * (g) - 18 * (b) + 128) >> 8) + 128) +/* Converts R8 G8 B8 color to YUV. */ +static av_always_inline void rgb_2_yuv(uint8_t r, uint8_t g, uint8_t b, uint8_t* y, uint8_t* u, uint8_t* v) +{ + *y = RGB2Y((int)r, (int)g, (int)b); + *u = RGB2U((int)r, (int)g, (int)b); + *v = RGB2V((int)r, (int)g, (int)b); +} + + +static av_always_inline void blend_yuv_8_8bits(AVFilterContext *ctx, AVFrame *dst, const AVSubtitleRect *src, int hsub, int vsub, int x, int y) +{ + OverlaySubsContext *s = ctx->priv; + const int src_w = src->w; + const int src_h = src->h; + const int dst_w = dst->width; + const int dst_h = dst->height; + const int sr = s->overlay_rgba_map[R]; + const int sg = s->overlay_rgba_map[G]; + const int sb = s->overlay_rgba_map[B]; + const int sa = s->overlay_rgba_map[A]; + uint32_t *pal = (uint32_t *)src->data[1]; + uint32_t yuvpal[256]; + + for (int i = 0; i < 256; ++i) { + const uint8_t *rgba = (uint8_t *)&pal[i]; + uint8_t *yuva = (uint8_t *)&yuvpal[i]; + rgb_2_yuv(rgba[sr], rgba[sg], rgba[sb], &yuva[Y], &yuva[U], &yuva[V]); + yuva[3] = rgba[sa]; + } + + blend_plane_8_8bits(ctx, dst, src, yuvpal, src_w, src_h, dst_w, dst_h, Y, 0, 0, x, y, s->main_desc->comp[Y].plane, s->main_desc->comp[Y].offset, s->main_desc->comp[Y].step); + blend_plane_8_8bits(ctx, dst, src, yuvpal, src_w, src_h, dst_w, dst_h, U, hsub, vsub, x, y, s->main_desc->comp[U].plane, s->main_desc->comp[U].offset, s->main_desc->comp[U].step); + blend_plane_8_8bits(ctx, dst, src, yuvpal, src_w, src_h, dst_w, dst_h, V, hsub, vsub, x, y, s->main_desc->comp[V].plane, s->main_desc->comp[V].offset, s->main_desc->comp[V].step); +} + +static int config_input_main(AVFilterLink *inlink) +{ + int ret; + AVFilterContext *ctx = inlink->dst; + OverlaySubsContext *s = inlink->dst->priv; + const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(inlink->format); + + av_image_fill_max_pixsteps(s->main_pix_step, NULL, pix_desc); + ff_fill_rgba_map(s->overlay_rgba_map, AV_PIX_FMT_RGB32); // it's actually AV_PIX_FMT_PAL8); + + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + + s->main_desc = pix_desc; + + s->main_is_packed_rgb = ff_fill_rgba_map(s->main_rgba_map, inlink->format) >= 0; + s->main_has_alpha = !(pix_desc->flags & AV_PIX_FMT_FLAG_ALPHA); + + /* Finish the configuration by evaluating the expressions + now when both inputs are configured. */ + s->var_values[VAR_MAIN_W ] = s->var_values[VAR_MW] = ctx->inputs[MAIN ]->w; + s->var_values[VAR_MAIN_H ] = s->var_values[VAR_MH] = ctx->inputs[MAIN ]->h; + s->var_values[VAR_OVERLAY_W] = s->var_values[VAR_OW] = ctx->inputs[OVERLAY]->w; + s->var_values[VAR_OVERLAY_H] = s->var_values[VAR_OH] = ctx->inputs[OVERLAY]->h; + s->var_values[VAR_HSUB] = 1<log2_chroma_w; + s->var_values[VAR_VSUB] = 1<log2_chroma_h; + s->var_values[VAR_X] = NAN; + s->var_values[VAR_Y] = NAN; + s->var_values[VAR_N] = 0; + s->var_values[VAR_T] = NAN; + s->var_values[VAR_POS] = NAN; + + if ((ret = set_expr(&s->x_pexpr, s->x_expr, "x", ctx)) < 0 || + (ret = set_expr(&s->y_pexpr, s->y_expr, "y", ctx)) < 0) + return ret; + + if (s->eval_mode == EVAL_MODE_INIT) { + eval_expr(ctx); + av_log(ctx, AV_LOG_VERBOSE, "x:%f xi:%d y:%f yi:%d\n", + s->var_values[VAR_X], s->x, + s->var_values[VAR_Y], s->y); + } + + av_log(ctx, AV_LOG_VERBOSE, + "main w:%d h:%d fmt:%s overlay w:%d h:%d fmt:%s\n", + ctx->inputs[MAIN]->w, ctx->inputs[MAIN]->h, + av_get_pix_fmt_name(ctx->inputs[MAIN]->format), + ctx->inputs[OVERLAY]->w, ctx->inputs[OVERLAY]->h, + av_get_pix_fmt_name(ctx->inputs[OVERLAY]->format)); + return 0; +} + +static int do_blend(FFFrameSync *fs) +{ + AVFilterContext *ctx = fs->parent; + AVFrame *mainpic, *second; + OverlaySubsContext *s = ctx->priv; + AVFilterLink *inlink = ctx->inputs[0]; + int ret; + + ret = ff_framesync_dualinput_get_writable(fs, &mainpic, &second); + if (ret < 0) + return ret; + if (!second) + return ff_filter_frame(ctx->outputs[0], mainpic); + + if (s->eval_mode == EVAL_MODE_FRAME) { + int64_t pos = mainpic->pkt_pos; + + s->var_values[VAR_N] = inlink->frame_count_out; + s->var_values[VAR_T] = mainpic->pts == AV_NOPTS_VALUE ? + NAN : mainpic->pts * av_q2d(inlink->time_base); + s->var_values[VAR_POS] = pos == -1 ? NAN : pos; + + s->var_values[VAR_OVERLAY_W] = s->var_values[VAR_OW] = second->width; + s->var_values[VAR_OVERLAY_H] = s->var_values[VAR_OH] = second->height; + s->var_values[VAR_MAIN_W ] = s->var_values[VAR_MW] = mainpic->width; + s->var_values[VAR_MAIN_H ] = s->var_values[VAR_MH] = mainpic->height; + + eval_expr(ctx); + av_log(ctx, AV_LOG_DEBUG, "n:%f t:%f pos:%f x:%f xi:%d y:%f yi:%d\n", + s->var_values[VAR_N], s->var_values[VAR_T], s->var_values[VAR_POS], + s->var_values[VAR_X], s->x, + s->var_values[VAR_Y], s->y); + } + + if (second->buf[0] && second->buf[0]->data) { + const AVSubtitle *sub = (AVSubtitle *)second->buf[0]->data; + unsigned num_rects, i; + + num_rects = sub->num_rects > 1 ? 1 : sub->num_rects; + + for (i = 0; i < num_rects; i++) { + const AVSubtitleRect *sub_rect = sub->rects[i]; + + if (sub_rect->type != AV_SUBTITLE_FMT_BITMAP) { + av_log(NULL, AV_LOG_WARNING, "sub2video: non-bitmap subtitle\n"); + return AVERROR_INVALIDDATA; + } + + switch (inlink->format) { + case AV_PIX_FMT_YUV420P: + blend_yuv_8_8bits(ctx, mainpic, sub_rect, 1, 1, sub_rect->x + s->x, sub_rect->y + s->y); + break; + case AV_PIX_FMT_YUV422P: + blend_yuv_8_8bits(ctx, mainpic, sub_rect, 1, 0, sub_rect->x + s->x, sub_rect->y + s->y); + break; + case AV_PIX_FMT_YUV444P: + blend_yuv_8_8bits(ctx, mainpic, sub_rect, 0, 0, sub_rect->x + s->x, sub_rect->y + s->y); + break; + case AV_PIX_FMT_RGB24: + case AV_PIX_FMT_BGR24: + case AV_PIX_FMT_ARGB: + case AV_PIX_FMT_RGBA: + case AV_PIX_FMT_BGRA: + case AV_PIX_FMT_ABGR: + blend_packed_rgb(ctx, mainpic, sub_rect, sub_rect->x + s->x, sub_rect->y + s->y, 1); + break; + default: + av_assert0(0); + break; + } + } + } + + return ff_filter_frame(ctx->outputs[0], mainpic); +} + +static av_cold int overlay_graphicsubs_init(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + s->fs.on_event = do_blend; + return 0; +} + +static int overlay_graphicsubs_activate(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + return ff_framesync_activate(&s->fs); +} + +static int graphicsub2video_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_BITMAP, AV_SUBTITLE_FMT_NONE }; + static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_RGB32, AV_PIX_FMT_NONE }; + int ret; + + /* set input subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output video formats */ + formats = ff_make_format_list(pix_fmts); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int graphicsub2video_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + OverlaySubsContext *s = ctx->priv; + + if (s->w <= 0 || s->h <= 0) { + s->w = inlink->w; + s->h = inlink->h; + } + return 0; +} + +static int graphicsub2video_config_output(AVFilterLink *outlink) +{ + const AVFilterContext *ctx = outlink->src; + OverlaySubsContext *s = ctx->priv; + const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(outlink->format); + + outlink->w = s->w; + outlink->h = s->h; + outlink->sample_aspect_ratio = (AVRational){1,1}; + + av_image_fill_max_pixsteps(s->main_pix_step, NULL, pix_desc); + ff_fill_rgba_map(s->overlay_rgba_map, AV_PIX_FMT_RGB32); + + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + + s->main_desc = pix_desc; + + s->main_is_packed_rgb = ff_fill_rgba_map(s->main_rgba_map, outlink->format) >= 0; + s->main_has_alpha = !(pix_desc->flags & AV_PIX_FMT_FLAG_ALPHA); + + return 0; +} + +static int graphicsub2video_filter_frame(AVFilterLink *inlink, AVFrame *src_frame) +{ + AVFilterLink *outlink = inlink->dst->outputs[0]; + AVSubtitle *sub; + AVFrame *out; + unsigned int i; + + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) + return AVERROR(ENOMEM); + + memset(out->data[0], 0, out->linesize[0] * out->height); + + out->pts = src_frame->pts; + + sub = (AVSubtitle *)src_frame->data[0]; + + if (sub) { + const unsigned num_rects = sub->num_rects; + + for (i = 0; i < num_rects; i++) { + const AVSubtitleRect *sub_rect = sub->rects[i]; + + if (sub_rect->type != AV_SUBTITLE_FMT_BITMAP) { + av_log(NULL, AV_LOG_WARNING, "sub2video: non-bitmap subtitle\n"); + return AVERROR_INVALIDDATA; + } + + blend_packed_rgb(inlink->dst, out, sub_rect, sub_rect->x, sub_rect->y, 1); + } + } + + av_frame_free(&src_frame); + return ff_filter_frame(outlink, out); +} + +#define OFFSET(x) offsetof(OverlaySubsContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption overlay_graphicsubs_options[] = { + { "x", "set the x expression", OFFSET(x_expr), AV_OPT_TYPE_STRING, {.str = "0"}, 0, 0, FLAGS }, + { "y", "set the y expression", OFFSET(y_expr), AV_OPT_TYPE_STRING, {.str = "0"}, 0, 0, FLAGS }, + { "eof_action", "Action to take when encountering EOF from secondary input ", + OFFSET(fs.opt_eof_action), AV_OPT_TYPE_INT, { .i64 = EOF_ACTION_REPEAT }, + EOF_ACTION_REPEAT, EOF_ACTION_PASS, .flags = FLAGS, "eof_action" }, + { "repeat", "Repeat the previous frame.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_REPEAT }, .flags = FLAGS, "eof_action" }, + { "endall", "End both streams.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_ENDALL }, .flags = FLAGS, "eof_action" }, + { "pass", "Pass through the main input.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_PASS }, .flags = FLAGS, "eof_action" }, + { "eval", "specify when to evaluate expressions", OFFSET(eval_mode), AV_OPT_TYPE_INT, {.i64 = EVAL_MODE_FRAME}, 0, EVAL_MODE_NB-1, FLAGS, "eval" }, + { "init", "eval expressions once during initialization", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_INIT}, .flags = FLAGS, .unit = "eval" }, + { "frame", "eval expressions per-frame", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_FRAME}, .flags = FLAGS, .unit = "eval" }, + { "shortest", "force termination when the shortest input terminates", OFFSET(fs.opt_shortest), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, + { "repeatlast", "repeat overlay of the last overlay frame", OFFSET(fs.opt_repeatlast), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, FLAGS }, + { NULL } +}; + +static const AVOption graphicsub2video_options[] = { + { "size", "set video size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, FLAGS }, + { "s", "set video size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, FLAGS }, + { NULL } +}; + +FRAMESYNC_DEFINE_CLASS(overlay_graphicsubs, OverlaySubsContext, fs); + +static const AVFilterPad overlay_graphicsubs_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input_main, + .flags = AVFILTERPAD_FLAG_NEEDS_WRITABLE, + }, + { + .name = "overlay", + .type = AVMEDIA_TYPE_SUBTITLE, + }, +}; + +static const AVFilterPad overlay_graphicsubs_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +const AVFilter ff_vf_overlay_graphicsubs = { + .name = "overlay_graphicsubs", + .description = NULL_IF_CONFIG_SMALL("Overlay graphical subtitles on top of the input."), + .preinit = overlay_graphicsubs_framesync_preinit, + .init = overlay_graphicsubs_init, + .uninit = overlay_graphicsubs_uninit, + .priv_size = sizeof(OverlaySubsContext), + .priv_class = &overlay_graphicsubs_class, + .query_formats = overlay_graphicsubs_query_formats, + .activate = overlay_graphicsubs_activate, + FILTER_INPUTS(overlay_graphicsubs_inputs), + FILTER_OUTPUTS(overlay_graphicsubs_outputs), +}; + +AVFILTER_DEFINE_CLASS(graphicsub2video); + +static const AVFilterPad graphicsub2video_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = graphicsub2video_filter_frame, + .config_props = graphicsub2video_config_input, + }, +}; + +static const AVFilterPad graphicsub2video_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = graphicsub2video_config_output, + }, +}; + +AVFilter ff_svf_graphicsub2video = { + .name = "graphicsub2video", + .description = NULL_IF_CONFIG_SMALL("Convert graphical subtitles to video"), + .query_formats = graphicsub2video_query_formats, + .priv_size = sizeof(OverlaySubsContext), + .priv_class = &graphicsub2video_class, + FILTER_INPUTS(graphicsub2video_inputs), + FILTER_OUTPUTS(graphicsub2video_outputs), +};