From patchwork Tue Oct 25 09:13:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 38983 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4a86:b0:9d:28a3:170e with SMTP id fn6csp2684963pzb; Tue, 25 Oct 2022 02:15:08 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4+hmCAA69LzdlHEuPmqzKvXVA6E12ibfwJ9rEjOLd4a5NFMo/0dtuebz6Vx9Gw+NV7L29k X-Received: by 2002:a17:907:97c7:b0:7a5:ad82:f31c with SMTP id js7-20020a17090797c700b007a5ad82f31cmr10201724ejc.497.1666689308049; Tue, 25 Oct 2022 02:15:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666689308; cv=none; d=google.com; s=arc-20160816; b=GWJQcnTUtj+oW3RrPv12jeuSingWLTr7T5c7Zv1NwqevEzlDi1epxNdOQxM3d4QAMt PXnVxTw67czBDw+IyfFNYWz7bQmRgGoJV5yEdhPbFYhJGl0vbZqtEDshk+stWcqp00oK Rj9DXEiDz0Vkf56A8aTKxc5DyT7cCqh6Q1bkzdbY18sHUu910WNGI2YyZZM1dE10itZW HAicH2Q9XM20Y7LPoTMC2XbRAVAjA4GaRC9HsLE512exDsS4/KsFLND97ldoQMAJzLdu 2atuoW3PCmCrckrslMVXEXI+qpCIZxMAuR0M+ADjYhywiiXfuLvdgesYfPv7Um+Mjisb cWGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=a2WaM0fzhmLu0TWHQp5BL9bIMHwq2Vrne/9e4xDNxsw=; b=Tr5DN4wm/H1BdcK16HTbGy/n/uG5BBMIv0VPag7LbVwXbCz+xHOtwBjRV5wy/acKYk lzX7o6TAZiZE1FXpFdKpxfZBslcieU10L0jTDeHcZoNG6RAHlRVt4vS0+tW3SQqhw9IH oYGKn0oLT5bOW9/w+Fz2XsUJkMPtRuOrvpUSSXld4mFBTbzO5p2kZvoJRqvMbAn1xw7D M+tzWBNBnl9NeM1hDnxoxIAMyPU4ZBKNpIiYy7WrAVowU+E/NelAmx4Hy8F6use+IZiJ E/KGSAbmGpnAXdaY2ZKNelz879AkRQvi7m7qvbsQPmiPD74g8lY9kKj7YtXcxuHLX66L xK/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=g3kQSNvB; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d17-20020a170906305100b007317ad1f9a4si1923902ejd.310.2022.10.25.02.15.07; Tue, 25 Oct 2022 02:15:08 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=g3kQSNvB; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 06B9E68BD18; Tue, 25 Oct 2022 12:14:07 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 20C6868BC71 for ; Tue, 25 Oct 2022 12:14:02 +0300 (EEST) Received: by mail-pl1-f176.google.com with SMTP id c2so2447263plz.11 for ; Tue, 25 Oct 2022 02:14:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date :references:in-reply-to:message-id:from:from:to:cc:subject:date :message-id:reply-to; bh=BaTjIvOE/dpwuL9yKzdaz3SevcpM3pY5YGZC9AhD9KQ=; b=g3kQSNvBwasdarjJ/+S+jyghvxFM1dapccieL9PyQEpC0yBxXmudTWSkKFpo1paBFA sW9eZOatJ4YD269Ez+pEwfkANzMv+Cwx8b0cx72JRlFnZiAafIWWEqS727VjaSF7s5/r xEZzzlL41XvXYlVEfUhhK2lrJE/5ec5YQ46wD1O5YaXlbCf3QDsLKhKtO6Nz0YN8awV6 tx/HQTV0Rwnm5Y+KyJaVSIRBhlj9WF4LSuXq3F6bSrWrmHAvWx4N5lkkI1veueLAmw3t wgt5owkigOX+REE7lMBqMMUvi9ffY0lcdYddRgqcwAb8UYQ/EXcckWwp+rG+TI/Nn+57 hb6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date :references:in-reply-to:message-id:from:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=BaTjIvOE/dpwuL9yKzdaz3SevcpM3pY5YGZC9AhD9KQ=; b=uryrvdNgLiHXn3spoUXX8Be/XQAKYqN07yFiiURsr0UFtlWVKicW0kzmT87mpc4EW/ 4ZMX0MF9KpZWA2L39Jc6JIloTlnekNOeyZThkzNPkztW7ebkFd7IZLuPNcCYbdHnaNkN ERd91LZ5rO7558PhKMLVW2C1S2WkEssjxJl3s9tR1k+bCAu6euXl7QX04OTpXlFNqWDM pgzGGOtaUPEgXLGj4YkGZLRH0FRVICTNmmlpKdtvIGCeyx6gqfEkJcGkzZSv/2n2fdzO S58JaOcG8Ah44qH5CZOHj1MwQsmzQEh7YEo9DA6jTPmaEPc72xowtkdmkj4d9FJxT7Uu ZgkA== X-Gm-Message-State: ACrzQf3kWqnDGB6i3LasSywTcrmENCM0NRIxiZGUDy4e2QQkul36Vmw6 x4yVQ6xveibO2JOX801pE1Jzn5ZIj2o= X-Received: by 2002:a17:90b:1e43:b0:213:1efe:9815 with SMTP id pi3-20020a17090b1e4300b002131efe9815mr8661481pjb.164.1666689241175; Tue, 25 Oct 2022 02:14:01 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id b9-20020a1709027e0900b0017f92d7fe2csm854236plm.288.2022.10.25.02.14.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Oct 2022 02:14:00 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: In-Reply-To: References: Date: Tue, 25 Oct 2022 09:13:31 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v9 10/25] avfilter/avfilter: Handle subtitle frames X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: softworkz Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Mf8XrM8UJLgg From: softworkz Signed-off-by: softworkz --- libavfilter/avfilter.c | 20 +++++++++++++++++--- libavfilter/avfilter.h | 11 +++++++++++ libavfilter/avfiltergraph.c | 5 +++++ libavfilter/formats.c | 16 ++++++++++++++++ libavfilter/formats.h | 3 +++ libavfilter/internal.h | 18 +++++++++++++++--- 6 files changed, 67 insertions(+), 6 deletions(-) diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index e49d76c14a..b40a2fc7cd 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -54,7 +54,8 @@ static void tlog_ref(void *ctx, AVFrame *ref, int end) ref->linesize[0], ref->linesize[1], ref->linesize[2], ref->linesize[3], ref->pts, ref->pkt_pos); - if (ref->width) { + switch(ref->type) { + case AVMEDIA_TYPE_VIDEO: ff_tlog(ctx, " a:%d/%d s:%dx%d i:%c iskey:%d type:%c", ref->sample_aspect_ratio.num, ref->sample_aspect_ratio.den, ref->width, ref->height, @@ -62,8 +63,8 @@ static void tlog_ref(void *ctx, AVFrame *ref, int end) ref->top_field_first ? 'T' : 'B', /* Top / Bottom */ ref->key_frame, av_get_picture_type_char(ref->pict_type)); - } - if (ref->nb_samples) { + break; + case AVMEDIA_TYPE_AUDIO: AVBPrint bprint; av_bprint_init(&bprint, 1, AV_BPRINT_SIZE_UNLIMITED); @@ -73,6 +74,7 @@ static void tlog_ref(void *ctx, AVFrame *ref, int end) ref->nb_samples, ref->sample_rate); av_bprint_finalize(&bprint, NULL); + break; } ff_tlog(ctx, "]%s", end ? "\n" : ""); @@ -356,6 +358,14 @@ int avfilter_config_links(AVFilterContext *filter) if (!link->time_base.num && !link->time_base.den) link->time_base = (AVRational) {1, link->sample_rate}; + + break; + + case AVMEDIA_TYPE_SUBTITLE: + if (!link->time_base.num && !link->time_base.den) + link->time_base = inlink ? inlink->time_base : AV_TIME_BASE_Q; + + break; } if (link->src->nb_inputs && link->src->inputs[0]->hw_frames_ctx && @@ -1023,6 +1033,10 @@ int ff_filter_frame(AVFilterLink *link, AVFrame *frame) av_assert1(frame->width == link->w); av_assert1(frame->height == link->h); } + } else if (link->type == AVMEDIA_TYPE_SUBTITLE) { + if (frame->format != link->format) { + av_log(link->dst, AV_LOG_WARNING, "Subtitle format change from %d to %d\n", link->format, frame->format); + } } else { if (frame->format != link->format) { av_log(link->dst, AV_LOG_ERROR, "Format change is not supported\n"); diff --git a/libavfilter/avfilter.h b/libavfilter/avfilter.h index 6d68ebece4..82f6b21520 100644 --- a/libavfilter/avfilter.h +++ b/libavfilter/avfilter.h @@ -45,6 +45,7 @@ #include "libavutil/log.h" #include "libavutil/samplefmt.h" #include "libavutil/pixfmt.h" +#include "libavutil/subfmt.h" #include "libavutil/rational.h" #include "libavfilter/version_major.h" @@ -356,6 +357,12 @@ typedef struct AVFilter { * and outputs use the same sample rate and channel count/layout. */ const enum AVSampleFormat *samples_list; + /** + * Analogous to pixels, but delimited by AV_SUBTITLE_FMT_NONE + * and restricted to filters that only have AVMEDIA_TYPE_SUBTITLE + * inputs and outputs. + */ + const enum AVSubtitleType *subs_list; /** * Equivalent to { pix_fmt, AV_PIX_FMT_NONE } as pixels_list. */ @@ -364,6 +371,10 @@ typedef struct AVFilter { * Equivalent to { sample_fmt, AV_SAMPLE_FMT_NONE } as samples_list. */ enum AVSampleFormat sample_fmt; + /** + * Equivalent to { sub_fmt, AV_SUBTITLE_FMT_NONE } as subs_list. + */ + enum AVSubtitleType sub_fmt; } formats; int priv_size; ///< size of private data to allocate for the filter diff --git a/libavfilter/avfiltergraph.c b/libavfilter/avfiltergraph.c index 53f468494d..f7547467f0 100644 --- a/libavfilter/avfiltergraph.c +++ b/libavfilter/avfiltergraph.c @@ -309,6 +309,8 @@ static int filter_link_check_formats(void *log, AVFilterLink *link, AVFilterForm return ret; break; + case AVMEDIA_TYPE_SUBTITLE: + return 0; default: av_assert0(!"reached"); } @@ -439,6 +441,9 @@ static int query_formats(AVFilterGraph *graph, void *log_ctx) if (!link) continue; + if (link->type == AVMEDIA_TYPE_SUBTITLE) + continue; + neg = ff_filter_get_negotiation(link); av_assert0(neg); for (neg_step = 1; neg_step < neg->nb_mergers; neg_step++) { diff --git a/libavfilter/formats.c b/libavfilter/formats.c index e8c2888c0c..12585ed428 100644 --- a/libavfilter/formats.c +++ b/libavfilter/formats.c @@ -19,6 +19,7 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include "libavutil/subfmt.h" #include "libavutil/avassert.h" #include "libavutil/channel_layout.h" #include "libavutil/common.h" @@ -491,6 +492,13 @@ AVFilterFormats *ff_all_formats(enum AVMediaType type) return NULL; fmt++; } + } else if (type == AVMEDIA_TYPE_SUBTITLE) { + if (ff_add_format(&ret, AV_SUBTITLE_FMT_BITMAP) < 0) + return NULL; + if (ff_add_format(&ret, AV_SUBTITLE_FMT_ASS) < 0) + return NULL; + if (ff_add_format(&ret, AV_SUBTITLE_FMT_TEXT) < 0) + return NULL; } return ret; @@ -774,6 +782,10 @@ int ff_default_query_formats(AVFilterContext *ctx) type = AVMEDIA_TYPE_AUDIO; formats = ff_make_format_list(f->formats.samples_list); break; + case FF_FILTER_FORMATS_SUBFMTS_LIST: + type = AVMEDIA_TYPE_SUBTITLE; + formats = ff_make_format_list(f->formats.subs_list); + break; case FF_FILTER_FORMATS_SINGLE_PIXFMT: type = AVMEDIA_TYPE_VIDEO; formats = ff_make_formats_list_singleton(f->formats.pix_fmt); @@ -782,6 +794,10 @@ int ff_default_query_formats(AVFilterContext *ctx) type = AVMEDIA_TYPE_AUDIO; formats = ff_make_formats_list_singleton(f->formats.sample_fmt); break; + case FF_FILTER_FORMATS_SINGLE_SUBFMT: + type = AVMEDIA_TYPE_SUBTITLE; + formats = ff_make_formats_list_singleton(f->formats.sub_fmt); + break; default: av_assert2(!"Unreachable"); /* Intended fallthrough */ diff --git a/libavfilter/formats.h b/libavfilter/formats.h index 22224dce2d..6cf952a059 100644 --- a/libavfilter/formats.h +++ b/libavfilter/formats.h @@ -183,6 +183,9 @@ av_warn_unused_result int ff_add_channel_layout(AVFilterChannelLayouts **l, const AVChannelLayout *channel_layout); +av_warn_unused_result +int ff_add_subtitle_type(AVFilterFormats **avff, int64_t fmt); + /** * Add *ref as a new reference to f. */ diff --git a/libavfilter/internal.h b/libavfilter/internal.h index 1a0752e4ee..dc56960eef 100644 --- a/libavfilter/internal.h +++ b/libavfilter/internal.h @@ -148,9 +148,11 @@ static av_always_inline int ff_filter_execute(AVFilterContext *ctx, avfilter_act enum FilterFormatsState { /** - * The default value meaning that this filter supports all formats - * and (for audio) sample rates and channel layouts/counts as long - * as these properties agree for all inputs and outputs. + * The default value meaning that this filter supports + * - For video: all formats + * - For audio: all sample rates and channel layouts/counts + * - For subtitles: all subtitle formats + * as long as these properties agree for all inputs and outputs. * This state is only allowed in case all inputs and outputs actually * have the same type. * The union is unused in this state. @@ -161,8 +163,10 @@ enum FilterFormatsState { FF_FILTER_FORMATS_QUERY_FUNC, ///< formats.query active. FF_FILTER_FORMATS_PIXFMT_LIST, ///< formats.pixels_list active. FF_FILTER_FORMATS_SAMPLEFMTS_LIST, ///< formats.samples_list active. + FF_FILTER_FORMATS_SUBFMTS_LIST, ///< formats.subs_list active. FF_FILTER_FORMATS_SINGLE_PIXFMT, ///< formats.pix_fmt active FF_FILTER_FORMATS_SINGLE_SAMPLEFMT, ///< formats.sample_fmt active. + FF_FILTER_FORMATS_SINGLE_SUBFMT, ///< formats.sub_fmt active. }; #define FILTER_QUERY_FUNC(func) \ @@ -174,16 +178,24 @@ enum FilterFormatsState { #define FILTER_SAMPLEFMTS_ARRAY(array) \ .formats.samples_list = array, \ .formats_state = FF_FILTER_FORMATS_SAMPLEFMTS_LIST +#define FILTER_SUBFMTS_ARRAY(array) \ + .formats.subs_list = array, \ + .formats_state = FF_FILTER_FORMATS_SUBFMTS_LIST #define FILTER_PIXFMTS(...) \ FILTER_PIXFMTS_ARRAY(((const enum AVPixelFormat []) { __VA_ARGS__, AV_PIX_FMT_NONE })) #define FILTER_SAMPLEFMTS(...) \ FILTER_SAMPLEFMTS_ARRAY(((const enum AVSampleFormat[]) { __VA_ARGS__, AV_SAMPLE_FMT_NONE })) +#define FILTER_SUBFMTS(...) \ + FILTER_SUBFMTS_ARRAY(((const enum AVSubtitleType[]) { __VA_ARGS__, AV_SUBTITLE_FMT_NONE })) #define FILTER_SINGLE_PIXFMT(pix_fmt_) \ .formats.pix_fmt = pix_fmt_, \ .formats_state = FF_FILTER_FORMATS_SINGLE_PIXFMT #define FILTER_SINGLE_SAMPLEFMT(sample_fmt_) \ .formats.sample_fmt = sample_fmt_, \ .formats_state = FF_FILTER_FORMATS_SINGLE_SAMPLEFMT +#define FILTER_SINGLE_SUBFMT(sub_fmt_) \ + .formats.sub_fmt = sub_fmt_, \ + .formats_state = FF_FILTER_FORMATS_SINGLE_SUBFMT #define FILTER_INOUTPADS(inout, array) \ .inout = array, \