From patchwork Thu Sep 30 19:30:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30796 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp984788iob; Thu, 30 Sep 2021 12:30:55 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwLT7r3FPF7MUeExkQiXZ8K3yB+KXApwFyikV5n7nFK2LCFib8fx8G0M0N3+e/hQSTTNsXr X-Received: by 2002:a50:e106:: with SMTP id h6mr9236816edl.295.1633030255228; Thu, 30 Sep 2021 12:30:55 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id g21si4217503ejp.758.2021.09.30.12.30.54; Thu, 30 Sep 2021 12:30:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=MKEb4QLs; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B2D4B68AA2A; Thu, 30 Sep 2021 22:30:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10olkn2105.outbound.protection.outlook.com [40.92.41.105]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C692B68AA16 for ; Thu, 30 Sep 2021 22:30:44 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=d6t4yYyWcCJEGtQ19X/BAaSYcrpreE/DTc0gCEHiM+xnmmgtJB30ESypGouVrRAxg1rvEt02aTK+sDJL12FKw6WSJLbyTgEfOYa7aOncKKbwiJLERkXxs9TDxWCDdhJzYZiRt4hJt+5Hd/BeG7H6IpnCZOZ1OSaM+W3t7urMzZCqSSlINfSu6wweA5xPnja/RLx125bH3gI2lbxLxuZXOydpOY3A5sU+NC+9ejw4Gq9/hyt52eArpo9rSAPUgR3gatMk3wY7xZGy+ganWBy/gbhVOXXR1AFMCX8FZYkEgENAZugYygalS1FC4jHixN9+8jdDajVxvxQj4lHfzSZcfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=CzmDgu/v3oAPz74d3CLg7N8Ujhrd3oTG0Np+8qKkOF0=; b=Q71kf6elaBB7Kg+rAL099Klq881APkAczIh2rVMOvRHPcVPUFY2FjnNB99omvw5EDDW5+hIVwMrhURpaiYUYCIMSh2lbYlwSBdYKEriWNegME+J1o8r0wPpuenQCqjjd2nFxkwMU64x4i7K84POTn1RDflzrPp3mXai57ZmIv+JR/DDLkjcSJfdM/Li9A8+SNj7eESE+2wt7kLMvRvNhqlU73TRM/yDXg/MfvN4NBgUthJM8KNK+5IO2i81OnbMcvprsXZ0jblJvaYPNPkHvHzuDnlEeP1BRJyc0mGURluvg41VG0/OntIWG6pbiX4SMFPJPqCw3awc5XPgSyT+RDA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CzmDgu/v3oAPz74d3CLg7N8Ujhrd3oTG0Np+8qKkOF0=; b=MKEb4QLsGIH7o4W3RppyTs1eMAaF4f+cELlTINg9mFhVzdlffjemHhuFNp7M+amEHHD6ploNWzzsf11hpB3fwTjiOiXotZ5l0Nh5/VLU9qXAZZGpCL+Vu3uqItTZpu52tNDBMCwCxRzi7/InFPBBpb9lYUrrGZHyqsjlkCoAWDaHZncIUaofpLow5cp/YO790juf7LJvZOmgoPq4oShPR2b88O8t1yD3nYWXBh3PiynNT5rar6pIDwV5qSESonvuAwh7jPUl9Cya39Ea2NcpnNKVQ2FDQikgjS5fJT70T97EyD+lWlblV6vW0Nocyvv3wgzz/Ncy472qOlYx9N+bTw== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5615.namprd04.prod.outlook.com (2603:10b6:208:fe::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:30:42 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:30:42 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 01/14] fftools/play,probe: Adjust for subtitle changes Thread-Index: AQHXtjGoFJ6ODqguoU28wlf7gcQ+YQ== Date: Thu, 30 Sep 2021 19:30:41 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [ozb5chFD6+JU58uCHMmSrzyJBWwZB0YW] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 58927164-d860-4a41-78fa-08d98448ca91 x-ms-traffictypediagnostic: MN2PR04MB5615: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: KX46GJqVJPpIGUl8l9735bi4TlLOUtahSzH8mLu06kVyURdsn44dLUtfk8QWMB69/4u+7kUbxDLMkUMei7JFSQQfj7RaW+nTYrBUPfV25+NQS17tzPmulaqVgLgPJvwOJLpz3i1/EK9D0bbRAPS38zqADCasw4GTL66qTQvMcYsBKk3Z6S11+V1JSTKHyGhK6UQ4txkGm7X671Dr4LnD3zZ56hHsG8cIw5oLlJqylMmOjv0JMgYHXWs4vGKHLa93M+PKasNWkge4Dj1wlDlG6LeED9dGs7Yojmnj7u9zs4jLFlN4vQJxm3Jt7ZSOji6YQOJ/+fUJN8PU2k75HjtvX7Ednl7dbOZr6jPrYgjBNEJM/t9EI+wrNDlqiQpL/jzhBlh+YU1PMggNz7lMJqFoxItIEJIQd6haI1cPQJQTlqKUO3yuCix8nNZsFWMDkuCg x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: Bt3GjOhpbDecP+WvEPp/8fLNqeTd6xXlrnkk5CDwWBQ2S2Ave0SCGPAJnr+k2aqVTZRphPEwe9IF3Cdq3WLXW5ZWDGnjj3E+G/GTimJOfxUMxV2TThcabpdG1zf+x6pV2kPJY2FLPHsGvL8D4TDHhw== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 58927164-d860-4a41-78fa-08d98448ca91 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:30:41.9909 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5615 Subject: [FFmpeg-devel] [PATCH v10 01/14] fftools/play, probe: Adjust for subtitle changes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: UJRURXqmljxI Signed-off-by: softworkz --- fftools/ffplay.c | 102 +++++++++++++++++++++------------------------- fftools/ffprobe.c | 48 ++++++++++++++-------- 2 files changed, 79 insertions(+), 71 deletions(-) diff --git a/fftools/ffplay.c b/fftools/ffplay.c index ccea0e4578..e8d36485ae 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -152,7 +152,6 @@ typedef struct Clock { /* Common struct for handling all types of decoded data and allocated render buffers. */ typedef struct Frame { AVFrame *frame; - AVSubtitle sub; int serial; double pts; /* presentation timestamp for the frame */ double duration; /* estimated duration of the frame */ @@ -586,7 +585,7 @@ static int decoder_init(Decoder *d, AVCodecContext *avctx, PacketQueue *queue, S return 0; } -static int decoder_decode_frame(Decoder *d, AVFrame *frame, AVSubtitle *sub) { +static int decoder_decode_frame(Decoder *d, AVFrame *frame) { int ret = AVERROR(EAGAIN); for (;;) { @@ -620,6 +619,9 @@ static int decoder_decode_frame(Decoder *d, AVFrame *frame, AVSubtitle *sub) { } } break; + case AVMEDIA_TYPE_SUBTITLE: + ret = avcodec_receive_frame(d->avctx, frame); + break; } if (ret == AVERROR_EOF) { d->finished = d->pkt_serial; @@ -652,25 +654,11 @@ static int decoder_decode_frame(Decoder *d, AVFrame *frame, AVSubtitle *sub) { av_packet_unref(d->pkt); } while (1); - if (d->avctx->codec_type == AVMEDIA_TYPE_SUBTITLE) { - int got_frame = 0; - ret = avcodec_decode_subtitle2(d->avctx, sub, &got_frame, d->pkt); - if (ret < 0) { - ret = AVERROR(EAGAIN); - } else { - if (got_frame && !d->pkt->data) { - d->packet_pending = 1; - } - ret = got_frame ? 0 : (d->pkt->data ? AVERROR(EAGAIN) : AVERROR_EOF); - } - av_packet_unref(d->pkt); + if (avcodec_send_packet(d->avctx, d->pkt) == AVERROR(EAGAIN)) { + av_log(d->avctx, AV_LOG_ERROR, "Receive_frame and send_packet both returned EAGAIN, which is an API violation.\n"); + d->packet_pending = 1; } else { - if (avcodec_send_packet(d->avctx, d->pkt) == AVERROR(EAGAIN)) { - av_log(d->avctx, AV_LOG_ERROR, "Receive_frame and send_packet both returned EAGAIN, which is an API violation.\n"); - d->packet_pending = 1; - } else { - av_packet_unref(d->pkt); - } + av_packet_unref(d->pkt); } } } @@ -683,7 +671,6 @@ static void decoder_destroy(Decoder *d) { static void frame_queue_unref_item(Frame *vp) { av_frame_unref(vp->frame); - avsubtitle_free(&vp->sub); } static int frame_queue_init(FrameQueue *f, PacketQueue *pktq, int max_size, int keep_last) @@ -981,7 +968,7 @@ static void video_image_display(VideoState *is) if (frame_queue_nb_remaining(&is->subpq) > 0) { sp = frame_queue_peek(&is->subpq); - if (vp->pts >= sp->pts + ((float) sp->sub.start_display_time / 1000)) { + if (vp->pts >= sp->pts + ((float) sp->frame->subtitle_start_time / 1000)) { if (!sp->uploaded) { uint8_t* pixels[4]; int pitch[4]; @@ -993,25 +980,27 @@ static void video_image_display(VideoState *is) if (realloc_texture(&is->sub_texture, SDL_PIXELFORMAT_ARGB8888, sp->width, sp->height, SDL_BLENDMODE_BLEND, 1) < 0) return; - for (i = 0; i < sp->sub.num_rects; i++) { - AVSubtitleRect *sub_rect = sp->sub.rects[i]; + for (i = 0; i < sp->frame->num_subtitle_areas; i++) { + AVSubtitleArea *area = sp->frame->subtitle_areas[i]; + SDL_Rect sdl_rect = { .x = area->x, .y = area->y, .w = area->w, .h = area->h }; - sub_rect->x = av_clip(sub_rect->x, 0, sp->width ); - sub_rect->y = av_clip(sub_rect->y, 0, sp->height); - sub_rect->w = av_clip(sub_rect->w, 0, sp->width - sub_rect->x); - sub_rect->h = av_clip(sub_rect->h, 0, sp->height - sub_rect->y); + area->x = av_clip(area->x, 0, sp->width ); + area->y = av_clip(area->y, 0, sp->height); + area->w = av_clip(area->w, 0, sp->width - area->x); + area->h = av_clip(area->h, 0, sp->height - area->y); is->sub_convert_ctx = sws_getCachedContext(is->sub_convert_ctx, - sub_rect->w, sub_rect->h, AV_PIX_FMT_PAL8, - sub_rect->w, sub_rect->h, AV_PIX_FMT_BGRA, + area->w, area->h, AV_PIX_FMT_PAL8, + area->w, area->h, AV_PIX_FMT_BGRA, 0, NULL, NULL, NULL); if (!is->sub_convert_ctx) { av_log(NULL, AV_LOG_FATAL, "Cannot initialize the conversion context\n"); return; } - if (!SDL_LockTexture(is->sub_texture, (SDL_Rect *)sub_rect, (void **)pixels, pitch)) { - sws_scale(is->sub_convert_ctx, (const uint8_t * const *)sub_rect->data, sub_rect->linesize, - 0, sub_rect->h, pixels, pitch); + if (!SDL_LockTexture(is->sub_texture, &sdl_rect, (void **)pixels, pitch)) { + const uint8_t* data[2] = { area->buf[0]->data, (uint8_t *)&area->pal }; + sws_scale(is->sub_convert_ctx, data, area->linesize, + 0, area->h, pixels, pitch); SDL_UnlockTexture(is->sub_texture); } } @@ -1038,16 +1027,18 @@ static void video_image_display(VideoState *is) #if USE_ONEPASS_SUBTITLE_RENDER SDL_RenderCopy(renderer, is->sub_texture, NULL, &rect); #else - int i; + unsigned i; double xratio = (double)rect.w / (double)sp->width; double yratio = (double)rect.h / (double)sp->height; - for (i = 0; i < sp->sub.num_rects; i++) { - SDL_Rect *sub_rect = (SDL_Rect*)sp->sub.rects[i]; - SDL_Rect target = {.x = rect.x + sub_rect->x * xratio, - .y = rect.y + sub_rect->y * yratio, - .w = sub_rect->w * xratio, - .h = sub_rect->h * yratio}; - SDL_RenderCopy(renderer, is->sub_texture, sub_rect, &target); + for (i = 0; i < sp->frame->num_subtitle_areas; i++) { + AVSubtitleArea *area = sp->frame->subtitle_areas[i]; + SDL_Rect sub_rect = { .x = area->x, .y = area->y, + .w = area->w, .h = area->h}; + SDL_Rect target = {.x = rect.x + sub_rect.x * xratio, + .y = rect.y + sub_rect.y * yratio, + .w = sub_rect.w * xratio, + .h = sub_rect.h * yratio}; + SDL_RenderCopy(renderer, is->sub_texture, &sub_rect, &target); } #endif } @@ -1651,19 +1642,20 @@ retry: sp2 = NULL; if (sp->serial != is->subtitleq.serial - || (is->vidclk.pts > (sp->pts + ((float) sp->sub.end_display_time / 1000))) - || (sp2 && is->vidclk.pts > (sp2->pts + ((float) sp2->sub.start_display_time / 1000)))) + || (is->vidclk.pts > (sp->pts + ((float) sp->frame->subtitle_end_time / 1000))) + || (sp2 && is->vidclk.pts > (sp2->pts + ((float) sp2->frame->subtitle_start_time / 1000)))) { if (sp->uploaded) { int i; - for (i = 0; i < sp->sub.num_rects; i++) { - AVSubtitleRect *sub_rect = sp->sub.rects[i]; + for (i = 0; i < sp->frame->num_subtitle_areas; i++) { + AVSubtitleArea *area = sp->frame->subtitle_areas[i]; + SDL_Rect sdl_rect = { .x = area->x, .y = area->y, .w = area->w, .h = area->h }; uint8_t *pixels; int pitch, j; - if (!SDL_LockTexture(is->sub_texture, (SDL_Rect *)sub_rect, (void **)&pixels, &pitch)) { - for (j = 0; j < sub_rect->h; j++, pixels += pitch) - memset(pixels, 0, sub_rect->w << 2); + if (!SDL_LockTexture(is->sub_texture, &sdl_rect, (void **)&pixels, &pitch)) { + for (j = 0; j < area->h; j++, pixels += pitch) + memset(pixels, 0, area->w << 2); SDL_UnlockTexture(is->sub_texture); } } @@ -1774,7 +1766,7 @@ static int get_video_frame(VideoState *is, AVFrame *frame) { int got_picture; - if ((got_picture = decoder_decode_frame(&is->viddec, frame, NULL)) < 0) + if ((got_picture = decoder_decode_frame(&is->viddec, frame)) < 0) return -1; if (got_picture) { @@ -2048,7 +2040,7 @@ static int audio_thread(void *arg) return AVERROR(ENOMEM); do { - if ((got_frame = decoder_decode_frame(&is->auddec, frame, NULL)) < 0) + if ((got_frame = decoder_decode_frame(&is->auddec, frame)) < 0) goto the_end; if (got_frame) { @@ -2246,14 +2238,14 @@ static int subtitle_thread(void *arg) if (!(sp = frame_queue_peek_writable(&is->subpq))) return 0; - if ((got_subtitle = decoder_decode_frame(&is->subdec, NULL, &sp->sub)) < 0) + if ((got_subtitle = decoder_decode_frame(&is->subdec, sp->frame)) < 0) break; pts = 0; - if (got_subtitle && sp->sub.format == 0) { - if (sp->sub.pts != AV_NOPTS_VALUE) - pts = sp->sub.pts / (double)AV_TIME_BASE; + if (got_subtitle && sp->frame->format == AV_SUBTITLE_FMT_BITMAP) { + if (sp->frame->subtitle_pts != AV_NOPTS_VALUE) + pts = sp->frame->subtitle_pts / (double)AV_TIME_BASE; sp->pts = pts; sp->serial = is->subdec.pkt_serial; sp->width = is->subdec.avctx->width; @@ -2263,7 +2255,7 @@ static int subtitle_thread(void *arg) /* now we can update the picture count */ frame_queue_push(&is->subpq); } else if (got_subtitle) { - avsubtitle_free(&sp->sub); + av_frame_free(&sp->frame); } } return 0; diff --git a/fftools/ffprobe.c b/fftools/ffprobe.c index 90e895bbf9..4e08807706 100644 --- a/fftools/ffprobe.c +++ b/fftools/ffprobe.c @@ -2208,22 +2208,43 @@ static void show_packet(WriterContext *w, InputFile *ifile, AVPacket *pkt, int p fflush(stdout); } -static void show_subtitle(WriterContext *w, AVSubtitle *sub, AVStream *stream, +static void show_subtitle(WriterContext *w, AVFrame *sub, AVStream *stream, AVFormatContext *fmt_ctx) { AVBPrint pbuf; + const char *s; av_bprint_init(&pbuf, 1, AV_BPRINT_SIZE_UNLIMITED); writer_print_section_header(w, SECTION_ID_SUBTITLE); print_str ("media_type", "subtitle"); - print_ts ("pts", sub->pts); - print_time("pts_time", sub->pts, &AV_TIME_BASE_Q); - print_int ("format", sub->format); - print_int ("start_display_time", sub->start_display_time); - print_int ("end_display_time", sub->end_display_time); - print_int ("num_rects", sub->num_rects); + print_ts ("pts", sub->subtitle_pts); + print_time("pts_time", sub->subtitle_pts, &AV_TIME_BASE_Q); + + // Remain compatible with previous outputs + switch (sub->format) { + case AV_SUBTITLE_FMT_BITMAP: + print_int ("format", 0); + break; + case AV_SUBTITLE_FMT_TEXT: + print_int ("format", 1); + break; + case AV_SUBTITLE_FMT_ASS: + print_int ("format", 1); + break; + default: + print_int ("format", -1); + break; + } + + s = av_get_subtitle_fmt_name(sub->format); + if (s) print_str ("format_str", s); + else print_str_opt("format_str", "unknown"); + + print_int ("start_display_time", sub->subtitle_start_time); + print_int ("end_display_time", sub->subtitle_end_time); + print_int ("num_subtitle_rects", sub->num_subtitle_areas); writer_print_section_footer(w); @@ -2388,7 +2409,7 @@ static av_always_inline int process_frame(WriterContext *w, AVFormatContext *fmt_ctx = ifile->fmt_ctx; AVCodecContext *dec_ctx = ifile->streams[pkt->stream_index].dec_ctx; AVCodecParameters *par = ifile->streams[pkt->stream_index].st->codecpar; - AVSubtitle sub; + AVFrame *sub = NULL; int ret = 0, got_frame = 0; clear_log(1); @@ -2396,6 +2417,7 @@ static av_always_inline int process_frame(WriterContext *w, switch (par->codec_type) { case AVMEDIA_TYPE_VIDEO: case AVMEDIA_TYPE_AUDIO: + case AVMEDIA_TYPE_SUBTITLE: if (*packet_new) { ret = avcodec_send_packet(dec_ctx, pkt); if (ret == AVERROR(EAGAIN)) { @@ -2414,12 +2436,6 @@ static av_always_inline int process_frame(WriterContext *w, } } break; - - case AVMEDIA_TYPE_SUBTITLE: - if (*packet_new) - ret = avcodec_decode_subtitle2(dec_ctx, &sub, &got_frame, pkt); - *packet_new = 0; - break; default: *packet_new = 0; } @@ -2434,11 +2450,11 @@ static av_always_inline int process_frame(WriterContext *w, nb_streams_frames[pkt->stream_index]++; if (do_show_frames) if (is_sub) - show_subtitle(w, &sub, ifile->streams[pkt->stream_index].st, fmt_ctx); + show_subtitle(w, sub, ifile->streams[pkt->stream_index].st, fmt_ctx); else show_frame(w, frame, ifile->streams[pkt->stream_index].st, fmt_ctx); if (is_sub) - avsubtitle_free(&sub); + av_frame_free(&sub); } return got_frame || *packet_new; } From patchwork Thu Sep 30 19:30:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30797 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp984964iob; Thu, 30 Sep 2021 12:31:06 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzrJFsBFjob5e8gAUTJM/G2JAqnMic3cOehcUFeyN8mTCIY1joeETq/IuGAt3YF2pT8e/cg X-Received: by 2002:a17:906:2cc9:: with SMTP id r9mr1245792ejr.138.1633030265927; Thu, 30 Sep 2021 12:31:05 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 16si5087990ejj.116.2021.09.30.12.31.05; Thu, 30 Sep 2021 12:31:05 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=GsogntQ6; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2556568AA25; Thu, 30 Sep 2021 22:30:58 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10olkn2105.outbound.protection.outlook.com [40.92.41.105]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B1FFA68AA0C for ; Thu, 30 Sep 2021 22:30:50 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NKIz8AY/WtNmk5wrSK+gV3nLgjFVr8UGYPBIpxxm0sB3m+ztoDgtuteX72DrZt1+pM3Q4M3hllc01mCpo95QQiKOtAK14xoFKrCML12S1L+shCO6fV/9OzBC1pSiBWgm1hM+0S3Z8GcXjAHesvGLD8zFPETU4idDmpyNpPjLD5sDWb1LvVfU/fX14MJSeebDG0d4P7Ojw/JACfxDDP8ZONhK00zcsmlt0GbjJ7wv+hw7XEvAvCRjepkwZ6ppHExUwlXCrCZFT4qWqVe0Tj92Pl+Wi3v1m0xM0SRtP0tSWfT96ZVzG1esKJ7JlAfjJHM+NjlKO4uP8B8Ul0D+Yfqafg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=QcUAVTJPD6zPqa+BkGSQ+9k2Y8V59MkC6hKY33R3VJA=; b=DbM0uMv2hGB4cuT5In1JanO+KTW8EjMUVFsIoKZa48qUOIE4vMWQE/gv3lOqrkZ7cdtE096GO26wsse4ph+tm4Kt+0eC7dM8HXFNo5VWGyCze612E6aGkqLUIDfWvlOAu3VmmAFGKZWLIsJV12fpXEBGAizGE5rpQWBBREDBUW+7hv0RVVU2j+2xw8vjW0mHFSjS2bP7f7vq6BhpiaiO7ZFu4YsASGZ1QRs/kr4+kz6wXA4az916lxajTgSR6VGvl/ln+rn+8iHyUiTwdyHQGtcmT8ubPq8zXRs8cMK2xj4XzWNnHrRbp+zZXVKBSexBmHe0bSzoakOh122G517j1A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QcUAVTJPD6zPqa+BkGSQ+9k2Y8V59MkC6hKY33R3VJA=; b=GsogntQ6XYhUBl8W4zJoLCBJPxg37rBAqf/8NPLj8H0DE/GgGE2qm6BEaIJc3KYeCbi7DwcHFJKtysEFekNoYjoR8DX1h1H13anGwRPmhOlwJhNqoFarHMATIwnwj8bQLKNqJdxQje+p2TRwTyXZ7TeC3LtThH+zucs8+Gs+eVHxMf6/MDeVdT3vcMxQBd+aS71XkGz5cGzmlnEmn3J/oi20Y1FeyugHPEYxcKdWuKkNUhgSPL7vmAzP4zixdr/VF3UjrLB7jd3XsC9y7SHQGCiw561n/MPLWAMdeDZ5s/gP5hc5j35oFPD9jNOFtiNs5GIicDTHTjVm6ZRWxBRGbA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5615.namprd04.prod.outlook.com (2603:10b6:208:fe::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:30:44 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:30:43 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 02/14] avfilter/subtitles: Add subtitles.c for subtitle frame allocation Thread-Index: AQHXtjGpZwRw3KLPXUiFLmupwWYwNA== Date: Thu, 30 Sep 2021 19:30:43 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [mgTm7mtu6rWSjPQQfotMhDeStZ3O9xIX] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b3580f20-047d-49de-a5ed-08d98448cbb7 x-ms-traffictypediagnostic: MN2PR04MB5615: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 6yR8kYMbTJEhWNC5Ybn+M2UJoBcFFb5VLP3tMhOJ4XpIi1grSPsajtKuFfnJXpS7SWYbd2On6R7coqDEYYbZ+O4e1+g/+WuzvE64VtPjw/RKljtFtEsYiYTFhs+/aRm+4PLHKay6lnI9zO7hAudvWVvJA0sT1uEONK+AS4R+cB7O2O5QNYJ4BP67+++SpiZW1yLx2MsjTbD6Nq4aZUMabkHrptzerjk+Ljbxk8kmgIXvCjmNCNex5QSWS9/NdRel1v9wj6gJ6pQaRBJYJ5N6f/W7QkeW2WC0MxI6gIpmYs0h4PBFU/MoXfuqCvjdP99JSxubDCSasj2tSKlqsvS8ikJMgKIJuMagn42wGL+iN/Lf+BFqVVf/ZzKNg8hzz2tmRaPETqx1hGKzt1OVnq459ofkKsTnUmWQfBCcsE3lJkFOCsP2OJoRv2JdQFkIpMEz x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: xiSlnqu++uOoPAaTjhShWbuAz7K1hvylDpBkHlwI/QhnLfCmX6f8gHpeCpswnfvPOCr2HOR62OP6SdUeV8Ir8BUk1vXc64LH5CLheoI0Dg9CablUq8Iy6eh6EnzovH7ygdnrig86n2m7cs3WmfGEfw== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: b3580f20-047d-49de-a5ed-08d98448cbb7 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:30:43.8718 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5615 Subject: [FFmpeg-devel] [PATCH v10 02/14] avfilter/subtitles: Add subtitles.c for subtitle frame allocation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: BwcKfN6lZTy/ Analog to avfilter/video.c and avfilter/audio.c Signed-off-by: softworkz --- libavfilter/Makefile | 1 + libavfilter/avfilter.c | 4 +++ libavfilter/internal.h | 1 + libavfilter/subtitles.c | 63 +++++++++++++++++++++++++++++++++++++++++ libavfilter/subtitles.h | 44 ++++++++++++++++++++++++++++ 5 files changed, 113 insertions(+) create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h diff --git a/libavfilter/Makefile b/libavfilter/Makefile index fa2d18e45f..357926624e 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -19,6 +19,7 @@ OBJS = allfilters.o \ framequeue.o \ graphdump.o \ graphparser.o \ + subtitles.o \ video.o \ OBJS-$(HAVE_THREADS) += pthread.o diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index f325918021..ef349a67f9 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -43,6 +43,7 @@ #include "formats.h" #include "framepool.h" #include "internal.h" +#include "subtitles.h" #include "libavutil/ffversion.h" const char av_filter_ffversion[] = "FFmpeg version " FFMPEG_VERSION; @@ -1476,6 +1477,9 @@ int ff_inlink_make_frame_writable(AVFilterLink *link, AVFrame **rframe) case AVMEDIA_TYPE_AUDIO: out = ff_get_audio_buffer(link, frame->nb_samples); break; + case AVMEDIA_TYPE_SUBTITLE: + out = ff_get_subtitles_buffer(link, link->format); + break; default: return AVERROR(EINVAL); } diff --git a/libavfilter/internal.h b/libavfilter/internal.h index e7c154aff0..8977dda2b3 100644 --- a/libavfilter/internal.h +++ b/libavfilter/internal.h @@ -90,6 +90,7 @@ struct AVFilterPad { union { AVFrame *(*video)(AVFilterLink *link, int w, int h); AVFrame *(*audio)(AVFilterLink *link, int nb_samples); + AVFrame *(*subtitle)(AVFilterLink *link, int format); } get_buffer; /** diff --git a/libavfilter/subtitles.c b/libavfilter/subtitles.c new file mode 100644 index 0000000000..951bfd612c --- /dev/null +++ b/libavfilter/subtitles.c @@ -0,0 +1,63 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/common.h" + +#include "subtitles.h" +#include "avfilter.h" +#include "internal.h" + + +AVFrame *ff_null_get_subtitles_buffer(AVFilterLink *link, int format) +{ + return ff_get_subtitles_buffer(link->dst->outputs[0], format); +} + +AVFrame *ff_default_get_subtitles_buffer(AVFilterLink *link, int format) +{ + AVFrame *frame; + + frame = av_frame_alloc(); + if (!frame) + return NULL; + + frame->format = format; + frame->type = AVMEDIA_TYPE_SUBTITLE; + + if (av_frame_get_buffer2(frame, 0) < 0) { + av_frame_free(&frame); + return NULL; + } + + return frame; +} + +AVFrame *ff_get_subtitles_buffer(AVFilterLink *link, int format) +{ + AVFrame *ret = NULL; + + if (link->dstpad->get_buffer.subtitle) + ret = link->dstpad->get_buffer.subtitle(link, format); + + if (!ret) + ret = ff_default_get_subtitles_buffer(link, format); + + return ret; +} diff --git a/libavfilter/subtitles.h b/libavfilter/subtitles.h new file mode 100644 index 0000000000..4a9115126e --- /dev/null +++ b/libavfilter/subtitles.h @@ -0,0 +1,44 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVFILTER_SUBTITLES_H +#define AVFILTER_SUBTITLES_H + +#include "avfilter.h" +#include "internal.h" + +/** default handler for get_subtitles_buffer() for subtitle inputs */ +AVFrame *ff_default_get_subtitles_buffer(AVFilterLink *link, int format); + +/** get_subtitles_buffer() handler for filters which simply pass subtitles along */ +AVFrame *ff_null_get_subtitles_buffer(AVFilterLink *link, int format); + +/** + * Request a subtitles frame with a specific set of permissions. + * + * @param link the output link to the filter from which the buffer will + * be requested + * @param format The subtitles format. + * @return A reference to the frame. This must be unreferenced with + * av_frame_free when you are finished with it. +*/ +AVFrame *ff_get_subtitles_buffer(AVFilterLink *link, int format); + +#endif /* AVFILTER_SUBTITLES_H */ From patchwork Thu Sep 30 19:30:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30810 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp985119iob; Thu, 30 Sep 2021 12:31:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwqViSzSnvY83GTx5TxvStbVHsROhX/zzo7JA3AGnZMDwbUuHaJTSFrWDH6ddTgDtbVOO3z X-Received: by 2002:a50:bf4a:: with SMTP id g10mr9282622edk.11.1633030277630; Thu, 30 Sep 2021 12:31:17 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id v18si4279196ejk.45.2021.09.30.12.31.17; Thu, 30 Sep 2021 12:31:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b="K/GfzWaf"; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B7CF268AA64; Thu, 30 Sep 2021 22:31:02 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10olkn2105.outbound.protection.outlook.com [40.92.41.105]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 10A6D68A867 for ; Thu, 30 Sep 2021 22:30:56 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Alhy3QetC0ZQ5jab6i8Cc/OquA5upJcHGlVtLed5Yt34O27mJ5xII0dCQutLZp2Dc2UapakMCD7KEDi7TVOeYWXNjQM5ZfD0OrPaRmKiwegIJvEDdxC/xXpI7FTznq+MnaXANCIcQGjzOJstqIXwRwiAwSSWYqep/jyXsX1M0IMvQJLVMTENShIitnQsAi6GmzcDtK+WV652DfPXEoAy2u3WEzXcHHa0k2uKYa6Q7+k/C6BqUdSNPdQ7u6arM7H+ZhEyS90zorP5zndAWBjA3CMDqGyWjZSlFtCUf8yNcgN2ihpuwj7Ezr+bun5XKZcVMLsATH5c8yMxCmiZC39wtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=pHcADS7MhKj8pKQGYw0N0qSHLwBcr94eTzWMq6nkf98=; b=hvMZG1bLZZfwG0/m9XWYxRtk7wZopX9+zbw9pkslxmDlmNj2VkcHpKhczes9yiIJy3WWwqrJCq1oEFJtflAZ5N1UrljJo6iefiCNgZ9Kbow7Gmayc8nW0EwgCS1InasB3GQoH+Z3Yt7IVE2o29xI0F6xKMz0oG24syGgrzViY0+yrn69JesX59qpWhw0uIz0A1ebu1v3PwiNImhX9PHOJoXdLlsYPTwWtwvSnmFGJOCJ+jInQ65nVrbCQ+jQEPLJbasUTa1nlorZ6DOdUpa9orK1sNVqL0Ttt8OzplrzBbRwFbLU7WbH7mGFQj8zMLlWELIhVnaGrkLyBUU3yZxexA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pHcADS7MhKj8pKQGYw0N0qSHLwBcr94eTzWMq6nkf98=; b=K/GfzWaf+zKQWyY8nglLPPPEqY2fzUh3JmhHtBHc0OONR9VZ2fceuf+2LWEp8f2uIZDb000XA4sIOpVIVc1yDuAhcqudCr55GnJBeWmKqeIXA7pQPt2petdJ+vHGnIfMdAmOq27sWphCSZC15eCxW4xVnlgRoeuvrMNnscUTuvITBXWywtbcHFmke3l8xdMqpfg+mDUEULliXT/zH3pbaBvLolAstLlyARKDVWPGDz7Mzdm/gR1Idjpq4H+FZotWlCpsl7lO2wuphs/QE78bkzFElpSU+OBKxQLP0K7kNT5+/mwf0iaXa3hvcnOVbgcQW8uD1cxyCJ6vt8+BbpsxeQ== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5615.namprd04.prod.outlook.com (2603:10b6:208:fe::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:30:54 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:30:54 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 03/14] avfilter/avfilter: Handle subtitle frames Thread-Index: AQHXtjGvYuU9ZpiiE0qHMy3UHogcqg== Date: Thu, 30 Sep 2021 19:30:54 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [GeHlfNrClhhpjfbdc5PtTc4Kq9Rm7sNn] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0524c1fc-b7ac-42cf-8263-08d98448d1f2 x-ms-traffictypediagnostic: MN2PR04MB5615: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: onntOn53UqHkxETGmy//YjKL9lWfZlHSfuJ0OwnG5FZ5GT/TvjqhKUteVQ2sE3gVgSxD3fYBH7BY/tCU+O8jtoB44/oY2kpbuMh3wF0e9r1bkzdL8EID7ogmsPQUBuHwXNjZSXppcLGCEh6al1SambylWtGTJDTZhSknJTMrtb1cc1IrK4KiHNH62BfsVxO+WL4w3G37NPn1xS93a6eCfjvQPwwrO4N0mIPbrMJB3ao3S3SQs+y3bZ91HzSwJuJKMdM4bs32rNwqZ4ISEn/LsbYKL7u46zpVAi9DkgCW3ClQzje0BOIpHG9ePtxl3OZ0Q3HD5IU25AHTEL+4SSMgF+FreOLjqeGNkLAjcsmuvfKuckm7K1sEAmj4Bxi/NoNDKLy0WkCOwWbVrwsdbAd1Lwryw4bIX2ZXpGvk+Rq6fUMVXSp3BwpXjzum9v5omkP+ x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: z6oneUdtStdqMB8jfpjB8u+X8vQVFKD+2TQYCYE+T1IG/TcpT9JNGSD03chmxQ5CUVXroifjqzqzpfFw0OghXVewNx6wu5XPNPECQg2CxEDxKtW/znt0lJLZHzHaCKIkfzrwITMAybh0SoOlzOU5dA== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 0524c1fc-b7ac-42cf-8263-08d98448d1f2 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:30:54.3549 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5615 Subject: [FFmpeg-devel] [PATCH v10 03/14] avfilter/avfilter: Handle subtitle frames X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: SdPxAqDrSOUf Signed-off-by: softworkz --- libavfilter/avfilter.c | 8 +++++--- libavfilter/avfiltergraph.c | 5 +++++ libavfilter/formats.c | 14 ++++++++++++++ libavfilter/formats.h | 3 +++ 4 files changed, 27 insertions(+), 3 deletions(-) diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index ef349a67f9..c567b7893b 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -57,7 +57,8 @@ void ff_tlog_ref(void *ctx, AVFrame *ref, int end) ref->linesize[0], ref->linesize[1], ref->linesize[2], ref->linesize[3], ref->pts, ref->pkt_pos); - if (ref->width) { + switch(ref->type) { + case AVMEDIA_TYPE_VIDEO: ff_tlog(ctx, " a:%d/%d s:%dx%d i:%c iskey:%d type:%c", ref->sample_aspect_ratio.num, ref->sample_aspect_ratio.den, ref->width, ref->height, @@ -65,12 +66,13 @@ void ff_tlog_ref(void *ctx, AVFrame *ref, int end) ref->top_field_first ? 'T' : 'B', /* Top / Bottom */ ref->key_frame, av_get_picture_type_char(ref->pict_type)); - } - if (ref->nb_samples) { + break; + case AVMEDIA_TYPE_AUDIO: ff_tlog(ctx, " cl:%"PRId64"d n:%d r:%d", ref->channel_layout, ref->nb_samples, ref->sample_rate); + break; } ff_tlog(ctx, "]%s", end ? "\n" : ""); diff --git a/libavfilter/avfiltergraph.c b/libavfilter/avfiltergraph.c index 45b028cd9c..7a5a4ea419 100644 --- a/libavfilter/avfiltergraph.c +++ b/libavfilter/avfiltergraph.c @@ -314,6 +314,8 @@ static int filter_link_check_formats(void *log, AVFilterLink *link, AVFilterForm return ret; break; + case AVMEDIA_TYPE_SUBTITLE: + return 0; default: av_assert0(!"reached"); } @@ -444,6 +446,9 @@ static int query_formats(AVFilterGraph *graph, void *log_ctx) if (!link) continue; + if (link->type == AVMEDIA_TYPE_SUBTITLE) + continue; + neg = ff_filter_get_negotiation(link); av_assert0(neg); for (neg_step = 1; neg_step < neg->nb_mergers; neg_step++) { diff --git a/libavfilter/formats.c b/libavfilter/formats.c index 1d2a51c0af..94c1891b78 100644 --- a/libavfilter/formats.c +++ b/libavfilter/formats.c @@ -19,6 +19,7 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include "libavcodec/avcodec.h" #include "libavutil/avassert.h" #include "libavutil/channel_layout.h" #include "libavutil/common.h" @@ -430,6 +431,12 @@ int ff_add_channel_layout(AVFilterChannelLayouts **l, uint64_t channel_layout) return 0; } +int ff_add_subtitle_type(AVFilterFormats **avff, int64_t fmt) +{ + ADD_FORMAT(avff, fmt, ff_formats_unref, int, formats, nb_formats); + return 0; +} + AVFilterFormats *ff_all_formats(enum AVMediaType type) { AVFilterFormats *ret = NULL; @@ -443,6 +450,13 @@ AVFilterFormats *ff_all_formats(enum AVMediaType type) return NULL; fmt++; } + } else if (type == AVMEDIA_TYPE_SUBTITLE) { + if (ff_add_subtitle_type(&ret, AV_SUBTITLE_FMT_BITMAP) < 0) + return NULL; + if (ff_add_subtitle_type(&ret, AV_SUBTITLE_FMT_ASS) < 0) + return NULL; + if (ff_add_subtitle_type(&ret, AV_SUBTITLE_FMT_TEXT) < 0) + return NULL; } return ret; diff --git a/libavfilter/formats.h b/libavfilter/formats.h index 42fe068765..7581fae01c 100644 --- a/libavfilter/formats.h +++ b/libavfilter/formats.h @@ -180,6 +180,9 @@ int ff_set_common_formats_from_list(AVFilterContext *ctx, const int *fmts); av_warn_unused_result int ff_add_channel_layout(AVFilterChannelLayouts **l, uint64_t channel_layout); +av_warn_unused_result +int ff_add_subtitle_type(AVFilterFormats **avff, int64_t fmt); + /** * Add *ref as a new reference to f. */ From patchwork Thu Sep 30 19:30:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30802 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp985278iob; Thu, 30 Sep 2021 12:31:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzxSIDhpwFfSV5x1UAGnNIDpS3/4JteClbZPMjvvsMLnaZgJ0gPWu4/Qa86YhMiar5G37x1 X-Received: by 2002:a17:906:1405:: with SMTP id p5mr1158103ejc.197.1633030289133; Thu, 30 Sep 2021 12:31:29 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z22si3706379edm.332.2021.09.30.12.31.28; Thu, 30 Sep 2021 12:31:29 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=BI8EYgiU; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A5B2068AA7B; Thu, 30 Sep 2021 22:31:03 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10olkn2105.outbound.protection.outlook.com [40.92.41.105]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6FDDA68A867 for ; Thu, 30 Sep 2021 22:31:01 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HKEdpNx1hGKN91hV44SVttbvz4wbaIFe+J7DHfGxXM+AuXs2NOIrmAJE355lMO4sInICKpdVYfSDf//fLfiaePfzsi+EzVeHBw2U+xUVIVRMYud6SKOLUvPYb2j6PnNOFmMr83/DveAikWgU35tiw40KFV+KfFwUAoFr45rPSj5HvQLXm5qVdmW8Tk8HDZdkoJcp01actm15eKIBnMDDEdZ7b/CdKXDTWG3m3pnBehGX/jP/WSgS6dtWxo5xXV1FIWrcESVc6dHonLXl2qxDSrpYWopCmw+Aocw3tD21XL7Q18Y05lzu1eNLkzbx35nxYj4XZPqFbrwNAtwdXL+s6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=6yLagHwLtu5hjyOOjcJm3FaExHPmbKKnwiUwxjGY4gM=; b=AzU5qMH84aAcBiYYLxoyVegL5ACV5bGswhn/ph5EFFX1eFmWhLfTPKEnA0YB7cdFxSQzKCPyJj7HuM8BuukE+XIxt9WbuQFDTcWfYfHb5G+DfyAkNZrR708WCvMYz7WVAiM3+1Ew8FVjR6AlUkM8s2QrsN00Rz2w/cEZnvpuyMaW/OI5f0ZxUxwRBMTQrrj+j2IPSggBB9TUYc957kI6AsUaQz51ehO8Emy7EdmHPHQjOEVRwn03yNSfQeSVXbC31/vbWNCLm24z6Z+vddpcvIvoq2YRgcEmq8Vg+VwtTmV3/Ub83EXj7Tokx7R9Hv8+cm3QeJGA4XH6dmYJARJfvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6yLagHwLtu5hjyOOjcJm3FaExHPmbKKnwiUwxjGY4gM=; b=BI8EYgiUQ+Hum/I+62VTzfixQxdsJEaSxswA2MpnOlXyYNyOOY7IXLC1sEh556XUHlACOE1SpUmaajjgv4O6+FpXj0DJ1VmJK5rgbaQLbnZOwz+8Hnn9tOqFAQY1FTF4zynIO+isqH0JYvALzBUuQ6kwsZibqUOsSLhIehSbWOooqq5bGogCIDFN/9kedrsh5OAG0R/FhuHRQ6YYSGVwXV6JK7Osx9WMLvILJAmNwK5At7nThfWju3g3kIrqFqh53hBQf1OWnqIeCBkNipM7K019Qr/X9HaTfXUQmIl5+mMnG92VeA0+n3Vz7QlVZ3LZDYYxUx8KIK48yjxTSH0WFQ== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5615.namprd04.prod.outlook.com (2603:10b6:208:fe::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:30:57 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:30:57 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 04/14] avfilter/sbuffer: Add sbuffersrc and sbuffersink filters Thread-Index: AQHXtjGxZxmTbs/7mkGXVNTyerrpHA== Date: Thu, 30 Sep 2021 19:30:57 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> In-Reply-To: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [lB42UZFfYlRHIyJK5PZnwNWrhSfJLo4V] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: c4faf010-c3de-4886-2c77-08d98448d3a1 x-ms-traffictypediagnostic: MN2PR04MB5615: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: UyyKzV4G6oPW6G1uUzskdLGDK0ChmYjszS/DJd7Ghae6qVz8CwvwJn9IykmOMvAhJLw4t9e3B1Lo8VBbAwa10BwHyMN6T7rZ3a7jBbLcL46k5YJhrff4PHNkmLw3CLFq0gNUW+vE8Jz8eWUAkBneasjePdTT2iBIpAN7mwCrDGFFhUc79tRIjqup6Uwo4P/5LzUFwT0Z7g6S0VtMQD/0zzlPhNlB/64ZoDIwlPjYPTaKmwWW9OO5VGmRIfRNjUC0+dNbfG1Pqep0RiM542YOGiJdmGiOt+efUyP1N7mesBq9pW76gh7wI+mL3c+61r2m7RaR/YScRy00zJF/aHuQFC2lraEzYyYJTcjOg7in2lehthJ9viJyHjX7Uv6qBjuupjAf2WwwJKCHQXikdlwUmGWBPf1gCT1xKk3Oz+4aSbyt7pAgTgsWDqj8JlUX3HpW x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: 0jxlnxuPfTUxehyclQA3572xB09EMviSA9WQoUFrQ/IgDJQ18SXL5WeO99eeVip4rdfzx1A2IGpLJfj2ttd9zTbMRAt5+kX0skMQbd1anWF7qKXLrMHRAjWCOlYP4ENke7GlmTQNu5eBDGYKjtU1pg== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: c4faf010-c3de-4886-2c77-08d98448d3a1 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:30:57.1853 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5615 Subject: [FFmpeg-devel] [PATCH v10 04/14] avfilter/sbuffer: Add sbuffersrc and sbuffersink filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: b+DXuTVtRq4S Signed-off-by: softworkz --- configure | 2 +- libavfilter/allfilters.c | 2 ++ libavfilter/buffersink.c | 63 +++++++++++++++++++++++++++++++++++ libavfilter/buffersink.h | 15 +++++++++ libavfilter/buffersrc.c | 72 ++++++++++++++++++++++++++++++++++++++++ libavfilter/buffersrc.h | 1 + 6 files changed, 154 insertions(+), 1 deletion(-) diff --git a/configure b/configure index 231d0398a8..d18bb73ed6 100755 --- a/configure +++ b/configure @@ -7720,7 +7720,7 @@ print_enabled_components(){ fi done if [ "$name" = "filter_list" ]; then - for c in asrc_abuffer vsrc_buffer asink_abuffer vsink_buffer; do + for c in asrc_abuffer vsrc_buffer ssrc_sbuffer asink_abuffer vsink_buffer ssink_sbuffer; do printf " &ff_%s,\n" $c >> $TMPH done fi diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 338561a446..51a1d09b47 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -536,8 +536,10 @@ extern const AVFilter ff_avsrc_movie; * being the same while having different 'types'). */ extern const AVFilter ff_asrc_abuffer; extern const AVFilter ff_vsrc_buffer; +extern const AVFilter ff_ssrc_sbuffer; extern const AVFilter ff_asink_abuffer; extern const AVFilter ff_vsink_buffer; +extern const AVFilter ff_ssink_sbuffer; extern const AVFilter ff_af_afifo; extern const AVFilter ff_vf_fifo; diff --git a/libavfilter/buffersink.c b/libavfilter/buffersink.c index 8b46dcb15e..cd038e1782 100644 --- a/libavfilter/buffersink.c +++ b/libavfilter/buffersink.c @@ -29,6 +29,8 @@ #include "libavutil/internal.h" #include "libavutil/opt.h" +#include "libavcodec/avcodec.h" + #define FF_INTERNAL_FIELDS 1 #include "framequeue.h" @@ -57,6 +59,10 @@ typedef struct BufferSinkContext { int *sample_rates; ///< list of accepted sample rates, terminated by -1 int sample_rates_size; + /* only used for subtitles */ + enum AVSubtitleType *subtitle_types; ///< list of accepted subtitle types, must be terminated with -1 + int subtitle_types_size; + AVFrame *peeked_frame; } BufferSinkContext; @@ -168,6 +174,15 @@ AVABufferSinkParams *av_abuffersink_params_alloc(void) return NULL; return params; } + +AVSBufferSinkParams *av_sbuffersink_params_alloc(void) +{ + AVSBufferSinkParams *params = av_mallocz(sizeof(AVSBufferSinkParams)); + + if (!params) + return NULL; + return params; +} #endif static av_cold int common_init(AVFilterContext *ctx) @@ -305,6 +320,28 @@ static int asink_query_formats(AVFilterContext *ctx) return 0; } +static int ssink_query_formats(AVFilterContext *ctx) +{ + BufferSinkContext *buf = ctx->priv; + AVFilterFormats *formats = NULL; + unsigned i; + int ret; + + CHECK_LIST_SIZE(subtitle_types) + if (buf->subtitle_types_size) { + for (i = 0; i < NB_ITEMS(buf->subtitle_types); i++) + if ((ret = ff_add_subtitle_type(&formats, buf->subtitle_types[i])) < 0) + return ret; + if ((ret = ff_set_common_formats(ctx, formats)) < 0) + return ret; + } else { + if ((ret = ff_default_query_formats(ctx)) < 0) + return ret; + } + + return 0; +} + #define OFFSET(x) offsetof(BufferSinkContext, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM static const AVOption buffersink_options[] = { @@ -322,9 +359,16 @@ static const AVOption abuffersink_options[] = { { NULL }, }; #undef FLAGS +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_SUBTITLE_PARAM +static const AVOption sbuffersink_options[] = { + { "subtitle_types", "set the supported subtitle formats", OFFSET(subtitle_types), AV_OPT_TYPE_BINARY, .flags = FLAGS }, + { NULL }, +}; +#undef FLAGS AVFILTER_DEFINE_CLASS(buffersink); AVFILTER_DEFINE_CLASS(abuffersink); +AVFILTER_DEFINE_CLASS(sbuffersink); static const AVFilterPad avfilter_vsink_buffer_inputs[] = { { @@ -363,3 +407,22 @@ const AVFilter ff_asink_abuffer = { FILTER_INPUTS(avfilter_asink_abuffer_inputs), .outputs = NULL, }; + +static const AVFilterPad avfilter_ssink_sbuffer_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + }, +}; + +const AVFilter ff_ssink_sbuffer = { + .name = "sbuffersink", + .description = NULL_IF_CONFIG_SMALL("Buffer subtitle frames, and make them available to the end of the filter graph."), + .priv_class = &sbuffersink_class, + .priv_size = sizeof(BufferSinkContext), + .init = common_init, + .query_formats = ssink_query_formats, + .activate = activate, + FILTER_INPUTS(avfilter_ssink_sbuffer_inputs), + .outputs = NULL, +}; diff --git a/libavfilter/buffersink.h b/libavfilter/buffersink.h index 69ed0f29a8..b439b586c5 100644 --- a/libavfilter/buffersink.h +++ b/libavfilter/buffersink.h @@ -129,6 +129,21 @@ typedef struct AVABufferSinkParams { */ attribute_deprecated AVABufferSinkParams *av_abuffersink_params_alloc(void); + +/** + * Deprecated and unused struct to use for initializing an sbuffersink context. + */ +typedef struct AVSBufferSinkParams { + const int *subtitle_type; +} AVSBufferSinkParams; + +/** + * Create an AVSBufferSinkParams structure. + * + * Must be freed with av_free(). + */ +attribute_deprecated +AVSBufferSinkParams *av_sbuffersink_params_alloc(void); #endif /** diff --git a/libavfilter/buffersrc.c b/libavfilter/buffersrc.c index 632bfc7ad8..902a09a721 100644 --- a/libavfilter/buffersrc.c +++ b/libavfilter/buffersrc.c @@ -39,6 +39,7 @@ #include "formats.h" #include "internal.h" #include "video.h" +#include "libavcodec/avcodec.h" typedef struct BufferSourceContext { const AVClass *class; @@ -63,6 +64,9 @@ typedef struct BufferSourceContext { uint64_t channel_layout; char *channel_layout_str; + /* subtitle only */ + enum AVSubtitleType subtitle_type; + int eof; } BufferSourceContext; @@ -130,6 +134,13 @@ int av_buffersrc_parameters_set(AVFilterContext *ctx, AVBufferSrcParameters *par if (param->channel_layout) s->channel_layout = param->channel_layout; break; + case AVMEDIA_TYPE_SUBTITLE: + s->subtitle_type = param->format; + if (param->width > 0) + s->w = param->width; + if (param->height > 0) + s->h = param->height; + break; default: return AVERROR_BUG; } @@ -197,6 +208,8 @@ int attribute_align_arg av_buffersrc_add_frame_flags(AVFilterContext *ctx, AVFra CHECK_AUDIO_PARAM_CHANGE(ctx, s, frame->sample_rate, frame->channel_layout, frame->channels, frame->format, frame->pts); break; + case AVMEDIA_TYPE_SUBTITLE: + break; default: return AVERROR(EINVAL); } @@ -269,6 +282,7 @@ unsigned av_buffersrc_get_nb_failed_requests(AVFilterContext *buffer_src) #define OFFSET(x) offsetof(BufferSourceContext, x) #define A AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_AUDIO_PARAM #define V AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM +#define S AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_SUBTITLE_PARAM static const AVOption buffer_options[] = { { "width", NULL, OFFSET(w), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, V }, @@ -298,6 +312,16 @@ static const AVOption abuffer_options[] = { AVFILTER_DEFINE_CLASS(abuffer); +static const AVOption sbuffer_options[] = { + { "time_base", NULL, OFFSET(time_base), AV_OPT_TYPE_RATIONAL, { .dbl = 0 }, 0, INT_MAX, S }, + { "subtitle_type", NULL, OFFSET(subtitle_type), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, S }, + { "width", NULL, OFFSET(w), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, V }, + { "height", NULL, OFFSET(h), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, V }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(sbuffer); + static av_cold int init_audio(AVFilterContext *ctx) { BufferSourceContext *s = ctx->priv; @@ -347,6 +371,21 @@ static av_cold int init_audio(AVFilterContext *ctx) return ret; } +static av_cold int init_subtitle(AVFilterContext *ctx) +{ + BufferSourceContext *c = ctx->priv; + + if (c->subtitle_type == AV_SUBTITLE_FMT_BITMAP) + av_log(ctx, AV_LOG_VERBOSE, "graphical subtitles - w:%d h:%d tb:%d/%d\n", + c->w, c->h, c->time_base.num, c->time_base.den); + else + av_log(ctx, AV_LOG_VERBOSE, "text subtitles - w:%d h:%d tb:%d/%d\n", + c->w, c->h, c->time_base.num, c->time_base.den); + + return 0; +} + + static av_cold void uninit(AVFilterContext *ctx) { BufferSourceContext *s = ctx->priv; @@ -381,6 +420,11 @@ static int query_formats(AVFilterContext *ctx) if ((ret = ff_set_common_channel_layouts(ctx, channel_layouts)) < 0) return ret; break; + case AVMEDIA_TYPE_SUBTITLE: + if ((ret = ff_add_format (&formats, c->subtitle_type)) < 0 || + (ret = ff_set_common_formats (ctx , formats )) < 0) + return ret; + break; default: return AVERROR(EINVAL); } @@ -408,6 +452,11 @@ static int config_props(AVFilterLink *link) if (!c->channel_layout) c->channel_layout = link->channel_layout; break; + case AVMEDIA_TYPE_SUBTITLE: + link->format = c->subtitle_type; + link->w = c->w; + link->h = c->h; + break; default: return AVERROR(EINVAL); } @@ -472,3 +521,26 @@ const AVFilter ff_asrc_abuffer = { FILTER_OUTPUTS(avfilter_asrc_abuffer_outputs), .priv_class = &abuffer_class, }; + +static const AVFilterPad avfilter_ssrc_sbuffer_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .request_frame = request_frame, + .config_props = config_props, + }, +}; + +const AVFilter ff_ssrc_sbuffer = { + .name = "sbuffer", + .description = NULL_IF_CONFIG_SMALL("Buffer subtitle frames, and make them accessible to the filterchain."), + .priv_size = sizeof(BufferSourceContext), + .query_formats = query_formats, + + .init = init_subtitle, + .uninit = uninit, + + .inputs = NULL, + FILTER_OUTPUTS(avfilter_ssrc_sbuffer_outputs), + .priv_class = &sbuffer_class, +}; diff --git a/libavfilter/buffersrc.h b/libavfilter/buffersrc.h index 08fbd18a47..929a2fa249 100644 --- a/libavfilter/buffersrc.h +++ b/libavfilter/buffersrc.h @@ -74,6 +74,7 @@ typedef struct AVBufferSrcParameters { /** * video: the pixel format, value corresponds to enum AVPixelFormat * audio: the sample format, value corresponds to enum AVSampleFormat + * subtitles: the subtitle format, value corresponds to enum AVSubtitleType */ int format; /** From patchwork Thu Sep 30 19:31:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30798 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp985493iob; Thu, 30 Sep 2021 12:31:42 -0700 (PDT) X-Google-Smtp-Source: ABdhPJziSDRKk7WjViBP4+HKtBBrPXtgPs9iUYqIFwzf4SJYxPMTDUlDgQNH58Z0g38pz/0cPp7d X-Received: by 2002:a05:6402:373:: with SMTP id s19mr9275314edw.353.1633030302670; Thu, 30 Sep 2021 12:31:42 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hr27si4051579ejc.306.2021.09.30.12.31.41; Thu, 30 Sep 2021 12:31:42 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b="YIs/gLul"; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9F3F668AA7F; Thu, 30 Sep 2021 22:31:06 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12olkn2031.outbound.protection.outlook.com [40.92.23.31]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7993A68AA6E for ; Thu, 30 Sep 2021 22:31:04 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bTc7nSfqAg1R6bc20iFbfEmrpC5mK3esP53QdifHNqu1fBNAXH0JCN1auWikH/wj9hF3r/LVnn6Hok1oyXuRLbr8M/xqnZTyKZP2iRubXoxmPLjVC2L6UjexWONplZnJtC3NnwQf7uX4+tfoZ27ZU3ZJVcapaL4zpM8v/6i6sLfs6d3E3wVaVTBQ3YQNCqGWtU4JyeRulySr7x0AqN83393cL8Uy0Mf1v2D0LOCnrrsSa53QBSmKOMxHcn9MyxScl+sSmZRs3G4lqnjnxk5E/1VjsHVeUAsy7/VDsniQ2V3q88e+VPUIOyLkOqJW8R+vbv0GHXJZDv5l9mR8+pVtRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=SURMdD2LMiNpS3PghnuN/pcDcxZNtb38/8R15g7ul44=; b=RJHEgTSchX6cyiQ0R72ZMFv6gaTULFH7XxjMlfxCRa6UVq4PMXB8BMZ6CdbOZzadG50Amu+JflF579B0EJtaoScYPsvbISbmXy31uMR+dWUnuySDlAwQpC9SCjUWFvbam2NFaQoOL6Up78eR7MAByAHMBsCY4Q0Apb2A0ml1hAmatbeSKi7gg6BBV+HP14dTU5F4Gk3gRHuonwyOtPhrXquyIeX6Ka7NBeloIXdXeb7H0NuiHw3tFdPHvHf6Yma/CQNhTD8gh989mg61PxbpvmeRK2a1P9Jp2srM38ACDtyubj5fFeRDyLcW+njojPsgqIKkoG2+WL07u3+8sgpOwg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SURMdD2LMiNpS3PghnuN/pcDcxZNtb38/8R15g7ul44=; b=YIs/gLulu3EYS/XPxt3+VzGuzgrTJQ4QqudkSx9OeFbTSGYalpwY3+wFosIrPpHFcvMj6AdkSKMnl4X1qcurmjGMyvidnQ7qk1DK2Ob+h7qwQjFRN9MQXzYnuP+/54JQeaEdsTwX/xZz3Cbczg+cFvRoJIx8DBIYoziNkiuYfiAjgewWd1vkfO3o/E3rXP4q5tEw6AzBe7EJirXHdfS7pIlYlpYTm/2O/hJ26b2F2+kve63ouVXaGUkkEJ62Fd9Ot4g+X3ZfeguPlucT07exDHu0cMsaNDWCTftl7fcmms7/A4Ya0xP7x67isihWAG7VMEyIn3s90s46I0xg/o1wwg== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5615.namprd04.prod.outlook.com (2603:10b6:208:fe::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:31:00 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:31:00 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 05/14] avfilter/overlay_graphicsubs: Add overlay_graphicsubs and graphicsub2video filters Thread-Index: AQHXtjGzVZJMble+O0e6SSg5+wUKPg== Date: Thu, 30 Sep 2021 19:31:00 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [iFvkN/Kxtb7A3xQn3Jq1WOAtCt1Cq3uX] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 220d29d8-22bc-4cef-68af-08d98448d5a3 x-ms-traffictypediagnostic: MN2PR04MB5615: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: +B+yHpz1BuRIISvSUewcM2r0rOV0Enu0aCChYdus6uz4z1NrN4VTzNBp+Bz+YeI1rqcS10iw6mrH6+3pdSSa6NAQYcQXlhXAcnPjGJsFW86CXwKRgL7PxvVkN96kgMxSnl/hNJ0EOvpcZtT23QbP5Xi7F5D6zIOuvEIkBMWvvWPjYfQMwaX3HGhiom83vVcvhUEQosTCx5a5fuvturRUzq9SzBn2oDKabnbD3/J84k85fH6bfaqKP3rqN74a1xxdznZPiuX70X+UniSvdtYcpOTdikiO/Q5A5sE5XT+jUb6uJxsy6rypOoOF0DdgUX3J9gCDHORJNUtd4NPNZn0uOso2F1i2o1pH5h4mQLpHpWT94+29d7sh2L7zQVATt0HZLQo+nB3vXMUMFzMkVv+ExdZ6VMLv8wfrk165+LR306ZQIWv62LcKNDKt1XycFfB1W0dEtVPSq2DpFgdlgTGqIb+uDmc64aojdzaQkyyCyhY= x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: UTgxY5406KIcNpwkJelTMOOJof9IxJDHmwCfDB/8p3t4VLzSRO7OMlczS4qZBNvd4/dE9O+8ZQ9RQI6F/hjZ02pcD5Z5GZWkzap6e7YCgoYhur6TMLYj0bMUZbw/9DFBd8qgqM2yqEEU0Uw17gxUyQ== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 220d29d8-22bc-4cef-68af-08d98448d5a3 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:31:00.5414 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5615 Subject: [FFmpeg-devel] [PATCH v10 05/14] avfilter/overlay_graphicsubs: Add overlay_graphicsubs and graphicsub2video filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 8trMdM7b1y1h - overlay_graphicsubs (VS -> V) Overlay graphic subtitles onto a video stream - graphicsub2video {S -> V) Converts graphic subtitles to video frames (with alpha) Gets auto-inserted for retaining compatibility with sub2video command lines Signed-off-by: softworkz --- doc/filters.texi | 104 ++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 2 + libavfilter/vf_overlay_graphicsubs.c | 724 +++++++++++++++++++++++++++ 4 files changed, 832 insertions(+) create mode 100644 libavfilter/vf_overlay_graphicsubs.c diff --git a/doc/filters.texi b/doc/filters.texi index 6328a8c563..18b5273e6e 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -25195,6 +25195,110 @@ tools. @c man end VIDEO SINKS +@chapter Subtitle Filters +@c man begin SUBTITLE FILTERS + +When you configure your FFmpeg build, you can disable any of the +existing filters using @code{--disable-filters}. + +Below is a description of the currently available subtitle filters. + +@section graphicsub2video + +Renders graphic subtitles as video frames. + +This filter replaces the previous "sub2video" hack which did the conversion implicitly and up-front as subtitle filtering wasn't possible at that time. +To retain compatibility with earlier sub2video command lines, this filter is being auto-inserted in those cases. + +For overlaying graphicsal subtitles it is recommended to use the 'overlay_graphicsubs' filter which is more efficient and takes less processing resources. + +This filter is still useful in cases where the overlay is done with hardware acceleration (e.g. overlay_qsv, overlay_vaapi, overlay_cuda) for preparing the overlay frames. + +It accepts the following parameters: + +@table @option +@item size, s +Set the size of the output video frame. + +@end table + +@subsection Examples + +@itemize +@item +Overlay PGS subtitles +(not recommended - better use overlay_graphicsubs) +@example +ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:1]graphicsub2video[subs];[0:0][subs]overlay" output.mp4 +@end example + +@item +Overlay PGS subtitles implicitly +The graphicsub2video is inserted automatically for compatibility with legacy command lines. +@example +ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:0][0:1]overlay" output.mp4 +@end example +@end itemize + +@section overlay_graphicsubs + +Overlay graphic subtitles onto a video stream. + +This filter can blend graphical subtitles on a video stream directly, i.e. without creating full-size alpha images first. +The blending operation is limited to the area of the subtitle rectangles, which also means that no processing is done at times where no subtitles are to be displayed. + + +It accepts the following parameters: + +@table @option +@item x +@item y +Set the expression for the x and y coordinates of the overlaid video +on the main video. Default value is "0" for both expressions. In case +the expression is invalid, it is set to a huge value (meaning that the +overlay will not be displayed within the output visible area). + +@item eof_action +See @ref{framesync}. + +@item eval +Set when the expressions for @option{x}, and @option{y} are evaluated. + +It accepts the following values: +@table @samp +@item init +only evaluate expressions once during the filter initialization or +when a command is processed + +@item frame +evaluate expressions for each incoming frame +@end table + +Default value is @samp{frame}. + +@item shortest +See @ref{framesync}. + +@end table + +@subsection Examples + +@itemize +@item +Overlay PGS subtitles +@example +ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:1]graphicsub2video[subs];[0:0][subs]overlay" output.mp4 +@end example + +@item +Overlay PGS subtitles implicitly +The graphicsub2video is inserted automatically for compatibility with legacy command lines. +@example +ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:0][0:1]overlay" output.mp4 +@end example +@end itemize +@c man end SUBTITLE FILTERS + @chapter Multimedia Filters @c man begin MULTIMEDIA FILTERS diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 357926624e..01c3926bf8 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -290,6 +290,7 @@ OBJS-$(CONFIG_FSPP_FILTER) += vf_fspp.o qp_table.o OBJS-$(CONFIG_GBLUR_FILTER) += vf_gblur.o OBJS-$(CONFIG_GEQ_FILTER) += vf_geq.o OBJS-$(CONFIG_GRADFUN_FILTER) += vf_gradfun.o +OBJS-$(CONFIG_GRAPHICSUB2VIDEO_FILTER) += vf_overlay_graphicsubs.o framesync.o OBJS-$(CONFIG_GRAPHMONITOR_FILTER) += f_graphmonitor.o OBJS-$(CONFIG_GRAYWORLD_FILTER) += vf_grayworld.o OBJS-$(CONFIG_GREYEDGE_FILTER) += vf_colorconstancy.o @@ -364,6 +365,7 @@ OBJS-$(CONFIG_OVERLAY_CUDA_FILTER) += vf_overlay_cuda.o framesync.o vf OBJS-$(CONFIG_OVERLAY_OPENCL_FILTER) += vf_overlay_opencl.o opencl.o \ opencl/overlay.o framesync.o OBJS-$(CONFIG_OVERLAY_QSV_FILTER) += vf_overlay_qsv.o framesync.o +OBJS-$(CONFIG_OVERLAY_GRAPHICSUBS_FILTER) += vf_overlay_graphicsubs.o framesync.o OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) += vf_overlay_vulkan.o vulkan.o OBJS-$(CONFIG_OWDENOISE_FILTER) += vf_owdenoise.o OBJS-$(CONFIG_PAD_FILTER) += vf_pad.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 51a1d09b47..c08145e274 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -346,6 +346,7 @@ extern const AVFilter ff_vf_oscilloscope; extern const AVFilter ff_vf_overlay; extern const AVFilter ff_vf_overlay_opencl; extern const AVFilter ff_vf_overlay_qsv; +extern const AVFilter ff_vf_overlay_graphicsubs; extern const AVFilter ff_vf_overlay_vulkan; extern const AVFilter ff_vf_overlay_cuda; extern const AVFilter ff_vf_owdenoise; @@ -525,6 +526,7 @@ extern const AVFilter ff_avf_showvolume; extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; +extern const AVFilter ff_svf_graphicsub2video; /* multimedia sources */ extern const AVFilter ff_avsrc_amovie; diff --git a/libavfilter/vf_overlay_graphicsubs.c b/libavfilter/vf_overlay_graphicsubs.c new file mode 100644 index 0000000000..c2e7f05e11 --- /dev/null +++ b/libavfilter/vf_overlay_graphicsubs.c @@ -0,0 +1,724 @@ +/* + * Copyright (c) 2021 softworkz (derived from vf_overlay) + * Copyright (c) 2010 Stefano Sabatini + * Copyright (c) 2010 Baptiste Coudurier + * Copyright (c) 2007 Bobby Bingham + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * overlay graphical subtitles on top of a video frame + */ + +#include "libavutil/eval.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "internal.h" +#include "drawutils.h" +#include "framesync.h" + +enum var_name { + VAR_MAIN_W, VAR_MW, + VAR_MAIN_H, VAR_MH, + VAR_OVERLAY_W, VAR_OW, + VAR_OVERLAY_H, VAR_OH, + VAR_HSUB, + VAR_VSUB, + VAR_X, + VAR_Y, + VAR_N, + VAR_POS, + VAR_T, + VAR_VARS_NB +}; + +typedef struct OverlaySubsContext { + const AVClass *class; + int x, y; ///< position of overlaid picture + int w, h; + AVFrame *outpicref; + + int main_is_packed_rgb; + uint8_t main_rgba_map[4]; + int main_has_alpha; + uint8_t overlay_rgba_map[4]; + int eval_mode; ///< EvalMode + + FFFrameSync fs; + + int main_pix_step[4]; ///< steps per pixel for each plane of the main output + int hsub, vsub; ///< chroma subsampling values + const AVPixFmtDescriptor *main_desc; ///< format descriptor for main input + + double var_values[VAR_VARS_NB]; + char *x_expr, *y_expr; + + AVExpr *x_pexpr, *y_pexpr; +} OverlaySubsContext; + +static const char *const var_names[] = { + "main_w", "W", ///< width of the main video + "main_h", "H", ///< height of the main video + "overlay_w", "w", ///< width of the overlay video + "overlay_h", "h", ///< height of the overlay video + "hsub", + "vsub", + "x", + "y", + "n", ///< number of frame + "pos", ///< position in the file + "t", ///< timestamp expressed in seconds + NULL +}; + +#define MAIN 0 +#define OVERLAY 1 + +#define R 0 +#define G 1 +#define B 2 +#define A 3 + +#define Y 0 +#define U 1 +#define V 2 + +enum EvalMode { + EVAL_MODE_INIT, + EVAL_MODE_FRAME, + EVAL_MODE_NB +}; + +static av_cold void overlay_graphicsubs_uninit(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + ff_framesync_uninit(&s->fs); + av_expr_free(s->x_pexpr); s->x_pexpr = NULL; + av_expr_free(s->y_pexpr); s->y_pexpr = NULL; +} + +static inline int normalize_xy(double d, int chroma_sub) +{ + if (isnan(d)) + return INT_MAX; + return (int)d & ~((1 << chroma_sub) - 1); +} + +static void eval_expr(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->var_values[VAR_Y] = av_expr_eval(s->y_pexpr, s->var_values, NULL); + /* It is necessary if x is expressed from y */ + s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->x = normalize_xy(s->var_values[VAR_X], s->hsub); + s->y = normalize_xy(s->var_values[VAR_Y], s->vsub); +} + +static int set_expr(AVExpr **pexpr, const char *expr, const char *option, void *log_ctx) +{ + int ret; + AVExpr *old = NULL; + + if (*pexpr) + old = *pexpr; + ret = av_expr_parse(pexpr, expr, var_names, + NULL, NULL, NULL, NULL, 0, log_ctx); + if (ret < 0) { + av_log(log_ctx, AV_LOG_ERROR, + "Error when evaluating the expression '%s' for %s\n", + expr, option); + *pexpr = old; + return ret; + } + + av_expr_free(old); + return 0; +} + +static int overlay_graphicsubs_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink0 = ctx->inputs[0]; + AVFilterLink *inlink1 = ctx->inputs[1]; + AVFilterLink *outlink = ctx->outputs[0]; + int ret; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_BITMAP, AV_SUBTITLE_FMT_NONE }; + static const enum AVPixelFormat supported_pix_fmts[] = { + AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV444P, + AV_PIX_FMT_ARGB, AV_PIX_FMT_RGBA, + AV_PIX_FMT_ABGR, AV_PIX_FMT_BGRA, + AV_PIX_FMT_RGB24, AV_PIX_FMT_BGR24, + AV_PIX_FMT_NONE + }; + + /* set input0 video formats */ + formats = ff_make_format_list(supported_pix_fmts); + if ((ret = ff_formats_ref(formats, &inlink0->outcfg.formats)) < 0) + return ret; + + /* set output0 video formats */ + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + /* set input1 subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink1->outcfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + OverlaySubsContext *s = ctx->priv; + int ret; + + if ((ret = ff_framesync_init_dualinput(&s->fs, ctx)) < 0) + return ret; + + outlink->w = ctx->inputs[MAIN]->w; + outlink->h = ctx->inputs[MAIN]->h; + outlink->time_base = ctx->inputs[MAIN]->time_base; + + return ff_framesync_configure(&s->fs); +} + +// divide by 255 and round to nearest +// apply a fast variant: (X+127)/255 = ((X+127)*257+257)>>16 = ((X+128)*257)>>16 +#define FAST_DIV255(x) ((((x) + 128) * 257) >> 16) + +// calculate the non-pre-multiplied alpha, applying the general equation: +// alpha = alpha_overlay / ( (alpha_main + alpha_overlay) - (alpha_main * alpha_overlay) ) +// (((x) << 16) - ((x) << 9) + (x)) is a faster version of: 255 * 255 * x +// ((((x) + (y)) << 8) - ((x) + (y)) - (y) * (x)) is a faster version of: 255 * (x + y) +#define UNPREMULTIPLY_ALPHA(x, y) ((((x) << 16) - ((x) << 9) + (x)) / ((((x) + (y)) << 8) - ((x) + (y)) - (y) * (x))) + +/** + * Blend image in src to destination buffer dst at position (x, y). + */ +static av_always_inline void blend_packed_rgb(const AVFilterContext *ctx, + const AVFrame *dst, const AVSubtitleArea *src, + int x, int y, + int is_straight) +{ + OverlaySubsContext *s = ctx->priv; + int i, imax, j, jmax; + const int src_w = src->w; + const int src_h = src->h; + const int dst_w = dst->width; + const int dst_h = dst->height; + uint8_t alpha; ///< the amount of overlay to blend on to main + const int dr = s->main_rgba_map[R]; + const int dg = s->main_rgba_map[G]; + const int db = s->main_rgba_map[B]; + const int da = s->main_rgba_map[A]; + const int dstep = s->main_pix_step[0]; + const int sr = s->overlay_rgba_map[R]; + const int sg = s->overlay_rgba_map[G]; + const int sb = s->overlay_rgba_map[B]; + const int sa = s->overlay_rgba_map[A]; + int slice_start, slice_end; + uint8_t *S, *sp, *d, *dp; + + i = FFMAX(-y, 0); + imax = FFMIN3(-y + dst_h, FFMIN(src_h, dst_h), y + src_h); + + slice_start = i; + slice_end = i + imax; + + sp = src->buf[0]->data + slice_start * src->linesize[0]; + dp = dst->data[0] + (slice_start + y) * dst->linesize[0]; + + for (i = slice_start; i < slice_end; i++) { + j = FFMAX(-x, 0); + S = sp + j; + d = dp + ((x + j) * dstep); + + for (jmax = FFMIN(-x + dst_w, src_w); j < jmax; j++) { + uint32_t val = src->pal[*S]; + const uint8_t *sval = (uint8_t *)&val; + alpha = sval[sa]; + + // if the main channel has an alpha channel, alpha has to be calculated + // to create an un-premultiplied (straight) alpha value + if (s->main_has_alpha && alpha != 0 && alpha != 255) { + const uint8_t alpha_d = d[da]; + alpha = UNPREMULTIPLY_ALPHA(alpha, alpha_d); + } + + switch (alpha) { + case 0: + break; + case 255: + d[dr] = sval[sr]; + d[dg] = sval[sg]; + d[db] = sval[sb]; + break; + default: + // main_value = main_value * (1 - alpha) + overlay_value * alpha + // since alpha is in the range 0-255, the result must divided by 255 + d[dr] = is_straight ? FAST_DIV255(d[dr] * (255 - alpha) + sval[sr] * alpha) : + FFMIN(FAST_DIV255(d[dr] * (255 - alpha)) + sval[sr], 255); + d[dg] = is_straight ? FAST_DIV255(d[dg] * (255 - alpha) + sval[sg] * alpha) : + FFMIN(FAST_DIV255(d[dg] * (255 - alpha)) + sval[sg], 255); + d[db] = is_straight ? FAST_DIV255(d[db] * (255 - alpha) + sval[sb] * alpha) : + FFMIN(FAST_DIV255(d[db] * (255 - alpha)) + sval[sb], 255); + } + + if (s->main_has_alpha) { + switch (alpha) { + case 0: + break; + case 255: + d[da] = sval[sa]; + break; + default: + // apply alpha compositing: main_alpha += (1-main_alpha) * overlay_alpha + d[da] += FAST_DIV255((255 - d[da]) * S[sa]); + } + } + d += dstep; + S += 1; + } + dp += dst->linesize[0]; + sp += src->linesize[0]; + } +} + +static av_always_inline void blend_plane_8_8bits(const AVFilterContext *ctx, const AVFrame *dst, const AVSubtitleArea *area, + const uint32_t *yuv_pal, int src_w, int src_h, int dst_w, int dst_h, int plane, int hsub, int vsub, + int x, int y, int dst_plane, int dst_offset, int dst_step) +{ + const int src_wp = AV_CEIL_RSHIFT(src_w, hsub); + const int src_hp = AV_CEIL_RSHIFT(src_h, vsub); + const int dst_wp = AV_CEIL_RSHIFT(dst_w, hsub); + const int dst_hp = AV_CEIL_RSHIFT(dst_h, vsub); + const int yp = y >> vsub; + const int xp = x >> hsub; + uint8_t *s, *sp, *d, *dp, *dap; + int imax, i, j, jmax; + int slice_start, slice_end; + + i = FFMAX(-yp, 0); \ + imax = FFMIN3(-yp + dst_hp, FFMIN(src_hp, dst_hp), yp + src_hp); \ + + slice_start = i; + slice_end = i + imax; + + sp = area->buf[0]->data + (slice_start << vsub) * area->linesize[0]; + dp = dst->data[dst_plane] + (yp + slice_start) * dst->linesize[dst_plane] + dst_offset; + + dap = dst->data[3] + ((yp + slice_start) << vsub) * dst->linesize[3]; + + for (i = slice_start; i < slice_end; i++) { + j = FFMAX(-xp, 0); + d = dp + (xp + j) * dst_step; + s = sp + (j << hsub); + jmax = FFMIN(-xp + dst_wp, src_wp); + + for (; j < jmax; j++) { + uint32_t val = yuv_pal[*s]; + const uint8_t *sval = (uint8_t *)&val; + const int alpha = sval[3]; + const int max = 255, mid = 128; + const int d_int = *d; + const int sval_int = sval[plane]; + + switch (alpha) { + case 0: + break; + case 255: + *d = sval[plane]; + break; + default: + if (plane > 0) + *d = av_clip(FAST_DIV255((d_int - mid) * (max - alpha) + (sval_int - mid) * alpha) , -mid, mid) + mid; + else + *d = FAST_DIV255(d_int * (max - alpha) + sval_int * alpha); + break; + } + + d += dst_step; + s += 1 << hsub; + } + dp += dst->linesize[dst_plane]; + sp += (1 << vsub) * area->linesize[0]; + dap += (1 << vsub) * dst->linesize[3]; + } +} + +#define RGB2Y(r, g, b) (uint8_t)(((66 * (r) + 129 * (g) + 25 * (b) + 128) >> 8) + 16) +#define RGB2U(r, g, b) (uint8_t)(((-38 * (r) - 74 * (g) + 112 * (b) + 128) >> 8) + 128) +#define RGB2V(r, g, b) (uint8_t)(((112 * (r) - 94 * (g) - 18 * (b) + 128) >> 8) + 128) +/* Converts R8 G8 B8 color to YUV. */ +static av_always_inline void rgb_2_yuv(uint8_t r, uint8_t g, uint8_t b, uint8_t* y, uint8_t* u, uint8_t* v) +{ + *y = RGB2Y((int)r, (int)g, (int)b); + *u = RGB2U((int)r, (int)g, (int)b); + *v = RGB2V((int)r, (int)g, (int)b); +} + + +static av_always_inline void blend_yuv_8_8bits(AVFilterContext *ctx, AVFrame *dst, const AVSubtitleArea *area, int hsub, int vsub, int x, int y) +{ + OverlaySubsContext *s = ctx->priv; + const int src_w = area->w; + const int src_h = area->h; + const int dst_w = dst->width; + const int dst_h = dst->height; + const int sr = s->overlay_rgba_map[R]; + const int sg = s->overlay_rgba_map[G]; + const int sb = s->overlay_rgba_map[B]; + const int sa = s->overlay_rgba_map[A]; + uint32_t yuvpal[256]; + + for (int i = 0; i < 256; ++i) { + const uint8_t *rgba = (const uint8_t *)&area->pal[i]; + uint8_t *yuva = (uint8_t *)&yuvpal[i]; + rgb_2_yuv(rgba[sr], rgba[sg], rgba[sb], &yuva[Y], &yuva[U], &yuva[V]); + yuva[3] = rgba[sa]; + } + + blend_plane_8_8bits(ctx, dst, area, yuvpal, src_w, src_h, dst_w, dst_h, Y, 0, 0, x, y, s->main_desc->comp[Y].plane, s->main_desc->comp[Y].offset, s->main_desc->comp[Y].step); + blend_plane_8_8bits(ctx, dst, area, yuvpal, src_w, src_h, dst_w, dst_h, U, hsub, vsub, x, y, s->main_desc->comp[U].plane, s->main_desc->comp[U].offset, s->main_desc->comp[U].step); + blend_plane_8_8bits(ctx, dst, area, yuvpal, src_w, src_h, dst_w, dst_h, V, hsub, vsub, x, y, s->main_desc->comp[V].plane, s->main_desc->comp[V].offset, s->main_desc->comp[V].step); +} + +static int config_input_main(AVFilterLink *inlink) +{ + int ret; + AVFilterContext *ctx = inlink->dst; + OverlaySubsContext *s = inlink->dst->priv; + const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(inlink->format); + + av_image_fill_max_pixsteps(s->main_pix_step, NULL, pix_desc); + ff_fill_rgba_map(s->overlay_rgba_map, AV_PIX_FMT_RGB32); // it's actually AV_PIX_FMT_PAL8); + + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + + s->main_desc = pix_desc; + + s->main_is_packed_rgb = ff_fill_rgba_map(s->main_rgba_map, inlink->format) >= 0; + s->main_has_alpha = !!(pix_desc->flags & AV_PIX_FMT_FLAG_ALPHA); + + /* Finish the configuration by evaluating the expressions + now when both inputs are configured. */ + s->var_values[VAR_MAIN_W ] = s->var_values[VAR_MW] = ctx->inputs[MAIN ]->w; + s->var_values[VAR_MAIN_H ] = s->var_values[VAR_MH] = ctx->inputs[MAIN ]->h; + s->var_values[VAR_OVERLAY_W] = s->var_values[VAR_OW] = ctx->inputs[OVERLAY]->w; + s->var_values[VAR_OVERLAY_H] = s->var_values[VAR_OH] = ctx->inputs[OVERLAY]->h; + s->var_values[VAR_HSUB] = 1<log2_chroma_w; + s->var_values[VAR_VSUB] = 1<log2_chroma_h; + s->var_values[VAR_X] = NAN; + s->var_values[VAR_Y] = NAN; + s->var_values[VAR_N] = 0; + s->var_values[VAR_T] = NAN; + s->var_values[VAR_POS] = NAN; + + if ((ret = set_expr(&s->x_pexpr, s->x_expr, "x", ctx)) < 0 || + (ret = set_expr(&s->y_pexpr, s->y_expr, "y", ctx)) < 0) + return ret; + + if (s->eval_mode == EVAL_MODE_INIT) { + eval_expr(ctx); + av_log(ctx, AV_LOG_VERBOSE, "x:%f xi:%d y:%f yi:%d\n", + s->var_values[VAR_X], s->x, + s->var_values[VAR_Y], s->y); + } + + av_log(ctx, AV_LOG_VERBOSE, + "main w:%d h:%d fmt:%s overlay w:%d h:%d fmt:%s\n", + ctx->inputs[MAIN]->w, ctx->inputs[MAIN]->h, + av_get_pix_fmt_name(ctx->inputs[MAIN]->format), + ctx->inputs[OVERLAY]->w, ctx->inputs[OVERLAY]->h, + av_get_pix_fmt_name(ctx->inputs[OVERLAY]->format)); + return 0; +} + +static int do_blend(FFFrameSync *fs) +{ + AVFilterContext *ctx = fs->parent; + AVFrame *mainpic, *second; + OverlaySubsContext *s = ctx->priv; + AVFilterLink *inlink = ctx->inputs[0]; + unsigned i; + int ret; + + ret = ff_framesync_dualinput_get_writable(fs, &mainpic, &second); + if (ret < 0) + return ret; + if (!second) + return ff_filter_frame(ctx->outputs[0], mainpic); + + if (s->eval_mode == EVAL_MODE_FRAME) { + int64_t pos = mainpic->pkt_pos; + + s->var_values[VAR_N] = (double)inlink->frame_count_out; + s->var_values[VAR_T] = mainpic->pts == AV_NOPTS_VALUE ? + NAN :(double)mainpic->pts * av_q2d(inlink->time_base); + s->var_values[VAR_POS] = pos == -1 ? NAN : (double)pos; + + s->var_values[VAR_OVERLAY_W] = s->var_values[VAR_OW] = second->width; + s->var_values[VAR_OVERLAY_H] = s->var_values[VAR_OH] = second->height; + s->var_values[VAR_MAIN_W ] = s->var_values[VAR_MW] = mainpic->width; + s->var_values[VAR_MAIN_H ] = s->var_values[VAR_MH] = mainpic->height; + + eval_expr(ctx); + av_log(ctx, AV_LOG_DEBUG, "n:%f t:%f pos:%f x:%f xi:%d y:%f yi:%d\n", + s->var_values[VAR_N], s->var_values[VAR_T], s->var_values[VAR_POS], + s->var_values[VAR_X], s->x, + s->var_values[VAR_Y], s->y); + } + + for (i = 0; i < second->num_subtitle_areas; i++) { + const AVSubtitleArea *sub_area = second->subtitle_areas[i]; + + if (sub_area->type != AV_SUBTITLE_FMT_BITMAP) { + av_log(NULL, AV_LOG_WARNING, "overlay_graphicsub: non-bitmap subtitle\n"); + return AVERROR_INVALIDDATA; + } + + switch (inlink->format) { + case AV_PIX_FMT_YUV420P: + blend_yuv_8_8bits(ctx, mainpic, sub_area, 1, 1, sub_area->x + s->x, sub_area->y + s->y); + break; + case AV_PIX_FMT_YUV422P: + blend_yuv_8_8bits(ctx, mainpic, sub_area, 1, 0, sub_area->x + s->x, sub_area->y + s->y); + break; + case AV_PIX_FMT_YUV444P: + blend_yuv_8_8bits(ctx, mainpic, sub_area, 0, 0, sub_area->x + s->x, sub_area->y + s->y); + break; + case AV_PIX_FMT_RGB24: + case AV_PIX_FMT_BGR24: + case AV_PIX_FMT_ARGB: + case AV_PIX_FMT_RGBA: + case AV_PIX_FMT_BGRA: + case AV_PIX_FMT_ABGR: + blend_packed_rgb(ctx, mainpic, sub_area, sub_area->x + s->x, sub_area->y + s->y, 1); + break; + default: + av_log(NULL, AV_LOG_ERROR, "Unsupported input pix fmt: %d\n", inlink->format); + return AVERROR(EINVAL); + } + } + + return ff_filter_frame(ctx->outputs[0], mainpic); +} + +static av_cold int overlay_graphicsubs_init(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + + s->fs.on_event = do_blend; + return 0; +} + +static int overlay_graphicsubs_activate(AVFilterContext *ctx) +{ + OverlaySubsContext *s = ctx->priv; + return ff_framesync_activate(&s->fs); +} + +static int graphicsub2video_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_BITMAP, AV_SUBTITLE_FMT_NONE }; + static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_RGB32, AV_PIX_FMT_NONE }; + int ret; + + /* set input subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output video formats */ + formats = ff_make_format_list(pix_fmts); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int graphicsub2video_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + OverlaySubsContext *s = ctx->priv; + + if (s->w <= 0 || s->h <= 0) { + s->w = inlink->w; + s->h = inlink->h; + } + return 0; +} + +static int graphicsub2video_config_output(AVFilterLink *outlink) +{ + const AVFilterContext *ctx = outlink->src; + OverlaySubsContext *s = ctx->priv; + const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(outlink->format); + + outlink->w = s->w; + outlink->h = s->h; + outlink->sample_aspect_ratio = (AVRational){1,1}; + + av_image_fill_max_pixsteps(s->main_pix_step, NULL, pix_desc); + ff_fill_rgba_map(s->overlay_rgba_map, AV_PIX_FMT_RGB32); + + s->hsub = pix_desc->log2_chroma_w; + s->vsub = pix_desc->log2_chroma_h; + + s->main_desc = pix_desc; + + s->main_is_packed_rgb = ff_fill_rgba_map(s->main_rgba_map, outlink->format) >= 0; + s->main_has_alpha = !!(pix_desc->flags & AV_PIX_FMT_FLAG_ALPHA); + + return 0; +} + +static int graphicsub2video_filter_frame(AVFilterLink *inlink, AVFrame *src_frame) +{ + AVFilterLink *outlink = inlink->dst->outputs[0]; + AVFrame *out; + const unsigned num_rects = src_frame->num_subtitle_areas; + unsigned int i; + + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) { + av_frame_free(&src_frame); + return AVERROR(ENOMEM); + } + + memset(out->data[0], 0, out->linesize[0] * out->height); + + out->pts = src_frame->pts; + + for (i = 0; i < num_rects; i++) { + const AVSubtitleArea *sub_rect = src_frame->subtitle_areas[i]; + + if (sub_rect->type != AV_SUBTITLE_FMT_BITMAP) { + av_log(NULL, AV_LOG_WARNING, "sub2video: non-bitmap subtitle\n"); + av_frame_free(&src_frame); + return AVERROR_INVALIDDATA; + } + + blend_packed_rgb(inlink->dst, out, sub_rect, sub_rect->x, sub_rect->y, 1); + } + + av_frame_free(&src_frame); + return ff_filter_frame(outlink, out); +} + +#define OFFSET(x) offsetof(OverlaySubsContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption overlay_graphicsubs_options[] = { + { "x", "set the x expression", OFFSET(x_expr), AV_OPT_TYPE_STRING, {.str = "0"}, 0, 0, FLAGS }, + { "y", "set the y expression", OFFSET(y_expr), AV_OPT_TYPE_STRING, {.str = "0"}, 0, 0, FLAGS }, + { "eof_action", "Action to take when encountering EOF from secondary input ", + OFFSET(fs.opt_eof_action), AV_OPT_TYPE_INT, { .i64 = EOF_ACTION_REPEAT }, + EOF_ACTION_REPEAT, EOF_ACTION_PASS, .flags = FLAGS, "eof_action" }, + { "repeat", "Repeat the previous frame.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_REPEAT }, .flags = FLAGS, "eof_action" }, + { "endall", "End both streams.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_ENDALL }, .flags = FLAGS, "eof_action" }, + { "pass", "Pass through the main input.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_PASS }, .flags = FLAGS, "eof_action" }, + { "eval", "specify when to evaluate expressions", OFFSET(eval_mode), AV_OPT_TYPE_INT, {.i64 = EVAL_MODE_FRAME}, 0, EVAL_MODE_NB-1, FLAGS, "eval" }, + { "init", "eval expressions once during initialization", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_INIT}, .flags = FLAGS, .unit = "eval" }, + { "frame", "eval expressions per-frame", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_FRAME}, .flags = FLAGS, .unit = "eval" }, + { "shortest", "force termination when the shortest input terminates", OFFSET(fs.opt_shortest), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, + { "repeatlast", "repeat overlay of the last overlay frame", OFFSET(fs.opt_repeatlast), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, FLAGS }, + { NULL } +}; + +static const AVOption graphicsub2video_options[] = { + { "size", "set video size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, FLAGS }, + { "s", "set video size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, FLAGS }, + { NULL } +}; + +FRAMESYNC_DEFINE_CLASS(overlay_graphicsubs, OverlaySubsContext, fs); + +static const AVFilterPad overlay_graphicsubs_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input_main, + .flags = AVFILTERPAD_FLAG_NEEDS_WRITABLE, + }, + { + .name = "overlay", + .type = AVMEDIA_TYPE_SUBTITLE, + }, +}; + +static const AVFilterPad overlay_graphicsubs_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +const AVFilter ff_vf_overlay_graphicsubs = { + .name = "overlay_graphicsubs", + .description = NULL_IF_CONFIG_SMALL("Overlay graphical subtitles on top of the input."), + .preinit = overlay_graphicsubs_framesync_preinit, + .init = overlay_graphicsubs_init, + .uninit = overlay_graphicsubs_uninit, + .priv_size = sizeof(OverlaySubsContext), + .priv_class = &overlay_graphicsubs_class, + .query_formats = overlay_graphicsubs_query_formats, + .activate = overlay_graphicsubs_activate, + FILTER_INPUTS(overlay_graphicsubs_inputs), + FILTER_OUTPUTS(overlay_graphicsubs_outputs), +}; + +AVFILTER_DEFINE_CLASS(graphicsub2video); + +static const AVFilterPad graphicsub2video_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = graphicsub2video_filter_frame, + .config_props = graphicsub2video_config_input, + }, +}; + +static const AVFilterPad graphicsub2video_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = graphicsub2video_config_output, + }, +}; + +const AVFilter ff_svf_graphicsub2video = { + .name = "graphicsub2video", + .description = NULL_IF_CONFIG_SMALL("Convert graphical subtitles to video"), + .query_formats = graphicsub2video_query_formats, + .priv_size = sizeof(OverlaySubsContext), + .priv_class = &graphicsub2video_class, + FILTER_INPUTS(graphicsub2video_inputs), + FILTER_OUTPUTS(graphicsub2video_outputs), +}; From patchwork Thu Sep 30 19:31:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30799 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp985691iob; Thu, 30 Sep 2021 12:31:57 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy9f+H1Vh9cbxAWjmiuA/KiFbh3fo+DxWFLDmQK3awYhmSXj5xR04ZQJRCgK0KE2ttvFqRL X-Received: by 2002:a05:6402:1e93:: with SMTP id f19mr7859378edf.88.1633030317011; Thu, 30 Sep 2021 12:31:57 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f2si5411207edl.338.2021.09.30.12.31.56; Thu, 30 Sep 2021 12:31:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=r0oddGhH; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EACA268AAA8; Thu, 30 Sep 2021 22:31:09 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12olkn2103.outbound.protection.outlook.com [40.92.23.103]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AD7BF68A867 for ; Thu, 30 Sep 2021 22:31:07 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IavzmkBndlZFm6s3gLLyj9zqIxDEr1dMHthaeg8Tdffg62hgTBQP/FIPDUU7o836F0zY1yRksWZ/PSviA/1Sxdd+3VUGccMZ8vz7EUNT1k7uYdnw3SyDKUljQKPaO6xMeuCNnDIYHr43eDaOIAzj/TP/r+Rq9xBzx3iu3DEgQbTSfFVoF61QIEC6wvhoOWuCRsZidKMS2xPE6v/AIoX3EbtuQQND5hRqw7OfJ6WevZ+5yPnHWsGLTKZQ4qVI3HjnyaSVQYp8t+3yGq04Yy+/aa7l2h1jYJtWwLKkQHtltgeJny2dHoWthmrRuduW6ko7/0GM4qK1Ry/6y+VMeiRI5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=XFj+GTsasenxNhhU34FN0SpMumymGASI/l9s1lPfJ6s=; b=gDdN3xFdRwAa49yO7eHGJM//P1aUb+B6H7bBLtJOupEtuYTGQgf7dh6/bLigpyTd9e8+kT5veyuA3nLjthLzXS+4wXJUZlczoMrKc2V/wkl893W1NZMHLCXdxECfNt3zJvUoMHSz85oT+o/YJ98x/dZaxwFmNnZKF5CbdnD5yjE6yTkSnXAdmqr5B+gBDgSUxtHZq3k8wdRw+jnR5c9ouBqb4//1a7+6pBc9oOFtYQEOOY4e4RYKIwFvKnrsSqMZEuj/We2YaFEk9hToTrxs7RiQ67caFpkoBINCc7b2Y69Irq90KXmOqZ3e0NveOchSb3HkjHjqu4SfSGxXef1mNg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XFj+GTsasenxNhhU34FN0SpMumymGASI/l9s1lPfJ6s=; b=r0oddGhHPXUpn0/PCtpF4y6WcwP2aFVrnGn7gh44bUZpwwYTyZxS0lH0yGuvClfJgdtq3IFkxurZxyW1qf6WFpLbJJQnBdFeZizaahDQDo8+/2L7Vj9Eq3MLDg++ifZIuZB/0SC6fw8vZhluFb6gls8n9CH7qmuoEp5gxAS/dupPJbu1OYZVFdEEcBsyST47rValZjNPNfCML510g8qEGSxWQ1v/y4eGQx3oKifjTPaPo4maI7khgThE8lwwa81fI3GAFe1qf2pZbtu64yj/3s3CqN47zLY91uS8zAhxL9hXVdde4q27OnznKXgDie+DCiRtnbVWqtqO6HVl+msc+w== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5615.namprd04.prod.outlook.com (2603:10b6:208:fe::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:31:03 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:31:03 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 06/14] fftools/ffmpeg: Replace sub2video with subtitle frame filtering Thread-Index: AQHXtjG0nrSZfDGVIU+NIUMZ2iMD6g== Date: Thu, 30 Sep 2021 19:31:03 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [7CG6o3b7wJ59wD/nB7Vls+40xKPltu4V] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3b21908c-6430-4e9e-82cc-08d98448d75d x-ms-traffictypediagnostic: MN2PR04MB5615: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: pfc3O4PHrCNqY2oAu2p7gcBvhX1a1p/ScseWFIrUEe8Mm67QgQOlIKTU70bk0BsySzRd9NyXkC/Qra3ao/c6yXDQPX9Q5B28WYiaMnE+jxQ9KY4MfjvYnsH8DH5uOlUls/qfCWf+WlfG5TW45OpdAt3PkjJJ9zSkin9CoyfijIpm3LROsaEAAUD8/cmsabbYIze06T37XCQ/7VkhjjF0ESkSlvCh1EPo6dIhbAUVL7gLDZI7ekT1jIdbEIpybm0FQeOC3YxtSj0Q6fCpvkpnjS3dDCuUnzikqKXtNNCwg3UxvseBCXg6Ndl+XmemCzGcFpyU/sdVnH37eH1B7RDh+ts3FJNyLsZuekEULIo5eCOs1iwlcCN/ttXOeLO3eHVKjTf9ePQwyOCqv6dwUrxdQ8qJkb0I61+79i3KmeCwt77YPiW6nuZVmNeoHjEfdEN6 x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: 388rxBTOhbN7cpD5NudDV8IrAQcr6HoYlSLQd/jOP//fTIusnA4vloT/ke8k1BASJxyEMapRIdg8PbTEKldlSA+OJwwxT9HV4G3cJagOo4ehx8DrQpUi1jboI/dL3X4Y4AkyFomg9HyiAN4gOeYCBg== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 3b21908c-6430-4e9e-82cc-08d98448d75d X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:31:03.4317 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5615 Subject: [FFmpeg-devel] [PATCH v10 06/14] fftools/ffmpeg: Replace sub2video with subtitle frame filtering X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: pk1giqkMtGlY This commit actually enables subtitle filtering in ffmpeg by sending and receiving subtitle frames to and from a filtergraph. The heartbeat functionality from the previous sub2video implementation is retained and applied to all subtitle frames (bitmap, text, ..). The other part of sub2video functionality is retained by auto-insertion of the new graphicsub2video filter. Justification for changed test refs: - sub2video The new results are identical excepting the last frame which is due to the implementation changes - sub2video_basic The previous results had some incorrect output because multiple frames had the same dts (even the first one was incorrect). The non-empty content frames are visually identical, the different CRC is due to the different blending algorithm that is being used. - sub2video_time_limited The previous results were incorrect. There was an initial empty frame with the same dts generated. This is no longer happening. The third frame was a repetition, which doesn't happen anymore with the new subtitle filtering. The source file contains a single subtitle event and the new and correct result is a single output frame - sub-dvb Running ffprobe -show_frames on the source file shows that there are 7 subtitle frames with 0 rects in the source at the start and 2 at the end. This translates to the 14 and 4 additional entries in the new test results. - filter-overlay-dvdsub-2397 Overlay results have slightly different CRCs due to different blending implementation Signed-off-by: softworkz --- fftools/ffmpeg.c | 507 ++++++++++------------ fftools/ffmpeg.h | 14 +- fftools/ffmpeg_filter.c | 209 ++++++--- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- tests/ref/fate/filter-overlay-dvdsub-2397 | 181 ++++---- tests/ref/fate/sub-dvb | 162 ++++--- tests/ref/fate/sub2video | 1 - tests/ref/fate/sub2video_basic | 93 ++-- tests/ref/fate/sub2video_time_limited | 4 +- 10 files changed, 622 insertions(+), 554 deletions(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index 98c2421938..6e7f621d61 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -169,163 +169,6 @@ static int restore_tty; static void free_input_threads(void); #endif -/* sub2video hack: - Convert subtitles to video with alpha to insert them in filter graphs. - This is a temporary solution until libavfilter gets real subtitles support. - */ - -static int sub2video_get_blank_frame(InputStream *ist) -{ - int ret; - AVFrame *frame = ist->sub2video.frame; - - av_frame_unref(frame); - ist->sub2video.frame->width = ist->dec_ctx->width ? ist->dec_ctx->width : ist->sub2video.w; - ist->sub2video.frame->height = ist->dec_ctx->height ? ist->dec_ctx->height : ist->sub2video.h; - ist->sub2video.frame->format = AV_PIX_FMT_RGB32; - if ((ret = av_frame_get_buffer(frame, 0)) < 0) - return ret; - memset(frame->data[0], 0, frame->height * frame->linesize[0]); - return 0; -} - -static void sub2video_copy_rect(uint8_t *dst, int dst_linesize, int w, int h, - AVSubtitleRect *r) -{ - uint32_t *pal, *dst2; - uint8_t *src, *src2; - int x, y; - - if (r->type != SUBTITLE_BITMAP) { - av_log(NULL, AV_LOG_WARNING, "sub2video: non-bitmap subtitle\n"); - return; - } - if (r->x < 0 || r->x + r->w > w || r->y < 0 || r->y + r->h > h) { - av_log(NULL, AV_LOG_WARNING, "sub2video: rectangle (%d %d %d %d) overflowing %d %d\n", - r->x, r->y, r->w, r->h, w, h - ); - return; - } - - dst += r->y * dst_linesize + r->x * 4; - src = r->data[0]; - pal = (uint32_t *)r->data[1]; - for (y = 0; y < r->h; y++) { - dst2 = (uint32_t *)dst; - src2 = src; - for (x = 0; x < r->w; x++) - *(dst2++) = pal[*(src2++)]; - dst += dst_linesize; - src += r->linesize[0]; - } -} - -static void sub2video_push_ref(InputStream *ist, int64_t pts) -{ - AVFrame *frame = ist->sub2video.frame; - int i; - int ret; - - av_assert1(frame->data[0]); - ist->sub2video.last_pts = frame->pts = pts; - for (i = 0; i < ist->nb_filters; i++) { - ret = av_buffersrc_add_frame_flags(ist->filters[i]->filter, frame, - AV_BUFFERSRC_FLAG_KEEP_REF | - AV_BUFFERSRC_FLAG_PUSH); - if (ret != AVERROR_EOF && ret < 0) - av_log(NULL, AV_LOG_WARNING, "Error while add the frame to buffer source(%s).\n", - av_err2str(ret)); - } -} - -void sub2video_update(InputStream *ist, int64_t heartbeat_pts, AVSubtitle *sub) -{ - AVFrame *frame = ist->sub2video.frame; - int8_t *dst; - int dst_linesize; - int num_rects, i; - int64_t pts, end_pts; - - if (!frame) - return; - if (sub) { - pts = av_rescale_q(sub->pts + sub->start_display_time * 1000LL, - AV_TIME_BASE_Q, ist->st->time_base); - end_pts = av_rescale_q(sub->pts + sub->end_display_time * 1000LL, - AV_TIME_BASE_Q, ist->st->time_base); - num_rects = sub->num_rects; - } else { - /* If we are initializing the system, utilize current heartbeat - PTS as the start time, and show until the following subpicture - is received. Otherwise, utilize the previous subpicture's end time - as the fall-back value. */ - pts = ist->sub2video.initialize ? - heartbeat_pts : ist->sub2video.end_pts; - end_pts = INT64_MAX; - num_rects = 0; - } - if (sub2video_get_blank_frame(ist) < 0) { - av_log(ist->dec_ctx, AV_LOG_ERROR, - "Impossible to get a blank canvas.\n"); - return; - } - dst = frame->data [0]; - dst_linesize = frame->linesize[0]; - for (i = 0; i < num_rects; i++) - sub2video_copy_rect(dst, dst_linesize, frame->width, frame->height, sub->rects[i]); - sub2video_push_ref(ist, pts); - ist->sub2video.end_pts = end_pts; - ist->sub2video.initialize = 0; -} - -static void sub2video_heartbeat(InputStream *ist, int64_t pts) -{ - InputFile *infile = input_files[ist->file_index]; - int i, j, nb_reqs; - int64_t pts2; - - /* When a frame is read from a file, examine all sub2video streams in - the same file and send the sub2video frame again. Otherwise, decoded - video frames could be accumulating in the filter graph while a filter - (possibly overlay) is desperately waiting for a subtitle frame. */ - for (i = 0; i < infile->nb_streams; i++) { - InputStream *ist2 = input_streams[infile->ist_index + i]; - if (!ist2->sub2video.frame) - continue; - /* subtitles seem to be usually muxed ahead of other streams; - if not, subtracting a larger time here is necessary */ - pts2 = av_rescale_q(pts, ist->st->time_base, ist2->st->time_base) - 1; - /* do not send the heartbeat frame if the subtitle is already ahead */ - if (pts2 <= ist2->sub2video.last_pts) - continue; - if (pts2 >= ist2->sub2video.end_pts || ist2->sub2video.initialize) - /* if we have hit the end of the current displayed subpicture, - or if we need to initialize the system, update the - overlayed subpicture and its start/end times */ - sub2video_update(ist2, pts2 + 1, NULL); - for (j = 0, nb_reqs = 0; j < ist2->nb_filters; j++) - nb_reqs += av_buffersrc_get_nb_failed_requests(ist2->filters[j]->filter); - if (nb_reqs) - sub2video_push_ref(ist2, pts2); - } -} - -static void sub2video_flush(InputStream *ist) -{ - int i; - int ret; - - if (ist->sub2video.end_pts < INT64_MAX) - sub2video_update(ist, INT64_MAX, NULL); - for (i = 0; i < ist->nb_filters; i++) { - ret = av_buffersrc_add_frame(ist->filters[i]->filter, NULL); - if (ret != AVERROR_EOF && ret < 0) - av_log(NULL, AV_LOG_WARNING, "Flush the frame error.\n"); - } -} - -/* end of sub2video hack */ - static void term_exit_sigsafe(void) { #if HAVE_TERMIOS_H @@ -526,7 +369,6 @@ static void ffmpeg_cleanup(int ret) avfilter_graph_free(&fg->graph); for (j = 0; j < fg->nb_inputs; j++) { InputFilter *ifilter = fg->inputs[j]; - struct InputStream *ist = ifilter->ist; while (av_fifo_size(ifilter->frame_queue)) { AVFrame *frame; @@ -536,15 +378,6 @@ static void ffmpeg_cleanup(int ret) } av_fifo_freep(&ifilter->frame_queue); av_freep(&ifilter->displaymatrix); - if (ist->sub2video.sub_queue) { - while (av_fifo_size(ist->sub2video.sub_queue)) { - AVSubtitle sub; - av_fifo_generic_read(ist->sub2video.sub_queue, - &sub, sizeof(sub), NULL); - avsubtitle_free(&sub); - } - av_fifo_freep(&ist->sub2video.sub_queue); - } av_buffer_unref(&ifilter->hw_frames_ctx); av_freep(&ifilter->name); av_freep(&fg->inputs[j]); @@ -636,12 +469,14 @@ static void ffmpeg_cleanup(int ret) av_frame_free(&ist->filter_frame); av_packet_free(&ist->pkt); av_dict_free(&ist->decoder_opts); - avsubtitle_free(&ist->prev_sub.subtitle); - av_frame_free(&ist->sub2video.frame); + av_frame_free(&ist->prev_sub.subtitle); av_freep(&ist->filters); av_freep(&ist->hwaccel_device); av_freep(&ist->dts_buffer); + av_frame_free(&ist->subtitle_heartbeat.recent_sub); + av_buffer_unref(&ist->subtitle_heartbeat.header); + avcodec_free_context(&ist->dec_ctx); av_freep(&input_streams[i]); @@ -1061,17 +896,21 @@ error: exit_program(1); } -static void do_subtitle_out(OutputFile *of, - OutputStream *ost, - AVSubtitle *sub) +static void do_subtitle_out(OutputFile *of, OutputStream *ost, AVFrame *frame) { - int subtitle_out_max_size = 1024 * 1024; + const int subtitle_out_max_size = 1024 * 1024; int subtitle_out_size, nb, i; AVCodecContext *enc; AVPacket *pkt = ost->pkt; + AVSubtitle out_sub = { 0 }; int64_t pts; - if (sub->pts == AV_NOPTS_VALUE) { + if (!frame) + return; + + av_log(NULL, AV_LOG_TRACE, "do_subtitle_out: sub->pts: %"PRId64" frame->pts: %"PRId64"\n", frame->subtitle_pts, frame->pts); + + if (frame->subtitle_pts == AV_NOPTS_VALUE) { av_log(NULL, AV_LOG_ERROR, "Subtitle packets must have a pts\n"); if (exit_on_error) exit_program(1); @@ -1097,30 +936,35 @@ static void do_subtitle_out(OutputFile *of, nb = 1; /* shift timestamp to honor -ss and make check_recording_time() work with -t */ - pts = sub->pts; + pts = frame->subtitle_pts; if (output_files[ost->file_index]->start_time != AV_NOPTS_VALUE) pts -= output_files[ost->file_index]->start_time; + + ost->sync_opts = av_rescale_q(pts, AV_TIME_BASE_Q, enc->time_base); + if (!check_recording_time(ost)) + return; + + frame->subtitle_pts = pts; + // subtitle_start_time is required to be 0 + frame->subtitle_pts += av_rescale_q(frame->subtitle_start_time, (AVRational){ 1, 1000 }, AV_TIME_BASE_Q); + frame->subtitle_end_time -= frame->subtitle_start_time; + frame->subtitle_start_time = 0; + + av_frame_get_subtitle(&out_sub, frame); + for (i = 0; i < nb; i++) { - unsigned save_num_rects = sub->num_rects; + const unsigned save_num_rects = out_sub.num_rects; - ost->sync_opts = av_rescale_q(pts, AV_TIME_BASE_Q, enc->time_base); - if (!check_recording_time(ost)) - return; + ost->frames_encoded++; - sub->pts = pts; - // start_display_time is required to be 0 - sub->pts += av_rescale_q(sub->start_display_time, (AVRational){ 1, 1000 }, AV_TIME_BASE_Q); - sub->end_display_time -= sub->start_display_time; - sub->start_display_time = 0; if (i == 1) - sub->num_rects = 0; + out_sub.num_rects = 0; - ost->frames_encoded++; + subtitle_out_size = avcodec_encode_subtitle(enc, subtitle_out, subtitle_out_max_size, &out_sub); - subtitle_out_size = avcodec_encode_subtitle(enc, subtitle_out, - subtitle_out_max_size, sub); if (i == 1) - sub->num_rects = save_num_rects; + out_sub.num_rects = save_num_rects; + if (subtitle_out_size < 0) { av_log(NULL, AV_LOG_FATAL, "Subtitle encoding failed\n"); exit_program(1); @@ -1129,19 +973,21 @@ static void do_subtitle_out(OutputFile *of, av_packet_unref(pkt); pkt->data = subtitle_out; pkt->size = subtitle_out_size; - pkt->pts = av_rescale_q(sub->pts, AV_TIME_BASE_Q, ost->mux_timebase); - pkt->duration = av_rescale_q(sub->end_display_time, (AVRational){ 1, 1000 }, ost->mux_timebase); + pkt->pts = av_rescale_q(frame->subtitle_pts, AV_TIME_BASE_Q, ost->mux_timebase); + pkt->duration = av_rescale_q(frame->subtitle_end_time, (AVRational){ 1, 1000 }, ost->mux_timebase); if (enc->codec_id == AV_CODEC_ID_DVB_SUBTITLE) { /* XXX: the pts correction is handled here. Maybe handling it in the codec would be better */ if (i == 0) - pkt->pts += av_rescale_q(sub->start_display_time, (AVRational){ 1, 1000 }, ost->mux_timebase); + pkt->pts += av_rescale_q(frame->subtitle_start_time, (AVRational){ 1, 1000 }, ost->mux_timebase); else - pkt->pts += av_rescale_q(sub->end_display_time, (AVRational){ 1, 1000 }, ost->mux_timebase); + pkt->pts += av_rescale_q(frame->subtitle_end_time, (AVRational){ 1, 1000 }, ost->mux_timebase); } pkt->dts = pkt->pts; output_packet(of, pkt, ost, 0); } + + avsubtitle_free(&out_sub); } static void do_video_out(OutputFile *of, @@ -1578,8 +1424,26 @@ static int reap_filters(int flush) } do_audio_out(of, ost, filtered_frame); break; + case AVMEDIA_TYPE_SUBTITLE: + + if (filtered_frame->format == AV_SUBTITLE_FMT_ASS && !enc->subtitle_header + && filtered_frame->subtitle_header) { + const char *subtitle_header = (char *)filtered_frame->subtitle_header->data; + enc->subtitle_header = (uint8_t *)av_strdup(subtitle_header); + if (!enc->subtitle_header) + return AVERROR(ENOMEM); + enc->subtitle_header_size = strlen(subtitle_header); + } + + if ((ost->enc_ctx->width == 0 || ost->enc_ctx->height == 0) + && filter->inputs[0]->w > 0 && filter->inputs[0]->h > 0 ) { + ost->enc_ctx->width = filter->inputs[0]->w; + ost->enc_ctx->height = filter->inputs[0]->h; + } + + do_subtitle_out(of, ost, filtered_frame); + break; default: - // TODO support subtitle filters av_assert0(0); } @@ -2175,7 +2039,8 @@ static int ifilter_has_all_input_formats(FilterGraph *fg) int i; for (i = 0; i < fg->nb_inputs; i++) { if (fg->inputs[i]->format < 0 && (fg->inputs[i]->type == AVMEDIA_TYPE_AUDIO || - fg->inputs[i]->type == AVMEDIA_TYPE_VIDEO)) + fg->inputs[i]->type == AVMEDIA_TYPE_VIDEO || + fg->inputs[i]->type == AVMEDIA_TYPE_SUBTITLE)) return 0; } return 1; @@ -2279,7 +2144,7 @@ static int ifilter_send_eof(InputFilter *ifilter, int64_t pts) // the filtergraph was never configured if (ifilter->format < 0) ifilter_parameters_from_codecpar(ifilter, ifilter->ist->st->codecpar); - if (ifilter->format < 0 && (ifilter->type == AVMEDIA_TYPE_AUDIO || ifilter->type == AVMEDIA_TYPE_VIDEO)) { + if (ifilter->format < 0 && (ifilter->type == AVMEDIA_TYPE_AUDIO || ifilter->type == AVMEDIA_TYPE_VIDEO || ifilter->type == AVMEDIA_TYPE_SUBTITLE)) { av_log(NULL, AV_LOG_ERROR, "Cannot determine format of input stream %d:%d after EOF\n", ifilter->ist->file_index, ifilter->ist->st->index); return AVERROR_INVALIDDATA; } @@ -2534,83 +2399,204 @@ fail: return err < 0 ? err : ret; } -static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, +static void subtitle_resend_current(InputStream *ist, int64_t heartbeat_pts) +{ + AVFrame *frame; + int64_t pts, end_pts; + + if (ist->subtitle_heartbeat.recent_sub) { + frame = av_frame_clone(ist->subtitle_heartbeat.recent_sub); + + pts = av_rescale_q(frame->subtitle_pts + frame->subtitle_start_time * 1000LL, AV_TIME_BASE_Q, ist->st->time_base); + end_pts = av_rescale_q(frame->subtitle_pts + frame->subtitle_end_time * 1000LL, AV_TIME_BASE_Q, ist->st->time_base); + } + else { + frame = av_frame_alloc(); + if (!frame) { + av_log(ist->dec_ctx, AV_LOG_ERROR, "Unable to alloc frame (out of memory).\n"); + return; + } + + frame->type = AVMEDIA_TYPE_SUBTITLE; + frame->format = av_get_subtitle_format_from_codecdesc(ist->dec_ctx->codec_descriptor); + + av_frame_get_buffer2(frame, 0); + + frame->width = ist->subtitle_heartbeat.w; + frame->height = ist->subtitle_heartbeat.h; + + pts = ist->subtitle_heartbeat.end_pts <= 0 ? heartbeat_pts : ist->subtitle_heartbeat.end_pts; + end_pts = INT64_MAX; + + frame->subtitle_pts = av_rescale_q(pts, ist->st->time_base, AV_TIME_BASE_Q); + frame->subtitle_end_time = 1000; + } + + frame->pts = pts; + ist->subtitle_heartbeat.last_pts = pts; + ist->subtitle_heartbeat.end_pts = end_pts; + + send_frame_to_filters(ist, frame); +} + +static void subtitle_heartbeat(InputStream *ist, int64_t pts) +{ + const InputFile *infile = input_files[ist->file_index]; + int i; + int64_t pts2; + + /* When a frame is read from a file, examine all subtitle streams in + the same file and send the subtitle frame again. Otherwise, decoded + video frames could be accumulating in the filter graph while a filter + (possibly overlay) is desperately waiting for a subtitle frame. */ + for (i = 0; i < infile->nb_streams; i++) { + InputStream *ist2 = input_streams[infile->ist_index + i]; + if (!ist2->subtitle_heartbeat.is_active) + continue; + /* subtitles seem to be usually muxed ahead of other streams; + if not, subtracting a larger time here is necessary */ + pts2 = av_rescale_q(pts, ist->st->time_base, ist2->st->time_base) - 1; + /* do not send the heartbeat frame if the subtitle is already ahead */ + if (pts2 <= ist2->subtitle_heartbeat.last_pts) + continue; + if (pts2 >= ist2->subtitle_heartbeat.end_pts) { + /* if we have hit the end of the current displayed subpicture, + or if we need to initialize the system, update the + overlayed subpicture and its start/end times */ + if (ist2->subtitle_heartbeat.recent_sub) + av_frame_free(&ist2->subtitle_heartbeat.recent_sub); + subtitle_resend_current(ist2, pts2 + 1); + } + //for (j = 0, nb_reqs = 0; j < ist2->nb_filters; j++) + // nb_reqs += av_buffersrc_get_nb_failed_requests(ist2->filters[j]->filter); + //if (nb_reqs) + // subtitle_heartbeat_resend_current(ist2, pts2); + } +} + +static InputStream *get_input_stream(OutputStream *ost) +{ + if (ost->source_index >= 0) + return input_streams[ost->source_index]; + return NULL; +} + +static int decode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, int *decode_failed) { - AVSubtitle subtitle; - int free_sub = 1; - int i, ret = avcodec_decode_subtitle2(ist->dec_ctx, - &subtitle, got_output, pkt); + AVFrame *decoded_frame; + AVCodecContext *avctx = ist->dec_ctx; + int i = 0, ret = 0, err = 0; + int64_t pts, end_pts; + + if (!ist->decoded_frame && !(ist->decoded_frame = av_frame_alloc())) + return AVERROR(ENOMEM); + if (!ist->filter_frame && !(ist->filter_frame = av_frame_alloc())) + return AVERROR(ENOMEM); + + decoded_frame = ist->decoded_frame; + decoded_frame->type = AVMEDIA_TYPE_SUBTITLE; + decoded_frame->format = av_get_subtitle_format_from_codecdesc(avctx->codec_descriptor); + + if (!ist->subtitle_heartbeat.header && avctx->subtitle_header && avctx->subtitle_header_size > 0) { + ist->subtitle_heartbeat.header = av_buffer_allocz(avctx->subtitle_header_size + 1); + if (!ist->subtitle_heartbeat.header) + return AVERROR(ENOMEM); + memcpy(ist->subtitle_heartbeat.header->data, avctx->subtitle_header, avctx->subtitle_header_size); + } + + ret = decode(avctx, decoded_frame, got_output, pkt); - check_decode_result(NULL, got_output, ret); + if (ret != AVERROR_EOF) + check_decode_result(NULL, got_output, ret); if (ret < 0 || !*got_output) { *decode_failed = 1; - if (!pkt->size) - sub2video_flush(ist); + if (!pkt->size) { + // Flush + for (i = 0; i < ist->nb_filters; i++) { + ret = av_buffersrc_add_frame(ist->filters[i]->filter, NULL); + if (ret != AVERROR_EOF && ret < 0) + av_log(NULL, AV_LOG_WARNING, "Flush the frame error.\n"); + } + } return ret; } if (ist->fix_sub_duration) { int end = 1; - if (ist->prev_sub.got_output) { - end = av_rescale(subtitle.pts - ist->prev_sub.subtitle.pts, + if (ist->prev_sub.got_output && ist->prev_sub.subtitle) { + end = av_rescale(decoded_frame->subtitle_pts - ist->prev_sub.subtitle->subtitle_pts, 1000, AV_TIME_BASE); - if (end < ist->prev_sub.subtitle.end_display_time) { - av_log(ist->dec_ctx, AV_LOG_DEBUG, + if (end < ist->prev_sub.subtitle->subtitle_end_time) { + av_log(avctx, AV_LOG_DEBUG, "Subtitle duration reduced from %"PRId32" to %d%s\n", - ist->prev_sub.subtitle.end_display_time, end, + ist->prev_sub.subtitle->subtitle_end_time, end, end <= 0 ? ", dropping it" : ""); - ist->prev_sub.subtitle.end_display_time = end; + ist->prev_sub.subtitle->subtitle_end_time = end; } } FFSWAP(int, *got_output, ist->prev_sub.got_output); FFSWAP(int, ret, ist->prev_sub.ret); - FFSWAP(AVSubtitle, subtitle, ist->prev_sub.subtitle); + FFSWAP(AVFrame*, decoded_frame, ist->prev_sub.subtitle); if (end <= 0) - goto out; + return end; } - if (!*got_output) + if (!*got_output || !decoded_frame) return ret; - if (ist->sub2video.frame) { - sub2video_update(ist, INT64_MIN, &subtitle); - } else if (ist->nb_filters) { - if (!ist->sub2video.sub_queue) - ist->sub2video.sub_queue = av_fifo_alloc(8 * sizeof(AVSubtitle)); - if (!ist->sub2video.sub_queue) - exit_program(1); - if (!av_fifo_space(ist->sub2video.sub_queue)) { - ret = av_fifo_realloc2(ist->sub2video.sub_queue, 2 * av_fifo_size(ist->sub2video.sub_queue)); - if (ret < 0) - exit_program(1); - } - av_fifo_generic_write(ist->sub2video.sub_queue, &subtitle, sizeof(subtitle), NULL); - free_sub = 0; - } + decoded_frame->type = AVMEDIA_TYPE_SUBTITLE; + decoded_frame->format = av_get_subtitle_format_from_codecdesc(avctx->codec_descriptor); - if (!subtitle.num_rects) - goto out; + ////if ((ret = av_frame_get_buffer2(decoded_frame, 0)) < 0) + //// return ret; - ist->frames_decoded++; + if ((ret = av_buffer_replace(&decoded_frame->subtitle_header, ist->subtitle_heartbeat.header)) < 0) + return ret; - for (i = 0; i < nb_output_streams; i++) { - OutputStream *ost = output_streams[i]; + pts = av_rescale_q(decoded_frame->subtitle_pts + decoded_frame->subtitle_start_time * 1000LL, + AV_TIME_BASE_Q, ist->st->time_base); + end_pts = av_rescale_q(decoded_frame->subtitle_pts + decoded_frame->subtitle_end_time * 1000LL, + AV_TIME_BASE_Q, ist->st->time_base); - if (!ost->pkt && !(ost->pkt = av_packet_alloc())) - exit_program(1); - if (!check_output_constraints(ist, ost) || !ost->encoding_needed - || ost->enc->type != AVMEDIA_TYPE_SUBTITLE) - continue; + ist->subtitle_heartbeat.last_pts = decoded_frame->pts = pts; + ist->subtitle_heartbeat.end_pts = end_pts; - do_subtitle_out(output_files[ost->file_index], ost, &subtitle); + if (ist->nb_filters > 0) { + AVFrame *filter_frame = av_frame_clone(decoded_frame); + if (!filter_frame) + err = AVERROR(ENOMEM); + else + err = send_frame_to_filters(ist, filter_frame); } -out: - if (free_sub) - avsubtitle_free(&subtitle); - return ret; + if (err >= 0) { + for (i = 0; i < nb_output_streams; i++) { + OutputStream *ost = output_streams[i]; + InputStream *ist_src = get_input_stream(ost); + + if (!ist_src || !check_output_constraints(ist, ost) + || ist_src != ist + || !ost->encoding_needed + || ost->enc->type != AVMEDIA_TYPE_SUBTITLE) + continue; + + if (ost->filter && ost->filter->filter->nb_inputs > 0) + continue; + + if (!ost->pkt && !((ost->pkt = av_packet_alloc()))) + exit_program(1); + do_subtitle_out(output_files[ost->file_index], ost, decoded_frame); + } + } + + av_frame_free(&ist->subtitle_heartbeat.recent_sub); + ist->subtitle_heartbeat.recent_sub = av_frame_clone(decoded_frame); + + av_frame_unref(ist->filter_frame); + av_frame_unref(decoded_frame); + return err < 0 ? err : ret; } static int send_filter_eof(InputStream *ist) @@ -2718,7 +2704,7 @@ static int process_input_packet(InputStream *ist, const AVPacket *pkt, int no_eo case AVMEDIA_TYPE_SUBTITLE: if (repeating) break; - ret = transcode_subtitles(ist, avpkt, &got_output, &decode_failed); + ret = decode_subtitles(ist, avpkt, &got_output, &decode_failed); if (!pkt && ret >= 0) ret = AVERROR_EOF; av_packet_unref(avpkt); @@ -3028,13 +3014,6 @@ FF_ENABLE_DEPRECATION_WARNINGS return 0; } -static InputStream *get_input_stream(OutputStream *ost) -{ - if (ost->source_index >= 0) - return input_streams[ost->source_index]; - return NULL; -} - static int compare_int64(const void *a, const void *b) { return FFDIFFSIGN(*(const int64_t *)a, *(const int64_t *)b); @@ -3535,7 +3514,7 @@ static int init_output_stream_encode(OutputStream *ost, AVFrame *frame) break; case AVMEDIA_TYPE_SUBTITLE: enc_ctx->time_base = AV_TIME_BASE_Q; - if (!enc_ctx->width) { + if (!enc_ctx->width && ost->source_index >= 0) { enc_ctx->width = input_streams[ost->source_index]->st->codecpar->width; enc_ctx->height = input_streams[ost->source_index]->st->codecpar->height; } @@ -3588,19 +3567,14 @@ static int init_output_stream(OutputStream *ost, AVFrame *frame, } if (ist && ist->dec->type == AVMEDIA_TYPE_SUBTITLE && ost->enc->type == AVMEDIA_TYPE_SUBTITLE) { - int input_props = 0, output_props = 0; - AVCodecDescriptor const *input_descriptor = - avcodec_descriptor_get(dec->codec_id); - AVCodecDescriptor const *output_descriptor = - avcodec_descriptor_get(ost->enc_ctx->codec_id); - if (input_descriptor) - input_props = input_descriptor->props & (AV_CODEC_PROP_TEXT_SUB | AV_CODEC_PROP_BITMAP_SUB); - if (output_descriptor) - output_props = output_descriptor->props & (AV_CODEC_PROP_TEXT_SUB | AV_CODEC_PROP_BITMAP_SUB); - if (input_props && output_props && input_props != output_props) { - snprintf(error, error_len, - "Subtitle encoding currently only possible from text to text " - "or bitmap to bitmap"); + AVCodecDescriptor const *input_descriptor = avcodec_descriptor_get(dec->codec_id); + AVCodecDescriptor const *output_descriptor = avcodec_descriptor_get(ost->enc_ctx->codec_id); + const enum AVSubtitleType in_subtitle_format = output_descriptor ? av_get_subtitle_format_from_codecdesc(input_descriptor) : AV_SUBTITLE_FMT_UNKNOWN; + const enum AVSubtitleType out_subtitle_format = output_descriptor ? av_get_subtitle_format_from_codecdesc(output_descriptor) : AV_SUBTITLE_FMT_UNKNOWN; + + if (ist->nb_filters == 0 && in_subtitle_format != AV_SUBTITLE_FMT_UNKNOWN && out_subtitle_format != AV_SUBTITLE_FMT_UNKNOWN + && in_subtitle_format != out_subtitle_format) { + snprintf(error, error_len, "Subtitle encoding is only possible from text to text or bitmap to bitmap"); return AVERROR_INVALIDDATA; } } @@ -4633,7 +4607,7 @@ static int process_input(int file_index) av_ts2timestr(input_files[ist->file_index]->ts_offset, &AV_TIME_BASE_Q)); } - sub2video_heartbeat(ist, pkt->pts); + subtitle_heartbeat(ist, pkt->pts); process_input_packet(ist, pkt, 0); @@ -4845,6 +4819,7 @@ static int transcode(void) /* at the end of stream, we must flush the decoder buffers */ for (i = 0; i < nb_input_streams; i++) { ist = input_streams[i]; + ist->subtitle_heartbeat.is_active = 0; if (!input_files[ist->file_index]->eof_reached) { process_input_packet(ist, NULL, 0); } diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index 30225e9ffe..ac965d72f3 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -349,17 +349,17 @@ typedef struct InputStream { struct { /* previous decoded subtitle and related variables */ int got_output; int ret; - AVSubtitle subtitle; + AVFrame *subtitle; } prev_sub; - struct sub2video { + struct subtitle_heartbeat { + int is_active; int64_t last_pts; int64_t end_pts; - AVFifoBuffer *sub_queue; ///< queue of AVSubtitle* before filter init - AVFrame *frame; + AVFrame *recent_sub; int w, h; - unsigned int initialize; ///< marks if sub2video_update should force an initialization - } sub2video; + AVBufferRef *header; + } subtitle_heartbeat; /* decoded data from this stream goes into all those filters * currently video and audio only */ @@ -661,8 +661,6 @@ int filtergraph_is_simple(FilterGraph *fg); int init_simple_filtergraph(InputStream *ist, OutputStream *ost); int init_complex_filtergraph(FilterGraph *fg); -void sub2video_update(InputStream *ist, int64_t heartbeat_pts, AVSubtitle *sub); - int ifilter_parameters_from_frame(InputFilter *ifilter, const AVFrame *frame); int ffmpeg_parse_options(int argc, char **argv); diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c index b798459946..ae033ef2f1 100644 --- a/fftools/ffmpeg_filter.c +++ b/fftools/ffmpeg_filter.c @@ -22,6 +22,8 @@ #include "ffmpeg.h" +#include + #include "libavfilter/avfilter.h" #include "libavfilter/buffersink.h" #include "libavfilter/buffersrc.h" @@ -30,11 +32,9 @@ #include "libavutil/avstring.h" #include "libavutil/bprint.h" #include "libavutil/channel_layout.h" -#include "libavutil/display.h" #include "libavutil/opt.h" #include "libavutil/pixdesc.h" #include "libavutil/pixfmt.h" -#include "libavutil/imgutils.h" #include "libavutil/samplefmt.h" // FIXME: YUV420P etc. are actually supported with full color range, @@ -221,9 +221,8 @@ static void init_input_filter(FilterGraph *fg, AVFilterInOut *in) enum AVMediaType type = avfilter_pad_get_type(in->filter_ctx->input_pads, in->pad_idx); int i; - // TODO: support other filter types - if (type != AVMEDIA_TYPE_VIDEO && type != AVMEDIA_TYPE_AUDIO) { - av_log(NULL, AV_LOG_FATAL, "Only video and audio filters supported " + if (type != AVMEDIA_TYPE_VIDEO && type != AVMEDIA_TYPE_AUDIO && type != AVMEDIA_TYPE_SUBTITLE) { + av_log(NULL, AV_LOG_FATAL, "Only video, audio and subtitle filters supported " "currently.\n"); exit_program(1); } @@ -244,8 +243,9 @@ static void init_input_filter(FilterGraph *fg, AVFilterInOut *in) for (i = 0; i < s->nb_streams; i++) { enum AVMediaType stream_type = s->streams[i]->codecpar->codec_type; if (stream_type != type && - !(stream_type == AVMEDIA_TYPE_SUBTITLE && - type == AVMEDIA_TYPE_VIDEO /* sub2video hack */)) + // in the followng case we auto-insert the graphicsub2video conversion filter + // for retaining compatibility with the previous sub2video hack + !(stream_type == AVMEDIA_TYPE_SUBTITLE && type == AVMEDIA_TYPE_VIDEO)) continue; if (check_stream_specifier(s, s->streams[i], *p == ':' ? p + 1 : p) == 1) { st = s->streams[i]; @@ -416,6 +416,39 @@ static int insert_filter(AVFilterContext **last_filter, int *pad_idx, return 0; } +static int configure_output_subtitle_filter(FilterGraph *fg, OutputFilter *ofilter, AVFilterInOut *out) +{ + OutputStream *ost = ofilter->ost; + AVFilterContext *last_filter = out->filter_ctx; + int pad_idx = out->pad_idx; + int ret; + char name[255]; + + snprintf(name, sizeof(name), "out_%d_%d", ost->file_index, ost->index); + ret = avfilter_graph_create_filter(&ofilter->filter, + avfilter_get_by_name("sbuffersink"), + name, NULL, NULL, fg->graph); + + if (ret < 0) { + av_log(NULL, AV_LOG_ERROR, "Unable to create filter sbuffersink\n"); + return ret; + } + + ////snprintf(name, sizeof(name), "trim_out_%d_%d", + //// ost->file_index, ost->index); + ////ret = insert_trim(of->start_time, of->recording_time, + //// &last_filter, &pad_idx, name); + ////if (ret < 0) + //// return ret; + + ////ost->st->codecpar->codec_tag = MKTAG('a', 's', 's', 's'); + + if ((ret = avfilter_link(last_filter, pad_idx, ofilter->filter, 0)) < 0) + return ret; + + return 0; +} + static int configure_output_video_filter(FilterGraph *fg, OutputFilter *ofilter, AVFilterInOut *out) { char *pix_fmts; @@ -594,7 +627,8 @@ static int configure_output_audio_filter(FilterGraph *fg, OutputFilter *ofilter, int i; for (i=0; ictx->nb_streams; i++) - if (of->ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) + if (of->ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO || + of->ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) break; if (ictx->nb_streams) { @@ -628,6 +662,7 @@ static int configure_output_filter(FilterGraph *fg, OutputFilter *ofilter, switch (avfilter_pad_get_type(out->filter_ctx->output_pads, out->pad_idx)) { case AVMEDIA_TYPE_VIDEO: return configure_output_video_filter(fg, ofilter, out); case AVMEDIA_TYPE_AUDIO: return configure_output_audio_filter(fg, ofilter, out); + case AVMEDIA_TYPE_SUBTITLE: return configure_output_subtitle_filter(fg, ofilter, out); default: av_assert0(0); return 0; } } @@ -647,51 +682,111 @@ void check_filter_outputs(void) } } -static int sub2video_prepare(InputStream *ist, InputFilter *ifilter) +static int configure_input_subtitle_filter(FilterGraph *fg, InputFilter *ifilter, + AVFilterInOut *in) { - AVFormatContext *avf = input_files[ist->file_index]->ctx; - int i, w, h; + AVFilterContext *last_filter; + const AVFilter *buffer_filt = avfilter_get_by_name("sbuffer"); + InputStream *ist = ifilter->ist; + AVBPrint args; + char name[255]; + int ret, pad_idx = 0; + int w, h; + AVBufferSrcParameters *par = av_buffersrc_parameters_alloc(); + enum AVMediaType media_type; + + if (!par) + return AVERROR(ENOMEM); + memset(par, 0, sizeof(*par)); + par->format = AV_PIX_FMT_NONE; + + if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_AUDIO) { + av_log(NULL, AV_LOG_ERROR, "Cannot connect subtitle filter to audio input\n"); + ret = AVERROR(EINVAL); + goto fail; + } + + if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_VIDEO) { + av_log(NULL, AV_LOG_ERROR, "Cannot connect subtitle filter to video input\n"); + ret = AVERROR(EINVAL); + goto fail; + } + + if (!ist->subtitle_heartbeat.header && ist->dec_ctx->subtitle_header && ist->dec_ctx->subtitle_header_size > 0) { + ist->subtitle_heartbeat.header = av_buffer_allocz(ist->dec_ctx->subtitle_header_size + 1); + if (!ist->subtitle_heartbeat.header) + return AVERROR(ENOMEM); + memcpy(ist->subtitle_heartbeat.header->data, ist->dec_ctx->subtitle_header, ist->dec_ctx->subtitle_header_size); + } + + ist->subtitle_heartbeat.is_active = 1; - /* Compute the size of the canvas for the subtitles stream. - If the subtitles codecpar has set a size, use it. Otherwise use the - maximum dimensions of the video streams in the same file. */ w = ifilter->width; h = ifilter->height; + if (!(w && h)) { - for (i = 0; i < avf->nb_streams; i++) { - if (avf->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) { - w = FFMAX(w, avf->streams[i]->codecpar->width); - h = FFMAX(h, avf->streams[i]->codecpar->height); - } - } - if (!(w && h)) { - w = FFMAX(w, 720); - h = FFMAX(h, 576); - } - av_log(avf, AV_LOG_INFO, "sub2video: using %dx%d canvas\n", w, h); + w = ist->dec_ctx->width; + h = ist->dec_ctx->height; } - ist->sub2video.w = ifilter->width = w; - ist->sub2video.h = ifilter->height = h; - ifilter->width = ist->dec_ctx->width ? ist->dec_ctx->width : ist->sub2video.w; - ifilter->height = ist->dec_ctx->height ? ist->dec_ctx->height : ist->sub2video.h; + if (!(w && h) && ist->dec_ctx->subtitle_header) { + ASSSplitContext *ass_ctx = avpriv_ass_split((char *)ist->dec_ctx->subtitle_header); + ASS *ass = (ASS *)ass_ctx; + w = ass->script_info.play_res_x; + h = ass->script_info.play_res_y; + avpriv_ass_split_free(ass_ctx); + } - /* rectangles are AV_PIX_FMT_PAL8, but we have no guarantee that the - palettes for all rectangles are identical or compatible */ - ifilter->format = AV_PIX_FMT_RGB32; + ist->subtitle_heartbeat.w = w; + ist->subtitle_heartbeat.h = h; + av_log(ifilter, AV_LOG_INFO, "subtitle input filter: decoding size %dx%d\n", ist->subtitle_heartbeat.w, ist->subtitle_heartbeat.h); - ist->sub2video.frame = av_frame_alloc(); - if (!ist->sub2video.frame) - return AVERROR(ENOMEM); - ist->sub2video.last_pts = INT64_MIN; - ist->sub2video.end_pts = INT64_MIN; + ifilter->width = w; + ifilter->height = h; + ist->dec_ctx->width = w; + ist->dec_ctx->height = h; + + ist->subtitle_heartbeat.last_pts = INT64_MIN; + + snprintf(name, sizeof(name), "graph %d subtitle input from stream %d:%d", fg->index, + ist->file_index, ist->st->index); + + + av_bprint_init(&args, 0, AV_BPRINT_SIZE_AUTOMATIC); + av_bprintf(&args, + "subtitle_type=%d:width=%d:height=%d:time_base=%d/%d:", + ifilter->format, ifilter->width, ifilter->height, + ist->st->time_base.num, ist->st->time_base.den); + if ((ret = avfilter_graph_create_filter(&ifilter->filter, buffer_filt, name, + args.str, NULL, fg->graph)) < 0) + goto fail; + + par->hw_frames_ctx = ifilter->hw_frames_ctx; + par->format = ifilter->format; + par->width = ifilter->width; + par->height = ifilter->height; + + ret = av_buffersrc_parameters_set(ifilter->filter, par); + if (ret < 0) + goto fail; + av_freep(&par); + last_filter = ifilter->filter; - /* sub2video structure has been (re-)initialized. - Mark it as such so that the system will be - initialized with the first received heartbeat. */ - ist->sub2video.initialize = 1; + media_type = avfilter_pad_get_type(in->filter_ctx->input_pads, in->pad_idx); + if (media_type == AVMEDIA_TYPE_VIDEO) { + ret = insert_filter(&last_filter, &pad_idx, "graphicsub2video", NULL); + if (ret < 0) + return ret; + } + + if ((ret = avfilter_link(last_filter, 0, in->filter_ctx, in->pad_idx)) < 0) + return ret; return 0; +fail: + av_freep(&par); + + return ret; } static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, @@ -710,8 +805,15 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, char name[255]; int ret, pad_idx = 0; int64_t tsoffset = 0; - AVBufferSrcParameters *par = av_buffersrc_parameters_alloc(); + AVBufferSrcParameters *par; + + if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_SUBTITLE) { + // Automatically insert conversion filter to retain compatibility + // with sub2video command lines + return configure_input_subtitle_filter(fg, ifilter, in); + } + par = av_buffersrc_parameters_alloc(); if (!par) return AVERROR(ENOMEM); memset(par, 0, sizeof(*par)); @@ -726,12 +828,6 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, if (!fr.num) fr = av_guess_frame_rate(input_files[ist->file_index]->ctx, ist->st, NULL); - if (ist->dec_ctx->codec_type == AVMEDIA_TYPE_SUBTITLE) { - ret = sub2video_prepare(ist, ifilter); - if (ret < 0) - goto fail; - } - sar = ifilter->sample_aspect_ratio; if(!sar.den) sar = (AVRational){0,1}; @@ -743,7 +839,7 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, tb.num, tb.den, sar.num, sar.den); if (fr.num && fr.den) av_bprintf(&args, ":frame_rate=%d/%d", fr.num, fr.den); - snprintf(name, sizeof(name), "graph %d input from stream %d:%d", fg->index, + snprintf(name, sizeof(name), "graph %d video input from stream %d:%d", fg->index, ist->file_index, ist->st->index); @@ -966,6 +1062,7 @@ static int configure_input_filter(FilterGraph *fg, InputFilter *ifilter, switch (avfilter_pad_get_type(in->filter_ctx->input_pads, in->pad_idx)) { case AVMEDIA_TYPE_VIDEO: return configure_input_video_filter(fg, ifilter, in); case AVMEDIA_TYPE_AUDIO: return configure_input_audio_filter(fg, ifilter, in); + case AVMEDIA_TYPE_SUBTITLE: return configure_input_subtitle_filter(fg, ifilter, in); default: av_assert0(0); return 0; } } @@ -1133,19 +1230,6 @@ int configure_filtergraph(FilterGraph *fg) } } - /* process queued up subtitle packets */ - for (i = 0; i < fg->nb_inputs; i++) { - InputStream *ist = fg->inputs[i]->ist; - if (ist->sub2video.sub_queue && ist->sub2video.frame) { - while (av_fifo_size(ist->sub2video.sub_queue)) { - AVSubtitle tmp; - av_fifo_generic_read(ist->sub2video.sub_queue, &tmp, sizeof(tmp), NULL); - sub2video_update(ist, INT64_MIN, &tmp); - avsubtitle_free(&tmp); - } - } - } - return 0; fail: @@ -1168,6 +1252,7 @@ int ifilter_parameters_from_frame(InputFilter *ifilter, const AVFrame *frame) ifilter->sample_rate = frame->sample_rate; ifilter->channels = frame->channels; ifilter->channel_layout = frame->channel_layout; + ifilter->type = frame->type; av_freep(&ifilter->displaymatrix); sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DISPLAYMATRIX); diff --git a/fftools/ffmpeg_hw.c b/fftools/ffmpeg_hw.c index 14e702bd92..be69d54aaf 100644 --- a/fftools/ffmpeg_hw.c +++ b/fftools/ffmpeg_hw.c @@ -449,7 +449,7 @@ int hw_device_setup_for_encode(OutputStream *ost) AVBufferRef *frames_ref = NULL; int i; - if (ost->filter) { + if (ost->filter && ost->filter->filter) { frames_ref = av_buffersink_get_hw_frames_ctx(ost->filter->filter); if (frames_ref && ((AVHWFramesContext*)frames_ref->data)->format == diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index 1d6d29cfc9..5a290c946f 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -2150,8 +2150,9 @@ static void init_output_filter(OutputFilter *ofilter, OptionsContext *o, switch (ofilter->type) { case AVMEDIA_TYPE_VIDEO: ost = new_video_stream(o, oc, -1); break; case AVMEDIA_TYPE_AUDIO: ost = new_audio_stream(o, oc, -1); break; + case AVMEDIA_TYPE_SUBTITLE: ost = new_subtitle_stream(o, oc, -1); break; default: - av_log(NULL, AV_LOG_FATAL, "Only video and audio filters are supported " + av_log(NULL, AV_LOG_FATAL, "Only video, audio and subtitle filters are supported " "currently.\n"); exit_program(1); } diff --git a/tests/ref/fate/filter-overlay-dvdsub-2397 b/tests/ref/fate/filter-overlay-dvdsub-2397 index 483e5fa4e0..99081b4519 100644 --- a/tests/ref/fate/filter-overlay-dvdsub-2397 +++ b/tests/ref/fate/filter-overlay-dvdsub-2397 @@ -490,368 +490,367 @@ 1, 3877, 3877, 10, 2013, 0x95a39f9c 1, 3887, 3887, 10, 2013, 0x4f7ea123 1, 3897, 3897, 10, 2013, 0x9efb9ba1 -0, 117, 117, 1, 518400, 0xbf8523da +0, 117, 117, 1, 518400, 0xc44e1d4c 1, 3907, 3907, 10, 2013, 0xf395b2cd 1, 3917, 3917, 10, 2013, 0x261a881e 1, 3927, 3927, 10, 2013, 0x7f2d9f72 1, 3937, 3937, 10, 2013, 0x0105b38d -0, 118, 118, 1, 518400, 0x41890ed6 +0, 118, 118, 1, 518400, 0xad1a084c 1, 3952, 3952, 10, 2013, 0x0e5db67e 1, 3962, 3962, 10, 2013, 0xfc9baf97 -0, 119, 119, 1, 518400, 0x588534fc +0, 119, 119, 1, 518400, 0x52d52e73 1, 3972, 3972, 10, 2013, 0x8e02a1b1 1, 3982, 3982, 10, 2013, 0x6eecaac8 1, 3992, 3992, 10, 2013, 0xf5558f0c 1, 4002, 4002, 10, 2013, 0x512ba99b -0, 120, 120, 1, 518400, 0x2145ebc1 +0, 120, 120, 1, 518400, 0xc732e546 1, 4012, 4012, 10, 2013, 0x932b9932 1, 4022, 4022, 10, 2013, 0xc01ea987 -0, 121, 121, 1, 518400, 0x28bca595 +0, 121, 121, 1, 518400, 0xc36f9f14 1, 4038, 4038, 10, 2013, 0x10879cf7 1, 4048, 4048, 10, 2013, 0x90679338 1, 4058, 4058, 10, 2013, 0x077d8a9e 1, 4068, 4068, 10, 2013, 0x969fa57c -0, 122, 122, 1, 518400, 0x77dc951e +0, 122, 122, 1, 518400, 0x78428e8b 1, 4078, 4078, 10, 2013, 0xe049ab07 1, 4088, 4088, 10, 2013, 0xf535b3b3 1, 4098, 4098, 10, 2013, 0xfe76bd37 -0, 123, 123, 1, 518400, 0xe8924c17 +0, 123, 123, 1, 518400, 0xf0f8458d 1, 4108, 4108, 10, 2013, 0xde79ad8c 1, 4123, 4123, 10, 2013, 0xe89b9c47 1, 4133, 4133, 10, 2013, 0xc570b0f0 -0, 124, 124, 1, 518400, 0xadb4cccc +0, 124, 124, 1, 518400, 0x7083c653 1, 4143, 4143, 10, 2013, 0xee709cd9 1, 4153, 4153, 10, 2013, 0xcfe5afab 1, 4163, 4163, 10, 2013, 0x98ff8ce4 -0, 125, 125, 1, 518400, 0x1d7b56ac +0, 125, 125, 1, 518400, 0xa105502c 1, 4173, 4173, 10, 2013, 0x9d19b44c 1, 4183, 4183, 10, 2013, 0x4349917a 1, 4193, 4193, 10, 2013, 0xbf54a59a -0, 126, 126, 1, 518400, 0xad5739a4 +0, 126, 126, 1, 518400, 0xd411331a 1, 4208, 4208, 10, 2013, 0xc4a399e0 1, 4218, 4218, 10, 2013, 0x1bf58ff0 1, 4228, 4228, 10, 2013, 0x3518ac56 -0, 127, 127, 1, 518400, 0x2733d35a +0, 127, 127, 1, 518400, 0x83b0ccdb 1, 4238, 4238, 10, 2013, 0xcd38c1de 1, 4248, 4248, 10, 2013, 0xbe7d9c4d 1, 4258, 4258, 10, 2013, 0xe113a306 1, 4268, 4268, 10, 2013, 0x083197ea -0, 128, 128, 1, 518400, 0x78e76da2 +0, 128, 128, 1, 518400, 0xa9be671a 1, 4278, 4278, 10, 2013, 0x1929b1eb 1, 4294, 4294, 10, 2013, 0x5d6ea5af 1, 4304, 4304, 10, 2013, 0x05519d53 -0, 129, 129, 1, 518400, 0x6c076013 +0, 129, 129, 1, 518400, 0xaeb75983 1, 4314, 4314, 10, 2013, 0x5773b380 1, 4324, 4324, 10, 2013, 0xaa70a8f5 1, 4334, 4334, 10, 2013, 0x990db0ec -0, 130, 130, 1, 518400, 0x7854f2b1 +0, 130, 130, 1, 518400, 0x81f8ec13 1, 4344, 4344, 10, 2013, 0x91d3a623 1, 4354, 4354, 10, 2013, 0xc91f9824 1, 4364, 4364, 10, 2013, 0x1d058abf -0, 131, 131, 1, 518400, 0xd2ae1ecd +0, 131, 131, 1, 518400, 0x8aaa1839 1, 4379, 4379, 10, 2013, 0x8de1b8d5 1, 4389, 4389, 10, 2013, 0x7872b06b 1, 4399, 4399, 10, 2013, 0xa084c203 -0, 132, 132, 1, 518400, 0xf5eab38d +0, 132, 132, 1, 518400, 0xc98bacf5 1, 4409, 4409, 10, 2013, 0xff90ae8d 1, 4419, 4419, 10, 2013, 0x61dead8e 1, 4429, 4429, 10, 2013, 0xee76b284 -0, 133, 133, 1, 518400, 0x994d3e9c +0, 133, 133, 1, 518400, 0x31083804 1, 4439, 4439, 10, 2013, 0xe888af7f 1, 4449, 4449, 10, 2013, 0x5d57b115 1, 4464, 4464, 10, 2013, 0xcdbfb1d0 -0, 134, 134, 1, 518400, 0x95ab705a +0, 134, 134, 1, 518400, 0x540a69dc 1, 4474, 4474, 10, 2013, 0x2e28a952 1, 4484, 4484, 10, 2013, 0x4795a994 1, 4494, 4494, 10, 2013, 0x7e7ea304 1, 4504, 4504, 10, 2013, 0x9502c1e1 -0, 135, 135, 1, 518400, 0x3c83c5ce +0, 135, 135, 1, 518400, 0x80d3bf46 1, 4514, 4514, 10, 2013, 0xf7c78ab2 1, 4524, 4524, 10, 2013, 0x24049816 1, 4534, 4534, 10, 2013, 0x52089dcf -0, 136, 136, 1, 518400, 0xfa22c508 +0, 136, 136, 1, 518400, 0x2967be7f 1, 4550, 4550, 10, 2013, 0x2150a0b1 1, 4560, 4560, 10, 2013, 0x3c2e9b93 1, 4570, 4570, 10, 2013, 0x491f932b -0, 137, 137, 1, 518400, 0xddda1712 +0, 137, 137, 1, 518400, 0x5a3b1092 1, 4580, 4580, 10, 2013, 0x31359cf8 1, 4590, 4590, 10, 2013, 0x1b00ac3f 1, 4600, 4600, 10, 2013, 0x8d7ab3cb -0, 138, 138, 1, 518400, 0x985a3b93 +0, 138, 138, 1, 518400, 0x8741350b 1, 4610, 4610, 10, 2013, 0xb2c2a4de 1, 4620, 4620, 10, 2013, 0x80a4abf2 1, 4635, 4635, 10, 2013, 0x0701a4ee -0, 139, 139, 1, 518400, 0xea63c5e7 +0, 139, 139, 1, 518400, 0xd5a9bf60 1, 4645, 4645, 10, 2013, 0xdc1ba5bc 1, 4655, 4655, 10, 2013, 0x6083a8a4 1, 4665, 4665, 10, 2013, 0x6226ad45 -0, 140, 140, 1, 518400, 0xef64983d +0, 140, 140, 1, 518400, 0xc05f91ba 1, 4675, 4675, 10, 2013, 0x2732a205 1, 4685, 4685, 10, 2013, 0x0f62a0d3 1, 4695, 4695, 10, 2013, 0xc1799249 -0, 141, 141, 1, 518400, 0x747bb193 +0, 141, 141, 1, 518400, 0x3fdaab0b 1, 4705, 4705, 10, 2013, 0xbccfa9c8 1, 4720, 4720, 10, 2013, 0xded096e7 1, 4730, 4730, 10, 2013, 0x7f0daf43 -0, 142, 142, 1, 518400, 0xb8748862 +0, 142, 142, 1, 518400, 0xab7281d9 1, 4740, 4740, 10, 2013, 0xc47ea682 1, 4750, 4750, 10, 2013, 0x5a72b07a 1, 4760, 4760, 10, 2013, 0x386faa8c 1, 4770, 4770, 10, 2013, 0xf9919a91 -0, 143, 143, 1, 518400, 0xaab55a5f +0, 143, 143, 1, 518400, 0xc80053d6 1, 4780, 4780, 10, 2013, 0x4908897e 1, 4790, 4790, 10, 2013, 0x4882b594 -0, 144, 144, 1, 518400, 0x7b468add +0, 144, 144, 1, 518400, 0x6526845c 1, 4806, 4806, 10, 2013, 0x113e98d1 1, 4816, 4816, 10, 2013, 0x5098b30d 1, 4826, 4826, 10, 2013, 0x0ef7b857 1, 4836, 4836, 10, 2013, 0x216ea176 -0, 145, 145, 1, 518400, 0xf2078707 +0, 145, 145, 1, 518400, 0x1b788089 1, 4846, 4846, 10, 2013, 0xf906944a 1, 4856, 4856, 10, 2013, 0xee9b92fb 1, 4866, 4866, 10, 2013, 0xd6029209 -0, 146, 146, 1, 518400, 0x6a2d931e +0, 146, 146, 1, 518400, 0xfa8e8ca9 1, 4876, 4876, 10, 2013, 0x2256a12e 1, 4891, 4891, 10, 2013, 0x89de8e4a 1, 4901, 4901, 10, 2013, 0x0bf0a584 -0, 147, 147, 1, 518400, 0xbbe3c417 +0, 147, 147, 1, 518400, 0xb278bda1 1, 4911, 4911, 10, 2013, 0x6a5ebd58 1, 4921, 4921, 10, 2013, 0x3edd9aa4 1, 4931, 4931, 10, 2013, 0xbd66ac26 -0, 148, 148, 1, 518400, 0x6294e449 +0, 148, 148, 1, 518400, 0xb0c3ddca 1, 4941, 4941, 10, 2013, 0x313896ea 1, 4951, 4951, 10, 2013, 0x6b83a6a0 1, 4961, 4961, 10, 2013, 0x9aafb109 -0, 149, 149, 1, 518400, 0xa05721e7 +0, 149, 149, 1, 518400, 0x10351b53 1, 4976, 4976, 10, 2013, 0x5192a85a 1, 4986, 4986, 10, 2013, 0x1f919f79 1, 4996, 4996, 10, 2013, 0xc0799c40 -0, 150, 150, 1, 518400, 0x37749183 +0, 150, 150, 1, 518400, 0xc1408aee 1, 5006, 5006, 10, 2013, 0x2988bcd8 1, 5016, 5016, 10, 2013, 0x1482913a 1, 5026, 5026, 10, 2013, 0x74da9a94 1, 5036, 5036, 10, 2013, 0x763eb709 -0, 151, 151, 1, 518400, 0xf9d9dca0 +0, 151, 151, 1, 518400, 0xf016d615 1, 5046, 5046, 10, 2013, 0x1285b405 1, 5062, 5062, 10, 2013, 0xb6ab9dfc -0, 152, 152, 1, 518400, 0x5f8ccf08 +0, 152, 152, 1, 518400, 0xa768c892 1, 5072, 5072, 10, 2013, 0xe4c8bf19 1, 5082, 5082, 10, 2013, 0xabbbade8 1, 5092, 5092, 10, 2013, 0xf8b69d89 1, 5102, 5102, 10, 2013, 0xce04a866 -0, 153, 153, 1, 518400, 0x7303f77b +0, 153, 153, 1, 518400, 0x11c3f11e 1, 5112, 5112, 10, 2013, 0x07528abf 1, 5122, 5122, 10, 2013, 0x74fb98bf 1, 5132, 5132, 10, 2013, 0x579fb1c9 -0, 154, 154, 1, 518400, 0x22b0513f +0, 154, 154, 1, 518400, 0xcd9a4ac4 1, 5147, 5147, 10, 2013, 0x7ddea2ed 1, 5157, 5157, 10, 2013, 0x296caa2c 1, 5167, 5167, 10, 2013, 0x346d9c4f -0, 155, 155, 1, 518400, 0x330485d2 +0, 155, 155, 1, 518400, 0x4ade7f5e 1, 5177, 5177, 10, 2013, 0x3e1fba15 1, 5187, 5187, 10, 2013, 0x48a2908f 1, 5197, 5197, 10, 2013, 0xc1938d09 -0, 156, 156, 1, 518400, 0x7f83daea +0, 156, 156, 1, 518400, 0x655dd46b 1, 5207, 5207, 10, 2013, 0x0e96a060 1, 5217, 5217, 10, 2013, 0x7b6a9e06 1, 5232, 5232, 10, 2013, 0x5b779d28 -0, 157, 157, 1, 518400, 0xee19f2df +0, 157, 157, 1, 518400, 0x5ab5ec61 1, 5242, 5242, 10, 2013, 0xf600aca1 1, 5252, 5252, 10, 2013, 0x3a6c9e68 1, 5262, 5262, 10, 2013, 0x0c8dc1b0 -0, 158, 158, 1, 518400, 0xb71b1c77 +0, 158, 158, 1, 518400, 0x45dc15e6 1, 5272, 5272, 10, 2013, 0x26beb245 1, 5282, 5282, 10, 2013, 0x2bc09557 1, 5292, 5292, 10, 2013, 0x27fc8845 1, 5302, 5302, 10, 2013, 0x1025aa47 -0, 159, 159, 1, 518400, 0xbffc1856 +0, 159, 159, 1, 518400, 0x201911d3 1, 5318, 5318, 10, 2013, 0xc2e69baa 1, 5328, 5328, 10, 2013, 0xdb249b92 1, 5338, 5338, 10, 2013, 0x6ccda29e -0, 160, 160, 1, 518400, 0xabc125aa +0, 160, 160, 1, 518400, 0x0fbc1f46 1, 5348, 5348, 10, 2013, 0xeaf6a1cf 1, 5358, 5358, 10, 2013, 0x509ba397 1, 5368, 5368, 10, 2013, 0xfaf8a2df -0, 161, 161, 1, 518400, 0x5ee467f8 +0, 161, 161, 1, 518400, 0x7e316179 1, 5378, 5378, 10, 2013, 0x41388f28 1, 5388, 5388, 10, 2013, 0xfe5eab39 1, 5403, 5403, 10, 2013, 0xd5ffa066 -0, 162, 162, 1, 518400, 0x6c2cf168 +0, 162, 162, 1, 518400, 0x73bbeaed 1, 5413, 5413, 10, 2013, 0x6813a30a 1, 5423, 5423, 10, 2013, 0x9be89718 1, 5433, 5433, 10, 2013, 0xaec3a27b -0, 163, 163, 1, 518400, 0x63996b26 +0, 163, 163, 1, 518400, 0x3a7c648a 1, 5446, 5446, 10, 2013, 0x579a983e 1, 5456, 5456, 10, 2013, 0x98cea21f 1, 5466, 5466, 10, 2013, 0xca77a58a -0, 164, 164, 1, 518400, 0xb34d789a +0, 164, 164, 1, 518400, 0x9f707209 1, 5476, 5476, 10, 2013, 0xcbc3b1ee 1, 5486, 5486, 10, 2013, 0xf3bb8f07 1, 5496, 5496, 10, 2013, 0x6aeebd92 -0, 165, 165, 1, 518400, 0xf49c030f +0, 165, 165, 1, 518400, 0x9f25fc5c 1, 5506, 5506, 10, 2013, 0xe955a449 1, 5516, 5516, 10, 2013, 0x9436aa5b 1, 5531, 5531, 10, 2013, 0x4f0a8f9f -0, 166, 166, 1, 518400, 0x092dc41a +0, 166, 166, 1, 518400, 0x2ed8bd75 1, 5541, 5541, 10, 2013, 0x3551b22d 1, 5551, 5551, 10, 2013, 0x0959a3d4 1, 5561, 5561, 10, 2013, 0x2ed5a11b 1, 5571, 5571, 10, 2013, 0x8f52a5c3 -0, 167, 167, 1, 518400, 0x4134c577 +0, 167, 167, 1, 518400, 0xb493becb 1, 5581, 5581, 10, 2013, 0x6552978d 1, 5591, 5591, 10, 2013, 0x7dcca0c1 1, 5601, 5601, 10, 2013, 0xbcd4a3c9 -0, 168, 168, 1, 518400, 0x261de1ed +0, 168, 168, 1, 518400, 0x7df6db57 1, 5616, 5616, 10, 2013, 0xfe41a8d8 1, 5626, 5626, 10, 2013, 0xc85aae14 1, 5636, 5636, 10, 2013, 0x1185b346 -0, 169, 169, 1, 518400, 0xcbc8566a +0, 169, 169, 1, 518400, 0x1cb94fca 1, 5646, 5646, 10, 2013, 0xf7429a0d 1, 5656, 5656, 10, 2013, 0x48c2a160 1, 5666, 5666, 10, 2013, 0x9d85a85d -0, 170, 170, 1, 518400, 0x407a5c76 +0, 170, 170, 1, 518400, 0x70db55d8 1, 5676, 5676, 10, 2013, 0xbbe89fe9 1, 5686, 5686, 10, 2013, 0xea429fe2 1, 5702, 5702, 10, 2013, 0x221ca1d4 -0, 171, 171, 1, 518400, 0x1ed73bb2 +0, 171, 171, 1, 518400, 0xc1d9351b 1, 5712, 5712, 10, 2013, 0x394b925b 1, 5722, 5722, 10, 2013, 0x556dc26f 1, 5732, 5732, 10, 2013, 0xce21a5e1 -0, 172, 172, 1, 518400, 0x8467ddb5 +0, 172, 172, 1, 518400, 0xa4b0d717 1, 5742, 5742, 10, 2013, 0xbc87c0a8 1, 5752, 5752, 10, 2013, 0xbac4ac07 1, 5762, 5762, 10, 2013, 0xdeefa4aa 1, 5772, 5772, 10, 2013, 0x1f15b362 -0, 173, 173, 1, 518400, 0x0523dc73 +0, 173, 173, 1, 518400, 0x3730d5e9 1, 5787, 5787, 10, 2013, 0x6406b7b2 1, 5797, 5797, 10, 2013, 0x8030a03d -0, 174, 174, 1, 518400, 0x81f5e895 +0, 174, 174, 1, 518400, 0x9673e1ec 1, 5807, 5807, 10, 2013, 0x0373a5b1 1, 5817, 5817, 10, 2013, 0x34ef93da 1, 5827, 5827, 10, 2013, 0x94c198fe 1, 5837, 5837, 10, 2013, 0xfefcabad -0, 175, 175, 1, 518400, 0xfc74608d +0, 175, 175, 1, 518400, 0x877959d5 1, 5847, 5847, 10, 2013, 0x8755b3ec 1, 5857, 5857, 10, 2013, 0xe436a6fd 1, 5872, 5872, 10, 2013, 0x9cf5a11e -0, 176, 176, 1, 518400, 0xc4e0dae0 +0, 176, 176, 1, 518400, 0x04f3d421 1, 5882, 5882, 10, 2013, 0x03b8a98c 1, 5892, 5892, 10, 2013, 0x6216a138 1, 5902, 5902, 10, 2013, 0xd87b9f12 -0, 177, 177, 1, 518400, 0x98367f5b +0, 177, 177, 1, 518400, 0x4f3078bc 1, 5912, 5912, 10, 2013, 0x4ce99653 1, 5922, 5922, 10, 2013, 0x6c2ea9e2 1, 5932, 5932, 10, 2013, 0x918cae4c -0, 178, 178, 1, 518400, 0x0f1a869d +0, 178, 178, 1, 518400, 0x8a127ff8 1, 5942, 5942, 10, 2013, 0xd19fa5f2 1, 5958, 5958, 10, 2013, 0x0bdda7c6 1, 5968, 5968, 10, 2013, 0x0f9ab0ca -0, 179, 179, 1, 518400, 0x45b6ccf2 +0, 179, 179, 1, 518400, 0x5864c64f 1, 5978, 5978, 10, 2013, 0x410a92b1 1, 5988, 5988, 10, 2013, 0xcfbe9d1c 1, 5998, 5998, 10, 2013, 0x59ed9d15 -0, 180, 180, 1, 518400, 0x5f9ccb77 +0, 180, 180, 1, 518400, 0xdaccc4c0 1, 6008, 6008, 10, 2013, 0x4e129e27 1, 6018, 6018, 10, 2013, 0x7bb9ac0a 1, 6028, 6028, 10, 2013, 0x826ca82b -0, 181, 181, 1, 518400, 0x5f15ea31 +0, 181, 181, 1, 518400, 0xd999e376 1, 6043, 6043, 10, 2013, 0x9ad5a74b 1, 6053, 6053, 10, 2013, 0x6c5f969a 1, 6063, 6063, 10, 2013, 0x8479a0e5 -0, 182, 182, 1, 518400, 0x86369f27 +0, 182, 182, 1, 518400, 0x8af39876 1, 6073, 6073, 10, 2013, 0x165298ef 1, 6083, 6083, 10, 2013, 0xdcadb4a1 1, 6093, 6093, 10, 2013, 0xa90e987c 1, 6103, 6103, 10, 2013, 0x1ac5b510 -0, 183, 183, 1, 518400, 0x2e27f9fa +0, 183, 183, 1, 518400, 0x5e72f33d 1, 6113, 6113, 10, 2013, 0x66728d85 1, 6128, 6128, 10, 2013, 0xe4859fc5 1, 6138, 6138, 10, 2013, 0x9901786e -0, 184, 184, 1, 518400, 0xc029a44d +0, 184, 184, 1, 518400, 0x14af9d92 1, 6148, 6148, 10, 2013, 0x6aebb406 1, 6158, 6158, 10, 2013, 0x7d13a2cc 1, 6168, 6168, 10, 2013, 0x99b7a8cc -0, 185, 185, 1, 518400, 0xebee33b0 +0, 185, 185, 1, 518400, 0x50b82d10 1, 6178, 6178, 10, 2013, 0x80b8a624 1, 6188, 6188, 10, 2013, 0xbb6aa271 1, 6198, 6198, 10, 2013, 0x17af9e4a -0, 186, 186, 1, 518400, 0x19e5494f +0, 186, 186, 1, 518400, 0xc068429c 1, 6214, 6214, 10, 2013, 0xfaf0a8f1 1, 6224, 6224, 10, 2013, 0xd6849b93 1, 6234, 6234, 10, 2013, 0xe9829669 -0, 187, 187, 1, 518400, 0xf697bd7c +0, 187, 187, 1, 518400, 0x8934b6d1 1, 6244, 6244, 10, 2013, 0x7ec98944 1, 6254, 6254, 10, 2013, 0x2b2099a4 1, 6264, 6264, 10, 2013, 0x1033a82f -0, 188, 188, 1, 518400, 0x82569002 +0, 188, 188, 1, 518400, 0x11d08947 1, 6274, 6274, 10, 2013, 0x5ec88990 1, 6284, 6284, 10, 2013, 0xd2a19b3d 1, 6299, 6299, 10, 2013, 0xa377b268 -0, 189, 189, 1, 518400, 0xfcb6d707 +0, 189, 189, 1, 518400, 0x8a27d041 1, 6309, 6309, 10, 2013, 0xfa859901 1, 6319, 6319, 10, 2013, 0x1713955a 1, 6329, 6329, 10, 2013, 0x70aab0da 1, 6339, 6339, 10, 2013, 0xcdaea422 -0, 190, 190, 1, 518400, 0x82a9662b +0, 190, 190, 1, 518400, 0xab265f7d 1, 6349, 6349, 10, 2013, 0x65c3bf80 1, 6359, 6359, 10, 2013, 0x1d75a55f 1, 6369, 6369, 10, 2013, 0xa5bea4de -0, 191, 191, 1, 518400, 0x212e16ee +0, 191, 191, 1, 518400, 0xff491040 1, 6384, 6384, 10, 2013, 0x184db71c 1, 6394, 6394, 10, 2013, 0x99858ec8 1, 6404, 6404, 10, 2013, 0xb8f2aee5 -0, 192, 192, 1, 518400, 0x2ca34dca +0, 192, 192, 1, 518400, 0x822b4704 1, 6414, 6414, 10, 2013, 0x4435b2ef 1, 6424, 6424, 10, 2013, 0x8acfa6c7 1, 6434, 6434, 10, 2013, 0x42b4c01f -0, 193, 193, 1, 518400, 0xe9ebe0a5 +0, 193, 193, 1, 518400, 0x4523d9f4 1, 6444, 6444, 10, 2013, 0x6e308c13 1, 6454, 6454, 10, 2013, 0x8227a0f6 1, 6470, 6470, 10, 2013, 0x6f12a7a2 -0, 194, 194, 1, 518400, 0x4e6b6917 +0, 194, 194, 1, 518400, 0xfc3c626e 1, 6480, 6480, 10, 2013, 0x785392be 1, 6490, 6490, 10, 2013, 0x81849c2b 1, 6500, 6500, 10, 2013, 0x5cf2af65 -0, 195, 195, 1, 518400, 0x7dcf20ab +0, 195, 195, 1, 518400, 0x237319e5 1, 6510, 6510, 10, 2013, 0x0c6ca6b4 1, 6520, 6520, 10, 2013, 0x412fab9f 1, 6530, 6530, 10, 2013, 0x08e792b4 -0, 196, 196, 1, 518400, 0xf30fac97 +0, 196, 196, 1, 518400, 0x892ca5d8 1, 6540, 6540, 10, 2013, 0x407aace3 1, 6555, 6555, 10, 2013, 0xd26bac16 1, 6565, 6565, 10, 2013, 0xac8bb295 -0, 197, 197, 1, 518400, 0xcb9fc692 +0, 197, 197, 1, 518400, 0xc4c0bfc7 1, 6575, 6575, 10, 2013, 0xddd1949c 1, 6585, 6585, 10, 2013, 0x6b26b868 1, 6595, 6595, 10, 2013, 0x5eaba587 1, 6605, 6605, 10, 2013, 0xef0793b9 -0, 198, 198, 1, 518400, 0x5d05601e +0, 198, 198, 1, 518400, 0x57c85956 1, 6615, 6615, 10, 2013, 0xdef19bd6 1, 6625, 6625, 10, 2013, 0xca98a635 -0, 199, 199, 1, 518400, 0x456c1417 +0, 199, 199, 1, 518400, 0xd6300d46 1, 6640, 6640, 10, 2013, 0x06269a5a 1, 6650, 6650, 10, 2013, 0x32cb9952 1, 6660, 6660, 10, 2013, 0xf01fa95a 1, 6670, 6670, 10, 2013, 0xefab9e55 -0, 200, 200, 1, 518400, 0x9a0fd1ad +0, 200, 200, 1, 518400, 0xd3dacaec 1, 6680, 6680, 10, 2013, 0x55a3b63a 1, 6690, 6690, 10, 2013, 0xcd36a553 1, 6700, 6700, 10, 2013, 0x2ec19877 -0, 201, 201, 1, 518400, 0x55db9716 +0, 201, 201, 1, 518400, 0x65429052 1, 6710, 6710, 10, 2013, 0xc18b924c 1, 6726, 6726, 10, 2013, 0xf132b04c 1, 6736, 6736, 10, 2013, 0x7975a44d -0, 202, 202, 1, 518400, 0x1f0d40d6 +0, 202, 202, 1, 518400, 0xec803a15 1, 6746, 6746, 10, 2013, 0x2aaf94cb 1, 6756, 6756, 10, 2013, 0x58cfa60f 1, 6766, 6766, 10, 2013, 0x9757a658 -0, 203, 203, 1, 518400, 0x73695c82 +0, 203, 203, 1, 518400, 0x7a9a55c9 1, 6776, 6776, 10, 2013, 0x67ebc0d5 1, 6786, 6786, 10, 2013, 0x3c50a70e 1, 6796, 6796, 10, 2013, 0x9c5799c6 -0, 204, 204, 1, 518400, 0xb0f10812 +0, 204, 204, 1, 518400, 0xcac30160 1, 6811, 6811, 10, 2013, 0x018d85b2 1, 6821, 6821, 10, 2013, 0x5367a956 -0, 205, 205, 1, 518400, 0xdec18505 -0, 208, 208, 1, 518400, 0xb147b947 -0, 240, 240, 1, 518400, 0x9d2e3977 +0, 205, 205, 1, 518400, 0x7e187e4f +0, 208, 208, 1, 518400, 0x0be0b2a2 diff --git a/tests/ref/fate/sub-dvb b/tests/ref/fate/sub-dvb index cbd1801d64..8f33c75d70 100644 --- a/tests/ref/fate/sub-dvb +++ b/tests/ref/fate/sub-dvb @@ -1,75 +1,93 @@ #tb 0: 1/1000000 #media_type 0: subtitle #codec_id 0: dvb_subtitle -0, 15600000, 15600000, 159000, 1168, 0xd0f89d82 -0, 15759000, 15759000, 159000, 14, 0x064900eb -0, 15760000, 15760000, 239000, 1544, 0xe60f1751 -0, 15999000, 15999000, 239000, 14, 0x0729010b -0, 16000000, 16000000, 339000, 1658, 0xbe343093 -0, 16339000, 16339000, 339000, 14, 0x0809012b -0, 16340000, 16340000, 599000, 2343, 0xc68f07ef -0, 16939000, 16939000, 599000, 14, 0x08e9014b -0, 16940000, 16940000, 459000, 2568, 0x0ee657b1 -0, 17399000, 17399000, 459000, 14, 0x09c9016b -0, 17400000, 17400000, 359000, 3422, 0xba5b63ce -0, 17759000, 17759000, 359000, 14, 0x0aa9018b -0, 17760000, 17760000, 219000, 5078, 0x95b19902 -0, 17979000, 17979000, 219000, 14, 0x0b8901ab -0, 17980000, 17980000, 959000, 5808, 0xc9717b89 -0, 18939000, 18939000, 959000, 14, 0x0c6901cb -0, 18940000, 18940000, 219000, 6015, 0x0becbfac -0, 19159000, 19159000, 219000, 14, 0x064900eb -0, 19160000, 19160000, 259000, 6519, 0xfcd24d26 -0, 19419000, 19419000, 259000, 14, 0x0729010b -0, 19420000, 19420000, 99000, 7061, 0xf0320408 -0, 19519000, 19519000, 99000, 14, 0x0809012b -0, 19520000, 19520000, 219000, 4773, 0x66c93074 -0, 19739000, 19739000, 219000, 14, 0x08e9014b -0, 19740000, 19740000, 219000, 5546, 0x06052c81 -0, 19959000, 19959000, 219000, 14, 0x09c9016b -0, 19960000, 19960000, 239000, 5754, 0x904f7325 -0, 20199000, 20199000, 239000, 14, 0x0aa9018b -0, 20200000, 20200000, 139000, 6099, 0xe30cde07 -0, 20339000, 20339000, 139000, 14, 0x0b8901ab -0, 20340000, 20340000, 799000, 6839, 0x770fcb6c -0, 21139000, 21139000, 799000, 14, 0x0c6901cb -0, 21140000, 21140000, 239000, 4744, 0xa91e1b41 -0, 21379000, 21379000, 239000, 14, 0x064900eb -0, 21380000, 21380000, 339000, 5824, 0xcf6d782b -0, 21719000, 21719000, 339000, 14, 0x0729010b -0, 21720000, 21720000, 1439000, 6212, 0xabf8f7cf -0, 23159000, 23159000, 1439000, 14, 0x0809012b -0, 23160000, 23160000, 1319000, 7082, 0xd7ca10f2 -0, 24479000, 24479000, 1319000, 14, 0x08e9014b -0, 24480000, 24480000, 219000, 5345, 0x12b2cae0 -0, 24699000, 24699000, 219000, 14, 0x09c9016b -0, 24700000, 24700000, 219000, 5765, 0xc7d46192 -0, 24919000, 24919000, 219000, 14, 0x0aa9018b -0, 24920000, 24920000, 599000, 6557, 0xcb995d30 -0, 25519000, 25519000, 599000, 14, 0x0b8901ab -0, 25520000, 25520000, 219000, 7091, 0xe6ea0559 -0, 25739000, 25739000, 219000, 14, 0x0c6901cb -0, 25740000, 25740000, 239000, 7305, 0xb66c404e -0, 25979000, 25979000, 239000, 14, 0x064900eb -0, 25980000, 25980000, 359000, 7590, 0x0cc2a481 -0, 26339000, 26339000, 359000, 14, 0x0729010b -0, 26340000, 26340000, 219000, 4629, 0xe18cfea8 -0, 26559000, 26559000, 219000, 14, 0x0809012b -0, 26560000, 26560000, 719000, 4785, 0x82043fc0 -0, 27279000, 27279000, 719000, 14, 0x08e9014b -0, 27280000, 27280000, 459000, 6061, 0xbde7d245 -0, 27739000, 27739000, 459000, 14, 0x09c9016b -0, 27740000, 27740000, 239000, 6301, 0x92d01a51 -0, 27979000, 27979000, 239000, 14, 0x0aa9018b -0, 27980000, 27980000, 99000, 6736, 0xbd25a134 -0, 28079000, 28079000, 99000, 14, 0x0b8901ab -0, 28080000, 28080000, 219000, 7214, 0x7ef93c13 -0, 28299000, 28299000, 219000, 14, 0x0c6901cb -0, 28300000, 28300000, 239000, 7366, 0x5bed7fcd -0, 28539000, 28539000, 239000, 14, 0x064900eb -0, 28540000, 28540000, 599000, 4564, 0x7f4c014b -0, 29139000, 29139000, 599000, 14, 0x0729010b -0, 29140000, 29140000, 219000, 4637, 0x682626b7 -0, 29359000, 29359000, 219000, 14, 0x0809012b -0, 29360000, 29360000, 1679000, 5358, 0x29e30c48 -0, 31039000, 31039000, 1679000, 14, 0x08e9014b +0, 0, 0, 279000, 14, 0x05d900db +0, 279000, 279000, 279000, 14, 0x064900eb +0, 280000, 280000, 4999000, 14, 0x06b900fb +0, 5279000, 5279000, 4999000, 14, 0x0729010b +0, 5280000, 5280000, 5019000, 14, 0x0799011b +0, 10299000, 10299000, 5019000, 14, 0x0809012b +0, 10300000, 10300000, 3599000, 14, 0x0879013b +0, 13899000, 13899000, 3599000, 14, 0x08e9014b +0, 13900000, 13900000, 219000, 14, 0x0959015b +0, 14119000, 14119000, 219000, 14, 0x09c9016b +0, 14120000, 14120000, 1439000, 14, 0x0a39017b +0, 15559000, 15559000, 1439000, 14, 0x0aa9018b +0, 15560000, 15560000, 39000, 14, 0x0b19019b +0, 15599000, 15599000, 39000, 14, 0x0b8901ab +0, 15600000, 15600000, 159000, 1168, 0xd69da022 +0, 15759000, 15759000, 159000, 14, 0x0c6901cb +0, 15760000, 15760000, 239000, 1544, 0xc5f116f1 +0, 15999000, 15999000, 239000, 14, 0x064900eb +0, 16000000, 16000000, 339000, 1658, 0x73563033 +0, 16339000, 16339000, 339000, 14, 0x0729010b +0, 16340000, 16340000, 599000, 2343, 0x7ac2078f +0, 16939000, 16939000, 599000, 14, 0x0809012b +0, 16940000, 16940000, 459000, 2568, 0x6eaa5751 +0, 17399000, 17399000, 459000, 14, 0x08e9014b +0, 17400000, 17400000, 359000, 3422, 0xd9d0636e +0, 17759000, 17759000, 359000, 14, 0x09c9016b +0, 17760000, 17760000, 219000, 5078, 0x722c9862 +0, 17979000, 17979000, 219000, 14, 0x0aa9018b +0, 17980000, 17980000, 959000, 5808, 0x38dd7ae9 +0, 18939000, 18939000, 959000, 14, 0x0b8901ab +0, 18940000, 18940000, 219000, 6015, 0xd4d2c40c +0, 19159000, 19159000, 219000, 14, 0x0c6901cb +0, 19160000, 19160000, 259000, 6519, 0x08af4c86 +0, 19419000, 19419000, 259000, 14, 0x064900eb +0, 19420000, 19420000, 99000, 7061, 0xecf10368 +0, 19519000, 19519000, 99000, 14, 0x0729010b +0, 19520000, 19520000, 219000, 4773, 0xbee42fd4 +0, 19739000, 19739000, 219000, 14, 0x0809012b +0, 19740000, 19740000, 219000, 5546, 0xdb822be1 +0, 19959000, 19959000, 219000, 14, 0x08e9014b +0, 19960000, 19960000, 239000, 5754, 0xfdcc7285 +0, 20199000, 20199000, 239000, 14, 0x09c9016b +0, 20200000, 20200000, 139000, 6099, 0xa409dd67 +0, 20339000, 20339000, 139000, 14, 0x0aa9018b +0, 20340000, 20340000, 799000, 6839, 0xc5eecacc +0, 21139000, 21139000, 799000, 14, 0x0b8901ab +0, 21140000, 21140000, 239000, 4744, 0x4e451fa1 +0, 21379000, 21379000, 239000, 14, 0x0c6901cb +0, 21380000, 21380000, 339000, 5824, 0x5299778b +0, 21719000, 21719000, 339000, 14, 0x064900eb +0, 21720000, 21720000, 1439000, 6212, 0x6d15f72f +0, 23159000, 23159000, 1439000, 14, 0x0729010b +0, 23160000, 23160000, 1319000, 7082, 0xe5c91052 +0, 24479000, 24479000, 1319000, 14, 0x0809012b +0, 24480000, 24480000, 219000, 5345, 0x2e5eca40 +0, 24699000, 24699000, 219000, 14, 0x08e9014b +0, 24700000, 24700000, 219000, 5765, 0x118060f2 +0, 24919000, 24919000, 219000, 14, 0x09c9016b +0, 24920000, 24920000, 599000, 6557, 0x89275c90 +0, 25519000, 25519000, 599000, 14, 0x0aa9018b +0, 25520000, 25520000, 219000, 7091, 0x996904b9 +0, 25739000, 25739000, 219000, 14, 0x0b8901ab +0, 25740000, 25740000, 239000, 7305, 0xc23e44ae +0, 25979000, 25979000, 239000, 14, 0x0c6901cb +0, 25980000, 25980000, 359000, 7590, 0xc5a3a3e1 +0, 26339000, 26339000, 359000, 14, 0x064900eb +0, 26340000, 26340000, 219000, 4629, 0x7ad6fe08 +0, 26559000, 26559000, 219000, 14, 0x0729010b +0, 26560000, 26560000, 719000, 4785, 0xcd3f3f20 +0, 27279000, 27279000, 719000, 14, 0x0809012b +0, 27280000, 27280000, 459000, 6061, 0x8b04d1a5 +0, 27739000, 27739000, 459000, 14, 0x08e9014b +0, 27740000, 27740000, 239000, 6301, 0xe7de19b1 +0, 27979000, 27979000, 239000, 14, 0x09c9016b +0, 27980000, 27980000, 99000, 6736, 0x38b3a094 +0, 28079000, 28079000, 99000, 14, 0x0aa9018b +0, 28080000, 28080000, 219000, 7214, 0x0b783b73 +0, 28299000, 28299000, 219000, 14, 0x0b8901ab +0, 28300000, 28300000, 239000, 7366, 0x98bf842d +0, 28539000, 28539000, 239000, 14, 0x0c6901cb +0, 28540000, 28540000, 599000, 4564, 0x3d9600ab +0, 29139000, 29139000, 599000, 14, 0x064900eb +0, 29140000, 29140000, 219000, 4637, 0x01f02617 +0, 29359000, 29359000, 219000, 14, 0x0729010b +0, 29360000, 29360000, 1679000, 5358, 0x5b0f0ba8 +0, 31039000, 31039000, 1679000, 14, 0x0809012b +0, 31040000, 31040000, 359000, 14, 0x0879013b +0, 31399000, 31399000, 359000, 14, 0x08e9014b +0, 31400000, 31400000, 479000, 14, 0x0959015b +0, 31879000, 31879000, 479000, 14, 0x09c9016b diff --git a/tests/ref/fate/sub2video b/tests/ref/fate/sub2video index 80abe9c905..ad8c0503a5 100644 --- a/tests/ref/fate/sub2video +++ b/tests/ref/fate/sub2video @@ -183,4 +183,3 @@ 0, 1139, 1139, 1, 518400, 0xbab197ea 1, 227834000, 227834000, 1262000, 1264, 0xc1d9fc57 0, 1140, 1140, 1, 518400, 0xb046dd30 -0, 1145, 1145, 1, 518400, 0xbab197ea diff --git a/tests/ref/fate/sub2video_basic b/tests/ref/fate/sub2video_basic index 5f72e292c9..31e64242a4 100644 --- a/tests/ref/fate/sub2video_basic +++ b/tests/ref/fate/sub2video_basic @@ -2,94 +2,89 @@ #media_type 0: video #codec_id 0: rawvideo #dimensions 0: 720x480 -#sar 0: 0/1 -0, 3312, 3312, 1, 1382400, 0x00000000 -0, 3312, 3312, 1, 1382400, 0x8c93c2ba +#sar 0: 1/1 +0, 3312, 3312, 1, 1382400, 0xc637b893 0, 3436, 3436, 1, 1382400, 0x00000000 -0, 3684, 3684, 1, 1382400, 0xb02e32ca +0, 3684, 3684, 1, 1382400, 0x4c2960ca 0, 3802, 3802, 1, 1382400, 0x00000000 -0, 4520, 4520, 1, 1382400, 0x83b71116 +0, 4520, 4520, 1, 1382400, 0x5fa18966 0, 4584, 4584, 1, 1382400, 0x00000000 -0, 4586, 4586, 1, 1382400, 0x85547fd1 +0, 4586, 4586, 1, 1382400, 0x55f4b7b1 0, 4645, 4645, 1, 1382400, 0x00000000 -0, 4648, 4648, 1, 1382400, 0x00000000 -0, 4648, 4648, 1, 1382400, 0xb6a8f181 +0, 4648, 4648, 1, 1382400, 0xdfa4cf32 0, 4715, 4715, 1, 1382400, 0x00000000 -0, 4717, 4717, 1, 1382400, 0xb64d1a2c +0, 4717, 4717, 1, 1382400, 0x35023df8 0, 4748, 4748, 1, 1382400, 0x00000000 -0, 4750, 4750, 1, 1382400, 0x7b37ecf3 +0, 4750, 4750, 1, 1382400, 0xed933219 0, 4792, 4792, 1, 1382400, 0x00000000 -0, 4993, 4993, 1, 1382400, 0xdc025bd1 +0, 4993, 4993, 1, 1382400, 0x1b26389a 0, 5027, 5027, 1, 1382400, 0x00000000 -0, 5029, 5029, 1, 1382400, 0x688b294d +0, 5029, 5029, 1, 1382400, 0xf0c7028b 0, 5068, 5068, 1, 1382400, 0x00000000 -0, 5070, 5070, 1, 1382400, 0xa2b33d1b +0, 5070, 5070, 1, 1382400, 0x395f521d 0, 5117, 5117, 1, 1382400, 0x00000000 -0, 5119, 5119, 1, 1382400, 0xb3e525e3 +0, 5119, 5119, 1, 1382400, 0x1ea87415 0, 5168, 5168, 1, 1382400, 0x00000000 -0, 5170, 5170, 1, 1382400, 0xaa8fbdd7 +0, 5170, 5170, 1, 1382400, 0xc6effdc1 0, 5216, 5216, 1, 1382400, 0x00000000 -0, 5218, 5218, 1, 1382400, 0x7b7f26dd +0, 5218, 5218, 1, 1382400, 0xba6846f8 0, 5249, 5249, 1, 1382400, 0x00000000 -0, 5251, 5251, 1, 1382400, 0x15e2f836 +0, 5251, 5251, 1, 1382400, 0x033c5d5b 0, 5289, 5289, 1, 1382400, 0x00000000 -0, 5291, 5291, 1, 1382400, 0x0fee9b0c +0, 5291, 5291, 1, 1382400, 0xef5abf66 0, 5358, 5358, 1, 1382400, 0x00000000 -0, 5360, 5360, 1, 1382400, 0x89d62791 +0, 5360, 5360, 1, 1382400, 0xec747954 0, 5429, 5429, 1, 1382400, 0x00000000 -0, 5431, 5431, 1, 1382400, 0xa6a9fd74 +0, 5431, 5431, 1, 1382400, 0xfa34bcaf 0, 5490, 5490, 1, 1382400, 0x00000000 -0, 5491, 5491, 1, 1382400, 0x7896178d +0, 5491, 5491, 1, 1382400, 0x8b7a709b 0, 5537, 5537, 1, 1382400, 0x00000000 -0, 5588, 5588, 1, 1382400, 0x01751a52 +0, 5588, 5588, 1, 1382400, 0xc333382f 0, 5647, 5647, 1, 1382400, 0x00000000 -0, 5688, 5688, 1, 1382400, 0xa3959c6f +0, 5688, 5688, 1, 1382400, 0xabe5dfcf 0, 5770, 5770, 1, 1382400, 0x00000000 -0, 5772, 5772, 1, 1382400, 0x3d3ea47b +0, 5772, 5772, 1, 1382400, 0x56948101 0, 5826, 5826, 1, 1382400, 0x00000000 -0, 5828, 5828, 1, 1382400, 0x593f8b24 +0, 5828, 5828, 1, 1382400, 0xb747834a 0, 5931, 5931, 1, 1382400, 0x00000000 -0, 5933, 5933, 1, 1382400, 0x171f05ba +0, 5933, 5933, 1, 1382400, 0x3448baad 0, 6001, 6001, 1, 1382400, 0x00000000 -0, 6003, 6003, 1, 1382400, 0xb014cdf1 +0, 6003, 6003, 1, 1382400, 0xaabe4f37 0, 6054, 6054, 1, 1382400, 0x00000000 -0, 6839, 6839, 1, 1382400, 0xd918e667 +0, 6839, 6839, 1, 1382400, 0x8a48cd6f 0, 6880, 6880, 1, 1382400, 0x00000000 -0, 7386, 7386, 1, 1382400, 0xc9406331 +0, 7386, 7386, 1, 1382400, 0x49518c43 0, 7419, 7419, 1, 1382400, 0x00000000 -0, 7501, 7501, 1, 1382400, 0xaf08b10d +0, 7501, 7501, 1, 1382400, 0x4a72fa21 0, 7549, 7549, 1, 1382400, 0x00000000 -0, 7551, 7551, 1, 1382400, 0x00000000 -0, 7551, 7551, 1, 1382400, 0x853a9d93 +0, 7551, 7551, 1, 1382400, 0xa82f7de8 0, 7589, 7589, 1, 1382400, 0x00000000 -0, 7605, 7605, 1, 1382400, 0x7491a87d +0, 7605, 7605, 1, 1382400, 0xeba0b5f3 0, 7647, 7647, 1, 1382400, 0x00000000 -0, 7649, 7649, 1, 1382400, 0xf7383c58 +0, 7649, 7649, 1, 1382400, 0xd6a91770 0, 7697, 7697, 1, 1382400, 0x00000000 -0, 7699, 7699, 1, 1382400, 0xe66be411 +0, 7699, 7699, 1, 1382400, 0x222f827c 0, 7743, 7743, 1, 1382400, 0x00000000 -0, 8032, 8032, 1, 1382400, 0xd6850362 +0, 8032, 8032, 1, 1382400, 0x3270f4ff 0, 8082, 8082, 1, 1382400, 0x00000000 -0, 8084, 8084, 1, 1382400, 0x3e1ed109 +0, 8084, 8084, 1, 1382400, 0x40813cb3 0, 8115, 8115, 1, 1382400, 0x00000000 -0, 8116, 8116, 1, 1382400, 0x39c1b7bd +0, 8116, 8116, 1, 1382400, 0x9d8fde41 0, 8160, 8160, 1, 1382400, 0x00000000 -0, 8180, 8180, 1, 1382400, 0x35b85f2e +0, 8180, 8180, 1, 1382400, 0xc6d7a701 0, 8207, 8207, 1, 1382400, 0x00000000 -0, 8209, 8209, 1, 1382400, 0x00000000 -0, 8209, 8209, 1, 1382400, 0x83f103e5 +0, 8209, 8209, 1, 1382400, 0x9d45f2dc 0, 8247, 8247, 1, 1382400, 0x00000000 -0, 8249, 8249, 1, 1382400, 0xbc1ca9b3 +0, 8249, 8249, 1, 1382400, 0x8525ee40 0, 8278, 8278, 1, 1382400, 0x00000000 -0, 8281, 8281, 1, 1382400, 0x94d4a51e +0, 8281, 8281, 1, 1382400, 0x5b26b98b 0, 8321, 8321, 1, 1382400, 0x00000000 -0, 8323, 8323, 1, 1382400, 0xf88cdfde +0, 8323, 8323, 1, 1382400, 0x51be311f 0, 8367, 8367, 1, 1382400, 0x00000000 -0, 8565, 8565, 1, 1382400, 0xdd51423b +0, 8565, 8565, 1, 1382400, 0x00a4f2a3 0, 8611, 8611, 1, 1382400, 0x00000000 -0, 8669, 8669, 1, 1382400, 0x08259fa4 +0, 8669, 8669, 1, 1382400, 0x40a445e8 0, 8708, 8708, 1, 1382400, 0x00000000 -0, 8941, 8941, 1, 1382400, 0x1663fa34 +0, 8941, 8941, 1, 1382400, 0x43ef5128 0, 8994, 8994, 1, 1382400, 0x00000000 -0, 8996, 8996, 1, 1382400, 0xda2ceb55 -0, 9027, 9027, 1, 1382400, 0x00000000 +0, 8996, 8996, 1, 1382400, 0x3c3e3819 diff --git a/tests/ref/fate/sub2video_time_limited b/tests/ref/fate/sub2video_time_limited index 9fb6fb06f9..715af02fee 100644 --- a/tests/ref/fate/sub2video_time_limited +++ b/tests/ref/fate/sub2video_time_limited @@ -2,7 +2,5 @@ #media_type 0: video #codec_id 0: rawvideo #dimensions 0: 1920x1080 -#sar 0: 0/1 -0, 2, 2, 1, 8294400, 0x00000000 +#sar 0: 1/1 0, 2, 2, 1, 8294400, 0xa87c518f -0, 10, 10, 1, 8294400, 0xa87c518f From patchwork Thu Sep 30 19:31:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30801 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp985922iob; Thu, 30 Sep 2021 12:32:12 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzQFGCMZRyA5prRIhmswwH/zlESgeRRr84Hdf9VZm10FYdQrv6KcaoaCNj2/zKdfgwGDVcK X-Received: by 2002:a17:906:6448:: with SMTP id l8mr1168406ejn.301.1633030332182; Thu, 30 Sep 2021 12:32:12 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hr27si4053157ejc.306.2021.09.30.12.32.10; Thu, 30 Sep 2021 12:32:12 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=RDqSA57B; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D2C3668AAA6; Thu, 30 Sep 2021 22:31:12 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12olkn2055.outbound.protection.outlook.com [40.92.22.55]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3D70868AA83 for ; Thu, 30 Sep 2021 22:31:08 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZGGBpU4Ihv1sq/iQqevqPCf2SVD4tXS83l6M6yXj5WMQrj3wdlQkdzAaFzdD0u9y3R9HLZx6kRjDOSlSodxJUX0OoOp+hdufoYzXxnPEByN75sq2C10Pj43Eb9AcEC2OgxVNJWMZBGTph6tXOhZ1uyWqz3d8XeY4tiIWWbdafk+Rouu6tV3O5T6xMlOj57Gq3eYVu1O+P5KTcAhCZFtokuCdZ8Qc5k8jjPBc/hp7/LtddCQE7UvYeWzafamRE6DEoFmmwkec0M6K0SGlDMU6cabU8+26XdDltzEHJmCB/S9gy+BNzEg2kJDuXRVPrsmjqyybj5LBkbTDw7rlwnlqRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=DysGA5xZcgZTLaloLJHAYyoxUEwJYnEynAv87+I1BCI=; b=cM+2lpP4GuSJxBVgNJyuWJT79jDu1dleA6OFSfVI2Wa21SiPtx8WjyhG3Q/GA0EQ6kwiylJNR5ZiEMR6IQhMXecsncfLEa4Bpcnhz5EZ0RxFffAxIVbuq2XnQ8JJ2fXbO6jPXIAljK9pOnkkhQtLQJt/2pjTBQC8h+e9aqd2YxKKS61gc9X9c7rZib9ABF52BAkjDRo3KvkjOIENH3sWX0tXG9KQyKRuvhM0ZTtSwfNVzfOwipNl9oxDNykp0SibDo2B2BV3NxkKfo748teqUpV7+OtnFia6YU20byNq37tXh0/en7s0eBUmTCdIBiCq2XmKJHZfrIRoA3fSCcp5Hw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DysGA5xZcgZTLaloLJHAYyoxUEwJYnEynAv87+I1BCI=; b=RDqSA57BJc3GNPAMKE6soMkp//Apb3OjZi3AkgNchqTrXLzdg/0zIRHhoBDhKA6GBVX/9H7pM0GFSQ1+D9/u7ELfEQP36pcpWc1iNE4sxFn/mTgVF3bQwAGFHDOL95ryPKIXFCoiARhg9PVZ7paT9cwy5M+66XSNALIAwpPn9Fh73l1hGJxZxLOrZTQMm0szhgpR/EzMj7JxwdNO85qWB8jobLPyMDkg7wfFpFmI5WV5FsYVsc9kQivbbr1IbmlX49ouUJ5GqNXTW1ZQtOiyVsSB5hSpXYJOI9wfHn/kteO553/PeySVbZSHAPl709fWkCdXZG2A2vsbKEllnPFV9g== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5519.namprd04.prod.outlook.com (2603:10b6:208:da::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:31:06 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:31:06 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 07/14] avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters Thread-Index: AQHXtjG2wY2tQEjpKUSdICaTHyC1uA== Date: Thu, 30 Sep 2021 19:31:05 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> <8a75051fd7be111f4ec52073a6da6679724a7127.1633027275.git.softworkz@hotmail.com> In-Reply-To: <8a75051fd7be111f4ec52073a6da6679724a7127.1633027275.git.softworkz@hotmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [LnGR5wrDtIPobjb0yktjwgfSVaqyiLPs] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: aafa7de1-0ecc-41f4-6d5f-08d98448d8d5 x-ms-traffictypediagnostic: MN2PR04MB5519: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 45SrweDCgpFFgJqlWjzQPOhxAgq9qdsYObVIsxHHAZtzzc7/Qnbcik/lVsmuiODedNf/U8Fn6MU96A1nQysU2skTgYclhltk6FjJxptValxf2wCicZ5Rig3ysS01vqt8jPLTmREWDwFVlY5F91o2X1wwVw82YQ2OTytuwgf346fvtGjZoeqH9XRShowQ3eKlutPrkAWJMwAWp+J7T1eO5cwB+546xYHkvkUE8p/mgDfdVqBC8oQniW5AqMkZROlV5C7H+Hahqjl6UMDoM9Meu2dMuu2ei2D3h2o0ladB7LHj+G+f+H1EqYlR8CmdiR8zDaUN7UvokSuCLy5uvZcER0u4zbeetOrXUiNWZCtqfEcO/J6wJIqo6SPYFzB109uDTtvcvMOT3Pw1qvGnk12X9+BwsXxT+8XR0Cmec5VhwBmuSFbAjCxGMuvWWvMBoqxTxsT7zD2wz3ldmchDWeyjIQl+wNqgQFdKB+AZZCkRolM= x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: rXTr2JfozwR75Y0xxPC8uruUsUOlPFfmcfvabgkE5emH7BURsxLkOBNZ6F5+AVwCcMK20Ad+Qc5YnHPRiFbrL3y5MRYKkEPot/1CDstyEBCCc2OC3VK+/Ng5ZbSe+lhreruoN6I4qOP8F3OWgEpFtQ== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: aafa7de1-0ecc-41f4-6d5f-08d98448d8d5 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:31:05.9473 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5519 Subject: [FFmpeg-devel] [PATCH v10 07/14] avfilter/overlay_textsubs: Add overlay_textsubs and textsubs2video filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Vj8H4OKUNf5v - overlay_textsubs {VS -> V) Overlay text subtitles onto a video stream. - textsubs2video {S -> V) Converts text subtitles to video frames Signed-off-by: softworkz --- configure | 2 + doc/filters.texi | 69 ++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 2 + libavfilter/avfilter.c | 18 +- libavfilter/vf_overlay_textsubs.c | 609 ++++++++++++++++++++++++++++++ 6 files changed, 697 insertions(+), 5 deletions(-) create mode 100644 libavfilter/vf_overlay_textsubs.c diff --git a/configure b/configure index d18bb73ed6..37fc4c20e7 100755 --- a/configure +++ b/configure @@ -3624,6 +3624,7 @@ openclsrc_filter_deps="opencl" overlay_opencl_filter_deps="opencl" overlay_qsv_filter_deps="libmfx" overlay_qsv_filter_select="qsvvpp" +overlay_textsubs_filter_deps="avcodec libass" overlay_vulkan_filter_deps="vulkan_lib libglslang" owdenoise_filter_deps="gpl" pad_opencl_filter_deps="opencl" @@ -3669,6 +3670,7 @@ superequalizer_filter_deps="avcodec" superequalizer_filter_select="rdft" surround_filter_deps="avcodec" surround_filter_select="rdft" +textsub2video_filter_deps="avcodec libass" tinterlace_filter_deps="gpl" tinterlace_merge_test_deps="tinterlace_filter" tinterlace_pad_test_deps="tinterlace_filter" diff --git a/doc/filters.texi b/doc/filters.texi index 18b5273e6e..2f81df6489 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -25297,6 +25297,75 @@ The graphicsub2video is inserted automatically for compatibility with legacy com ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:0][0:1]overlay" output.mp4 @end example @end itemize + +@section overlay_textsubs + +Overlay text subtitles onto a video stream. + +This filter supersedes the classic @ref{subtitles} filter opposed to which it does no longer require to open and access the source stream separately, which is often causing problems or doesn't even work for non-local or slow sources. + +Inputs: +- 0: Video [YUV420P, YUV422P, YUV444P, ARGB, RGBA, ABGR, BGRA, RGB24, BGR24] +- 1: Subtitles [text] + +Outputs: +- 0: Video (same as input) + +It accepts the following parameters: + +@table @option + +@item alpha +Process alpha channel, by default alpha channel is untouched. + +@item fonts_dir +Set a directory path containing fonts that can be used by the filter. +These fonts will be used in addition to whatever the font provider uses. + +@item force_style +Override default style or script info parameters of the subtitles. It accepts a +string containing ASS style format @code{KEY=VALUE} couples separated by ",". + +@end table + +@section textsub2video + +Converts text subtitles to video frames. + +For overlaying text subtitles onto video frames it is recommended to use the overlay_textsubs filter. +The textsub2video is useful for for creating transparent text-frames when overlay is done via hw acceleration + +Inputs: +- 0: Subtitles [text] + +Outputs: +- 0: Video [ARGB, RGBA, ABGR, BGRA] + +It accepts the following parameters: + +@table @option + +@item rate, r +Set the framerate for updating overlay frames. +Normally, overlay frames will only be updated each time when the subtitles to display are changing. +In cases where subtitles include advanced features (like animation), this parameter determines the frequency by which the overlay frames should be updated. + +@item size, s +Set the output frame size. +Allows to override the size of output video frames. + +@item alpha +Process alpha channel, by default alpha channel is untouched. + +@item fonts_dir +Set a directory path containing fonts that can be used by the filter. +These fonts will be used in addition to whatever the font provider uses. + +@item force_style +Override default style or script info parameters of the subtitles. It accepts a +string containing ASS style format @code{KEY=VALUE} couples separated by ",". + +@end table @c man end SUBTITLE FILTERS @chapter Multimedia Filters diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 01c3926bf8..584a97c9a3 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -366,6 +366,7 @@ OBJS-$(CONFIG_OVERLAY_OPENCL_FILTER) += vf_overlay_opencl.o opencl.o \ opencl/overlay.o framesync.o OBJS-$(CONFIG_OVERLAY_QSV_FILTER) += vf_overlay_qsv.o framesync.o OBJS-$(CONFIG_OVERLAY_GRAPHICSUBS_FILTER) += vf_overlay_graphicsubs.o framesync.o +OBJS-$(CONFIG_OVERLAY_TEXTSUBS_FILTER) += vf_overlay_textsubs.o OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) += vf_overlay_vulkan.o vulkan.o OBJS-$(CONFIG_OWDENOISE_FILTER) += vf_owdenoise.o OBJS-$(CONFIG_PAD_FILTER) += vf_pad.o @@ -454,6 +455,7 @@ OBJS-$(CONFIG_SWAPRECT_FILTER) += vf_swaprect.o OBJS-$(CONFIG_SWAPUV_FILTER) += vf_swapuv.o OBJS-$(CONFIG_TBLEND_FILTER) += vf_blend.o framesync.o OBJS-$(CONFIG_TELECINE_FILTER) += vf_telecine.o +OBJS-$(CONFIG_TEXTSUB2VIDEO_FILTER) += vf_overlay_textsubs.o OBJS-$(CONFIG_THISTOGRAM_FILTER) += vf_histogram.o OBJS-$(CONFIG_THRESHOLD_FILTER) += vf_threshold.o framesync.o OBJS-$(CONFIG_THUMBNAIL_FILTER) += vf_thumbnail.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index c08145e274..8c3702aeea 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -347,6 +347,7 @@ extern const AVFilter ff_vf_overlay; extern const AVFilter ff_vf_overlay_opencl; extern const AVFilter ff_vf_overlay_qsv; extern const AVFilter ff_vf_overlay_graphicsubs; +extern const AVFilter ff_vf_overlay_textsubs; extern const AVFilter ff_vf_overlay_vulkan; extern const AVFilter ff_vf_overlay_cuda; extern const AVFilter ff_vf_owdenoise; @@ -527,6 +528,7 @@ extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_svf_graphicsub2video; +extern const AVFilter ff_svf_textsub2video; /* multimedia sources */ extern const AVFilter ff_avsrc_amovie; diff --git a/libavfilter/avfilter.c b/libavfilter/avfilter.c index c567b7893b..639a54096b 100644 --- a/libavfilter/avfilter.c +++ b/libavfilter/avfilter.c @@ -464,7 +464,7 @@ static int64_t guess_status_pts(AVFilterContext *ctx, int status, AVRational lin return AV_NOPTS_VALUE; } -static int ff_request_frame_to_filter(AVFilterLink *link) +static int ff_request_frame_to_filter(AVFilterLink *link, int input_index) { int ret = -1; @@ -473,8 +473,8 @@ static int ff_request_frame_to_filter(AVFilterLink *link) link->frame_blocked_in = 1; if (link->srcpad->request_frame) ret = link->srcpad->request_frame(link); - else if (link->src->inputs[0]) - ret = ff_request_frame(link->src->inputs[0]); + else if (link->src->inputs[input_index]) + ret = ff_request_frame(link->src->inputs[input_index]); if (ret < 0) { if (ret != AVERROR(EAGAIN) && ret != link->status_in) ff_avfilter_link_set_in_status(link, ret, guess_status_pts(link->src, ret, link->time_base)); @@ -1173,6 +1173,14 @@ static int forward_status_change(AVFilterContext *filter, AVFilterLink *in) { unsigned out = 0, progress = 0; int ret; + int input_index = 0; + + for (int i = 0; i < in->dst->nb_inputs; i++) { + if (&in->dst->input_pads[i] == in->dstpad) { + input_index = i; + break; + } + } av_assert0(!in->status_out); if (!filter->nb_outputs) { @@ -1182,7 +1190,7 @@ static int forward_status_change(AVFilterContext *filter, AVFilterLink *in) while (!in->status_out) { if (!filter->outputs[out]->status_in) { progress++; - ret = ff_request_frame_to_filter(filter->outputs[out]); + ret = ff_request_frame_to_filter(filter->outputs[out], input_index); if (ret < 0) return ret; } @@ -1219,7 +1227,7 @@ static int ff_filter_activate_default(AVFilterContext *filter) for (i = 0; i < filter->nb_outputs; i++) { if (filter->outputs[i]->frame_wanted_out && !filter->outputs[i]->frame_blocked_in) { - return ff_request_frame_to_filter(filter->outputs[i]); + return ff_request_frame_to_filter(filter->outputs[i], 0); } } return FFERROR_NOT_READY; diff --git a/libavfilter/vf_overlay_textsubs.c b/libavfilter/vf_overlay_textsubs.c new file mode 100644 index 0000000000..ac99943d57 --- /dev/null +++ b/libavfilter/vf_overlay_textsubs.c @@ -0,0 +1,609 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * overlay text subtitles on top of a video frame + */ + +#include +#include + +#include "drawutils.h" +#include "filters.h" + +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" + +typedef struct TextSubsContext { + const AVClass *class; + ASS_Library *library; + ASS_Renderer *renderer; + ASS_Track *track; + + char *default_font_path; + char *fonts_dir; + char *fc_file; + double font_scale; + double font_size; + char *force_style; + char *language; + int margin; + + int alpha; + FFDrawContext draw; + + int got_header; + int out_w, out_h; + AVRational frame_rate; + AVFrame *last_frame; + int eof; +} TextSubsContext; + +/* libass supports a log level ranging from 0 to 7 */ +static const int ass_libavfilter_log_level_map[] = { + AV_LOG_QUIET, /* 0 */ + AV_LOG_PANIC, /* 1 */ + AV_LOG_FATAL, /* 2 */ + AV_LOG_ERROR, /* 3 */ + AV_LOG_WARNING, /* 4 */ + AV_LOG_INFO, /* 5 */ + AV_LOG_VERBOSE, /* 6 */ + AV_LOG_DEBUG, /* 7 */ +}; + +static void ass_log(int ass_level, const char *fmt, va_list args, void *ctx) +{ + const int ass_level_clip = av_clip(ass_level, 0, FF_ARRAY_ELEMS(ass_libavfilter_log_level_map) - 1); + const int level = ass_libavfilter_log_level_map[ass_level_clip]; + + av_vlog(ctx, level, fmt, args); + av_log(ctx, level, "\n"); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + TextSubsContext *s = ctx->priv; + + if (s->track) + ass_free_track(s->track); + if (s->renderer) + ass_renderer_done(s->renderer); + if (s->library) + ass_library_done(s->library); + + s->track = NULL; + s->renderer = NULL; + s->library = NULL; + + av_frame_free(&s->last_frame); +} + +static int overlay_textsubs_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink0 = ctx->inputs[0]; + AVFilterLink *inlink1 = ctx->inputs[1]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input0 video formats */ + formats = ff_draw_supported_pixel_formats(0); + if ((ret = ff_formats_ref(formats, &inlink0->outcfg.formats)) < 0) + return ret; + + /* set input1 subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink1->outcfg.formats)) < 0) + return ret; + + /* set output0 video formats */ + formats = ff_draw_supported_pixel_formats(0); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + + outlink->w = ctx->inputs[0]->w; + outlink->h = ctx->inputs[0]->h; + outlink->time_base = ctx->inputs[0]->time_base; + + return 0; +} + +static int config_input_main(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = inlink->dst->priv; + int ret; + + ret = ff_draw_init(&s->draw, inlink->format, s->alpha ? FF_DRAW_PROCESS_ALPHA : 0); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize ff_draw.\n"); + return ret; + } + + ass_set_frame_size (s->renderer, inlink->w, inlink->h); + ass_set_pixel_aspect(s->renderer, av_q2d(inlink->sample_aspect_ratio)); + + av_log(ctx, AV_LOG_VERBOSE, "Subtitle screen: %dx%d\n\n\n\n", inlink->w, inlink->h); + + return 0; +} + +/* libass stores an RGBA color in the format RRGGBBTT, where TT is the transparency level */ +#define AR(c) ( (c)>>24) +#define AG(c) (((c)>>16)&0xFF) +#define AB(c) (((c)>>8) &0xFF) +#define AA(c) ((0xFF-(c)) &0xFF) + +static void overlay_ass_image(TextSubsContext *s, AVFrame *picref, + const ASS_Image *image) +{ + for (; image; image = image->next) { + uint8_t rgba_color[] = {AR(image->color), AG(image->color), AB(image->color), AA(image->color)}; + FFDrawColor color; + ff_draw_color(&s->draw, &color, rgba_color); + ff_blend_mask(&s->draw, &color, + picref->data, picref->linesize, + picref->width, picref->height, + image->bitmap, image->stride, image->w, image->h, + 3, 0, image->dst_x, image->dst_y); + } +} + +static void process_header(AVFilterContext *link, AVFrame *sub) +{ + TextSubsContext *s = link->priv; + ASS_Track *track = s->track; + ASS_Style *style; + int sid = 0; + + if (!track) + return; + + if (sub && sub->subtitle_header) { + char *subtitle_header = (char *)sub->subtitle_header->data; + ass_process_codec_private(s->track, subtitle_header, strlen(subtitle_header)); + } + else { + char* subtitle_header = avpriv_ass_get_subtitle_header_default(0); + if (!subtitle_header) + return; + + ass_process_codec_private(s->track, subtitle_header, strlen(subtitle_header)); + av_free(subtitle_header); + } + + if (s->language) + s->track->Language = strdup(s->language); + + if (!s->track->event_format) { + s->track->event_format = strdup("ReadOrder, Layer, Style, Name, MarginL, MarginR, MarginV, Effect, Text"); + } + + if (s->track->n_styles == 0) { + sid = ass_alloc_style(track); + style = &s->track->styles[sid]; + style->Name = strdup("Default"); + style->PrimaryColour = 0xffffff00; + style->SecondaryColour = 0x00ffff00; + style->OutlineColour = 0x00000000; + style->BackColour = 0x00000080; + style->Bold = 200; + style->ScaleX = 1.0; + style->ScaleY = 1.0; + style->Spacing = 0; + style->BorderStyle = 1; + style->Outline = 2; + style->Shadow = 3; + style->Alignment = 2; + } + else + style = &s->track->styles[sid]; + + style->FontSize = s->font_size; + style->MarginL = style->MarginR = style->MarginV = s->margin; + + track->default_style = sid; + + s->got_header = 1; +} + +static int filter_video_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + int detect_change = 0; + + int64_t time_ms = frame->pts * av_q2d(inlink->time_base) * 1000; + + ASS_Image *image = ass_render_frame(s->renderer, s->track, time_ms, &detect_change); + + if (detect_change) + av_log(ctx, AV_LOG_DEBUG, "Change happened at time ms:%lld\n", time_ms); + + overlay_ass_image(s, frame, image); + + return ff_filter_frame(ctx->outputs[0], frame); +} + +static int filter_subtitle_frame(AVFilterLink *inlink, AVFrame *sub) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + + if (!s->got_header) + process_header(ctx, sub); + + const int64_t start_time = av_rescale_q(sub->subtitle_pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t duration = sub->subtitle_end_time; + + for (unsigned i = 0; i < sub->num_subtitle_areas; i++) { + char *ass_line = sub->subtitle_areas[i]->ass; + if (!ass_line) + break; + ass_process_chunk(s->track, ass_line, strlen(ass_line), start_time, duration); + } + + av_frame_free(&sub); + return 0; +} + +static av_cold int init(AVFilterContext *ctx) +{ + TextSubsContext *s = ctx->priv; + + s->library = ass_library_init(); + + if (!s->library) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize libass.\n"); + return AVERROR(EINVAL); + } + + ass_set_message_cb(s->library, ass_log, ctx); + + /* Initialize fonts */ + if (s->fonts_dir) + ass_set_fonts_dir(s->library, s->fonts_dir); + + ass_set_extract_fonts(s->library, 1); + + s->renderer = ass_renderer_init(s->library); + if (!s->renderer) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize libass renderer.\n"); + return AVERROR(EINVAL); + } + + s->track = ass_new_track(s->library); + if (!s->track) { + av_log(ctx, AV_LOG_ERROR, "ass_new_track() failed!\n"); + return AVERROR(EINVAL); + } + + ass_set_fonts(s->renderer, s->default_font_path, NULL, 1, s->fc_file, 1); + + if (s->force_style) { + char **list = NULL; + char *temp = NULL; + char *ptr = av_strtok(s->force_style, ",", &temp); + int i = 0; + while (ptr) { + av_dynarray_add(&list, &i, ptr); + if (!list) { + return AVERROR(ENOMEM); + } + ptr = av_strtok(NULL, ",", &temp); + } + av_dynarray_add(&list, &i, NULL); + if (!list) { + return AVERROR(ENOMEM); + } + ass_set_style_overrides(s->library, list); + av_free(list); + } + + return 0; +} + +static int textsub2video_query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input0 subtitle format */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output0 video format */ + formats = ff_draw_supported_pixel_formats(AV_PIX_FMT_FLAG_ALPHA); + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int textsub2video_config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + + if (s->out_w <= 0 || s->out_h <= 0) { + s->out_w = inlink->w; + s->out_h = inlink->h; + } + + return 0; +} + +static int textsub2video_config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + TextSubsContext *s = ctx->priv; + int ret; + + ret = ff_draw_init(&s->draw, outlink->format, FF_DRAW_PROCESS_ALPHA); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Could not initialize ff_draw.\n"); + return ret; + } + + if (s->out_w <= 0 || s->out_h <= 0) { + av_log(ctx, AV_LOG_ERROR, "No output image size set.\n"); + return AVERROR(EINVAL); + } + + ass_set_frame_size (s->renderer, s->out_w, s->out_h); + + outlink->w = s->out_w; + outlink->h = s->out_h; + outlink->sample_aspect_ratio = (AVRational){1,1}; + outlink->frame_rate = s->frame_rate; + + return 0; +} + +static int textsub2video_filter_frame(AVFilterLink *inlink, AVFrame *sub) +{ + AVFilterContext *ctx = inlink->dst; + TextSubsContext *s = ctx->priv; + + av_log(ctx, AV_LOG_VERBOSE, "textsub2video_filter_frame num_subtitle_rects: %d\n", sub->num_subtitle_areas); + + if (!s->got_header) + process_header(ctx, sub); + + const int64_t start_time = av_rescale_q(sub->subtitle_pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t duration = sub->subtitle_end_time; + + for (unsigned i = 0; i < sub->num_subtitle_areas; i++) { + char *ass_line = sub->subtitle_areas[i]->ass; + if (!ass_line) + break; + ass_process_chunk(s->track, ass_line, strlen(ass_line), start_time, duration); + } + + av_frame_free(&sub); + return 0; +} + +static int textsub2video_request_frame(AVFilterLink *outlink) +{ + TextSubsContext *s = outlink->src->priv; + const AVFilterLink *inlink = outlink->src->inputs[0]; + int64_t last_pts = outlink->current_pts; + int64_t next_pts; + int i, detect_change = 0; + AVFrame *out; + ASS_Image *image; + + if (last_pts == AV_NOPTS_VALUE) + last_pts = inlink->current_pts * av_q2d(inlink->time_base) / av_q2d(outlink->time_base); + + next_pts = last_pts + (1.0 / av_q2d(outlink->frame_rate) / av_q2d(outlink->time_base)); + + int64_t time_ms = next_pts * av_q2d(outlink->time_base) * 1000; + + image = ass_render_frame(s->renderer, s->track, time_ms, &detect_change); + + if (detect_change) + av_log(outlink->src, AV_LOG_DEBUG, "Change happened at time ms:%lld\n", time_ms); + else if (s->last_frame) { + out = av_frame_clone(s->last_frame); + if (!out) + return AVERROR(ENOMEM); + + out->pts = next_pts; + return ff_filter_frame(outlink, out); + } + + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) + return AVERROR(ENOMEM); + + for (i = 2; i < AV_NUM_DATA_POINTERS; i++) { + if (out->buf[i] && i != 1) + memset(out->buf[i]->data, 0, out->buf[i]->size); + } + + out->pts = next_pts; + + if (image) + overlay_ass_image(s, out, image); + + av_frame_free(&s->last_frame); + + s->last_frame = av_frame_clone(out); + + return ff_filter_frame(outlink, out); +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + TextSubsContext *s = ctx->priv; + AVFrame *in; + int ret; + + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + ret = ff_inlink_consume_frame(inlink, &in); + if (ret < 0) + return ret; + if (ret > 0) + return textsub2video_filter_frame(inlink, in); + + if (ff_outlink_frame_wanted(outlink)) { + if (!s->eof && ff_outlink_get_status(ctx->inputs[0])) { + s->eof = 1; + } + if (!s->eof && ff_inlink_queued_frames(ctx->inputs[0]) == 0) + ff_inlink_request_frame(ctx->inputs[0]); + if (s->eof && ff_inlink_queued_frames(ctx->inputs[0]) <= 0) { + ff_outlink_set_status(outlink, AVERROR_EOF, AV_NOPTS_VALUE); + } + else { + return textsub2video_request_frame(outlink); + } + + return 0; + } + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(TextSubsContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption overlay_textsubs_options[] = { + {"alpha", "enable processing of alpha channel", OFFSET(alpha), AV_OPT_TYPE_BOOL, {.i64 = 0 }, 0, 1, FLAGS}, + {"font_scale", "font scale factor", OFFSET(font_scale), AV_OPT_TYPE_DOUBLE, {.dbl = 1.0 }, 0.0, 100.0, FLAGS}, + {"font_size", "default font size", OFFSET(font_size), AV_OPT_TYPE_DOUBLE, {.dbl = 18.0}, 0.0, 100.0, FLAGS}, + {"force_style", "force subtitle style", OFFSET(force_style), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"margin", "default margin", OFFSET(margin), AV_OPT_TYPE_INT, {.i64 = 20 }, 0, INT_MAX, FLAGS}, + {"default_font_path", "path to default font", OFFSET(default_font_path), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fonts_dir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontsdir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontconfig_file", "fontconfig file to load", OFFSET(fc_file), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"language", "default language", OFFSET(language), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + { NULL } +}; + +static const AVOption textsub2video_options[] = { + {"rate", "set frame rate", OFFSET(frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="8"}, 0, INT_MAX, FLAGS}, + {"r", "set frame rate", OFFSET(frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="8"}, 0, INT_MAX, FLAGS}, + {"size", "set video size", OFFSET(out_w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, FLAGS}, + {"s", "set video size", OFFSET(out_w), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, FLAGS}, + {"font_scale", "font scale factor", OFFSET(font_scale), AV_OPT_TYPE_DOUBLE, {.dbl = 1.0 }, 0.0, 100.0, FLAGS}, + {"font_size", "default font size", OFFSET(font_size), AV_OPT_TYPE_DOUBLE, {.dbl = 18.0}, 0.0, 100.0, FLAGS}, + {"force_style", "force subtitle style", OFFSET(force_style), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"margin", "default margin", OFFSET(margin), AV_OPT_TYPE_INT, {.i64 = 20 }, 0, INT_MAX, FLAGS}, + {"default_font_path", "path to default font", OFFSET(default_font_path), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fonts_dir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontsdir", "directory to scan for fonts", OFFSET(fonts_dir), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"fontconfig_file", "fontconfig file to load", OFFSET(fc_file), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + {"language", "default language", OFFSET(language), AV_OPT_TYPE_STRING, {.str = NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + { NULL } +}; + +#if CONFIG_OVERLAY_TEXTSUBS_FILTER + +AVFILTER_DEFINE_CLASS(overlay_textsubs); + +static const AVFilterPad overlay_textsubs_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input_main, + .flags = AVFILTERPAD_FLAG_NEEDS_WRITABLE, + .filter_frame = filter_video_frame, + }, + { + .name = "overlay", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_subtitle_frame, + }, +}; + +static const AVFilterPad overlay_textsubs_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +const AVFilter ff_vf_overlay_textsubs = { + .name = "overlay_textsubs", + .description = NULL_IF_CONFIG_SMALL("Overlay textual subtitles on top of the input."), + .init = init, + .uninit = uninit, + .priv_size = sizeof(TextSubsContext), + .priv_class = &overlay_textsubs_class, + .query_formats = overlay_textsubs_query_formats, + FILTER_INPUTS(overlay_textsubs_inputs), + FILTER_OUTPUTS(overlay_textsubs_outputs), +}; +#endif + +#if CONFIG_TEXTSUB2VIDEO_FILTER + +AVFILTER_DEFINE_CLASS(textsub2video); + +static const AVFilterPad textsub2video_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .config_props = textsub2video_config_input, + }, +}; + +static const AVFilterPad textsub2video_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = textsub2video_config_output, + }, +}; + +const AVFilter ff_svf_textsub2video = { + .name = "textsub2video", + .description = NULL_IF_CONFIG_SMALL("Convert textual subtitles to video frames"), + .init = init, + .uninit = uninit, + .query_formats = textsub2video_query_formats, + .priv_size = sizeof(TextSubsContext), + .priv_class = &textsub2video_class, + .activate = activate, + FILTER_INPUTS(textsub2video_inputs), + FILTER_OUTPUTS(textsub2video_outputs), +}; +#endif From patchwork Thu Sep 30 19:31:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30809 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp986108iob; Thu, 30 Sep 2021 12:32:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzuJk+3+okn2S58leOEgna3gRCYUMznjVw9BJlQukvTsZA+a4HjiqSrGTOO9SCCO2CLamcZ X-Received: by 2002:a17:906:38db:: with SMTP id r27mr1208294ejd.338.1633030344211; Thu, 30 Sep 2021 12:32:24 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x36si4313986ede.190.2021.09.30.12.32.23; Thu, 30 Sep 2021 12:32:24 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=bPSuEifK; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CAF3E68AABC; Thu, 30 Sep 2021 22:31:14 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12olkn2021.outbound.protection.outlook.com [40.92.22.21]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D552568AABE for ; Thu, 30 Sep 2021 22:31:12 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I097SIaNycYQdNKd6OMkbQ3cW5h1apKIjcA7otG7DX4mMmZnulcfLB1PR0Li+EztmyWoMCHT5B65JIGLc+tosIQdAhctMMGQih9rPtiRRSJHZbVjcsiGNcl41ow7A3hsc1/oP7n9zF45aMAAl2SnpuXK0cZLFQ6tp3tb0aAhV4C8sBBFchGB7taDzk8W1qOINJ1L2ej4eGfzgm0B/HZLkHi4I4LsaFmqaT7TgUqti99fDTREK7XEMpxnQOS9naQf2FCkvtJ/9h2gG1CWyJu4M6BzWVWSlQz1SqZu0XndJyz8eOmiNGl0IFqSxntvkEHfKGZ9jsCwPsbO6oCqGw80rg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=mMt5xVoKtpgYj8OEpF+5ViF4lUlAtBXf9ndG4n3QBYw=; b=GxBaNHVGV9dx5FakrAEvYOWg02R68rVVHOOEP9fahmEg92leWwG0CQw6PVCLhGzOtg7Xe4SghsBI4nK2xgBcZtvldbrtxxuejr3teVhH+ZmBTnTS5IlWwa/RvQB/iNDEvuhuJss7bASwA8aQhoWvaVXjB/Pci1W6E3eA6YyekdAZdxbCXIh+mu4DCp2lTj6Nh0Xw1Ayb/wRX6/uhGU1DSJtdoD44TfSq0qPT0DaXRUcyhFL3Xy3BdsZ3KfFTI9Ub6s/I8mhFS/DiWZn9E02Q4m0ROj0hYCJJL6XDf/vVKtqJ+cctMPslbDIooIXFbIYAc9wbDz7GMrMUAKAxNxSnCQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mMt5xVoKtpgYj8OEpF+5ViF4lUlAtBXf9ndG4n3QBYw=; b=bPSuEifKATmTZmhtIZlioVBQ2X1vzY5znbjYuRj9PsV2UUsRViGOuHtvnTIbUgMtlqC3xCyABjAVBPVOO1SwaUh9/6SSZ1ul5r8Vm4QIN7ntwdUGuOM2WcQulpC/3MP+/H/Gknn5g5qq2OZ+5drtJ52hRmnAEyNFlbQG8hYQbGTh2zy42RTbkZorDIdhLlZa4FrUkf3lGo2V/GQnIQgUpzWz2mZR8GNO6NMRB/yx+aaiEqAszHlvdG1StP+P+YZwXrGI+S4xSUXqiqhaEzxKi06Lr8J2wFJ+RYib7H7H17VJJ0NAAEUN6EM7Bq+DR4gwqhqNn7sAnbdFOIljDA1udA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5519.namprd04.prod.outlook.com (2603:10b6:208:da::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:31:09 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:31:09 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 08/14] avfilter/textmod: Add textmod, censor and show_speaker filters Thread-Index: AQHXtjG4RTjUpFMdcEeITLAjVGP+sA== Date: Thu, 30 Sep 2021 19:31:09 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> <8a75051fd7be111f4ec52073a6da6679724a7127.1633027275.git.softworkz@hotmail.com> <48414b2bba60c4be4c41a1bd3ba8edae069f1fc9.1633027275.git.softworkz@hotmail.com> In-Reply-To: <48414b2bba60c4be4c41a1bd3ba8edae069f1fc9.1633027275.git.softworkz@hotmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [mAQ4B/EOENLMflbfPj5nyRRcYZn8luX4] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e5eafc04-b273-44d1-8a47-08d98448dae5 x-ms-traffictypediagnostic: MN2PR04MB5519: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: v5+NMoiynCT9fdP9TT09FfTszOwwuLkXWLTwKJHxtB73sPDEKwtEMRSW/d+2tSTgfPgEPTGRBrbZrx9ptlhWsXzayBRyjuPO+JXOWvVISyPu/xDwfyVJJCmqmFNJcFQSaAUFRzLz/UuHUQu31ZTYKSK/yY6lUz1GiY7qedpvR9zPyVEk15XKDzxVioq6O32KCIsjtQSJnaVBFFXEsUOlkcEUbvQjbkbWdgknnDyfLFzhHph9IAxp36uwNQrDkcxZ7y+DJr01nq0f1xT2iUOo3JK7DJ2dQvrYSdVHx8/a8Skt3j8FSvzgyK3LBikhDAKCC+An+dA2PlNgIfnCOGkMAvMMvFTi4osk2pZEiBqjiN7G7diSVrRe4ZKSKbs9ExKefnUAQU5DPaEA4oyLnR74voY8azOoVeEPZXCtrne/SxfUeMrmzsx6Y4Me8kdkiC+g1JBUYaj0x9I71FYFakbWzh5uzs09kNR1W7JKHnJA2E5i/U1oEapw8aVtlbMjf5ZgwGpbwpAoo83rO5xecjEk/A== x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: EelM+Msf1G6O+5JA83h0PvNb66E2RcZ6/eukn0Xx0pl44gisClZ1LP5IxMdwcu2FwkU2ufuJFDqAcWb3sD0KPqtVCFG82lqNAZEJCOnIr1OaTgwj5iY6uEDYFrcBwuz6Lat+EW8rq0YnDXyzluT1HA== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: e5eafc04-b273-44d1-8a47-08d98448dae5 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:31:09.3984 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5519 Subject: [FFmpeg-devel] [PATCH v10 08/14] avfilter/textmod: Add textmod, censor and show_speaker filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 4jLm9OnHhYYY - textmod {S -> S) Modify subtitle text in a number of ways - censor {S -> S) Censor subtitles using a word list - show_speaker {S -> S) Prepend speaker names from ASS subtitles to the visible text lines Signed-off-by: softworkz --- doc/filters.texi | 229 +++++++++++++ libavfilter/Makefile | 5 + libavfilter/allfilters.c | 3 + libavfilter/sf_textmod.c | 719 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 956 insertions(+) create mode 100644 libavfilter/sf_textmod.c diff --git a/doc/filters.texi b/doc/filters.texi index 2f81df6489..c751211017 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -25195,6 +25195,7 @@ tools. @c man end VIDEO SINKS + @chapter Subtitle Filters @c man begin SUBTITLE FILTERS @@ -25203,6 +25204,166 @@ existing filters using @code{--disable-filters}. Below is a description of the currently available subtitle filters. + +@section censor + +Censor selected words in text subtitles. + +Inputs: +- 0: Subtitles [text] + +Outputs: +- 0: Subtitles [text] + +It accepts the following parameters: + +@table @option +@item mode +The censoring mode to apply. + +Supported censoring modes are: + +@table @var +@item 0, keep_first_last +Replace all characters with the 'censor_char' except the first and the last character of a word. +For words with less than 4 characters, the last character will be replaced as well. +For words with less than 3 characters, the first character will be replaced as well. +@item 1, keep_first +Replace all characters with the 'censor_char' except the first character of a word. +For words with less than 3 characters, the first character will be replaced as well. +@item 2, all +Replace all characters with the 'censor_char'. +@end table + +@item words +A list of words to censor, separated by 'separator'. + +@item words_file +Specify a file from which to load the contents for the 'words' parameter. + +@item censor_char +Single character used as replacement for censoring. + +@item separator +Delimiter character for words. Used with replace_words and remove_words- Must be a single character. +The default is '.'. + +@end table + +@subsection Examples + +@itemize +@item +Change all characters to upper case while keeping all styles and animations: +@example +ffmpeg -i "https://streams.videolan.org/ffmpeg/mkv_subtitles.mkv" -filter_complex "[0:s]textmod=mode=to_upper" -map 0 -y out.mkv +@end example +@item +Remove a set of symbol characters for am improved and smoother visual apperance: +@example +ffmpeg -i "https://streams.videolan.org/ffmpeg/mkv_subtitles.mkv" -filter_complex "[0:s]textmod=mode=remove_chars:find='$&#@*§'" -map 0 -y out.mkv +@end example +@end itemize + + +@section stripstyles + +Remove all inline styles from subtitle events. + +It accepts the following parameters: + +@table @option +@item remove_animated +Also remove text which is subject to animation (default: true) +Usually, animated text elements are used used in addition to static subtitle lines for creating effects, so in most cases it is safe to remove the animation content. +If subtitle text is missing, try setting this to false. + +@end table + +@subsection Examples + +@itemize +@item +Remove all styles and animations from subtitles: +@example +ffmpeg -i "https://streams.videolan.org/samples/sub/SSA/subtitle_testing_complex.mkv" -filter_complex "[0:1]stripstyles" -map 0 output.mkv +@end example +@end itemize + +@section textmod + +Modify subtitle text in a number of ways. + +Inputs: +- 0: Subtitles [text] + +Outputs: +- 0: Subtitles [text] + +It accepts the following parameters: + +@table @option +@item mode +The kind of text modification to apply + +Supported operation modes are: + +@table @var +@item 0, leet +Convert subtitle text to 'leet speak'. It's primarily useful for testing as the modification will be visible with almost all text lines. +@item 1, to_upper +Change all text to upper case. Might improve readability. +@item 2, to_lower +Change all text to lower case. +@item 3, replace_chars +Replace one or more characters. Requires the find and replace parameters to be specified. +Both need to be equal in length. +The first char in find is replaced by the first char in replace, same for all subsequent chars. +@item 4, remove_chars +Remove certain characters. Requires the find parameter to be specified. +All chars in the find parameter string will be removed from all subtitle text. +@item 5, replace_words +Replace one or more words. Requires the find and replace parameters to be specified. Multiple words must be separated by the delimiter char specified vie the separator parameter (default: ','). +The number of words in the find and replace parameters needs to be equal. +The first word in find is replaced by the first word in replace, same for all subsequent words +@item 6, remove_words +Remove certain words. Requires the find parameter to be specified. Multiple words must be separated by the delimiter char specified vie the separator parameter (default: ','). +All words in the find parameter string will be removed from all subtitle text. +@end table + +@item find +Required for replace_chars, remove_chars, replace_words and remove_words. + +@item find_file +Specify a file from which to load the contents for the 'find' parameter. + +@item replace +Required for replace_chars and replace_words. + +@item replace_file +Specify a file from which to load the contents for the 'replace' parameter. + +@item separator +Delimiter character for words. Used with replace_words and remove_words- Must be a single character. +The default is '.'. + +@end table + +@subsection Examples + +@itemize +@item +Change all characters to upper case while keeping all styles and animations: +@example +ffmpeg -i "https://streams.videolan.org/ffmpeg/mkv_subtitles.mkv" -filter_complex "[0:s]textmod=mode=to_upper" -map 0 -y out.mkv +@end example +@item +Remove a set of symbol characters for am improved and smoother visual apperance: +@example +ffmpeg -i "https://streams.videolan.org/ffmpeg/mkv_subtitles.mkv" -filter_complex "[0:s]textmod=mode=remove_chars:find='$&#@*§'" -map 0 -y out.mkv +@end example +@end itemize + @section graphicsub2video Renders graphic subtitles as video frames. @@ -25247,6 +25408,11 @@ Overlay graphic subtitles onto a video stream. This filter can blend graphical subtitles on a video stream directly, i.e. without creating full-size alpha images first. The blending operation is limited to the area of the subtitle rectangles, which also means that no processing is done at times where no subtitles are to be displayed. +- 0: Video [YUV420P, YUV422P, YUV444P, ARGB, RGBA, ABGR, BGRA, RGB24, BGR24] +- 1: Subtitles [bitmap] + +Outputs: +- 0: Video (same as input) It accepts the following parameters: @@ -25328,6 +25494,69 @@ string containing ASS style format @code{KEY=VALUE} couples separated by ",". @end table +@section show_speaker + +Prepend speaker names to subtitle lines (when available). + +Subtitles in ASS/SSA format are often including the names of the persons +or character for each subtitle line. The show_speaker filter adds those names +to the actual subtitle text to make it visible on playback. + +Inputs: +- 0: Subtitles [text] + +Outputs: +- 0: Subtitles [text] + +It accepts the following parameters: + +@table @option +@item format +The format for prepending speaker names. Default is 'square_brackets'. + +Supported operation modes are: + +@table @var +@item 0, square_brackets +Enclose the speaker name in square brackets, followed by space ('[speaker] text'). +@item 1, round_brackets +Enclose the speaker name in round brackets, followed by space ('(speaker) text'). +@item 2, colon +Separate the speaker name with a colon and space ('speaker: text'). +@item 3, plain +Separate the speaker name with a space only ('speaker text'). +@end table + +@item line_break +Set thís parameter to insert a line break between speaker name and text instead of the space character. + +@item style +Allows to set a specific style for the speaker name text. + +This can be either a named style that exists in the ass subtitle header script (e.g. 'Default') or an ass style override code. +Example: @{\\c&HDD0000&\\be1\\i1\\bord10@} +This sets the color to blue, enables edge blurring, italic font and a border of size 10. + +The behavior is as follows: + +- When the style parameter is not provided, the filter will find the first position in the event string that is actual text. + The speaker name will be inserted at this position. This allows to have the speaker name shown in the same style like the + regular text, in case the string would start with a sequence of style codes. +- When the style parameter is provided, everything will be prepended to the original text: + Style Code or Style name >> Speaker Name >> Style Reset Code >> Original Text + +@end table + +@subsection Examples + +@itemize +@item +Prepend speaker names with blue text, smooth edges and blend/overlay that onto the video. +@example +ffmpeg -i INPUT -filter_complex "show_speaker=format=colon:style='@{\\c&HDD0000&\\be1@}',[0:v]overlay_textsubs" +@end example +@end itemize + @section textsub2video Converts text subtitles to video frames. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 584a97c9a3..df250ed07b 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -537,6 +537,11 @@ OBJS-$(CONFIG_YUVTESTSRC_FILTER) += vsrc_testsrc.o OBJS-$(CONFIG_NULLSINK_FILTER) += vsink_nullsink.o +# subtitle filters +OBJS-$(CONFIG_CENSOR_FILTER) += sf_textmod.o +OBJS-$(CONFIG_SHOW_SPEAKER_FILTER) += sf_textmod.o +OBJS-$(CONFIG_TEXTMOD_FILTER) += sf_textmod.o + # multimedia filters OBJS-$(CONFIG_ABITSCOPE_FILTER) += avf_abitscope.o OBJS-$(CONFIG_ADRAWGRAPH_FILTER) += f_drawgraph.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 8c3702aeea..232eac4ef6 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -527,6 +527,9 @@ extern const AVFilter ff_avf_showvolume; extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; +extern const AVFilter ff_sf_censor; +extern const AVFilter ff_sf_show_speaker; +extern const AVFilter ff_sf_textmod; extern const AVFilter ff_svf_graphicsub2video; extern const AVFilter ff_svf_textsub2video; diff --git a/libavfilter/sf_textmod.c b/libavfilter/sf_textmod.c new file mode 100644 index 0000000000..7397194f4b --- /dev/null +++ b/libavfilter/sf_textmod.c @@ -0,0 +1,719 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * text subtitle filter which allows to modify subtitle text in several ways + */ + +#include + +#include "libavutil/opt.h" +#include "internal.h" +#include "libavutil/ass_split_internal.h" +#include "libavutil/bprint.h" +#include "libavutil/file.h" + +static const char* leet_src = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"; +static const char* leet_dst = "abcd3f6#1jklmn0pq257uvwxyzAB(D3F6#1JKLMN0PQ257UVWXYZ"; + +enum TextModFilterType { + TM_TEXTMOD, + TM_CENSOR, + TM_SHOW_SPEAKER, +}; + +enum TextModOperation { + OP_LEET, + OP_TO_UPPER, + OP_TO_LOWER, + OP_REPLACE_CHARS, + OP_REMOVE_CHARS, + OP_REPLACE_WORDS, + OP_REMOVE_WORDS, + NB_OPS, +}; + +enum CensorMode { + CM_KEEP_FIRST_LAST, + CM_KEEP_FIRST, + CM_ALL, +}; + +enum ShowSpeakerMode { + SM_SQUARE_BRACKETS, + SM_ROUND_BRACKETS, + SM_COLON, + SM_PLAIN, +}; + +typedef struct TextModContext { + const AVClass *class; + enum AVSubtitleType format; + enum TextModFilterType filter_type; + enum TextModOperation operation; + enum CensorMode censor_mode; + enum ShowSpeakerMode speaker_mode; + char *find; + char *find_file; + char *style; + char *replace; + char *replace_file; + char *separator; + char *censor_char; + char **find_list; + int line_break; + int nb_find_list; + char **replace_list; + int nb_replace_list; +} TextModContext; + +static char **split_string(char *source, int *nb_elems, const char *delim) +{ + char **list = NULL; + char *temp = NULL; + char *ptr = av_strtok(source, delim, &temp); + + while (ptr) { + if (strlen(ptr)) { + av_dynarray_add(&list, nb_elems, ptr); + if (!list) + return NULL; + } + + ptr = av_strtok(NULL, delim, &temp); + } + + if (!list) + return NULL; + + for (int i = 0; i < *nb_elems; i++) { + list[i] = av_strdup(list[i]); + if (!list[i]) { + for (int n = 0; n < i; n++) + av_free(list[n]); + av_free(list); + return NULL; + } + } + + return list; +} + +static int load_text_from_file(AVFilterContext *ctx, const char *file_name, char **text, char separator) +{ + int err; + uint8_t *textbuf; + char *tmp; + size_t textbuf_size; + int offset = 0; + + if ((err = av_file_map(file_name, &textbuf, &textbuf_size, 0, ctx)) < 0) { + av_log(ctx, AV_LOG_ERROR, "The text file '%s' could not be read or is empty\n", file_name); + return err; + } + + if (textbuf_size > 1 && + (textbuf[0] == 0xFF && textbuf[1] == 0xFE + || textbuf[0] == 0xFE && textbuf[1] == 0xFF)) { + av_log(ctx, AV_LOG_ERROR, "UTF text files are not supported. File: %s\n", file_name); + return AVERROR(EINVAL); + } + + if (textbuf_size > 2 && textbuf[0] == 0xEF && textbuf[1] == 0xBB && textbuf[2] == 0xBF) + offset = 3; // UTF-8 + + if (textbuf_size > SIZE_MAX - 1 || !((tmp = av_strndup((char *)textbuf + offset, textbuf_size - offset)))) { + av_file_unmap(textbuf, textbuf_size); + return AVERROR(ENOMEM); + } + + av_file_unmap(textbuf, textbuf_size); + + for (size_t i = 0; i < strlen(tmp); i++) { + switch (tmp[i]) { + case '\n': + case '\r': + case '\f': + case '\v': + tmp[i] = separator; + } + } + + *text = tmp; + + return 0; +} + +static int load_files(AVFilterContext *ctx) +{ + TextModContext *s = ctx->priv; + int ret; + + if (!s->separator || strlen(s->separator) != 1) { + av_log(ctx, AV_LOG_ERROR, "A single character needs to be specified for the separator parameter.\n"); + return AVERROR(EINVAL); + } + + if (s->find_file && strlen(s->find_file)) { + ret = load_text_from_file(ctx, s->find_file, &s->find, s->separator[0]); + if (ret < 0 ) + return ret; + } + + if (s->replace_file && strlen(s->replace_file)) { + ret = load_text_from_file(ctx, s->replace_file, &s->replace, s->separator[0]); + if (ret < 0 ) + return ret; + } + + return 0; +} + +static int init_censor(AVFilterContext *ctx) +{ + TextModContext *s = ctx->priv; + int ret; + + s->filter_type = TM_CENSOR; + s->operation = OP_REPLACE_WORDS; + + ret = load_files(ctx); + if (ret < 0 ) + return ret; + + if (!s->find || !strlen(s->find)) { + av_log(ctx, AV_LOG_ERROR, "Either the 'words' or the 'words_file' parameter needs to be specified\n"); + return AVERROR(EINVAL); + } + + if (!s->censor_char || strlen(s->censor_char) != 1) { + av_log(ctx, AV_LOG_ERROR, "A single character needs to be specified for the censor_char parameter\n"); + return AVERROR(EINVAL); + } + + s->find_list = split_string(s->find, &s->nb_find_list, s->separator); + if (!s->find_list) + return AVERROR(ENOMEM); + + s->replace_list = av_calloc(s->nb_find_list, sizeof(char *)); + if (!s->replace_list) + return AVERROR(ENOMEM); + + for (int i = 0; i < s->nb_find_list; i++) { + size_t len, start = 0, end; + char *item = av_strdup(s->find_list[i]); + if (!item) + return AVERROR(ENOMEM); + + len = end = strlen(item); + + switch (s->censor_mode) { + case CM_KEEP_FIRST_LAST: + + if (len > 2) + start = 1; + if (len > 3) + end--; + + break; + case CM_KEEP_FIRST: + + if (len > 2) + start = 1; + + break; + } + + for (size_t n = start; n < end; n++) + item[n] = s->censor_char[0]; + + s->replace_list[i] = item; + } + + return 0; +} + +static int init_show_speaker(AVFilterContext *ctx) +{ + TextModContext *s = ctx->priv; + s->filter_type = TM_SHOW_SPEAKER; + + return 0; +} + +static int init(AVFilterContext *ctx) +{ + TextModContext *s = ctx->priv; + int ret; + + ret = load_files(ctx); + if (ret < 0 ) + return ret; + + switch (s->operation) { + case OP_REPLACE_CHARS: + case OP_REMOVE_CHARS: + case OP_REPLACE_WORDS: + case OP_REMOVE_WORDS: + if (!s->find || !strlen(s->find)) { + av_log(ctx, AV_LOG_ERROR, "Selected mode requires the 'find' parameter to be specified\n"); + return AVERROR(EINVAL); + } + break; + } + + switch (s->operation) { + case OP_REPLACE_CHARS: + case OP_REPLACE_WORDS: + if (!s->replace || !strlen(s->replace)) { + av_log(ctx, AV_LOG_ERROR, "Selected mode requires the 'replace' parameter to be specified\n"); + return AVERROR(EINVAL); + } + break; + } + + if (s->operation == OP_REPLACE_CHARS && strlen(s->find) != strlen(s->replace)) { + av_log(ctx, AV_LOG_ERROR, "Selected mode requires the 'find' and 'replace' parameters to have the same length\n"); + return AVERROR(EINVAL); + } + + if (s->operation == OP_REPLACE_WORDS || s->operation == OP_REMOVE_WORDS) { + if (!s->separator || strlen(s->separator) != 1) { + av_log(ctx, AV_LOG_ERROR, "Selected mode requires a single separator char to be specified\n"); + return AVERROR(EINVAL); + } + + s->find_list = split_string(s->find, &s->nb_find_list, s->separator); + if (!s->find_list) + return AVERROR(ENOMEM); + + if (s->operation == OP_REPLACE_WORDS) { + + s->replace_list = split_string(s->replace, &s->nb_replace_list, s->separator); + if (!s->replace_list) + return AVERROR(ENOMEM); + + if (s->nb_find_list != s->nb_replace_list) { + av_log(ctx, AV_LOG_ERROR, "The number of words in 'find' and 'replace' needs to be equal\n"); + return AVERROR(EINVAL); + } + } + } + + return 0; +} + +static void uninit(AVFilterContext *ctx) +{ + TextModContext *s = ctx->priv; + + for (int i = 0; i < s->nb_find_list; i++) + av_freep(&s->find_list[i]); + + s->nb_find_list = 0; + av_freep(&s->find_list); + + for (int i = 0; i < s->nb_replace_list; i++) + av_freep(&s->replace_list[i]); + + s->nb_replace_list = 0; + av_freep(&s->replace_list); +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output formats */ + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + + + +static char *process_text(TextModContext *s, char *text) +{ + const char *char_src = s->find; + const char *char_dst = s->replace; + char *result = NULL; + int escape_level = 0, k = 0; + + switch (s->operation) { + case OP_LEET: + case OP_REPLACE_CHARS: + + if (s->operation == OP_LEET) { + char_src = leet_src; + char_dst = leet_dst; + } + + result = av_strdup(text); + if (!result) + return NULL; + + for (size_t n = 0; n < strlen(result); n++) { + if (result[n] == '{') + escape_level++; + + if (!escape_level) { + size_t len = strlen(char_src); + for (size_t t = 0; t < len; t++) { + if (result[n] == char_src[t]) { + result[n] = char_dst[t]; + break; + } + } + } + + if (result[n] == '}') + escape_level--; + } + + break; + case OP_TO_UPPER: + case OP_TO_LOWER: + + result = av_strdup(text); + if (!result) + return NULL; + + for (size_t n = 0; n < strlen(result); n++) { + if (result[n] == '{') + escape_level++; + if (!escape_level) + result[n] = s->operation == OP_TO_LOWER ? av_tolower(result[n]) : av_toupper(result[n]); + if (result[n] == '}') + escape_level--; + } + + break; + case OP_REMOVE_CHARS: + + result = av_strdup(text); + if (!result) + return NULL; + + for (size_t n = 0; n < strlen(result); n++) { + int skip_char = 0; + + if (result[n] == '{') + escape_level++; + + if (!escape_level) { + size_t len = strlen(char_src); + for (size_t t = 0; t < len; t++) { + if (result[n] == char_src[t]) { + skip_char = 1; + break; + } + } + } + + if (!skip_char) + result[k++] = result[n]; + + if (result[n] == '}') + escape_level--; + } + + result[k] = 0; + + break; + case OP_REPLACE_WORDS: + case OP_REMOVE_WORDS: + + result = av_strdup(text); + if (!result) + return NULL; + + for (int n = 0; n < s->nb_find_list; n++) { + char *tmp = result; + const char *replace = (s->operation == OP_REPLACE_WORDS) ? s->replace_list[n] : ""; + + result = av_strireplace(result, s->find_list[n], replace); + if (!result) + return NULL; + + av_free(tmp); + } + + break; + } + + return result; +} + +static char *process_dialog_show_speaker(TextModContext *s, char *ass_line) +{ + ASSDialog *dialog = avpriv_ass_split_dialog(NULL, ass_line); + int escape_level = 0; + unsigned pos = 0, len; + char *result, *text; + AVBPrint pbuf; + + if (!dialog) + return NULL; + + text = process_text(s, dialog->text); + if (!text) + return NULL; + + if (!dialog->name || !strlen(dialog->name) || !dialog->text || !strlen(dialog->text)) + return av_strdup(ass_line); + + // Find insertion point in case the line starts with style codes + len = (unsigned)strlen(dialog->text); + for (unsigned i = 0; i < len; i++) { + + if (dialog->text[i] == '{') + escape_level++; + + if (dialog->text[i] == '}') + escape_level--; + + if (escape_level == 0) { + pos = i; + break; + } + } + + if (s->style && strlen(s->style)) + // When a style is specified reset the insertion point + // (always add speaker plus style at the start in that case) + pos = 0; + + if (pos >= len - 1) + return av_strdup(ass_line); + + av_bprint_init(&pbuf, 1, AV_BPRINT_SIZE_UNLIMITED); + + if (pos > 0) { + av_bprint_append_data(&pbuf, dialog->text, pos); + } + + if (s->style && strlen(s->style)) { + if (s->style[0] == '{') + // Assume complete and valid style code, e.g. {\c&HFF0000&} + av_bprintf(&pbuf, "%s", s->style); + else + // Otherwise it must be a style name + av_bprintf(&pbuf, "{\\r%s}", s->style); + } + + switch (s->speaker_mode) { + case SM_SQUARE_BRACKETS: + av_bprintf(&pbuf, "[%s]", dialog->name); + break; + case SM_ROUND_BRACKETS: + av_bprintf(&pbuf, "(%s)", dialog->name); + break; + case SM_COLON: + av_bprintf(&pbuf, "%s:", dialog->name); + break; + case SM_PLAIN: + av_bprintf(&pbuf, "%s", dialog->name); + break; + } + + if (s->style && strlen(s->style)) { + // Reset line style + if (dialog->style && strlen(dialog->style) && !av_strcasecmp(dialog->style, "default")) + av_bprintf(&pbuf, "{\\r%s}", dialog->style); + else + av_bprintf(&pbuf, "{\\r}"); + } + + if (s->line_break) + av_bprintf(&pbuf, "\\N"); + else + av_bprintf(&pbuf, " "); + + av_bprint_append_data(&pbuf, dialog->text + pos, len - pos); + + av_bprint_finalize(&pbuf, &text); + + result = avpriv_ass_get_dialog(dialog->readorder, dialog->layer, dialog->style, dialog->name, text); + + av_free(text); + avpriv_ass_free_dialog(&dialog); + return result; +} + +static char *process_dialog(TextModContext *s, char *ass_line) +{ + ASSDialog *dialog; + char *result, *text; + + if (s->filter_type == TM_SHOW_SPEAKER) + return process_dialog_show_speaker(s, ass_line); + + dialog = avpriv_ass_split_dialog(NULL, ass_line); + if (!dialog) + return NULL; + + text = process_text(s, dialog->text); + if (!text) + return NULL; + + result = avpriv_ass_get_dialog(dialog->readorder, dialog->layer, dialog->style, dialog->name, text); + + av_free(text); + avpriv_ass_free_dialog(&dialog); + return result; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + TextModContext *s = inlink->dst->priv; + AVFilterLink *outlink = inlink->dst->outputs[0]; + int ret; + + outlink->format = inlink->format; + + ret = av_frame_make_writable(frame); + if (ret < 0) + return ret; + + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + + AVSubtitleArea *area = frame->subtitle_areas[i]; + + if (area->ass) { + char *tmp = area->ass; + area->ass = process_dialog(s, area->ass); + av_free(tmp); + if (!area->ass) + return AVERROR(ENOMEM); + } + } + + return ff_filter_frame(outlink, frame); +} + +#define OFFSET(x) offsetof(TextModContext, x) +#define FLAGS (AV_OPT_FLAG_SUBTITLE_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption textmod_options[] = { + { "mode", "set operation mode", OFFSET(operation), AV_OPT_TYPE_INT, {.i64=OP_LEET}, OP_LEET, NB_OPS-1, FLAGS, "mode" }, + { "leet", "convert text to 'leet speak'", 0, AV_OPT_TYPE_CONST, {.i64=OP_LEET}, 0, 0, FLAGS, "mode" }, + { "to_upper", "change to upper case", 0, AV_OPT_TYPE_CONST, {.i64=OP_TO_UPPER}, 0, 0, FLAGS, "mode" }, + { "to_lower", "change to lower case", 0, AV_OPT_TYPE_CONST, {.i64=OP_TO_LOWER}, 0, 0, FLAGS, "mode" }, + { "replace_chars", "replace characters", 0, AV_OPT_TYPE_CONST, {.i64=OP_REPLACE_CHARS}, 0, 0, FLAGS, "mode" }, + { "remove_chars", "remove characters", 0, AV_OPT_TYPE_CONST, {.i64=OP_REMOVE_CHARS}, 0, 0, FLAGS, "mode" }, + { "replace_words", "replace words", 0, AV_OPT_TYPE_CONST, {.i64=OP_REPLACE_WORDS}, 0, 0, FLAGS, "mode" }, + { "remove_words", "remove words", 0, AV_OPT_TYPE_CONST, {.i64=OP_REMOVE_WORDS}, 0, 0, FLAGS, "mode" }, + { "find", "chars/words to find or remove", OFFSET(find), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "find_file", "load find param from file", OFFSET(find_file), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "replace", "chars/words to replace", OFFSET(replace), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "replace_file", "load replace param from file", OFFSET(replace_file), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "separator", "word separator", OFFSET(separator), AV_OPT_TYPE_STRING, {.str = ","}, 0, 0, FLAGS, NULL }, + { NULL }, +}; + + +static const AVOption censor_options[] = { + { "mode", "set censoring mode", OFFSET(censor_mode), AV_OPT_TYPE_INT, {.i64=CM_KEEP_FIRST_LAST}, 0, 2, FLAGS, "mode" }, + { "keep_first_last", "censor inner chars", 0, AV_OPT_TYPE_CONST, {.i64=CM_KEEP_FIRST_LAST}, 0, 0, FLAGS, "mode" }, + { "keep_first", "censor all but first char", 0, AV_OPT_TYPE_CONST, {.i64=CM_KEEP_FIRST}, 0, 0, FLAGS, "mode" }, + { "all", "censor all chars", 0, AV_OPT_TYPE_CONST, {.i64=CM_ALL}, 0, 0, FLAGS, "mode" }, + { "words", "list of words to censor", OFFSET(find), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "words_file", "path to word list file", OFFSET(find_file), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "separator", "word separator", OFFSET(separator), AV_OPT_TYPE_STRING, {.str = ","}, 0, 0, FLAGS, NULL }, + { "censor_char", "replacement character", OFFSET(censor_char), AV_OPT_TYPE_STRING, {.str = "*"}, 0, 0, FLAGS, NULL }, + { NULL }, +}; + +static const AVOption show_speaker_options[] = { + { "format", "speaker name formatting", OFFSET(speaker_mode), AV_OPT_TYPE_INT, {.i64=SM_SQUARE_BRACKETS}, 0, 2, FLAGS, "format" }, + { "square_brackets", "[speaker] text", 0, AV_OPT_TYPE_CONST, {.i64=SM_SQUARE_BRACKETS}, 0, 0, FLAGS, "format" }, + { "round_brackets", "(speaker) text", 0, AV_OPT_TYPE_CONST, {.i64=SM_ROUND_BRACKETS}, 0, 0, FLAGS, "format" }, + { "colon", "speaker: text", 0, AV_OPT_TYPE_CONST, {.i64=SM_COLON}, 0, 0, FLAGS, "format" }, + { "plain", "speaker text", 0, AV_OPT_TYPE_CONST, {.i64=SM_PLAIN}, 0, 0, FLAGS, "format" }, + { "line_break", "insert line break", OFFSET(line_break), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, FLAGS, NULL }, + { "style", "ass type name or style code", OFFSET(style), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(textmod); +AVFILTER_DEFINE_CLASS(censor); +AVFILTER_DEFINE_CLASS(show_speaker); + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_frame, + }, +}; + +static const AVFilterPad outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + }, +}; + +const AVFilter ff_sf_textmod = { + .name = "textmod", + .description = NULL_IF_CONFIG_SMALL("Modify subtitle text in several ways"), + .init = init, + .uninit = uninit, + .query_formats = query_formats, + .priv_size = sizeof(TextModContext), + .priv_class = &textmod_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), +}; + +const AVFilter ff_sf_censor = { + .name = "censor", + .description = NULL_IF_CONFIG_SMALL("Censor words in subtitle text"), + .init = init_censor, + .uninit = uninit, + .query_formats = query_formats, + .priv_size = sizeof(TextModContext), + .priv_class = &censor_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), +}; + +const AVFilter ff_sf_show_speaker = { + .name = "show_speaker", + .description = NULL_IF_CONFIG_SMALL("Prepend speaker names to text subtitles (when available)"), + .init = init_show_speaker, + .uninit = uninit, + .query_formats = query_formats, + .priv_size = sizeof(TextModContext), + .priv_class = &show_speaker_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), +}; From patchwork Thu Sep 30 19:31:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30804 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp986246iob; Thu, 30 Sep 2021 12:32:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwvbMcduBP8aPh47MkxZCJZSodPadV/WhiIrkfUNLvZ/zktsUiG+jD2ZJzInZvVO6vUGAXF X-Received: by 2002:a17:906:3157:: with SMTP id e23mr1204417eje.29.1633030354380; Thu, 30 Sep 2021 12:32:34 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c7si4653421eja.538.2021.09.30.12.32.34; Thu, 30 Sep 2021 12:32:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b="h9X+/v68"; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BD78E68AB0D; Thu, 30 Sep 2021 22:31:16 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12olkn2013.outbound.protection.outlook.com [40.92.23.13]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D372468AAC1 for ; Thu, 30 Sep 2021 22:31:14 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QEZzVdTy72jJ3mgooy1z6MIK8D67H1UGoBOh04KZoJm40OLLwrbqXYS/AhViWrh/jeUWQMAa39wJFY0vOutQFIgfpaPi2DBJ30IsUOFBI0d3AiCwsp1lF0kVjy9PxqdCjV+TAlOeWBx85gUVovEC/vEswvs9zTr7i0XPkMXi9odHrS92PhefgDEA8avxiAhOXyH5jfYnpfZlknVEo/GYs77AHeDcUxaIxbITT7cevRbzAn1zhPuHtvBQHtIXLpn2bmzws2Ad9xvHwJRkAvlu9RCHszYreJ0d3dA0kCn7FZ6ubun0JHsIHF8C3d7uqWrZAT+7psmDI6cqPTCxpf8VCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=flvh5mItrxhHLnWVXapdxbX6IEa8LIfASw/0e0u6b3A=; b=KVBYEaJhioUwMBCpmmFRwXQdPufxlvGYwqA4lEbn6hZpZKpMJ8d/Tf/Sd/e6TIJlmP85JZtiaimq2EbjXce1mk7EOYAoDG+wusJS09oSGKtDHjADaEa9oubPOA0bYt1KgyZ/BhLCNjpfQZXBVtcL+ibAfJVAdF/kvxpuI02/2U8V7McNDo9x3sGoLo3oQgKPpXVUJHjquzzkXmW7EtSt02Ujd0gRaMj6hIQAIYZcJaVUDTGbA3Eo+DT6gjXZ2NO3sKSUBpbFMX5dvZ03BgsJKj6vfeCYW21T+bYuLKL5UF/grga8jEvi0x77UMLSyGr04fXjvLXjGpfP5pJufhlPlA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=flvh5mItrxhHLnWVXapdxbX6IEa8LIfASw/0e0u6b3A=; b=h9X+/v68l9ZBMNdhWkKKpwEfB7MBzE0qriZf3wsHhdhB01/69zNNr0hpqgSx+0FoFlYq4eFDrFNudbfsASbCpWF+e4zwopwzrCc3c810sxlJIEd0N7e5kHx/8MUpUlC6VpRNqR1DA8VaMXsKXmAcwvY1xamyXI7F9a1rDpYG4J3DETXCdCSlgBQB/zE2pBVp8aLFaUHZ/hGAuwfbeOoYWT3Lw/Nu4CwlzkkzlGUuiO+YrhkvI+MNAcozsI7d0hpBTVfMcSpM83bFiPcONSH/5SJiPiQMaJStJNJDtCRj25765Hq8JAXmQY5+42Isyl7S7DwqCQ+Xviq1DYyD1my42Q== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5615.namprd04.prod.outlook.com (2603:10b6:208:fe::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:31:12 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:31:12 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 09/14] avfilter/stripstyles: Add stripstyles filter Thread-Index: AQHXtjG6VAvyGtoSME+BPCUwiBXgTA== Date: Thu, 30 Sep 2021 19:31:12 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> <8a75051fd7be111f4ec52073a6da6679724a7127.1633027275.git.softworkz@hotmail.com> <48414b2bba60c4be4c41a1bd3ba8edae069f1fc9.1633027275.git.softworkz@hotmail.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [BwiGGFm/GpoRrfhV5AOE8GOZavTboC19] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 86c3a80a-5067-45a0-578a-08d98448dc8d x-ms-traffictypediagnostic: MN2PR04MB5615: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: XncwIHs3UC2qvfHeOv7vewj4IyMNKizmuHR3cDpBEu3l8vagw8xOC+I5v3ATpMck4yXKgUZAtotYlmYID6zAfPYZJ/gdyslvFU590u1oJZQbGDgMqZgGUUMaLLe/M4D/AgJ6Fl4+VHaTionJC1tnbUMcYhl/8dPr9+EVsEm1mn/yUAYGwOdRYAVD9xUt2ER7B1rHLG46663ERcNM1ZNF7H5bKWeelwzYer4Yg9vXvx3zYYxMcoFGdyPSinxZrCSW97tFqzs7sb9vDjIeyJ/MVI55MlxXrzztbDBkJ0lx3+wuae0ixId35VhktpO/xbr4hgFzgBeEuBe74n7Q90QQpjJjSfKFsHBlyyNcaetdNK4JQnMVt6cVO/wA6Z83IwJiE5mwgbH88Kxva++nHWoE0I36wAYfTR1iIfXp5Bg/uUwrt8n6ThVNDzxb0xYYkZWL x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: iyUGW4zD0LIVyfx7vshacoyVC3wxT4oMYoBxSs+UtYsT22nDdMy6hsqRNkW/ZdSNlgskEJdTZ4lUznIo/RB9RM1rXp3IxZGsrtu9sqaqyzFnJltnzGeM+LBgsHGhtuz0WZmTzqw+UcNMYTHmv8r89w== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 86c3a80a-5067-45a0-578a-08d98448dc8d X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:31:12.1698 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5615 Subject: [FFmpeg-devel] [PATCH v10 09/14] avfilter/stripstyles: Add stripstyles filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ts80nU82/JrH - stripstyles {S -> S) Remove all inline styles from subtitle events Signed-off-by: softworkz --- libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/sf_stripstyles.c | 211 +++++++++++++++++++++++++++++++++++ 3 files changed, 213 insertions(+) create mode 100644 libavfilter/sf_stripstyles.c diff --git a/libavfilter/Makefile b/libavfilter/Makefile index df250ed07b..c991a525f0 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -541,6 +541,7 @@ OBJS-$(CONFIG_NULLSINK_FILTER) += vsink_nullsink.o OBJS-$(CONFIG_CENSOR_FILTER) += sf_textmod.o OBJS-$(CONFIG_SHOW_SPEAKER_FILTER) += sf_textmod.o OBJS-$(CONFIG_TEXTMOD_FILTER) += sf_textmod.o +OBJS-$(CONFIG_STRIPSTYLES_FILTER) += sf_stripstyles.o # multimedia filters OBJS-$(CONFIG_ABITSCOPE_FILTER) += avf_abitscope.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 232eac4ef6..bdf4c2ae52 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -529,6 +529,7 @@ extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_sf_censor; extern const AVFilter ff_sf_show_speaker; +extern const AVFilter ff_sf_stripstyles; extern const AVFilter ff_sf_textmod; extern const AVFilter ff_svf_graphicsub2video; extern const AVFilter ff_svf_textsub2video; diff --git a/libavfilter/sf_stripstyles.c b/libavfilter/sf_stripstyles.c new file mode 100644 index 0000000000..bdde28577f --- /dev/null +++ b/libavfilter/sf_stripstyles.c @@ -0,0 +1,211 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * text subtitle filter which removes inline-styles from subtitles + */ + +#include "libavutil/opt.h" +#include "internal.h" +#include "libavutil/ass_split_internal.h" +#include "libavutil/bprint.h" + +typedef struct StripStylesContext { + const AVClass *class; + enum AVSubtitleType format; + int remove_animated; +} StripStylesContext; + +typedef struct DialogContext { + StripStylesContext* ss_ctx; + AVBPrint buffer; + int drawing_scale; + int is_animated; +} DialogContext; + +static void dialog_text_cb(void *priv, const char *text, int len) +{ + DialogContext *s = priv; + + av_log(s->ss_ctx, AV_LOG_DEBUG, "dialog_text_cb: %s\n", text); + + if (!s->drawing_scale && (!s->is_animated || !s->ss_ctx->remove_animated)) + av_bprint_append_data(&s->buffer, text, len); +} + +static void dialog_new_line_cb(void *priv, int forced) +{ + DialogContext *s = priv; + if (!s->drawing_scale && !s->is_animated) + av_bprint_append_data(&s->buffer, forced ? "\\N" : "\\n", 2); +} + +static void dialog_drawing_mode_cb(void *priv, int scale) +{ + DialogContext *s = priv; + s->drawing_scale = scale; +} + +static void dialog_animate_cb(void *priv, int t1, int t2, int accel, char *style) +{ + DialogContext *s = priv; + s->is_animated = 1; +} + +static void dialog_move_cb(void *priv, int x1, int y1, int x2, int y2, int t1, int t2) +{ + DialogContext *s = priv; + if (t1 >= 0 || t2 >= 0) + s->is_animated = 1; +} + +static const ASSCodesCallbacks dialog_callbacks = { + .text = dialog_text_cb, + .new_line = dialog_new_line_cb, + .drawing_mode = dialog_drawing_mode_cb, + .animate = dialog_animate_cb, + .move = dialog_move_cb, +}; + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input subtitle format */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output video format */ + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static char *ass_get_line(int readorder, int layer, const char *style, + const char *speaker, const char *effect, const char *text) +{ + return av_asprintf("%d,%d,%s,%s,0,0,0,%s,%s", + readorder, layer, style ? style : "Default", + speaker ? speaker : "", effect, text); +} + +static char *process_dialog(StripStylesContext *s, const char *ass_line) +{ + DialogContext dlg_ctx = { .ss_ctx = s }; + ASSDialog *dialog = avpriv_ass_split_dialog(NULL, ass_line); + char *result = NULL; + + if (!dialog) + return NULL; + + dlg_ctx.ss_ctx = s; + + av_bprint_init(&dlg_ctx.buffer, 0, AV_BPRINT_SIZE_UNLIMITED); + + avpriv_ass_split_override_codes(&dialog_callbacks, &dlg_ctx, dialog->text); + + if (av_bprint_is_complete(&dlg_ctx.buffer) + && dlg_ctx.buffer.len > 0) + result = ass_get_line(dialog->readorder, dialog->layer, dialog->style, dialog->name, dialog->effect, dlg_ctx.buffer.str); + + av_bprint_finalize(&dlg_ctx.buffer, NULL); + avpriv_ass_free_dialog(&dialog); + + return result; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + StripStylesContext *s = inlink->dst->priv; + AVFilterLink *outlink = inlink->dst->outputs[0]; + int ret; + + outlink->format = inlink->format; + + ret = av_frame_make_writable(frame); + if (ret <0 ) { + av_frame_free(&frame); + return AVERROR(ENOMEM); + } + + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + + AVSubtitleArea *area = frame->subtitle_areas[i]; + + if (area->ass) { + char *tmp = area->ass; + area->ass = process_dialog(s, area->ass); + + if (area->ass) { + av_log(inlink->dst, AV_LOG_DEBUG, "original: %s\n", tmp); + av_log(inlink->dst, AV_LOG_DEBUG, "stripped: %s\n", area->ass); + } + else + area->ass = NULL; + + av_free(tmp); + } + } + + return ff_filter_frame(outlink, frame); +} + +#define OFFSET(x) offsetof(StripStylesContext, x) +#define FLAGS (AV_OPT_FLAG_SUBTITLE_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption stripstyles_options[] = { + { "remove_animated", "remove animated text (default: yes)", OFFSET(remove_animated), AV_OPT_TYPE_BOOL, {.i64 = 1 }, 0, 1, FLAGS, 0 }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(stripstyles); + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_frame, + }, +}; + +static const AVFilterPad outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + }, +}; + +const AVFilter ff_sf_stripstyles = { + .name = "stripstyles", + .description = NULL_IF_CONFIG_SMALL("Strip subtitle inline styles"), + .query_formats = query_formats, + .priv_size = sizeof(StripStylesContext), + .priv_class = &stripstyles_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), +}; + From patchwork Thu Sep 30 19:31:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30805 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp986413iob; Thu, 30 Sep 2021 12:32:44 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwN9wS4sI3NZNjuqyBSMZpGH0m6ZuhB6S/k4cXecZBnA/wP7cIB+1iZg7yuDOB3LV0TAaVv X-Received: by 2002:a50:becf:: with SMTP id e15mr9245893edk.114.1633030364120; Thu, 30 Sep 2021 12:32:44 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x13si5471369edd.531.2021.09.30.12.32.43; Thu, 30 Sep 2021 12:32:44 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=ZlJ1xehL; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B151468AB51; Thu, 30 Sep 2021 22:31:19 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12olkn2082.outbound.protection.outlook.com [40.92.22.82]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B5C0C68A9EA for ; Thu, 30 Sep 2021 22:31:17 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WOnwHQICSi9/4GsA1KR+XorlU03JLhcCSlAEijj6QJraV6gw1dLdfvTDe3vbUv5JAz1eo53VRiwYLL0knvEdoVxNDGc5hjhZXqVKDfIHLEIcfb7VLS1pbj3/mQWsm9kTZklouMrZ08S86iccvUL8rc2/bSFd5yFgrjWsPMAv6CRQwKQsUyc40A6lT0R5mWWb3aYyGWn069mR4Ap+Lt/TGVEeQ5zMQID592Tbhg89XUR1ABw2428zfgolM3onouwcPo02HA8Ss7jRDztq3ufRPfyQmrkYRaXsne145iC9umr1wHhWO2WmtP2w7g1y4RqkY+vcEhD1TklzA2EHmkMI7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=LrgTywqa0I9/zoWe5uijvCGyVGDQOjkKbYgo1wUoTGg=; b=Ncmoihu0XrJ9z8hNO741EelcSRcys0Dy5Lp+jwAOPFqc8BXEGxJ36W3jpOXvHweclhjQ7MTQHtbNXNwDtjksgnr+xFUSBm3+yDfY/9riuTyDw+YLBb72XWvhGHTH9OycibJuLN6kGf2mTJO5FnlmSDNoboslfyf7ffr+3XZ1HJA7RmJHUcd1sCKck1mKSQiSh6HFVmmC67qcV8TVZ96du+x7jOnYcLcrCa+qRxwpgwnaMH8Xyj+P1IOqzFFzDSCsFAYyBiM5EozgpXj34/IVrsELq2OuoJ1mP1Uwc1/QYcQvesnlH4WR3IaxtfzeRnY+xBZqiGwT9wBtMXctKcJIvQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LrgTywqa0I9/zoWe5uijvCGyVGDQOjkKbYgo1wUoTGg=; b=ZlJ1xehL2mTyZuWzMUkG63elcqx5jbQpxctJiZjfZEje04Pvlk5kxzHVA5ipx0fo5yjKXicFBDHSXQVWt98MfaZkR77FbYruC7yaqqV24InjBz9z0JU4MSJFsY/XW8uL/Fxmisrn+wcSMiDzg4QNC04b6cUhHY2q7GFwCs4+lz0a0uv2jYls63FQemVmux+pXJkPoa7/mJZakRS2G4nsoKP4mjfetMjHaOjUDVgLq0LOCU3f6dOMny8meQBy0uG0V17w7CZhdyrs4OUbw/9Zz7nzbibRqYY97iFPwjl4DUHUvx5nA/ZQxrPjrW8KMWYMADfXGAj6rSlV3tv+m6unSQ== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5519.namprd04.prod.outlook.com (2603:10b6:208:da::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:31:14 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:31:14 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 10/14] avfilter/split_cc: Add split_cc filter for closed caption handling Thread-Index: AQHXtjG7SxG8s2XqukKds6sNUfkROQ== Date: Thu, 30 Sep 2021 19:31:14 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> <8a75051fd7be111f4ec52073a6da6679724a7127.1633027275.git.softworkz@hotmail.com> <48414b2bba60c4be4c41a1bd3ba8edae069f1fc9.1633027275.git.softworkz@hotmail.com> <94fd41cac821f78e13de073ea8e69d37a3bc0b33.1633027275.git.softworkz@hotmail.com> In-Reply-To: <94fd41cac821f78e13de073ea8e69d37a3bc0b33.1633027275.git.softworkz@hotmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [SO5Gu+NE4C5JrKOfIuwXvfcEmvPpe2HP] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: aa5ca26a-6ca3-4a09-9b29-08d98448de23 x-ms-traffictypediagnostic: MN2PR04MB5519: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ZU/TaFXkHyPugn5i69ikhJXKFcy8b6jSS3J5LFTt4s+FdQacMRHk9Jf3rV4D0ZKrlDzirkKn+tNxs1t+LC6E/vpLHyIV6kmNFF9vWiARZTegUqAa64a5+jItGEePkFSaGH1+Q61k8mlRJSvEsFkpzkApMDnFXA8YlDZl/f88P1bC8EFTzdp7gpxrTlAJweIGc7SnFbRZhpVtsuU5Zje2tk+nzCqZwch0r6GBdn10hzopSPguBFQYZokLpwFTd+6gjnlVu5edPmtz3U6yH7KWDYWj4Zk/GKY2uWRnCz77Ut3uuP5zy4SpAJacYtbba/nudwAzHNZhx6ivTcpAJRDNv1YHphwNnfgmfDAFM73pRkXd7Pc2VJjkwkYFz/lcMHUj5oflGQGTsKo502PIk/BKLgHxaVw06EPe47Le1vY9Of98Qo3W3fHKudhaS5WjCa2KyDFzN0aLbPdouv1jjQvPwbiYU/CTjIP8IepIgHvIy3DwIa7q/p4fz4U8VB+tpxMAU8tJezzrrUE1+3i8M5gYUw== x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: ViaPjex50TFMVGnF4OJIEU3FQH4R8vHKLZye6ThEy3ltLftFzgv5zPtpU4jk548UNiRrDoCxylZckfJ7dGoHyDvXav19ESsIwXEEw2Sa0hnEM8cU4l0pac8Ww1dZmRoDBGfp3WZj10bI3dVxiQlKpg== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: aa5ca26a-6ca3-4a09-9b29-08d98448de23 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:31:14.8543 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5519 Subject: [FFmpeg-devel] [PATCH v10 10/14] avfilter/split_cc: Add split_cc filter for closed caption handling X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: glAS+J8+a9jc - split_cc {V -> VS) Extract closed-caption (A53) data from video frames as subtitle Frames ffmpeg -y -loglevel verbose -i "https://streams.videolan.org/streams/ts/CC/NewsStream-608-ac3.ts" -filter_complex "[0:v]split_cc[vid1],textmod=mode=remove_chars:find='@',[vid1]overlay_textsubs" output.mkv Signed-off-by: softworkz --- configure | 1 + doc/filters.texi | 45 ++++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/sf_split_cc.c | 298 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 346 insertions(+) create mode 100644 libavfilter/sf_split_cc.c diff --git a/configure b/configure index 37fc4c20e7..6da059e6ee 100755 --- a/configure +++ b/configure @@ -3663,6 +3663,7 @@ spp_filter_select="fft idctdsp fdctdsp me_cmp pixblockdsp" sr_filter_deps="avformat swscale" sr_filter_select="dnn" stereo3d_filter_deps="gpl" +split_cc_filter_deps="cc_dec_decoder" subtitles_filter_deps="avformat avcodec libass" super2xsai_filter_deps="gpl" pixfmts_super2xsai_test_deps="super2xsai_filter" diff --git a/doc/filters.texi b/doc/filters.texi index c751211017..001d2baa8f 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -25494,6 +25494,16 @@ string containing ASS style format @code{KEY=VALUE} couples separated by ",". @end table +@subsection Examples + +@itemize +@item +Overlay ASS subtitles with animations: +@example +ffmpeg -i "http://streams.videolan.org/samples/sub/SSA/subtitle_testing_complex.mkv" -filter_complex "[0:v]overlay_textsubs" -map 0 -y out.mkv +@end example +@end itemize + @section show_speaker Prepend speaker names to subtitle lines (when available). @@ -25557,6 +25567,41 @@ ffmpeg -i INPUT -filter_complex "show_speaker=format=colon:style='@{\\c&HDD0000& @end example @end itemize + +@section split_cc + +Split-out closed-caption/A53 subtitles from video frame side data. + +This filter provides an input and an output for video frames, which are just passed through without modification. +The second out provides subtitle frames which are extracted from video frame side data. + +Inputs: +- 0: Video + +Outputs: +- 0: Video (same as input) +- 1: Subtitles [text] + +It accepts the following parameters: + +@table @option + +@item use_cc_styles +Emit closed caption style header. This will make closed captions look like on normal TV devices. +(white font on black background rectangles) + +@end table + +@subsection Examples + +@itemize +@item +Extract closed captions as text subtitle stream and overlay it onto the video: +@example +ffmpeg -i "https://streams.videolan.org/streams/ts/CC/NewsStream-608-ac3.ts" -filter_complex "[0:v:0]split_cc[vid1][sub1];[vid1][sub1]overlay_textsubs" output.mkv +@end example +@end itemize + @section textsub2video Converts text subtitles to video frames. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index c991a525f0..b260e7e0a2 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -541,6 +541,7 @@ OBJS-$(CONFIG_NULLSINK_FILTER) += vsink_nullsink.o OBJS-$(CONFIG_CENSOR_FILTER) += sf_textmod.o OBJS-$(CONFIG_SHOW_SPEAKER_FILTER) += sf_textmod.o OBJS-$(CONFIG_TEXTMOD_FILTER) += sf_textmod.o +OBJS-$(CONFIG_SPLIT_CC_FILTER) += sf_split_cc.o OBJS-$(CONFIG_STRIPSTYLES_FILTER) += sf_stripstyles.o # multimedia filters diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index bdf4c2ae52..40bb3c229e 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -529,6 +529,7 @@ extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_sf_censor; extern const AVFilter ff_sf_show_speaker; +extern const AVFilter ff_sf_split_cc; extern const AVFilter ff_sf_stripstyles; extern const AVFilter ff_sf_textmod; extern const AVFilter ff_svf_graphicsub2video; diff --git a/libavfilter/sf_split_cc.c b/libavfilter/sf_split_cc.c new file mode 100644 index 0000000000..d96912a90b --- /dev/null +++ b/libavfilter/sf_split_cc.c @@ -0,0 +1,298 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * subtitle filter for splitting out closed-caption/A53 subtitles from video frame side data + */ + +#include "libavutil/opt.h" +#include "subtitles.h" +#include "libavcodec/avcodec.h" + +typedef struct SplitCaptionsContext { + const AVClass *class; + enum AVSubtitleType format; + AVCodecContext *cc_dec; + int eof; + AVFrame *next_sub_frame; + int new_frame; + AVBufferRef *subtitle_header; + int use_cc_styles; +} SplitCaptionsContext; + +static int init(AVFilterContext *ctx) +{ + SplitCaptionsContext *s = ctx->priv; + + int ret; + const AVCodec *codec = avcodec_find_decoder(AV_CODEC_ID_EIA_608); + if (!codec) { + av_log(ctx, AV_LOG_ERROR, "failed to find EIA-608/708 decoder\n"); + return AVERROR_DECODER_NOT_FOUND; + } + + if (!((s->cc_dec = avcodec_alloc_context3(codec)))) { + av_log(ctx, AV_LOG_ERROR, "failed to allocate EIA-608/708 decoder\n"); + return AVERROR(ENOMEM); + } + + if ((ret = avcodec_open2(s->cc_dec, codec, NULL)) < 0) { + av_log(ctx, AV_LOG_ERROR, "failed to open EIA-608/708 decoder: %i\n", ret); + return ret; + } + + if (s->use_cc_styles && s->cc_dec->subtitle_header && s->cc_dec->subtitle_header[0] != 0) { + char* subtitle_header = av_strdup((char *)s->cc_dec->subtitle_header); + if (!subtitle_header) + return AVERROR(ENOMEM); + s->subtitle_header = av_buffer_create((uint8_t *)subtitle_header, strlen(subtitle_header) + 1, NULL, NULL, 0); + if (!s->subtitle_header) { + av_free(subtitle_header); + return AVERROR(ENOMEM); + } + } + + return 0; +} + +static void uninit(AVFilterContext *ctx) +{ + SplitCaptionsContext *s = ctx->priv; + av_frame_free(&s->next_sub_frame); +} + +static int config_input(AVFilterLink *link) +{ + const SplitCaptionsContext *context = link->dst->priv; + + if (context->cc_dec) + context->cc_dec->pkt_timebase = link->time_base; + + return 0; +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink0 = ctx->inputs[0]; + AVFilterLink *outlink0 = ctx->outputs[0]; + AVFilterLink *outlink1 = ctx->outputs[1]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NB }; + int ret; + + /* set input0 video formats */ + formats = ff_all_formats(AVMEDIA_TYPE_VIDEO); + if ((ret = ff_formats_ref(formats, &inlink0->outcfg.formats)) < 0) + return ret; + + /* set output0 video formats */ + if ((ret = ff_formats_ref(formats, &outlink0->incfg.formats)) < 0) + return ret; + + /* set output1 subtitle formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &outlink1->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_sub_output(AVFilterLink *outlink) +{ + const AVFilterLink *inlink = outlink->src->inputs[0]; + + outlink->time_base = inlink->time_base; + outlink->format = AV_SUBTITLE_FMT_ASS; + + return 0; +} + +static int request_sub_frame(AVFilterLink *outlink) +{ + SplitCaptionsContext *s = outlink->src->priv; + + if (s->eof) + return AVERROR_EOF; + + if (s->next_sub_frame && s->new_frame) { + AVFrame *out; + s->next_sub_frame->pts++; + + out = av_frame_clone(s->next_sub_frame); + if (!out) + return AVERROR(ENOMEM); + + out->subtitle_pts = av_rescale_q(s->next_sub_frame->pts, outlink->time_base, AV_TIME_BASE_Q); + s->new_frame = 0; + + return ff_filter_frame(outlink, out); + } + + return 0; +} + +static int decode(AVCodecContext *avctx, AVFrame *frame, int *got_frame, AVPacket *pkt) +{ + int ret; + + *got_frame = 0; + + if (pkt) { + ret = avcodec_send_packet(avctx, pkt); + // In particular, we don't expect AVERROR(EAGAIN), because we read all + // decoded frames with avcodec_receive_frame() until done. + if (ret < 0 && ret != AVERROR_EOF) + return ret; + } + + ret = avcodec_receive_frame(avctx, frame); + if (ret < 0 && ret != AVERROR(EAGAIN)) + return ret; + if (ret >= 0) + *got_frame = 1; + + return 0; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFrameSideData *sd; + SplitCaptionsContext *s = inlink->dst->priv; + AVFilterLink *outlink0 = inlink->dst->outputs[0]; + AVFilterLink *outlink1 = inlink->dst->outputs[1]; + AVPacket *pkt; + AVFrame *sub_out; + + int ret; + + outlink0->format = inlink->format; + + sd = av_frame_get_side_data(frame, AV_FRAME_DATA_A53_CC); + + if (sd) { + int got_output = 0; + + pkt = av_packet_alloc(); + pkt->buf = av_buffer_ref(sd->buf); + if (!pkt->buf) { + ret = AVERROR(ENOMEM); + goto fail; + } + + pkt->data = sd->data; + pkt->size = (int)sd->size; + pkt->pts = frame->pts; + + sub_out = ff_get_subtitles_buffer(outlink1, AV_SUBTITLE_FMT_ASS); + if (!sub_out) { + ret = AVERROR(ENOMEM); + goto fail; + } + + if ((ret = av_buffer_replace(&sub_out->subtitle_header, s->subtitle_header)) < 0) + goto fail; + + ret = decode(s->cc_dec, sub_out, &got_output, pkt); + + if (ret < 0) + goto fail; + + if (got_output) { + sub_out->pts = frame->pts; + av_frame_free(&s->next_sub_frame); + s->next_sub_frame = sub_out; + sub_out = NULL; + s->new_frame = 1; + + ret = request_sub_frame(outlink1); + if (ret < 0) + goto fail; + } + } + + if (!s->next_sub_frame) { + s->next_sub_frame = ff_get_subtitles_buffer(outlink1, outlink1->format); + if (!s->next_sub_frame) { + ret = AVERROR(ENOMEM); + goto fail; + } + + s->next_sub_frame->subtitle_end_time = 100; + s->next_sub_frame->pts = frame->pts; + s->new_frame = 1; + + if ((ret = av_buffer_replace(&s->next_sub_frame->subtitle_header, s->subtitle_header)) < 0) + goto fail; + } + else + s->next_sub_frame->pts = frame->pts; + + ret = ff_filter_frame(outlink0, frame); + +fail: + av_packet_free(&pkt); + av_frame_free(&sub_out); + return ret; +} + +#define OFFSET(x) offsetof(SplitCaptionsContext, x) +#define FLAGS (AV_OPT_FLAG_SUBTITLE_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption split_cc_options[] = { + { "use_cc_styles", "Emit closed caption style header", OFFSET(use_cc_styles), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, FLAGS, NULL }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(split_cc); + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + .config_props = config_input, + }, +}; + +static const AVFilterPad outputs[] = { + { + .name = "video_passthrough", + .type = AVMEDIA_TYPE_VIDEO, + }, + { + .name = "subtitles", + .type = AVMEDIA_TYPE_SUBTITLE, + .request_frame = request_sub_frame, + .config_props = config_sub_output, + }, +}; + +const AVFilter ff_sf_split_cc = { + .name = "split_cc", + .description = NULL_IF_CONFIG_SMALL("Extract closed-caption (A53) data from video as subtitle stream."), + .init = init, + .uninit = uninit, + .query_formats = query_formats, + .priv_size = sizeof(SplitCaptionsContext), + .priv_class = &split_cc_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), +}; From patchwork Thu Sep 30 19:31:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30806 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp986569iob; Thu, 30 Sep 2021 12:32:55 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzuCWbe6h8/cfHRKVFkQekmFpqemLmgkOUybILiRvv1zVNxZzW+e/1WVnNhfFej1jlcPHOK X-Received: by 2002:a17:906:15cc:: with SMTP id l12mr1216891ejd.110.1633030374806; Thu, 30 Sep 2021 12:32:54 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l11si4366716eje.503.2021.09.30.12.32.54; Thu, 30 Sep 2021 12:32:54 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=PnMiU7Gz; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1326F68AB17; Thu, 30 Sep 2021 22:31:23 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10olkn2085.outbound.protection.outlook.com [40.92.42.85]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 304C268AB17 for ; Thu, 30 Sep 2021 22:31:20 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VtMfn7aTS9L2XZCOP0OakIBP9vw/8hSKxaNryAJpSlxMsIBkgR+OFXM0fRkdBssDfkbOh9H3ph6JfphG0NLOG/uhUijU3izykrKcTRyB537Mm8dyk3oRxvMzMIObzdCaW9w8Myq+kdMqsx8+UgOC9fpZaf4ugQZkqnsqw/Rwe1LuxMLafrMIlBHHz7lRII7d+oAh/s4apiY7KkXKdZXivpHg7ncRDuNqvf60G6N6VjmqjICtGKInINJvIXM7PrWM8pRdNe8C2Zc6ELltitztCXnSua8uhPAY0t+xEzZwYKXJde0cfzP5S1I0LvqnlVFxoH+Lp6rjhzkZef1W6HJX3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=Hzw3BVIlQry9wx1jPEY1q7Ycox3z0zdV411EMGTxGBc=; b=WrsmzQV+AWZgPhs2JCnH/7h10ZGD70RbBoQl2A3pb66PVWdGK8UtafpuT/i5aK6KSoGqCwMsYRVKMNNsDs+QhzOLtJ0D3kAuRCt6/FHQ3foOVVgNMUMDTvQioVNj3H4GFD/kSxsVpT1lfOgN5XjfLqWgHIf+cW1TxOLooHLsRh9VEZQvG6P2a7erSh146efRhhXxUaqWi5O61MsRAAwRyko+ksk6Qzmze/tIHjk7hR4fT6awheuPpvM0zQdUKCPuFJFVWeZrc9aRmzyHTMMld6Y1Dpad1VohnFiASyUlJBMn4gBlrCAYfJgjZW9v3DXq9a5Q9K6Yrb6aZauqEafDkA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Hzw3BVIlQry9wx1jPEY1q7Ycox3z0zdV411EMGTxGBc=; b=PnMiU7GzIJUYtRiDkaQHt+SHeAsR3cBNolWMg1yygxTPUvgIwEjWIwijsZqRb1b81+82YqyBp7/gkXXDvD+KErmTeofjXD1Slc2TaYSohjtFJ5Zg+3WXki2EkOtQnGw6tOdPQFoZQFnHl85ySLQCK4TN/FHkyDB0vkawQrTl7C2w7yUxQ8C6+DN7wZ3oVQVEZu3aYvN5b5GU8vK7NRSoF7fjCoFqYNFWeyi4xszdsnOlm1DrkrAZoYFooOA5iyFsIbHEH6S2O2Hc6mD53rN3UankL6XJXrqEd4WxN6g4WxXi06/Y5C4x9hJ25tUucFYaavwIyLlMVO1GzVWDX4crhw== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5519.namprd04.prod.outlook.com (2603:10b6:208:da::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:31:17 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:31:17 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 11/14] avfilter/graphicsub2text: Add new graphicsub2text filter (OCR) Thread-Index: AQHXtjG9y3ClZ1nDwEmHWoPcHvRHMw== Date: Thu, 30 Sep 2021 19:31:17 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> <8a75051fd7be111f4ec52073a6da6679724a7127.1633027275.git.softworkz@hotmail.com> <48414b2bba60c4be4c41a1bd3ba8edae069f1fc9.1633027275.git.softworkz@hotmail.com> <94fd41cac821f78e13de073ea8e69d37a3bc0b33.1633027275.git.softworkz@hotmail.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [EUyxAbl7i1M3UVEW1Y6Ntgs2dlPaPsbo] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0b2987f6-8641-4cfb-1028-08d98448dfe1 x-ms-traffictypediagnostic: MN2PR04MB5519: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 34zLQDAWn4UMuCqnB1ZGg5Nf/MNocMgL6M3GFSMu7LcDBoJgaWCrydsFuubc3xe+yHJT/kYzxr9uEaLGz/g02RUGqLDqxx9p2onMaWRolUlfnlVsgOvfbhxyjxjgNqcGjKnyUcbPbfIJBzIfDZ013IIpbMgRNTZrHbvsnwsUW65I/AffMKIySHMMLqhhOYfl+Bkc9A4/3gjcmqiMcuvu0gqJuq8yqbKiYCmwF2ZwGdSfvWauNJO+GePmrXgYu0emLMrgaXBdhWd9LGOLI6qI7EQBTotcWb4CLAr9codinjf1WMrXeTuiBIsd0Vgrz67L+K2APOHAtJQc4vx+vXDI0PELh5lShGCWPoJrKHxtAbzOuDjplpoTQmovxyklo+W9NHH6DPheQB7zQ391yNyOLAP54n1z8qJHCF/SWQv23nzYo88PqSEky5NzUUAJD/Lk9efisbHRs2QWs55OtklDcBPXzglHp6T3wQ0jB3rmloE8yppYDdYsYl+64ME+Dk/GYk38U1DiIfBk2ECa/Rzkjw== x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: WHc1ZT0DMXHYPpSk2uEuhIt72A7alA1HTJwM8Ye0tb8UP6PhHdIp2jx6dLhDSCRIyPeg0duNtYSxJwJ/BQwlbnFwQBuN2IKNFEBqXurOFtOreyjNbJMVObDiAvyZD6MzyAGCGWeSJ0AuDUPt0shPpg== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 0b2987f6-8641-4cfb-1028-08d98448dfe1 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:31:17.7376 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5519 Subject: [FFmpeg-devel] [PATCH v10 11/14] avfilter/graphicsub2text: Add new graphicsub2text filter (OCR) X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 4j4Y1Qabr+B8 Signed-off-by: softworkz --- configure | 1 + doc/filters.texi | 55 ++++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/sf_graphicsub2text.c | 323 +++++++++++++++++++++++++++++++ 5 files changed, 381 insertions(+) create mode 100644 libavfilter/sf_graphicsub2text.c diff --git a/configure b/configure index 6da059e6ee..9108742122 100755 --- a/configure +++ b/configure @@ -3601,6 +3601,7 @@ frei0r_deps_any="libdl LoadLibrary" frei0r_filter_deps="frei0r" frei0r_src_filter_deps="frei0r" fspp_filter_deps="gpl" +graphicsub2text_filter_deps="libtesseract" histeq_filter_deps="gpl" hqdn3d_filter_deps="gpl" interlace_filter_deps="gpl" diff --git a/doc/filters.texi b/doc/filters.texi index 001d2baa8f..e709fcbc03 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -25364,6 +25364,61 @@ ffmpeg -i "https://streams.videolan.org/ffmpeg/mkv_subtitles.mkv" -filter_comple @end example @end itemize +@section graphicsub2text + +Converts graphic subtitles to text subtitles by performing OCR. + +For this filter to be available, ffmpeg needs to be compiled with libtesseract (see https://github.com/tesseract-ocr/tesseract). +Language models need to be downloaded from https://github.com/tesseract-ocr/tessdata and put into as subfolder named 'tessdata' or into a folder specified via the environment variable 'TESSDATA_PREFIX'. +The path can also be specified via filter option (see below). + +Note: These models are including the data for both OCR modes. + +Inputs: +- 0: Subtitles [bitmap] + +Outputs: +- 0: Subtitles [text] + +It accepts the following parameters: + +@table @option +@item ocr_mode +The character recognition mode to use. + +Supported OCR modes are: + +@table @var +@item 0, tesseract +This is the classic libtesseract operation mode. It is fast but less accurate than LSTM. +@item 1, lstm +Newer OCR implementation based on ML models. Provides usually better results, requires more processing resources. +@item 2, both +Use a combination of both modes. +@end table + +@item tessdata_path +The path to a folder containing the language models to be used. + +@item language +The recognition language. It needs to match the first three characters of a language model file in the tessdata path. + +@end table + + +@subsection Examples + +@itemize +@item +Convert DVB graphic subtitles to ASS (text) subtitles + +Note: For this to work, you need to have the data file 'eng.traineddata' in a 'tessdata' subfolder (see above). +@example +ffmpeg ffmpeg -loglevel verbose -i "https://streams.videolan.org/streams/ts/video_subs_ttxt%2Bdvbsub.ts" -filter_complex "[0:13]graphicsub2text=ocr_mode=both" -c:s ass -y output.mkv +@end example +@end itemize + + @section graphicsub2video Renders graphic subtitles as video frames. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index b260e7e0a2..fc12664c56 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -290,6 +290,7 @@ OBJS-$(CONFIG_FSPP_FILTER) += vf_fspp.o qp_table.o OBJS-$(CONFIG_GBLUR_FILTER) += vf_gblur.o OBJS-$(CONFIG_GEQ_FILTER) += vf_geq.o OBJS-$(CONFIG_GRADFUN_FILTER) += vf_gradfun.o +OBJS-$(CONFIG_GRAPHICSUB2TEXT_FILTER) += sf_graphicsub2text.o OBJS-$(CONFIG_GRAPHICSUB2VIDEO_FILTER) += vf_overlay_graphicsubs.o framesync.o OBJS-$(CONFIG_GRAPHMONITOR_FILTER) += f_graphmonitor.o OBJS-$(CONFIG_GRAYWORLD_FILTER) += vf_grayworld.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 40bb3c229e..72ab388518 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -528,6 +528,7 @@ extern const AVFilter ff_avf_showwaves; extern const AVFilter ff_avf_showwavespic; extern const AVFilter ff_vaf_spectrumsynth; extern const AVFilter ff_sf_censor; +extern const AVFilter ff_sf_graphicsub2text; extern const AVFilter ff_sf_show_speaker; extern const AVFilter ff_sf_split_cc; extern const AVFilter ff_sf_stripstyles; diff --git a/libavfilter/sf_graphicsub2text.c b/libavfilter/sf_graphicsub2text.c new file mode 100644 index 0000000000..6c5c3d14ce --- /dev/null +++ b/libavfilter/sf_graphicsub2text.c @@ -0,0 +1,323 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * subtitle filter to convert graphical subs to text subs via OCR + */ + +#include +#include + +#include "libavutil/opt.h" +#include "subtitles.h" + +typedef struct SubOcrContext { + const AVClass *class; + int w, h; + + TessBaseAPI *tapi; + TessOcrEngineMode ocr_mode; + char *tessdata_path; + char *language; + + int readorder_counter; + + AVFrame *pending_frame; +} SubOcrContext; + + +static int init(AVFilterContext *ctx) +{ + SubOcrContext *s = ctx->priv; + const char* tver = TessVersion(); + int ret; + + s->tapi = TessBaseAPICreate(); + + if (!s->tapi || !tver || !strlen(tver)) { + av_log(ctx, AV_LOG_ERROR, "Failed to access libtesseract\n"); + return AVERROR(ENOSYS); + } + + av_log(ctx, AV_LOG_VERBOSE, "Initializing libtesseract, version: %s\n", tver); + + ret = TessBaseAPIInit4(s->tapi, s->tessdata_path, s->language, s->ocr_mode, NULL, 0, NULL, NULL, 0, 1); + if (ret < 0 ) { + av_log(ctx, AV_LOG_ERROR, "Failed to initialize libtesseract. Error: %d\n", ret); + return AVERROR(ENOSYS); + } + + ret = TessBaseAPISetVariable(s->tapi, "tessedit_char_blacklist", "|"); + if (ret < 0 ) { + av_log(ctx, AV_LOG_ERROR, "Failed to set 'tessedit_char_blacklist'. Error: %d\n", ret); + return AVERROR(EINVAL); + } + + return 0; +} + +static void uninit(AVFilterContext *ctx) +{ + SubOcrContext *s = ctx->priv; + + if (s->tapi) { + TessBaseAPIEnd(s->tapi); + TessBaseAPIDelete(s->tapi); + } +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats, *formats2; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType in_fmts[] = { AV_SUBTITLE_FMT_BITMAP, AV_SUBTITLE_FMT_NONE }; + static const enum AVSubtitleType out_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input format */ + formats = ff_make_format_list(in_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output format */ + formats2 = ff_make_format_list(out_fmts); + if ((ret = ff_formats_ref(formats2, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + SubOcrContext *s = ctx->priv; + + if (s->w <= 0 || s->h <= 0) { + s->w = inlink->w; + s->h = inlink->h; + } + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + const AVFilterContext *ctx = outlink->src; + SubOcrContext *s = ctx->priv; + + outlink->format = AV_SUBTITLE_FMT_ASS; + outlink->w = s->w; + outlink->h = s->h; + + return 0; +} + +static uint8_t* create_grayscale_image(AVFilterContext *ctx, AVSubtitleArea *area) +{ + uint8_t gray_pal[256]; + const size_t img_size = area->buf[0]->size; + const uint8_t* img = area->buf[0]->data; + uint8_t* gs_img = av_malloc(img_size); + + if (!gs_img) + return NULL; + + for (unsigned i = 0; i < 256; i++) { + const uint8_t *col = (uint8_t*)&area->pal[i]; + const int val = (int)col[3] * FFMAX3(col[0], col[1], col[2]); + gray_pal[i] = (uint8_t)(val >> 8); + } + + for (unsigned i = 0; i < img_size; i++) + gs_img[i] = 255 - gray_pal[img[i]]; + + return gs_img; +} + +static int convert_area(AVFilterContext *ctx, AVSubtitleArea *area) +{ + SubOcrContext *s = ctx->priv; + char *ocr_text = NULL; + int ret; + uint8_t *gs_img = create_grayscale_image(ctx, area); + + if (!gs_img) + return AVERROR(ENOMEM); + + area->type = AV_SUBTITLE_FMT_ASS; + TessBaseAPISetImage(s->tapi, gs_img, area->w, area->h, 1, area->linesize[0]); + TessBaseAPISetSourceResolution(s->tapi, 70); + + ret = TessBaseAPIRecognize(s->tapi, NULL); + if (ret == 0) + ocr_text = TessBaseAPIGetUTF8Text(s->tapi); + + if (!ocr_text) { + av_log(ctx, AV_LOG_WARNING, "OCR didn't return a text. ret=%d\n", ret); + area->ass = NULL; + } + else { + const size_t len = strlen(ocr_text); + + if (len > 0 && ocr_text[len - 1] == '\n') + ocr_text[len - 1] = 0; + + av_log(ctx, AV_LOG_VERBOSE, "OCR Result: %s\n", ocr_text); + + area->ass = av_strdup(ocr_text); + + TessDeleteText(ocr_text); + } + + av_freep(&gs_img); + av_buffer_unref(&area->buf[0]); + area->type = AV_SUBTITLE_FMT_ASS; + + return 0; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + SubOcrContext *s = ctx->priv; + AVFilterLink *outlink = inlink->dst->outputs[0]; + int ret, frame_sent = 0; + + if (s->pending_frame) { + const uint64_t pts_diff = frame->subtitle_pts - s->pending_frame->subtitle_pts; + s->pending_frame->subtitle_end_time = (uint32_t)(pts_diff / 1000); + + ret = ff_filter_frame(outlink, s->pending_frame); + s->pending_frame = NULL; + if (ret < 0) + return ret; + + frame_sent = 1; + + if (frame->num_subtitle_areas == 0) { + // No need to forward this empty frame + av_frame_free(&frame); + return 0; + } + } + + ret = av_frame_make_writable(frame); + + if (ret < 0) { + av_frame_free(&frame); + return ret; + } + + frame->format = AV_SUBTITLE_FMT_ASS; + + av_log(ctx, AV_LOG_DEBUG, "filter_frame sub_pts: %"PRIu64", start_time: %d, end_time: %d, num_areas: %d\n", + frame->subtitle_pts, frame->subtitle_start_time, frame->subtitle_end_time, frame->num_subtitle_areas); + + if (frame->num_subtitle_areas > 1 && + frame->subtitle_areas[0]->y > frame->subtitle_areas[frame->num_subtitle_areas - 1]->y) { + + for (unsigned i = 0; i < frame->num_subtitle_areas / 2; i++) + FFSWAP(AVSubtitleArea*, frame->subtitle_areas[i], frame->subtitle_areas[frame->num_subtitle_areas - i - 1]); + } + + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + AVSubtitleArea *area = frame->subtitle_areas[i]; + + ret = convert_area(ctx, area); + if (ret < 0) + return ret; + + if (area->ass && area->ass[0] != '\0') { + char *tmp = area->ass; + + if (i == 0) + area->ass = avpriv_ass_get_dialog(s->readorder_counter++, 0, "Default", NULL, tmp); + else + area->ass = av_asprintf("\\N%s", tmp); + + av_free(tmp); + } + } + + // When decoders can't determine the end time, they are setting it either to UINT32_NAX + // or 30s (dvbsub). + if (frame->num_subtitle_areas > 0 && frame->subtitle_end_time >= 30000) { + // Can't send it without end time, wait for the next frame to determine the end_display time + s->pending_frame = frame; + + if (frame_sent) + return 0; + + // To keep all going, send an empty frame instead + frame = ff_get_subtitles_buffer(outlink, AV_SUBTITLE_FMT_ASS); + if (!frame) + return AVERROR(ENOMEM); + + av_frame_copy_props(frame, s->pending_frame); + frame->subtitle_end_time = 1; + } + + return ff_filter_frame(outlink, frame); +} + +#define OFFSET(x) offsetof(SubOcrContext, x) +#define FLAGS (AV_OPT_FLAG_SUBTITLE_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption graphicsub2text_options[] = { + { "ocr_mode", "set ocr mode", OFFSET(ocr_mode), AV_OPT_TYPE_INT, {.i64=OEM_TESSERACT_ONLY}, OEM_TESSERACT_ONLY, 2, FLAGS, "ocr_mode" }, + { "tesseract", "classic tesseract ocr", 0, AV_OPT_TYPE_CONST, {.i64=OEM_TESSERACT_ONLY}, 0, 0, FLAGS, "ocr_mode" }, + { "lstm", "lstm (ML based)", 0, AV_OPT_TYPE_CONST, {.i64=OEM_LSTM_ONLY}, 0, 0, FLAGS, "ocr_mode" }, + { "both", "use both models combined", 0, AV_OPT_TYPE_CONST, {.i64=OEM_TESSERACT_LSTM_COMBINED}, 0, 0, FLAGS, "ocr_mode" }, + { "tessdata_path", "path to tesseract data", OFFSET(tessdata_path), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS, NULL }, + { "language", "ocr language", OFFSET(language), AV_OPT_TYPE_STRING, {.str = "eng"}, 0, 0, FLAGS, NULL }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(graphicsub2text); + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_frame, + .config_props = config_input, + }, +}; + +static const AVFilterPad outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .config_props = config_output, + }, +}; + +const AVFilter ff_sf_graphicsub2text = { + .name = "graphicsub2text", + .description = NULL_IF_CONFIG_SMALL("Convert graphical subtitles to text subtitles via OCR"), + .init = init, + .uninit = uninit, + .query_formats = query_formats, + .priv_size = sizeof(SubOcrContext), + .priv_class = &graphicsub2text_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), +}; From patchwork Thu Sep 30 19:31:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30808 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp986770iob; Thu, 30 Sep 2021 12:33:06 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyHtiocr2Zmi0BNSpi8xM3VMKSJ7axmf2RCOcnCtarOjFFHd8F96hvPRoDT/2j1gCzQVQdQ X-Received: by 2002:a50:9552:: with SMTP id v18mr8953633eda.395.1633030386379; Thu, 30 Sep 2021 12:33:06 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ox7si4140389ejb.360.2021.09.30.12.33.05; Thu, 30 Sep 2021 12:33:06 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=g3tZL7qp; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1AAFA68AB27; Thu, 30 Sep 2021 22:31:31 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10olkn2053.outbound.protection.outlook.com [40.92.42.53]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9531068AA8F for ; Thu, 30 Sep 2021 22:31:23 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GeAlvo+lcHv4nX4hDrDse6yRV6Yt75I0unHMobJtMtqGluNkGEWswXRG22Uka9YpSECXSxQCpO+z53Wk5FnFCd9P055QNwpjqQzA5wdrOVz5S6f9VmboeoUoMkUi5a7gFmzxYpaEIySGCYU8NcibkYIwnifVFHevpZ8uzkWQYfkyBI77URHcwdotyL3mflVpgifdJR5HecZsTbagSQ1MkQ40YEVItI3y2bOYgzDZm3byWvG1BCvkuzLFYWK2sL3nCuUp13Wy4xTOfgQIWjievxhUQl5j8yH4EJcx1PMpA07Ex87J/dDRxzLL80KvRkbX0TaOlt7UCXYwG25rIYzmng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=G7VJzhbG/K+9bgjkapMYSjkMTYCMHRNUIFAn26yIzRQ=; b=KqAzb/xLqHwqa9TGpRsRlzMrjCds1AhwihqARfT7BqGttC4Q8cR3W0UNkK2LgSCH43bMivV5SVeJ5jQH9Sv29crtdglhdXwlrI+CYCiJR128ZP//8c/Qf3EZ6oggesGwGPAXbCqZblPDyS8LANwBAa/gVJkk7yY4L8zvAJkl0tUXblN1eHHNxD9QoE/oaE3sheFKNItCFl7MMU/K8OwWeEFquqAVKWTOVHL9arLfFahQ0QKBNFngmBynqTZU3bvQ4NJ0plWMTdCC+p3kGxvWh8kexE4XpJrFpDU+qIFABaSm6abAfGJE5W1MCrm8n1zV05Aai73RNbLTqv1yjk7PMg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G7VJzhbG/K+9bgjkapMYSjkMTYCMHRNUIFAn26yIzRQ=; b=g3tZL7qpk1+Rqq7os+4c5l9s0YOesIy2cgGLtfpC7HJRJ7w8i6SsyvswSwHU8yoUWkLFkDJj/GVsRZibpE97/TAQxqDDGP6C2D3Kun3z8dfwUyv2xB+X0jRPhtjRdLc/t4y6Z2sK2Ya+RpveSEwbWBOIx7AwIwMhb55+j7MVio+ylUoQL69HPRe7yICMX541EzGpPvC4PUb+DVSi7nebKdWu86o3hQoSSuKuFVAea0hizKfBrXf43o+v3kxvjYWZh59WxgacSaHlEEaWHDI3Q8/n1i+EoSeo/7Or9RX5bxKZwV1IMiHwcYChzoCAglqmd4xzbXB6Z+e0a3JUmGPCPA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5519.namprd04.prod.outlook.com (2603:10b6:208:da::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:31:20 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:31:20 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 12/14] avfilter/subscale: Add filter for scaling and/or re-arranging graphical subtitles Thread-Index: AQHXtjG/aYpCK4nUqkWxCTwq3rVr5A== Date: Thu, 30 Sep 2021 19:31:20 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> <8a75051fd7be111f4ec52073a6da6679724a7127.1633027275.git.softworkz@hotmail.com> <48414b2bba60c4be4c41a1bd3ba8edae069f1fc9.1633027275.git.softworkz@hotmail.com> <94fd41cac821f78e13de073ea8e69d37a3bc0b33.1633027275.git.softworkz@hotmail.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [VYPltTvojDVnz17CLZXRkUEYHTq7Ko9N] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 06bedd8a-f54d-4fa8-ccea-08d98448e1bb x-ms-traffictypediagnostic: MN2PR04MB5519: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: lGuM9D+t3bxsYsX3msNJVfizX9zQUazMPKbzw/OU9XcUWzKYFPa1auKaeCBPxHzs0HhsLgIzHNpXVEjJvRg7c8jnsY225Q2cuFMC7k/jr4qWcJT5Ni9hcYz3cBCEToOaZJNmOMf5QP/WmpB6lmuNoqIjh9+gUX6R+5A321ZSuH3/Qk8N2rHuY6oiFwUWWjQ/EpEr+Tk2VfjAs5I28MQjo9zTpgwUsXw15+DrcS9/xf7NRdKW61XcPOZBxef3AsVO+0/JlCUZML0Xef5ssxNCS4EApK4QRhBcCjgCKDRkpz/JDLUa+ty9tkXF8LXQY+bSho672ButsyO89eiZxN1+6GdFDGrlq/GrVhKdMbPwQER+wROccbf9QOaY6VtYUCpOsuiXZRlazKhT0Mf9+J8XPKsv+f4oRL0qfQktGpFPdG7JZvYPgKdG2sOIBvttH+5aOL8/nOvhBTZVhI500wsGANOdMm1vrvV2n99Mxs/baC0= x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: nQq/1JpeQS0KKfv83fxrsi5Hx2RDFGJL+XtsiqAkdz3QhqNj52HuU4LfnpQNhOVHLVvvsx/+en1Kn3RWt1EiJGr3KQPi73HT2/eGL/iCFagkpwKeRi02iVfxHNdk4NnDxjELVCkn9nc5uhKMMjjXdw== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 06bedd8a-f54d-4fa8-ccea-08d98448e1bb X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:31:20.8069 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5519 Subject: [FFmpeg-devel] [PATCH v10 12/14] avfilter/subscale: Add filter for scaling and/or re-arranging graphical subtitles X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 72QsXLx4ulLE Signed-off-by: softworkz --- configure | 1 + doc/filters.texi | 160 +++++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/sf_subscale.c | 867 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 1030 insertions(+) create mode 100644 libavfilter/sf_subscale.c diff --git a/configure b/configure index 9108742122..93b26da269 100755 --- a/configure +++ b/configure @@ -3665,6 +3665,7 @@ sr_filter_deps="avformat swscale" sr_filter_select="dnn" stereo3d_filter_deps="gpl" split_cc_filter_deps="cc_dec_decoder" +subscale_filter_deps="swscale avcodec" subtitles_filter_deps="avformat avcodec libass" super2xsai_filter_deps="gpl" pixfmts_super2xsai_test_deps="super2xsai_filter" diff --git a/doc/filters.texi b/doc/filters.texi index e709fcbc03..9cc6b05835 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -25695,6 +25695,166 @@ Override default style or script info parameters of the subtitles. It accepts a string containing ASS style format @code{KEY=VALUE} couples separated by ",". @end table + +@section subscale + +Provides high-quality scaling and rearranging functionality for graphical subtitles. + +The subscale filter provides multiple approaches for manipulating +the size and position of graphical subtitle rectangles wich can +be combined or used separately. +Scaling is performed by converting the palettized subtitle bitmaps +to RGBA and re-quantization to palette colors afterwards via elbg algorithm. + +The two major operations are 'scale' and 're-arrange' with the +latter being separated as 'arrange_h' and 'arrange_v'. + + +Inputs: +- 0: Subtitles [bitmap] + +Outputs: +- 0: Subtitles [bitmap] + +It accepts the following parameters: + +@table @option + +@item w, width +Set the width of the output. +Width and height in case of graphical subtitles are just indicating +a virtual size for which the output (consisting of 0-n bitmap rectangles) +is intended to be displayed on. + +@item h, height +Set the height of the output. + +@item margin_h +Sets a horizontal margin to be preserverved when using any +of the arrange modes. + +@item margin_v +Sets a vertical margin to be preserverved when using any +of the arrange modes. + +@item force_original_aspect_ratio +Enable decreasing or increasing output video width or height if necessary to +keep the original aspect ratio. Possible values: + +@table @samp +@item disable +Scale the video as specified and disable this feature. + +@item decrease +The output video dimensions will automatically be decreased if needed. + +@item increase +The output video dimensions will automatically be increased if needed. + +@end table + + +@item scale_mode +Specifies how subtitle bitmaps should be scaled. +The scale factor is determined by the the factor between input +and output size. + +@table @samp +@item none +Do not apply any common scaling. + +@item uniform +Uniformly scale all subtitle bitmaps including their positions. + +@item uniform_no_reposition +Uniformly scale all subtitle bitmaps without changing positions. + +@end table + + +@item arrange_h +Specifies how subtitle bitmaps should be arranged horizontally. + +@item arrange_v +Specifies how subtitle bitmaps should be arranged vertically. + + +@table @samp +@item none +Do not rearrange subtitle bitmaps. + +@item margin_no_scale +Move subtitle bitmaps to be positioned inside the specified +margin (margin_h or margin_v) when possible and without scaling. + +@item margin_and_scale +Move subtitle bitmaps to be positioned inside the specified +margin (margin_h or margin_v) and scale in case it doesn't fit. + +@item snapalign_no_scale +Categorize subtitle bitmap positions as one of left/center/right +or top/bottom/middle based on original positioning and apply +these alignments for the target positioning. +No scaling will be applied. + +@item snapalign_and_scale +Categorize subtitle bitmap positions as one of left/center/right +or top/bottom/middle based on original positioning and apply +these alignments for the target positioning. +Bitmaps that do not fit inside the margins borders are +scaled to fit. +@end table + +@item eval +Set evaluation mode for the expressions (@option{width}, @option{height}). + +It accepts the following values: +@table @samp +@item init +Evaluate expressions only once during the filter initialization. + +@item frame +Evaluate expressions for each incoming frame. This is way slower than the +@samp{init} mode since it requires all the scalers to be re-computed, but it +allows advanced dynamic expressions. +@end table + +Default value is @samp{init}. + + +@item num_colors +Set the number of palette colors for output images. +Choose the maximum (256) when further processing is done (e.g. +overlaying on a video). +When subtitles will be encoded as bitmap subtitles (e.g. dvbsub), +a smaller number of palette colors (e.g. 4-16) might need to be used, depending +on the target format and codec. + +@item bitmap_width_align +@item bitmap_height_align +Make sure that subtitle bitmap sizes are a multiple of this +value. Default is 2. + +@end table + +@subsection Examples + +@itemize +@item +Uniformly scale down video and bitmap subtitles and encode +subtitles as dvbsub. +@example +ffmpeg -y -loglevel verbose -ss 32 -i "https://streams.videolan.org/samples/sub/largeres_vobsub.mkv" -filter_complex "[0:0]scale=480x270[vid1];[0:10]subscale=w=480:h=270:num_colors=16[sub1]" -map [vid1] -map [sub1] -c:s dvbsub -c:v libx265 output.ts +@end example +@item +Squeeze video vertically and arrange subtitle bitmaps +inside the video area without scaling, then overlay +subtitles onto the video. +@example +ffmpeg -y -loglevel verbose -ss 32 -i "https://streams.videolan.org/samples/sub/largeres_vobsub.mkv" -filter_complex "[0:0]scale=1920x400,setsar=1[vid1];[0:10]subscale=w=1920:h=400:scale_mode=none:margin_h=50:arrange_h=margin_no_scale:arrange_v=margin_no_scale:margin_v=50:num_colors=256[sub1];[vid1][sub1]overlay_graphicsubs" -c:v libx265 output.ts +@end example +@end itemize + @c man end SUBTITLE FILTERS @chapter Multimedia Filters diff --git a/libavfilter/Makefile b/libavfilter/Makefile index fc12664c56..4b13a4ffe4 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -544,6 +544,7 @@ OBJS-$(CONFIG_SHOW_SPEAKER_FILTER) += sf_textmod.o OBJS-$(CONFIG_TEXTMOD_FILTER) += sf_textmod.o OBJS-$(CONFIG_SPLIT_CC_FILTER) += sf_split_cc.o OBJS-$(CONFIG_STRIPSTYLES_FILTER) += sf_stripstyles.o +OBJS-$(CONFIG_SUBSCALE_FILTER) += sf_subscale.o # multimedia filters OBJS-$(CONFIG_ABITSCOPE_FILTER) += avf_abitscope.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 72ab388518..c4552b944d 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -532,6 +532,7 @@ extern const AVFilter ff_sf_graphicsub2text; extern const AVFilter ff_sf_show_speaker; extern const AVFilter ff_sf_split_cc; extern const AVFilter ff_sf_stripstyles; +extern const AVFilter ff_sf_subscale; extern const AVFilter ff_sf_textmod; extern const AVFilter ff_svf_graphicsub2video; extern const AVFilter ff_svf_textsub2video; diff --git a/libavfilter/sf_subscale.c b/libavfilter/sf_subscale.c new file mode 100644 index 0000000000..9f14f9fee4 --- /dev/null +++ b/libavfilter/sf_subscale.c @@ -0,0 +1,867 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * scale graphical subtitles filter + */ + +#include + +#include "drawutils.h" +#include "internal.h" +#include "scale_eval.h" +#include "libavutil/eval.h" +#include "libavutil/opt.h" +#include "libswscale/swscale.h" + +#include "libavcodec/elbg.h" + +static const char *const var_names[] = { + "in_w", "iw", + "in_h", "ih", + "out_w", "ow", + "out_h", "oh", + "a", + "sar", + "dar", + "margin_h", + "margin_v", + NULL +}; + +enum var_name { + VAR_IN_W, VAR_IW, + VAR_IN_H, VAR_IH, + VAR_OUT_W, VAR_OW, + VAR_OUT_H, VAR_OH, + VAR_A, + VAR_SAR, + VAR_DAR, + VARS_B_H, + VARS_B_V, + VARS_NB +}; + +enum EvalMode { + EVAL_MODE_INIT, + EVAL_MODE_FRAME, + EVAL_MODE_NB +}; + +enum SubScaleMode { + SSM_NONE, + SSM_UNIFORM, + SSM_UNIFORM_NO_REPOSITION, +}; + +enum SubArrangeMode { + SAM_NONE, + SAM_ENSUREMARGIN_NO_SCALE, + SAM_ENSUREMARGIN_AND_SCALE, + SAM_SNAPALIGNMENT_NO_SCALE, + SAM_SNAPALIGNMENT_AND_SCALE, +}; + +typedef struct SubScaleContext { + const AVClass *class; + struct SwsContext *sws; + AVDictionary *opts; + + int w, h; + + char *w_expr; ///< width expression string + char *h_expr; ///< height expression string + AVExpr *w_pexpr; + AVExpr *h_pexpr; + double var_values[VARS_NB]; + + int force_original_aspect_ratio; + int eval_mode; ///< expression evaluation mode + + // Scale Options + enum SubScaleMode scale_mode; + + // Arrange Options + enum SubArrangeMode arrange_mode_h; + enum SubArrangeMode arrange_mode_v; + int margin_h; + int margin_v; + char *margin_h_expr; + char *margin_v_expr; + AVExpr *margin_h_pexpr; + AVExpr *margin_v_pexpr; + + // Bitmap Options + int num_output_colors; + int bitmap_width_align; + int bitmap_height_align; + + // Color Quantization Fields + struct ELBGContext *ctx; + AVLFG lfg; + int *codeword; + int *codeword_closest_codebook_idxs; + int *codebook; + int r_idx, g_idx, b_idx, a_idx; +} SubScaleContext; + + +static int config_output(AVFilterLink *outlink); + +static int check_exprs(AVFilterContext *ctx) +{ + const SubScaleContext *s = ctx->priv; + unsigned vars_w[VARS_NB] = { 0 }, vars_h[VARS_NB] = { 0 }; + + if (!s->w_pexpr && !s->h_pexpr) + return AVERROR(EINVAL); + + if (s->w_pexpr) + av_expr_count_vars(s->w_pexpr, vars_w, VARS_NB); + if (s->h_pexpr) + av_expr_count_vars(s->h_pexpr, vars_h, VARS_NB); + + if (vars_w[VAR_OUT_W] || vars_w[VAR_OW]) { + av_log(ctx, AV_LOG_ERROR, "Width expression cannot be self-referencing: '%s'.\n", s->w_expr); + return AVERROR(EINVAL); + } + + if (vars_h[VAR_OUT_H] || vars_h[VAR_OH]) { + av_log(ctx, AV_LOG_ERROR, "Height expression cannot be self-referencing: '%s'.\n", s->h_expr); + return AVERROR(EINVAL); + } + + if ((vars_w[VAR_OUT_H] || vars_w[VAR_OH]) && + (vars_h[VAR_OUT_W] || vars_h[VAR_OW])) { + av_log(ctx, AV_LOG_WARNING, "Circular references detected for width '%s' and height '%s' - possibly invalid.\n", s->w_expr, s->h_expr); + } + + if (s->margin_h_pexpr) + av_expr_count_vars(s->margin_h_pexpr, vars_w, VARS_NB); + if (s->margin_v_pexpr) + av_expr_count_vars(s->margin_v_pexpr, vars_h, VARS_NB); + + return 0; +} + +static int scale_parse_expr(AVFilterContext *ctx, char *str_expr, AVExpr **pexpr_ptr, const char *var, const char *args) +{ + SubScaleContext *s = ctx->priv; + int ret, is_inited = 0; + char *old_str_expr = NULL; + AVExpr *old_pexpr = NULL; + + if (str_expr) { + old_str_expr = av_strdup(str_expr); + if (!old_str_expr) + return AVERROR(ENOMEM); + av_opt_set(s, var, args, 0); + } + + if (*pexpr_ptr) { + old_pexpr = *pexpr_ptr; + *pexpr_ptr = NULL; + is_inited = 1; + } + + ret = av_expr_parse(pexpr_ptr, args, var_names, + NULL, NULL, NULL, NULL, 0, ctx); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Cannot parse expression for %s: '%s'\n", var, args); + goto revert; + } + + ret = check_exprs(ctx); + if (ret < 0) + goto revert; + + if (is_inited && (ret = config_output(ctx->outputs[0])) < 0) + goto revert; + + av_expr_free(old_pexpr); + av_freep(&old_str_expr); + + return 0; + +revert: + av_expr_free(*pexpr_ptr); + *pexpr_ptr = NULL; + if (old_str_expr) { + av_opt_set(s, var, old_str_expr, 0); + av_free(old_str_expr); + } + if (old_pexpr) + *pexpr_ptr = old_pexpr; + + return ret; +} + +static av_cold int init_dict(AVFilterContext *ctx, AVDictionary **opts) +{ + SubScaleContext *s = ctx->priv; + uint8_t rgba_map[4]; + int ret; + + if (!s->w_expr) + av_opt_set(s, "w", "iw", 0); + if (!s->h_expr) + av_opt_set(s, "h", "ih", 0); + + ret = scale_parse_expr(ctx, NULL, &s->w_pexpr, "width", s->w_expr); + if (ret < 0) + return ret; + + ret = scale_parse_expr(ctx, NULL, &s->h_pexpr, "height", s->h_expr); + if (ret < 0) + return ret; + + av_log(ctx, AV_LOG_VERBOSE, "w:%s h:%s\n", + s->w_expr, s->h_expr); + + if (!s->margin_h_expr) + av_opt_set(s, "margin_h", "0", 0); + if (!s->margin_v_expr) + av_opt_set(s, "margin_v", "0", 0); + + ret = scale_parse_expr(ctx, NULL, &s->margin_h_pexpr, "margin_h", s->margin_h_expr); + if (ret < 0) + return ret; + + ret = scale_parse_expr(ctx, NULL, &s->margin_v_pexpr, "margin_v", s->margin_v_expr); + if (ret < 0) + return ret; + + s->opts = *opts; + *opts = NULL; + + ff_fill_rgba_map(&rgba_map[0], AV_PIX_FMT_RGB32); + + s->r_idx = rgba_map[0]; // R + s->g_idx = rgba_map[1]; // G + s->b_idx = rgba_map[2]; // B + s->a_idx = rgba_map[3]; // A + + av_lfg_init(&s->lfg, 123456789); + + + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + SubScaleContext *s = ctx->priv; + av_expr_free(s->w_pexpr); + av_expr_free(s->h_pexpr); + s->w_pexpr = s->h_pexpr = NULL; + + av_expr_free(s->margin_h_pexpr); + av_expr_free(s->margin_v_pexpr); + s->margin_h_pexpr = s->margin_v_pexpr = NULL; + + sws_freeContext(s->sws); + s->sws = NULL; + av_dict_free(&s->opts); + + avpriv_elbg_free(&s->ctx); + + av_freep(&s->codebook); + av_freep(&s->codeword); + av_freep(&s->codeword_closest_codebook_idxs); +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_BITMAP, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input formats */ + formats = ff_make_format_list(subtitle_fmts); + if ((ret = ff_formats_ref(formats, &inlink->outcfg.formats)) < 0) + return ret; + + /* set output formats */ + if ((ret = ff_formats_ref(formats, &outlink->incfg.formats)) < 0) + return ret; + + return 0; +} + +static int config_input(AVFilterLink *inlink) +{ + ////const AVFilterContext *ctx = inlink->dst; + ////SubScaleContext *s = ctx->priv; + + ////if (s->w <= 0 || s->h <= 0) { + //// s->w = inlink->w; + //// s->h = inlink->h; + ////} + return 0; +} + +static int scale_eval_dimensions(AVFilterContext *ctx) +{ + SubScaleContext *s = ctx->priv; + const AVFilterLink *inlink = ctx->inputs[0]; + char *expr; + int eval_w, eval_h, margin_h, margin_v; + int ret; + double res; + + s->var_values[VAR_IN_W] = s->var_values[VAR_IW] = inlink->w; + s->var_values[VAR_IN_H] = s->var_values[VAR_IH] = inlink->h; + s->var_values[VAR_OUT_W] = s->var_values[VAR_OW] = NAN; + s->var_values[VAR_OUT_H] = s->var_values[VAR_OH] = NAN; + s->var_values[VARS_B_H] = s->var_values[VARS_B_V] = 0; + s->var_values[VAR_A] = (double) inlink->w / inlink->h; + s->var_values[VAR_SAR] = inlink->sample_aspect_ratio.num ? + (double) inlink->sample_aspect_ratio.num / inlink->sample_aspect_ratio.den : 1; + s->var_values[VAR_DAR] = s->var_values[VAR_A] * s->var_values[VAR_SAR]; + + res = av_expr_eval(s->w_pexpr, s->var_values, NULL); + s->var_values[VAR_OUT_W] = s->var_values[VAR_OW] = (int) res == 0 ? inlink->w : (int) res; + + res = av_expr_eval(s->h_pexpr, s->var_values, NULL); + if (isnan(res)) { + expr = s->h_expr; + ret = AVERROR(EINVAL); + goto fail; + } + eval_h = s->var_values[VAR_OUT_H] = s->var_values[VAR_OH] = (int) res == 0 ? inlink->h : (int) res; + + res = av_expr_eval(s->w_pexpr, s->var_values, NULL); + if (isnan(res)) { + expr = s->w_expr; + ret = AVERROR(EINVAL); + goto fail; + } + eval_w = s->var_values[VAR_OUT_W] = s->var_values[VAR_OW] = (int) res == 0 ? inlink->w : (int) res; + + s->w = eval_w; + s->h = eval_h; + + res = av_expr_eval(s->margin_h_pexpr, s->var_values, NULL); + s->var_values[VARS_B_H] = (int)res; + + res = av_expr_eval(s->margin_v_pexpr, s->var_values, NULL); + if (isnan(res)) { + expr = s->margin_v_expr; + ret = AVERROR(EINVAL); + goto fail; + } + margin_v = s->var_values[VARS_B_V] = (int)res; + + res = av_expr_eval(s->margin_h_pexpr, s->var_values, NULL); + if (isnan(res)) { + expr = s->margin_h_expr; + ret = AVERROR(EINVAL); + goto fail; + } + margin_h = s->var_values[VARS_B_H] = (int)res; + + s->margin_h = margin_h; + s->margin_v = margin_v; + + return 0; + +fail: + av_log(ctx, AV_LOG_ERROR, + "Error when evaluating the expression '%s'.\n", expr); + return ret; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + AVFilterLink *inlink = outlink->src->inputs[0]; + SubScaleContext *s = ctx->priv; + int ret; + + outlink->format = AV_SUBTITLE_FMT_BITMAP; + outlink->time_base = ctx->inputs[0]->time_base; + + if ((ret = scale_eval_dimensions(ctx)) < 0) + goto fail; + + ff_scale_adjust_dimensions(inlink, &s->w, &s->h, + s->force_original_aspect_ratio, 2); + + if (s->w > INT_MAX || + s->h > INT_MAX || + (s->h * inlink->w) > INT_MAX || + (s->w * inlink->h) > INT_MAX) + av_log(ctx, AV_LOG_ERROR, "Rescaled value for width or height is too big.\n"); + + outlink->w = s->w; + outlink->h = s->h; + + if (s->sws) + sws_freeContext(s->sws); + + s->sws = sws_alloc_context(); + if (!s->sws) + return AVERROR(ENOMEM); + + av_opt_set_pixel_fmt(s->sws, "src_format", AV_PIX_FMT_PAL8, 0); + av_opt_set_int(s->sws, "dst_format", AV_PIX_FMT_RGB32, 0); + av_opt_set_int(s->sws, "threads", ff_filter_get_nb_threads(ctx), 0); + + if (s->opts) { + const AVDictionaryEntry *e = NULL; + while ((e = av_dict_get(s->opts, "", e, AV_DICT_IGNORE_SUFFIX))) { + if ((ret = av_opt_set(s->sws, e->key, e->value, 0)) < 0) + return ret; + } + } + + if ((ret = sws_init_context(s->sws, NULL, NULL)) < 0) + return ret; + + if (inlink->sample_aspect_ratio.num){ + outlink->sample_aspect_ratio = av_mul_q((AVRational){outlink->h * inlink->w, outlink->w * inlink->h}, inlink->sample_aspect_ratio); + } else + outlink->sample_aspect_ratio = inlink->sample_aspect_ratio; + + av_log(ctx, AV_LOG_VERBOSE, "Output size set to %dx%d.\n", outlink->w, outlink->h); + + return 0; +fail: + return ret; +} + +static int palettize_image(SubScaleContext *const s, const int w, const int h, const int src_linesize, uint8_t *src_data, + int dst_linesize, uint8_t *dst_data, uint32_t *dst_pal) +{ + int k, ret; + const int codeword_length = w * h; + + /* Re-Initialize */ + s->codeword = av_realloc_f(s->codeword, codeword_length, 4 * sizeof(*s->codeword)); + if (!s->codeword) + return AVERROR(ENOMEM); + + s->codeword_closest_codebook_idxs = + av_realloc_f(s->codeword_closest_codebook_idxs, codeword_length, + sizeof(*s->codeword_closest_codebook_idxs)); + if (!s->codeword_closest_codebook_idxs) + return AVERROR(ENOMEM); + + s->codebook = av_realloc_f(s->codebook, s->num_output_colors, 4 * sizeof(*s->codebook)); + if (!s->codebook) + return AVERROR(ENOMEM); + + /* build the codeword */ + k = 0; + for (int i = 0; i < h; i++) { + uint8_t *p = src_data; + for (int j = 0; j < w; j++) { + s->codeword[k++] = p[s->b_idx]; + s->codeword[k++] = p[s->g_idx]; + s->codeword[k++] = p[s->r_idx]; + s->codeword[k++] = p[s->a_idx]; + p += 4; + } + src_data += src_linesize; + } + + /* compute the codebook */ + ret = avpriv_elbg_do(&s->ctx, s->codeword, 4, + codeword_length, s->codebook, + s->num_output_colors, 1, + s->codeword_closest_codebook_idxs, &s->lfg, 0); + + if (ret < 0) + return ret; + + /* Write Palette */ + for (int i = 0; i < s->num_output_colors; i++) { + dst_pal[i] = s->codebook[i*4+3] << 24 | + (s->codebook[i*4+2] << 16) | + (s->codebook[i*4+1] << 8) | + s->codebook[i*4 ]; + } + + /* Write Image */ + k = 0; + for (int i = 0; i < h; i++) { + uint8_t *p = dst_data; + for (int j = 0; j < w; j++, p++) { + p[0] = s->codeword_closest_codebook_idxs[k++]; + } + + dst_data += dst_linesize; + } + + return ret; +} + +static int rescale_size(int64_t a, AVRational factor) +{ + const int64_t res = av_rescale_rnd(a, factor.num, factor.den, AV_ROUND_NEAR_INF); + if (res > INT32_MAX || res < 0) + return 0; + + return (int)res; +} + + +static int scale_area(AVFilterLink *link, AVSubtitleArea *area, const int target_width, const int target_height) +{ + const AVFilterContext *ctx = link->dst; + SubScaleContext *s = ctx->priv; + int ret; + + AVBufferRef *dst_buffer; + const uint8_t* data[2] = { area->buf[0]->data, (uint8_t *)&area->pal }; + const int dstW = FFALIGN(target_width, s->bitmap_width_align); + const int dstH = FFALIGN(target_height, s->bitmap_height_align); + const int tmp_linesize[2] = { FFALIGN(dstW * 4, 32), 0 }; + const int dst_linesize[2] = { dstW, 0 }; + uint8_t* tmp[2] = { 0, 0 }; + + AVBufferRef *tmp_buffer = av_buffer_allocz(tmp_linesize[0] * dstH); + if (!tmp_buffer) + return AVERROR(ENOMEM); + + if (!s->sws) + return 0; + + tmp[0] = tmp_buffer->data; + + s->sws = sws_getCachedContext(s->sws, area->w, area->h, AV_PIX_FMT_PAL8, + dstW, dstH, AV_PIX_FMT_RGB32, SWS_BICUBIC, NULL, NULL, NULL); + if (!s->sws) { + av_log(NULL, AV_LOG_FATAL, "Cannot initialize the conversion context. dstW=%d dstH=%d\n", dstW, dstH); + return AVERROR(EINVAL); + } + + // Rescale to ARGB + ret = sws_scale(s->sws, data, area->linesize, 0, area->h, tmp, tmp_linesize); + if (ret < 0) { + av_buffer_unref(&tmp_buffer); + return ret; + } + + // Alloc output buffer + dst_buffer = av_buffer_allocz(dst_linesize[0] * dstH); + if (!dst_buffer) { + av_buffer_unref(&tmp_buffer); + return AVERROR(ENOMEM); + } + + // Quantize to palettized image + ret = palettize_image(s, dstW, dstH, tmp_linesize[0], tmp[0], dst_linesize[0], dst_buffer->data, area->pal); + av_buffer_unref(&tmp_buffer); + + if (ret < 0) { + av_buffer_unref(&dst_buffer); + return ret; + } + + av_buffer_unref(&area->buf[0]); + ret = av_buffer_replace(&area->buf[0], dst_buffer); + if (ret < 0) { + av_buffer_unref(&dst_buffer); + return ret; + } + + area->w = dstW; + area->h = dstH; + area->linesize[0] = dst_linesize[0]; + area->nb_colors = s->num_output_colors; + + return ret; +} + +static int process_area(AVFilterLink *inlink, AVSubtitleArea *area, AVRational x_factor, AVRational y_factor) +{ + AVFilterContext *ctx = inlink->dst; + const SubScaleContext *s = ctx->priv; + int target_w, target_h, target_x, target_y; + const int border_l = s->margin_h; + const int border_r = s->w - s->margin_h; + const int border_t = s->margin_v; + const int border_b = s->h - s->margin_v; + + av_log(ctx, AV_LOG_VERBOSE, "process_area - start: x/y: (%d:%d) size: %dx%d scale_mode: %d x-factor: %d:%d y-factor: %d:%d\n", + area->x, area->y, area->w, area->h, s->scale_mode, x_factor.num, x_factor.den, y_factor.num, y_factor.den); + + switch (s->scale_mode) { + case SSM_NONE: + target_w = area->w; + target_h = area->h; + target_x = area->x; + target_y = area->y; + break; + case SSM_UNIFORM: + target_w = rescale_size(area->w, x_factor); + target_h = rescale_size(area->h, y_factor); + target_x = rescale_size(area->x, x_factor); + target_y = rescale_size(area->y, y_factor); + break; + case SSM_UNIFORM_NO_REPOSITION: + target_w = rescale_size(area->w, x_factor); + target_h = rescale_size(area->h, y_factor); + target_x = area->x; + target_y = area->y; + break; + default: + av_log(ctx, AV_LOG_ERROR, "Invalid scale_mode: %d\n", s->scale_mode); + return AVERROR(EINVAL); + } + + av_log(ctx, AV_LOG_VERBOSE, "process_area - scaled: x/y: (%d:%d) size: %dx%d.\n", target_x, target_y, target_w, target_h); + + + switch (s->arrange_mode_h) { + case SAM_ENSUREMARGIN_AND_SCALE: + case SAM_SNAPALIGNMENT_AND_SCALE: + { + // IF it doesn't fit - scale it + int max_width = s->w - 2 * s->margin_h; + + if (max_width < 2) + max_width = 2; + + if (target_w > max_width) { + target_h = (int)av_rescale(target_h, max_width, target_w); + target_w = max_width; + target_x = s->margin_h; + } + } + break; + } + + switch (s->arrange_mode_h) { + case SAM_NONE: + break; + case SAM_ENSUREMARGIN_NO_SCALE: + case SAM_ENSUREMARGIN_AND_SCALE: + // Left border + if (target_x < border_l) + target_x = border_l; + + // Right border + if (target_x + target_w > border_r) + target_x = border_r - target_w; + + break; + case SAM_SNAPALIGNMENT_NO_SCALE: + case SAM_SNAPALIGNMENT_AND_SCALE: + { + // Use original values to detect alignment + const int left_margin = area->x; + const int right_margin = inlink->w - area->x - area->w; + const AVRational diff_factor_r = { left_margin - right_margin, area->w }; + const float diff_factor = (float)av_q2d(diff_factor_r); + + if (diff_factor > 0.2f) { + // Right aligned + target_x = border_r - target_w; + } else if (diff_factor < -0.2f) { + // Left aligned + target_x = border_l; + } else { + // Centered + target_x = (inlink->w - area->w) / 2; + } + } + + break; + default: + av_log(ctx, AV_LOG_ERROR, "Invalid arrange_mode_h: %d\n", s->arrange_mode_h); + return AVERROR(EINVAL); + } + + av_log(ctx, AV_LOG_VERBOSE, "process_area - arr_h: x/y: (%d:%d) size: %dx%d.\n", target_x, target_y, target_w, target_h); + + + switch (s->arrange_mode_v) { + case SAM_ENSUREMARGIN_AND_SCALE: + case SAM_SNAPALIGNMENT_AND_SCALE: + { + // IF it doesn't fit - scale it + int max_height = s->h - 2 * s->margin_v; + + if (max_height < 2) + max_height = 2; + + if (target_h > max_height) { + target_w = (int)av_rescale(target_w, max_height, target_h); + target_h = max_height; + target_y = s->margin_v; + } + } + break; + } + + switch (s->arrange_mode_v) { + case SAM_NONE: + break; + case SAM_ENSUREMARGIN_NO_SCALE: + case SAM_ENSUREMARGIN_AND_SCALE: + // Top border + if (target_y < border_t) + target_y = border_t; + + // Bottom border + if (target_y + target_h > border_b) + target_y = border_b - target_h; + + break; + case SAM_SNAPALIGNMENT_NO_SCALE: + case SAM_SNAPALIGNMENT_AND_SCALE: + { + // Use original values to detect alignment + const int top_margin = area->y; + const int bottom_margin = inlink->h - area->y - area->h; + const AVRational diff_factor_r = { top_margin - bottom_margin, area->h }; + const float diff_factor = (float)av_q2d(diff_factor_r); + + if (diff_factor > 0.2f) { + // Bottom aligned + target_y = border_b - target_h; + } else if (diff_factor < -0.2f) { + // Top aligned + target_y = border_t; + } else { + // Centered + target_y = (inlink->h - area->h) / 2; + } + } + + break; + default: + av_log(ctx, AV_LOG_ERROR, "Invalid arrange_mode_v: %d\n", s->arrange_mode_v); + return AVERROR(EINVAL); + } + + av_log(ctx, AV_LOG_VERBOSE, "process_area - arr_v: x/y: (%d:%d) size: %dx%d.\n", target_x, target_y, target_w, target_h); + + area->x = target_x; + area->y = target_y; + + if (area->w != target_w || area->h != target_h) + return scale_area(inlink, area, target_w, target_h); + + return 0; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + const AVFilterContext *ctx = inlink->dst; + AVFilterLink *outlink = ctx->outputs[0]; + int ret; + + ret = av_frame_make_writable(frame); + if (ret >= 0) { + const AVRational x_factor = { .num = outlink->w, .den = inlink->w} ; + const AVRational y_factor = { .num = outlink->h, .den = inlink->h} ; + + for (unsigned i = 0; i < frame->num_subtitle_areas; ++i) { + AVSubtitleArea *area = frame->subtitle_areas[i]; + + ret = process_area(inlink, area, x_factor, y_factor); + if (ret < 0) + return ret; + } + + return ff_filter_frame(outlink, frame); + } + + return ret; +} + +#define OFFSET(x) offsetof(SubScaleContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_FILTERING_PARAM + +static const AVOption subscale_options[] = { + { "margin_h", "horizontal border", OFFSET(margin_h_expr), AV_OPT_TYPE_STRING, .flags = FLAGS }, + { "margin_v", "vertical border", OFFSET(margin_v_expr), AV_OPT_TYPE_STRING, .flags = FLAGS }, + { "w", "Output video width", OFFSET(w_expr), AV_OPT_TYPE_STRING, .flags = FLAGS }, + { "width", "Output video width", OFFSET(w_expr), AV_OPT_TYPE_STRING, .flags = FLAGS }, + { "h", "Output video height", OFFSET(h_expr), AV_OPT_TYPE_STRING, .flags = FLAGS }, + { "height","Output video height", OFFSET(h_expr), AV_OPT_TYPE_STRING, .flags = FLAGS }, + { "force_original_aspect_ratio", "decrease or increase w/h if necessary to keep the original AR", OFFSET(force_original_aspect_ratio), AV_OPT_TYPE_INT, { .i64 = 0}, 0, 2, FLAGS, "force_oar" }, + { "disable", NULL, 0, AV_OPT_TYPE_CONST, {.i64 = 0 }, 0, 0, FLAGS, "force_oar" }, + { "decrease", NULL, 0, AV_OPT_TYPE_CONST, {.i64 = 1 }, 0, 0, FLAGS, "force_oar" }, + { "increase", NULL, 0, AV_OPT_TYPE_CONST, {.i64 = 2 }, 0, 0, FLAGS, "force_oar" }, + { "scale_mode", "specify how to scale subtitles", OFFSET(scale_mode), AV_OPT_TYPE_INT, {.i64 = SSM_UNIFORM}, 0, SSM_UNIFORM_NO_REPOSITION, FLAGS, "scale_mode" }, + { "none", "no common scaling", 0, AV_OPT_TYPE_CONST, {.i64=SSM_NONE}, .flags = FLAGS, .unit = "scale_mode" }, + { "uniform", "uniformly scale and reposition", 0, AV_OPT_TYPE_CONST, {.i64=SSM_UNIFORM}, .flags = FLAGS, .unit = "scale_mode" }, + { "uniform_no_reposition", "uniformly scale but keep positions", 0, AV_OPT_TYPE_CONST, {.i64=SSM_UNIFORM_NO_REPOSITION}, .flags = FLAGS, .unit = "scale_mode" }, + + { "arrange_h", "specify how to arrange subtitles horizontally", OFFSET(arrange_mode_h), AV_OPT_TYPE_INT, {.i64 = SAM_NONE}, 0, SAM_SNAPALIGNMENT_AND_SCALE, FLAGS, "arrange" }, + { "arrange_v", "specify how to arrange subtitles vertically", OFFSET(arrange_mode_v), AV_OPT_TYPE_INT, {.i64 = SAM_NONE}, 0, SAM_SNAPALIGNMENT_AND_SCALE, FLAGS, "arrange" }, + { "none", "no repositioning", 0, AV_OPT_TYPE_CONST, {.i64=SAM_NONE}, .flags = FLAGS, .unit = "arrange" }, + { "margin_no_scale", "move subs inside border when possible", 0, AV_OPT_TYPE_CONST, {.i64=SAM_ENSUREMARGIN_NO_SCALE,}, .flags = FLAGS, .unit = "arrange" }, + { "margin_and_scale", "move subs inside border and scale as needed", 0, AV_OPT_TYPE_CONST, {.i64=SAM_ENSUREMARGIN_AND_SCALE,}, .flags = FLAGS, .unit = "arrange" }, + { "snapalign_no_scale", "snap subs to near/far/center when possible", 0, AV_OPT_TYPE_CONST, {.i64=SAM_SNAPALIGNMENT_NO_SCALE,}, .flags = FLAGS, .unit = "arrange" }, + { "snapalign_and_scale", "snap subs to near/far/center and scale as needed", 0, AV_OPT_TYPE_CONST, {.i64=SAM_SNAPALIGNMENT_AND_SCALE,}, .flags = FLAGS, .unit = "arrange" }, + + { "eval", "specify when to evaluate expressions", OFFSET(eval_mode), AV_OPT_TYPE_INT, {.i64 = EVAL_MODE_INIT}, 0, EVAL_MODE_NB-1, FLAGS, "eval" }, + { "init", "eval expressions once during initialization", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_INIT}, .flags = FLAGS, .unit = "eval" }, + { "frame", "eval expressions during initialization and per-frame", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_FRAME}, .flags = FLAGS, .unit = "eval" }, + { "num_colors", "number of palette colors in output", OFFSET(num_output_colors), AV_OPT_TYPE_INT, {.i64 = 256 }, 2, 256, FLAGS }, + { "bitmap_width_align", "Bitmap width alignment", OFFSET(bitmap_width_align), AV_OPT_TYPE_INT, {.i64 = 2 }, 1, 256, FLAGS }, + { "bitmap_height_align", "Bitmap height alignment", OFFSET(bitmap_height_align), AV_OPT_TYPE_INT, {.i64 = 2 }, 1, 256, FLAGS }, + { NULL } +}; + +static const AVClass subscale_class = { + .class_name = "subscale", + .item_name = av_default_item_name, + .option = subscale_options, + .version = LIBAVUTIL_VERSION_INT, + .category = AV_CLASS_CATEGORY_FILTER, +}; + +static const AVFilterPad inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .filter_frame = filter_frame, + .config_props = config_input, + }, +}; + +static const AVFilterPad outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_SUBTITLE, + .config_props = config_output, + }, +}; + +const AVFilter ff_sf_subscale = { + .name = "subscale", + .description = NULL_IF_CONFIG_SMALL("Scale graphical subtitles."), + .init_dict = init_dict, + .uninit = uninit, + .query_formats = query_formats, + .priv_size = sizeof(SubScaleContext), + .priv_class = &subscale_class, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), +}; + From patchwork Thu Sep 30 19:31:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30803 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp986952iob; Thu, 30 Sep 2021 12:33:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwKHE1crqCq9hUtCI40Yd9AaMTQml08mL0kdcFqjbFpCXuGqR6kTpFuBHo8fWxujRgWjdrs X-Received: by 2002:a05:6402:1437:: with SMTP id c23mr9327714edx.247.1633030397735; Thu, 30 Sep 2021 12:33:17 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ga1si5217316ejc.128.2021.09.30.12.33.17; Thu, 30 Sep 2021 12:33:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=dktopkCG; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1642E68AC09; Thu, 30 Sep 2021 22:31:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12olkn2093.outbound.protection.outlook.com [40.92.22.93]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B87DC68ABAC for ; Thu, 30 Sep 2021 22:31:34 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DnBmaXaGLVJ5dqpujv8+E5vpVs4ZQ7p0FZdFXJbCs1/Z4DHI+d6BgOfDK8QlJ2Jw/Msr8oLvPh8Braiv4nGRKbxDFjkyaLah+nLTjgPOdj3fBPQfP4QRhpHg+sKfIePeNWb2hmMqjMEuTGET2BcvlWvyifvKhxYXfGft3zMEK2cNmvQLCQ4CtOe2lhptRH7fjpFJaK6uUhp+8ntC1DPPGDg6hknRHeIz3YRHqr1RoUvb8L14iefFbnDtNZjd5NGYmw5mptZ3vhJfzkOKlJa5OEMwoi0KYj86NhVaMyDyFAGnXPWXqSt+fA5vGYq26P6xEr1eVYrP/Y1HrV7OLj35OA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=FoDF7eD3FIWRUF0QfzF/XIxFf/MjeAUmQ2HOkoPVYmk=; b=JeTJmtx9Kl6WYOEhtxHWUIILrlUgW6D+RmiGJGdj7MgO9fH5dGFrluIZBnBQr7TyTBf1zZv7SzLFP1sVCp8sKLdXN+3ySIrFsrz35GIey10DvFx6cvXAyrdRqhfv6FXVxIKz3Y+WWYAUJ4Dy63fZpPqslTsHsKqqkCzMMZuDvimFC8Aughbylw8PcVrfBWlzI8KuP1dhVS0iGjeENSstlU4OzkXWlEEWzTQPt5xghwfemscjAg+kJ2gUZjiaclKNo7mmGbks+emGU/Vf4lRmMugsFQrlI1nuM60op0GTZ2zhUsoMGbDvOeCG5Vi1khcyRWfsEADhyA7KNcLEeBHCzw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FoDF7eD3FIWRUF0QfzF/XIxFf/MjeAUmQ2HOkoPVYmk=; b=dktopkCGcx78dBFaT5BHkpA9a2gShVIVv7nbYodDgBfPhJFpmc9ReZ0ViK2ZGrm+9eVkDwa3gu9nnuOyYhYaRSEzbc0MDOK2Pd4oWCQL+3UoGfTWuz8rHE/Ot/hczh0ETzIfZiH8+GMGRcB2m3NDMl85MLBP6JAIO0IHkikd5ETpRJoVIwVwynjuVt2gahQ5Ml+yacAz9kmaU+2NnN6+GPyvJLNt5cqSQo/ai4iEOpTz+f9u7cHANYIj4luN6Pg4nzj9/td9pqk+P2jJEXrPA9YAqpzVRp/joaOZeWtWtIoULrPvNrYnVBKTdYKa1UowTF+AmpFerXxG/T2QklZkgQ== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5853.namprd04.prod.outlook.com (2603:10b6:208:a6::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:31:32 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:31:32 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 13/14] avformat/utils: Add av_stream_get_codec_properties() Thread-Index: AQHXtjHFGFSRlMimRk+34LfYYvHSKw== Date: Thu, 30 Sep 2021 19:31:32 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> <8a75051fd7be111f4ec52073a6da6679724a7127.1633027275.git.softworkz@hotmail.com> <48414b2bba60c4be4c41a1bd3ba8edae069f1fc9.1633027275.git.softworkz@hotmail.com> <94fd41cac821f78e13de073ea8e69d37a3bc0b33.1633027275.git.softworkz@hotmail.com> <1c07cefd3cf9ff5aeced727c0005957d0a7f833d.1633027275.git.softworkz@hotmail.com> In-Reply-To: <1c07cefd3cf9ff5aeced727c0005957d0a7f833d.1633027275.git.softworkz@hotmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [YPriBdVWGiI4Uz/0rOAOg5durtaIEXjA] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a0070dd8-c153-46b6-45de-08d98448e871 x-ms-traffictypediagnostic: MN2PR04MB5853: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: yl2i3KaRDSK/DaVM82S+h6c0bjk9XNLszM4lUDYmwXfh5uAjWd0ocujV7F5fsl1SPpogjD3hg7HhD2roLZ9Fyc1UNrjOeKr2fzoevl1Z59DCsRT6JHH32xoERmstUpRr8zKWciKo7pyBWkMsz0ihCvRIDw/7Yxq5QQbBPdaZfDCEPJc8F2IlQcAc4nkm309wTRIQAmSnOKYp+TVLYYdoQpFeD9PPvt+b9MglcfpMMZRe4WhA3y2rd8D5IOorSLzgBrOTT2yQhLV7p7SEophpdVAVsrmptc4fu8N4DWl1jbYDragr0a/ogPuWwYb9qh2wTZ/V8R6ZBaSCGL5hIYFJ19Te5gWCTcZOMo++7Cfmgtpsa1/C8Y1RxO3tmOaMNQQirWnMPJWPDAnP/69H2eWvriwPGXl2c4ne8IuFZ1RGaeF8hxByG0EmgSPlW9+KelIf x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: aEH89KrzJDI8U/Emxl/lUJ4ABhqRGuC0EPmdCf3Z7JmbKTgjPsN3q3wyPegZK9xUl8W3ZPjUWjq1v9TGqmi76MtELxhzGJpV6PQNGw8x+5+Ay6DkS7oF9IRvwmIG1yvkBYb3k/MOA8Rz+BqtNtPQ3Q== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: a0070dd8-c153-46b6-45de-08d98448e871 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:31:32.1325 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5853 Subject: [FFmpeg-devel] [PATCH v10 13/14] avformat/utils: Add av_stream_get_codec_properties() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 1yti0C0uonxS Signed-off-by: softworkz --- doc/APIchanges | 3 +++ libavformat/avformat.h | 9 +++++++++ libavformat/utils.c | 5 +++++ libavformat/version.h | 2 +- 4 files changed, 18 insertions(+), 1 deletion(-) diff --git a/doc/APIchanges b/doc/APIchanges index 7b267a79ac..20848fe93f 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -14,6 +14,9 @@ libavutil: 2021-04-27 API changes, most recent first: +2021-04-27 - xxxxxxxxxx - lavf 59.6.100 - avformat.h + Add av_stream_get_codec_properties() + 2021-09-21 - xxxxxxxxxx - lavu 57.7.100 - pixfmt.h Add AV_PIX_FMT_X2BGR10. diff --git a/libavformat/avformat.h b/libavformat/avformat.h index 3a5bc8a06d..89ed984d1d 100644 --- a/libavformat/avformat.h +++ b/libavformat/avformat.h @@ -2763,6 +2763,15 @@ int avformat_transfer_internal_stream_timing_info(const AVOutputFormat *ofmt, */ AVRational av_stream_get_codec_timebase(const AVStream *st); +/** + * Get the internal codec properties from a stream. + * + * See @ref AVCodecContext.properties. + * + * @param st input stream to extract the timebase from + */ +unsigned av_stream_get_codec_properties(const AVStream *st); + /** * @} */ diff --git a/libavformat/utils.c b/libavformat/utils.c index f1ec4c4c2f..64b90e785d 100644 --- a/libavformat/utils.c +++ b/libavformat/utils.c @@ -1945,6 +1945,11 @@ AVRational av_stream_get_codec_timebase(const AVStream *st) return cffstream(st)->avctx->time_base; } +unsigned av_stream_get_codec_properties(const AVStream *st) +{ + return cffstream(st)->avctx->properties; +} + void ff_format_set_url(AVFormatContext *s, char *url) { av_assert0(url); diff --git a/libavformat/version.h b/libavformat/version.h index 13df244d97..d5dd22059b 100644 --- a/libavformat/version.h +++ b/libavformat/version.h @@ -32,7 +32,7 @@ // Major bumping may affect Ticket5467, 5421, 5451(compatibility with Chromium) // Also please add any ticket numbers that you believe might be affected here #define LIBAVFORMAT_VERSION_MAJOR 59 -#define LIBAVFORMAT_VERSION_MINOR 5 +#define LIBAVFORMAT_VERSION_MINOR 6 #define LIBAVFORMAT_VERSION_MICRO 100 #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \ From patchwork Thu Sep 30 19:31:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30800 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp987096iob; Thu, 30 Sep 2021 12:33:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwlfIxF1E/E7GrBGnAaXqHyMQ3GsO1GrpWOhCRfosp+DRtgjCRGxhEgKVcSnX1MgPPx65Uv X-Received: by 2002:a05:6402:484:: with SMTP id k4mr9257035edv.303.1633030408047; Thu, 30 Sep 2021 12:33:28 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s4si4320046ejm.735.2021.09.30.12.33.27; Thu, 30 Sep 2021 12:33:28 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=OajMpXSb; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D705468AC25; Thu, 30 Sep 2021 22:31:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12olkn2093.outbound.protection.outlook.com [40.92.22.93]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2718168ABAC for ; Thu, 30 Sep 2021 22:31:35 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ja4XZ+QXZDpOHlC4MArdSmFMCU6YMEtjIHm8W8d213KxcHHWgJ4dE/9yGRsutDB2mwTMlvoVTC31tx/eCLruJxAO9WrdOdDU+tXqPuvWyEunF7ut2ThFIkimysNc1dNJ8jtPgdo58j72hiyJm3+K5aFDA1Ei/36iCn4i4LuW1yFOxyNf0H+JQlE9VomYMqXNuLU/FKBYpKWsY+kbaBmrALYH1qFaCHY/QflH2z0vaeP9T78Vy8YIHufLLFQtvaG/bAw/yYC1wzE2ziwTq8xk588RSD7xadiI1oOB/ll2sTV6xNRyTlD7fZWU1Kmboe7mTA4uDH7m6xhsnHZDqLosuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=BLZW2r/rWa1at/oFipOlkclKfJmFFwcaz9L/9mn7J7o=; b=TByNdqCc8yCdqax94ifGiJpCJYG4PQLpDJr8pAIocTjcgxwKQANrtVcpzz9ZNhuUJ9g8KII//vfYfs88rL9TUIv1Pxc4SKj+R3AcZN591DFCBRD1VugayJxdKAaGBDe1XBXl6WMjMQywJqaUO4KX9WOTVR2ZY10yZjcg+IIywJuR22lsRosTuvyRohxBsx1SrBmWZEyzGmnGUn9LfC/itzUQqhstNqxHAkh4FtBz46VpzferemaOBmygIFFFTNkAgmuzhBzxvE0A1Va9y2keHk6f5C4HTt95clOd08pQ3cEauk/6Z8yJeqxcLmrTGmhnVQ5o/8rdZEUKfyaNXDHACA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BLZW2r/rWa1at/oFipOlkclKfJmFFwcaz9L/9mn7J7o=; b=OajMpXSbq4BmhIiUj1c6dIGqeF59wxjF917imIbH7iiNefc0MPxSReT8pYJ7WBbI2MJeC4/e7HQt3YJ7qxhCNQVHH3o7CO8VOJl4glmvRWC/CExQCRJZsZnLeN46h+QfYqJpT8BnK1e1siBR/SJKdzBJJMcEU4EyKR35g+u+SMbFjBHnV2U+YJPy8+malHT5iBEuARyVUlUW4R/3QLDW979nb8qT/boicHcMnZkE7RxlpdOkcGHDiXE54wW8v0jx0zb27DZaMiTnRQbpcKuFERA/g1pcjM9+KwQpVKB0B+RQXQ4tOD+JJL+3XvJTOUqNRXYO8/fPKueTyWVMSHhqDg== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB5853.namprd04.prod.outlook.com (2603:10b6:208:a6::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Thu, 30 Sep 2021 19:31:33 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4544.023; Thu, 30 Sep 2021 19:31:33 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v10 14/14] ffprobe: Fix incorrect display of closed_captions property Thread-Index: AQHXtjHGDsoUwFPFSEutXUKNPHa+gQ== Date: Thu, 30 Sep 2021 19:31:33 +0000 Message-ID: References: <48e5eb4d4c53473e2c34dd12061f2faa67522a89.1633027274.git.softworkz@hotmail.com> <8a75051fd7be111f4ec52073a6da6679724a7127.1633027275.git.softworkz@hotmail.com> <48414b2bba60c4be4c41a1bd3ba8edae069f1fc9.1633027275.git.softworkz@hotmail.com> <94fd41cac821f78e13de073ea8e69d37a3bc0b33.1633027275.git.softworkz@hotmail.com> <1c07cefd3cf9ff5aeced727c0005957d0a7f833d.1633027275.git.softworkz@hotmail.com> <4105f813c06a87b2ed60355177669b54f3e30773.1633027275.git.softworkz@hotmail.com> In-Reply-To: <4105f813c06a87b2ed60355177669b54f3e30773.1633027275.git.softworkz@hotmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [1NtCooi8XgxrAL6PJXl6UH7OFITj0ekF] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 196c75e5-f572-4aba-f293-08d98448e93b x-ms-traffictypediagnostic: MN2PR04MB5853: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: cz1tqPlnf4qb4evXnno1p7CYK0HipEqw/cmw3t0D/gnZRqIdL27gl8E0V9qzfquKMHTc8GkqDLWL7z93jqC2kIG4sjd6GgmtMa4H+JaDJjiNwdfLm8dzpXhy5Rpp2LN8U7qzx2oK5psI7vvcdjrjSOHn+samCVTV+POys6I0p1xiTfpUFLsz9pMS1Lp6j3feaNKBtvIgo/MJYXQ/StRp2O8111Rc2+KrUa9gcsvl/187mopX9c6bvj9KRTbOy58G5NdQ7C2t1/V++HNBkVJoljiS+RJZI2ukjkrS5s06Ei6nmz96vYSiFLT3HNTTzPmJWMGHvyLgQGcgmN72dX1+zLGs1Epd+wi9ybpep93dbhvvfzGemkR/4y728YYLlfzC20qAFzxN2zvgHKRbADZu186JtDrIHfaJjjM0yxsd3I4T4/vc8HsljKHI9MwKhWpCC4/dwRzYTKfZCHJtZFRAMCggYieSla5oVRSkMiIcV40= x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: 5Kt8nDtTrpUQsB/K+WxZJ6oa73+EhvJNMJ85otHi5wUnrifAPV+R6jbahQN/I56bneAbmmd8E4S48mWkYnFMmHpLTxdoDEgLFR5AXA1aMGEgYTe35xfqtx8U7zM0NpR+HA1usx5rAx1LCoeegs+RSg== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 196c75e5-f572-4aba-f293-08d98448e93b X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 19:31:33.4158 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB5853 Subject: [FFmpeg-devel] [PATCH v10 14/14] ffprobe: Fix incorrect display of closed_captions property X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 2dS/LsVxzCZt Repro Example: ffprobe -show_entries stream=closed_captions:disposition=:side_data= "http://streams.videolan.org/streams/ts/CC/NewsStream-608-ac3.ts" While the codec string includes "Closed Captions", the stream data is showing: closed_captions=0 The test ref was incorrect as the test media file actually does have cc. Signed-off-by: softworkz --- fftools/ffprobe.c | 5 +++-- tests/ref/fate/ts-demux | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) stream|index=1|codec_name=ac3|profile=unknown|codec_type=audio|codec_tag_string=[4][0][0][0]|codec_tag=0x0004|sample_fmt=fltp|sample_rate=48000|channels=6|channel_layout=5.1(side)|bits_per_sample=0|id=0x34|r_frame_rate=0/0|avg_frame_rate=0/0|time_base=1/90000|start_pts=3912633305|start_time=43473.703389|duration_ts=14400|duration=0.160000|bit_rate=384000|max_bit_rate=N/A|bits_per_raw_sample=N/A|nb_frames=N/A|nb_read_frames=N/A|nb_read_packets=5|disposition:default=0|disposition:dub=0|disposition:original=0|disposition:comment=0|disposition:lyrics=0|disposition:karaoke=0|disposition:forced=0|disposition:hearing_impaired=0|disposition:visual_impaired=0|disposition:clean_effects=0|disposition:attached_pic=0|disposition:timed_thumbnails=0|disposition:captions=0|disposition:descriptions=0|disposition:metadata=0|disposition:dependent=0|disposition:still_image=0|tag:language=eng stream|index=2|codec_name=ac3|profile=unknown|codec_type=audio|codec_tag_string=[4][0][0][0]|codec_tag=0x0004|sample_fmt=fltp|sample_rate=48000|channels=2|channel_layout=stereo|bits_per_sample=0|id=0x35|r_frame_rate=0/0|avg_frame_rate=0/0|time_base=1/90000|start_pts=3912634060|start_time=43473.711778|duration_ts=14400|duration=0.160000|bit_rate=192000|max_bit_rate=N/A|bits_per_raw_sample=N/A|nb_frames=N/A|nb_read_frames=N/A|nb_read_packets=5|disposition:default=0|disposition:dub=0|disposition:original=0|disposition:comment=0|disposition:lyrics=0|disposition:karaoke=0|disposition:forced=0|disposition:hearing_impaired=0|disposition:visual_impaired=0|disposition:clean_effects=0|disposition:attached_pic=0|disposition:timed_thumbnails=0|disposition:captions=0|disposition:descriptions=0|disposition:metadata=0|disposition:dependent=0|disposition:still_image=0|tag:language=es diff --git a/fftools/ffprobe.c b/fftools/ffprobe.c index 4e08807706..201f02bee1 100644 --- a/fftools/ffprobe.c +++ b/fftools/ffprobe.c @@ -2669,10 +2669,11 @@ static int show_stream(WriterContext *w, AVFormatContext *fmt_ctx, int stream_id print_int("width", par->width); print_int("height", par->height); if (dec_ctx) { + unsigned codec_properties = av_stream_get_codec_properties(ist->st); print_int("coded_width", dec_ctx->coded_width); print_int("coded_height", dec_ctx->coded_height); - print_int("closed_captions", !!(dec_ctx->properties & FF_CODEC_PROPERTY_CLOSED_CAPTIONS)); - print_int("film_grain", !!(dec_ctx->properties & FF_CODEC_PROPERTY_FILM_GRAIN)); + print_int("closed_captions", !!(codec_properties & FF_CODEC_PROPERTY_CLOSED_CAPTIONS)); + print_int("film_grain", !!(codec_properties & FF_CODEC_PROPERTY_FILM_GRAIN)); } print_int("has_b_frames", par->video_delay); sar = av_guess_sample_aspect_ratio(fmt_ctx, stream, NULL); diff --git a/tests/ref/fate/ts-demux b/tests/ref/fate/ts-demux index 8e7a81da41..1d1382cf37 100644 --- a/tests/ref/fate/ts-demux +++ b/tests/ref/fate/ts-demux @@ -41,7 +41,7 @@ packet|codec_type=audio|stream_index=2|pts=3912642700|pts_time=43473.807778|dts= packet|codec_type=video|stream_index=0|pts=3912686363|pts_time=43474.292922|dts=3912686363|dts_time=43474.292922|duration=1501|duration_time=0.016678|size=4944|pos=506660|flags=__|data_hash=CRC32:54a86cbb packet|codec_type=audio|stream_index=1|pts=3912644825|pts_time=43473.831389|dts=3912644825|dts_time=43473.831389|duration=2880|duration_time=0.032000|size=906|pos=474888|flags=K_|data_hash=CRC32:0893d398 packet|codec_type=audio|stream_index=2|pts=3912645580|pts_time=43473.839778|dts=3912645580|dts_time=43473.839778|duration=2880|duration_time=0.032000|size=354|pos=491808|flags=K_|data_hash=CRC32:f5963fa6 -stream|index=0|codec_name=mpeg2video|profile=4|codec_type=video|codec_tag_string=[2][0][0][0]|codec_tag=0x0002|width=1280|height=720|coded_width=0|coded_height=0|closed_captions=0|film_grain=0|has_b_frames=1|sample_aspect_ratio=1:1|display_aspect_ratio=16:9|pix_fmt=yuv420p|level=4|color_range=tv|color_space=unknown|color_transfer=unknown|color_primaries=unknown|chroma_location=left|field_order=progressive|refs=1|id=0x31|r_frame_rate=60000/1001|avg_frame_rate=60000/1001|time_base=1/90000|start_pts=3912669846|start_time=43474.109400|duration_ts=19519|duration=0.216878|bit_rate=15000000|max_bit_rate=N/A|bits_per_raw_sample=N/A|nb_frames=N/A|nb_read_frames=N/A|nb_read_packets=15|extradata_hash=CRC32:53134fa8|disposition:default=0|disposition:dub=0|disposition:original=0|disposition:comment=0|disposition:lyrics=0|disposition:karaoke=0|disposition:forced=0|disposition:hearing_impaired=0|disposition:visual_impaired=0|disposition:clean_effects=0|disposition:attached_pic=0|disposition:timed_ thumbnails=0|disposition:captions=0|disposition:descriptions=0|disposition:metadata=0|disposition:dependent=0|disposition:still_image=0 +stream|index=0|codec_name=mpeg2video|profile=4|codec_type=video|codec_tag_string=[2][0][0][0]|codec_tag=0x0002|width=1280|height=720|coded_width=0|coded_height=0|closed_captions=1|film_grain=0|has_b_frames=1|sample_aspect_ratio=1:1|display_aspect_ratio=16:9|pix_fmt=yuv420p|level=4|color_range=tv|color_space=unknown|color_transfer=unknown|color_primaries=unknown|chroma_location=left|field_order=progressive|refs=1|id=0x31|r_frame_rate=60000/1001|avg_frame_rate=60000/1001|time_base=1/90000|start_pts=3912669846|start_time=43474.109400|duration_ts=19519|duration=0.216878|bit_rate=15000000|max_bit_rate=N/A|bits_per_raw_sample=N/A|nb_frames=N/A|nb_read_frames=N/A|nb_read_packets=15|extradata_hash=CRC32:53134fa8|disposition:default=0|disposition:dub=0|disposition:original=0|disposition:comment=0|disposition:lyrics=0|disposition:karaoke=0|disposition:forced=0|disposition:hearing_impaired=0|disposition:visual_impaired=0|disposition:clean_effects=0|disposition:attached_pic=0|disposition:timed_ thumbnails=0|disposition:captions=0|disposition:descriptions=0|disposition:metadata=0|disposition:dependent=0|disposition:still_image=0 side_data|side_data_type=CPB properties|max_bitrate=15000000|min_bitrate=0|avg_bitrate=0|buffer_size=9781248|vbv_delay=-1