From patchwork Tue May 3 16:13:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Traian Coza X-Patchwork-Id: 35566 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:a885:b0:7f:4be2:bd17 with SMTP id ca5csp496126pzb; Tue, 3 May 2022 09:14:00 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyr9V0A86jo8UZm7w/09ZKuBmGdBoUN77pJy1fxLABOP3PlIkyXpXeTwHUNRi2ir5is49HC X-Received: by 2002:a17:907:6090:b0:6e8:abda:8933 with SMTP id ht16-20020a170907609000b006e8abda8933mr16616350ejc.46.1651594439796; Tue, 03 May 2022 09:13:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1651594439; cv=none; d=google.com; s=arc-20160816; b=EwJlOfGyD3PgHW6O3jtYvQwGqm0AO9de5stmtGb1b+K+f9pbtDMnNz7OEuhKkwV+aN 4XpYp2WIOsjlktnTOQU30sESaXoCn9PSiFeVU3Cy7ByfLyWRbQHxxgcdHTWIF0nwURw5 AoeyfJPqTJWh8C61MUG8klileGRn9GYq+Ac6AFs5jJ09pS1vyWe72ODfBvwZkgxdJN0S KYGFbkDz7yGeANba7jzbMt0P25yLdvjwO8+/Dc1ubhPb+GfntOE9sgLX+RDby2N2zb+r zz8aGHm+9zSqFmwoVAmcKqJgCyWXehUzJJUWfnYB8yIE7Q10UL6BzLz5lTegL/vIyj+i OlVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=PGjlCPgi8e7Vc5sYDPVjUMhNUptyUKqKuJEypaensBQ=; b=r/WJsI4gzYDL4J4l2VBk8odi6wGxf/8qA89QR89lKjtYzU6x4+UJU9joT5vSK63QTq jzWnrBvmNNO/9gRytjz3zJ6Y+KpKDNS5E1MZtbe1xVoHzb9Kgt4N1tTW3BZ1mszcWYjT YPJqrwOEKzwwqVsQrdpZlazx5r6F1oa+YyJSR6OccUNdQbCAk02xE3IihpL3UlwJkgRZ tP3PdrzLkEw6SM04ZfCOfkA81MZOZYS7eJPu2E71a23cEzjk3Ku3AjmfF2Fw0/sYPymy q1F3BiE2liixJIc2IjlqrxNXdGqyofrwTRj7xMNAvr2UiWYBSjJ9CCuBQjFZ8Td4+KBA 04MA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=Jzq1buNb; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id du8-20020a17090772c800b006f45fa75d7esi4411391ejc.894.2022.05.03.09.13.59; Tue, 03 May 2022 09:13:59 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=Jzq1buNb; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EAE7D68B379; Tue, 3 May 2022 19:13:55 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 91DFE680725 for ; Tue, 3 May 2022 19:13:48 +0300 (EEST) Received: by mail-qk1-f176.google.com with SMTP id s4so14144421qkh.0 for ; Tue, 03 May 2022 09:13:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HrxyjLR/H2MyzNj8ERRPZO+zGKPI9lN7b2Pbm4gX3Yw=; b=Jzq1buNbsBOU3ue1vBep7OgOqUxF2l74WZ02NPCxBEBbajFwUScpOh4KMLWFk8YDjI jC0JtqfTPcEQVrZiC0N8CIhSC3dAU1K8+QVkeqeKiONV00Xztm7hm57gAO3PjxmUiQ86 vsLkdbsiy2ZCYczhWghK6cP3pdHkCBTCnsOjow+woNm9Lyfio9nUSzY2vUU3PyrVbhjO KkWzu0TsheLBijanaLMbCyPPbggHC8DAhkWq5DJdVVncF8edAWp2cGto1T5vIuPorUxs gnx5caYaVgTpaXQRCHXydqOA9qGWy7s32JGfW8gxYP5uuRORmVwLk+2sOatq8QKN+K8N Y3mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HrxyjLR/H2MyzNj8ERRPZO+zGKPI9lN7b2Pbm4gX3Yw=; b=Vykq5hisHLpnxKc3NyELyB4jlSn6bO0AuCmlP+CiDz27utc/jHgXrTLR9vvU9DPZC/ jk9dwkMpj9S7w8hy9b7vuOVtdb5aae4/XuDe5hjl0AFCkfrlAuN8siUqQmQ7sxdePPkU Vevd2DOXaYYHyJqeTT/rLJcJ+GoT1F6lti6eacPiLfv3PCFwOx85nmU+vdxTVvhhk1JH UoUYrlwlNr5cITVe//aCrMDmOyUpZ2aEqYzRNXyb+c++i3Ui/reBmOsuOy7zU5ahcH4a KbIdDMxIcOMP1aAlF/KdM4K+3uHh6yo9AinLpl6ZYzZabgcbHBahxyMQft4hUdjzJU81 HMgA== X-Gm-Message-State: AOAM533zfojFhENNpKyT1wc63RDwYU/d0vnCpwCURvbz9DfwIJgJDRmz dnQzkwQivM5CAJr51MTzAKawOQx05tfS X-Received: by 2002:a37:b147:0:b0:69f:9b0c:2cb1 with SMTP id a68-20020a37b147000000b0069f9b0c2cb1mr12803688qkf.546.1651594426894; Tue, 03 May 2022 09:13:46 -0700 (PDT) Received: from thinkpad.. ([66.220.203.136]) by smtp.gmail.com with ESMTPSA id fp35-20020a05622a50a300b002f39b99f67asm5812731qtb.20.2022.05.03.09.13.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 May 2022 09:13:46 -0700 (PDT) From: Traian Coza To: ffmpeg-devel@ffmpeg.org Date: Tue, 3 May 2022 12:13:17 -0400 Message-Id: <20220503161328.842587-2-traian.coza@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220503161328.842587-1-traian.coza@gmail.com> References: <20220503161328.842587-1-traian.coza@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 01/12] Implemented text to bitmap subtitles! X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Traian Coza Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Vy5T3w+vhV4q --- fftools/ffmpeg.c | 47 ++++++++++++++- libavfilter/vf_subtitles.c | 117 +++++++++++++++++++++++++++++++++++++ 2 files changed, 163 insertions(+), 1 deletion(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index a85ed18b08..53717d3ebb 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -2325,6 +2325,45 @@ fail: return err < 0 ? err : ret; } +void render_avsub_ass(InputStream *, AVSubtitle *); + +static void print_subtitle(AVSubtitle sub) +{ + printf("sub.format: %u\n", sub.format); + printf("sub.start_display_time: %u\n", sub.start_display_time); + printf("sub.end_display_time: %u\n", sub.end_display_time); + printf("sub.num_rects: %u\n", sub.num_rects); + printf("sub.pts: %ld\n", sub.pts); + for (int i = 0; i < sub.num_rects; i++) + { + printf("sub.rects[%d]->type: %d\n", i, sub.rects[i]->type); + printf("sub.rects[%d]->nb_colors: %d\n", i, sub.rects[i]->nb_colors); + printf("sub.rects[%d]->(x,y,w,h): (%d,%d,%d,%d)\n", i, sub.rects[i]->x, sub.rects[i]->y, sub.rects[i]->w, sub.rects[i]->h); + printf("sub.rects[%d]->linesize: [%d,%d,%d,%d]\n", i, sub.rects[i]->linesize[0], sub.rects[i]->linesize[1], sub.rects[i]->linesize[1], sub.rects[i]->linesize[1]); + switch (sub.rects[i]->type) + { + case SUBTITLE_TEXT: + printf("sub.rects[%d]->text: %s\n", i, sub.rects[i]->text); + break; + case SUBTITLE_ASS: + printf("sub.rects[%d]->ass: %s\n", i, sub.rects[i]->ass); + break; + case SUBTITLE_BITMAP: + for (int c = 0; c < sub.rects[i]->nb_colors; c++) + printf("color %d: [%u,%u,%u,%u]\n", c, + sub.rects[i]->data[1][c * 4 + 0], + sub.rects[i]->data[1][c * 4 + 1], + sub.rects[i]->data[1][c * 4 + 2], + sub.rects[i]->data[1][c * 4 + 3]); + for (int y = 0; y < sub.rects[i]->h; y++, printf("\n")) + for (int x = 0; x < sub.rects[i]->w; x++) + printf("%d", sub.rects[i]->data[0][y * sub.rects[i]->w + x]); + break; + } + } + printf("\n"); +} + static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, int *decode_failed) { @@ -2383,6 +2422,8 @@ static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, goto out; ist->frames_decoded++; + + render_avsub_ass(ist, &subtitle); for (i = 0; i < nb_output_streams; i++) { OutputStream *ost = output_streams[i]; @@ -3213,11 +3254,15 @@ static int init_output_stream(OutputStream *ost, AVFrame *frame, input_props = input_descriptor->props & (AV_CODEC_PROP_TEXT_SUB | AV_CODEC_PROP_BITMAP_SUB); if (output_descriptor) output_props = output_descriptor->props & (AV_CODEC_PROP_TEXT_SUB | AV_CODEC_PROP_BITMAP_SUB); - if (input_props && output_props && input_props != output_props) { + /*if (input_props && output_props && input_props != output_props) { snprintf(error, error_len, "Subtitle encoding currently only possible from text to text " "or bitmap to bitmap"); return AVERROR_INVALIDDATA; + }*/ + if (input_props == AV_CODEC_PROP_BITMAP_SUB && output_props == AV_CODEC_PROP_TEXT_SUB) { + snprintf(error, error_len, "Subtitle encoding from bitmap to text currently not possible"); + return AVERROR_INVALIDDATA; } } diff --git a/libavfilter/vf_subtitles.c b/libavfilter/vf_subtitles.c index 82e140e986..f751786033 100644 --- a/libavfilter/vf_subtitles.c +++ b/libavfilter/vf_subtitles.c @@ -181,6 +181,24 @@ static void overlay_ass_image(AssContext *ass, AVFrame *picref, } } +static void print_ass_image(const ASS_Image *image) +{ + int index = 0; + for (; image != NULL; image = image->next, index++) + { + printf("index: %d\n", index); + printf("image->(dst_x,dst_y): (%d,%d)\n", image->dst_x, image->dst_y); + printf("image->(w,h): (%d,%d)\n", image->w, image->h); + printf("image->stride: %d\n", image->stride); + printf("image->type: %d\n", image->type); + printf("image->color: [%u,%u,%u,%u]\n", AR(image->color), AG(image->color), AB(image->color), AA(image->color)); + for (int y = 0; y < image->h; y++, printf("\n")) + for (int x = 0; x < image->w; x++) + printf("%02X", image->bitmap[y * image->stride + x]); + printf("\n"); + } +} + static int filter_frame(AVFilterLink *inlink, AVFrame *picref) { AVFilterContext *ctx = inlink->dst; @@ -199,6 +217,105 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *picref) return ff_filter_frame(outlink, picref); } +#include "../fftools/ffmpeg.h" +#include "libavcodec/avcodec.h" +#include "../libavcodec/ass_split.h" + +#define ALPHA_THRESHOLD 0b10000000 + +void render_avsub_ass(InputStream *, AVSubtitle *); +void render_avsub_ass(InputStream *ist, AVSubtitle *sub) +{ + ASS_Library *library = ass_library_init(); + ass_set_extract_fonts(library, 1); + ASS_Renderer *renderer = ass_renderer_init(library); + int w = 1920, h = 1080; + ass_set_frame_size(renderer, w, h); + ass_set_pixel_aspect(renderer, 1); + ass_set_storage_size(renderer, w, h); + ass_set_shaper(renderer, 0); + ass_set_fonts(renderer, NULL, NULL, 1, NULL, 1); + + for (int r = 0; r < sub->num_rects; r++) + { + AVSubtitleRect *rect = sub->rects[r]; + if (rect->data[0]) continue; + + ASSSplitContext *ass_context = ff_ass_split((char *)ist->dec_ctx->subtitle_header); + ASSDialog *dialog = ff_ass_split_dialog(ass_context, rect->ass); + + ASS_Track *track = ass_read_memory(library, (char *)ist->dec_ctx->subtitle_header, ist->dec_ctx->subtitle_header_size, NULL); + track->n_events = track->max_events = 1; + track->events = (ASS_Event *)malloc(sizeof(ASS_Event)); + track->events[0].Start = sub->start_display_time + sub->pts / (AV_TIME_BASE / 1000); + track->events[0].Duration = sub->end_display_time - sub->start_display_time; + track->events[0].Effect = strdup(dialog->effect); + track->events[0].Layer = dialog->layer; + track->events[0].MarginL = dialog->margin_l; + track->events[0].MarginR = dialog->margin_r; + track->events[0].MarginV = dialog->margin_v; + track->events[0].Name = strdup(dialog->name); + track->events[0].Text = strdup(dialog->text); + track->events[0].ReadOrder = dialog->readorder; + track->events[0].Style = 0; + for (int style = 0; style < track->n_styles; style++) + if (!strcmp(track->styles[style].Name, dialog->style)) + track->events[0].Style = style; + track->events[0].render_priv = NULL; + ff_ass_free_dialog(&dialog); + ff_ass_split_free(ass_context); + + int change; + ASS_Image *image = ass_render_frame(renderer, track, track->events[0].Start + 1, &change); // Don't have to free it for some reason + printf("image: %p\n", image); + + rect->x = image->dst_x; rect->w = 0; + rect->y = image->dst_y; rect->h = 0; + rect->nb_colors = 1; // Transparent background counts as a color + for (ASS_Image *img = image; img != NULL; img = img->next) + { + // Set image bounds to encompass all images + if (img->dst_x < rect->x) rect->x = img->dst_x; + if (img->dst_y < rect->y) rect->y = img->dst_y; + if (img->dst_x + img->w > rect->x + rect->w) + rect->w = img->dst_x + img->w - rect->x; + if (img->dst_y + img->h > rect->y + rect->h) + rect->h = img->dst_y + img->h - rect->y; + rect->nb_colors++; + } + rect->linesize[0] = rect->w; + rect->data[0] = (uint8_t *)malloc(rect->w * rect->h * sizeof(uint8_t)); + rect->data[1] = (uint8_t *)malloc(4 * rect->nb_colors * sizeof(uint8_t)); + memset(rect->data[0], 0, rect->w * rect->h); // Set all to transparent + memset(rect->data[1], 0, 4); // Set transparent color + memset(&rect->linesize[1], 0, 3 * sizeof(int)); + rect->data[2] = rect->data[3] = NULL; + for (int color = 1; image != NULL; image = image->next, color++) + { + // Set color + rect->data[1][4 * color + 0] = AR(image->color); + rect->data[1][4 * color + 1] = AG(image->color); + rect->data[1][4 * color + 2] = AB(image->color); + rect->data[1][4 * color + 3] = AA(image->color); + // Set pixels + for (int y = 0; y < image->h; y++) + for (int x = 0; x < image->w; x++) + if (image->bitmap[y * image->stride + x] >= ALPHA_THRESHOLD) + { + int x_rect = image->dst_x + x - rect->x; + int y_rect = image->dst_y + y - rect->y; + rect->data[0][y_rect * rect->w + x_rect] = color; + } + } + rect->type = SUBTITLE_BITMAP; + + ass_free_track(track); + } + + ass_renderer_done(renderer); + ass_library_done(library); +} + static const AVFilterPad ass_inputs[] = { { .name = "default",