From patchwork Tue Dec 21 07:56:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 32776 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp5182049iog; Mon, 20 Dec 2021 23:57:17 -0800 (PST) X-Google-Smtp-Source: ABdhPJyM7XgMSKAQYrkBruPHxSHI1YZc3VdWwIRmvzXOBgR86dGB6slmK8KyVGK53uXKnJu+hRXP X-Received: by 2002:aa7:d299:: with SMTP id w25mr2096599edq.269.1640073437561; Mon, 20 Dec 2021 23:57:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1640073437; cv=none; d=google.com; s=arc-20160816; b=u1FuvsByLe/Q1+bK3SeugWQFQ/prxqr8aQJIWYqdLbqATY6dfNkOaonJ6+RgkzMyGM O+1Ulo7q+FlgLOgA6qcMtu4KZwpRT9jPWdW2Vmt0rgWza+Om0MXlq+qlxtGIcR8hy6WJ Ld1rR7hQa+/h/PQ3xwf1aD5S1cS+GdFcj1jpnUfYwBkyfCyIj7D8IdPMjHrVFZsDibtm TTaaBxQcg6FyU+oo7NBNypXqCTA54FsII0oZRzW/JPpZ7Gff7Z7/Z80NEHtIJNk/2IYd wUUOWtC47zQ6qePBHx5w3dCwd4bAd+fCJO1V5ZVjg0dI/XTNzRWQyIKOqnY9LHbAnp5F p4/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=A4SeFj3FdN54aB2djTsIRUtGWwW9W/MmjWUk85kESos=; b=AUAnY3Ui8RW3MreYlyZUBGLWaKX74vCP5vWnJg1qDcqsu0uEdLugKxN9fyIQ61ikRy vVIAO4HLIVPNKwzV5/eKJrgMlZkE094TMtllPq40Em7RVWFuS64V0C/mBs3wdZiTpAZC oo2uPhl6a9G1WEsR7vllBtr8eIF8+N6oh7qcuabd0yj4Iyt/CNTc8RnG/WjRqqvo1SWu IawmZRHTrTwd/uIf2SUJjVteD9wLUQ+KpKh1HCinZqtMrDlGEpMK8dDmCYaBJOvgphJB n+lIL9fqwhZLVYb3zgwsoJGFfnR4ihf3WDySHykf/d3Z05cpinmNifExht+3qEXDLLjp u0RA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b="qAX/j6V7"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id nd27si12212351ejc.85.2021.12.20.23.57.17; Mon, 20 Dec 2021 23:57:17 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b="qAX/j6V7"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 40DC968AF2C; Tue, 21 Dec 2021 09:57:05 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8DF8868AB2E for ; Tue, 21 Dec 2021 09:56:57 +0200 (EET) Received: by mail-lf1-f46.google.com with SMTP id g26so11627438lfv.11 for ; Mon, 20 Dec 2021 23:56:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ymNxm4BYK24CFTh53P1IrRiRqYUFdFuHYFgc8ZyeT8w=; b=qAX/j6V7yhBPEE4dWc+vECbFOhJ83b/DLhNBXeqHmwQ+qylmTQpB0DIFS7ztT/7g9G 2uE4P1PLV80G5zrAyokqb2abFx9YjMB+p8TGJ8rjQzxMhhkelOqt66SOuvsjyFxTPxda tv1M91l53SecCWTZmDB/Ggq2xFObntZTHBT+EPazskx6gUjvLxBHmN1eQjXhTzZ4XpjY YFtrqBk5W8zaeBo98EsSOOIgnRty/PJ5WfbvymFPWuhGiljXHmLw0f02/4vDqrJA7Hg3 rnMrxTzT5Ee7t0JA2EtkNl0WH/OE2cY3s7CbwxVSv6dwfkpBvpCjUns4peZwInN2gcSH M/hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ymNxm4BYK24CFTh53P1IrRiRqYUFdFuHYFgc8ZyeT8w=; b=wiGPsCqWgLskERbHofRDLDGjAv4W3PvQCwudmNqHEGBFZ3u1iuZjRnEChtFRdtwWcV VvdBifMUdu2kJnyAoQOOcbp1cbp+1K+z3pfHmyRIyAqFG61JOPFAleX8Oh5WM9w/v1hs jCW7jrQe/pIQMA1DdhNdqWK4fZFwz3P0IkWCStAD+OuO2WrVBob8dqFGLOFm7/5iI49P DBbz/lRvw+rUFpvlT6dcWgWUlxMZ7aC5xIqprbFQ3Qrblmg5gZLjm17yxjLGqMUikbe7 O6kfXW/upXHK276n2FUTWjX2jBZMnG2qbIvpyGK0pNmSvvuYlTjpud9/UkaXZCOMEeSJ ckTQ== X-Gm-Message-State: AOAM530D+/UtAJUj/XR44laH6HhgFOkYtvHQxzW6i9ue1oH94Yy+s0Db +bxS13axeZBmCJBUnmFjI4maZCwkOK0= X-Received: by 2002:ac2:5ca4:: with SMTP id e4mr2103847lfq.263.1640073416936; Mon, 20 Dec 2021 23:56:56 -0800 (PST) Received: from localhost.localdomain (91-153-198-187.elisa-laajakaista.fi. [91.153.198.187]) by smtp.gmail.com with ESMTPSA id j18sm2631729lfg.60.2021.12.20.23.56.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Dec 2021 23:56:56 -0800 (PST) From: =?utf-8?q?Jan_Ekstr=C3=B6m?= To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 Dec 2021 09:56:48 +0200 Message-Id: <20211221075648.13059-2-jeebjp@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211221075648.13059-1-jeebjp@gmail.com> References: <20211221075648.13059-1-jeebjp@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] ffmpeg: add video heartbeat capability to fix_sub_duration X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Q64gb0XFFrKa From: Jan Ekström Splits the currently handled subtitle at random access point packets that can be configured to follow a specific output stream. This way the subtitle - which is known to be shown at this time can be split and passed to muxer before its full duration is yet known. Signed-off-by: Jan Ekström --- doc/ffmpeg.texi | 11 ++++ fftools/ffmpeg.c | 142 +++++++++++++++++++++++++++++++++++++++++++ fftools/ffmpeg.h | 8 +++ fftools/ffmpeg_opt.c | 9 +++ 4 files changed, 170 insertions(+) diff --git a/doc/ffmpeg.texi b/doc/ffmpeg.texi index 164419cad3..7ceb5532ce 100644 --- a/doc/ffmpeg.texi +++ b/doc/ffmpeg.texi @@ -1297,6 +1297,17 @@ List all hardware acceleration components enabled in this build of ffmpeg. Actual runtime availability depends on the hardware and its suitable driver being installed. +@item -fix_sub_duration_heartbeat[:@var{stream_specifier}] +Set a specific output video stream as the heartbeat stream according to which +to split and push through currently in-progress subtitle upon receipt of a +random access packet. + +This lowers the latency of subtitles for which the end packet or the following +subtitle has not yet been received. + +Requires @option{-fix_sub_duration} to be set for the relevant input subtitle +stream for this to have any effect. + @end table @section Audio Options diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index 1697891a99..f8cc11d1f4 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -126,6 +126,7 @@ typedef struct BenchmarkTimeStamps { int64_t sys_usec; } BenchmarkTimeStamps; +static int trigger_fix_sub_duration_heartbeat(OutputStream *ost, const AVPacket *pkt); static void do_video_stats(OutputStream *ost, int frame_size); static BenchmarkTimeStamps get_benchmark_time_stamps(void); static int64_t getmaxrss(void); @@ -1382,6 +1383,13 @@ static void do_video_out(OutputFile *of, av_ts2str(pkt->dts), av_ts2timestr(pkt->dts, &ost->mux_timebase)); } + if ((ret = trigger_fix_sub_duration_heartbeat(ost, pkt)) < 0) { + av_log(NULL, AV_LOG_ERROR, + "Subtitle heartbeat logic failed in %s! (%s)\n", + __func__, av_err2str(ret)); + exit_program(1); + } + frame_size = pkt->size; output_packet(of, pkt, ost, 0); @@ -2091,6 +2099,16 @@ static void do_streamcopy(InputStream *ist, OutputStream *ost, const AVPacket *p ost->sync_opts += opkt->duration; + if (ost->st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) { + int ret; + if ((ret = trigger_fix_sub_duration_heartbeat(ost, pkt)) < 0) { + av_log(NULL, AV_LOG_ERROR, + "Subtitle heartbeat logic failed in %s! (%s)\n", + __func__, av_err2str(ret)); + exit_program(1); + } + } + output_packet(of, opkt, ost, 0); } @@ -2548,6 +2566,130 @@ out: return ret; } +static int copy_av_subtitle(AVSubtitle *dst, AVSubtitle *src) +{ + int ret = AVERROR_BUG; + AVSubtitle tmp = { + .format = src->format, + .start_display_time = src->start_display_time, + .end_display_time = src->end_display_time, + .num_rects = 0, + .rects = NULL, + .pts = src->pts + }; + + if (!src->num_rects) + goto success; + + if (!(tmp.rects = av_calloc(src->num_rects, sizeof(*tmp.rects)))) + return AVERROR(ENOMEM); + + for (int i = 0; i < src->num_rects; i++) { + AVSubtitleRect *src_rect = src->rects[i]; + AVSubtitleRect *dst_rect; + + if (!(dst_rect = tmp.rects[i] = av_mallocz(sizeof(*tmp.rects[0])))) { + ret = AVERROR(ENOMEM); + goto cleanup; + } + + tmp.num_rects++; + + dst_rect->type = src_rect->type; + dst_rect->flags = src_rect->flags; + + dst_rect->x = src_rect->x; + dst_rect->y = src_rect->y; + dst_rect->w = src_rect->w; + dst_rect->h = src_rect->h; + dst_rect->nb_colors = src_rect->nb_colors; + + if (src_rect->text) + if (!(dst_rect->text = av_strdup(src_rect->text))) { + ret = AVERROR(ENOMEM); + goto cleanup; + } + + if (src_rect->ass) + if (!(dst_rect->ass = av_strdup(src_rect->ass))) { + ret = AVERROR(ENOMEM); + goto cleanup; + } + + for (int j = 0; j < 4; j++) { + // SUBTITLE_BITMAP images are special in the sense that they + // are like PAL8 images. first pointer to data, second to + // palette. This makes the size calculation match this. + size_t buf_size = src_rect->type == SUBTITLE_BITMAP && j == 1 ? + AVPALETTE_SIZE : + src_rect->h * src_rect->linesize[j]; + + if (!src_rect->data[j]) + continue; + + if (!(dst_rect->data[j] = av_memdup(src_rect->data[j], buf_size))) { + ret = AVERROR(ENOMEM); + goto cleanup; + } + dst_rect->linesize[j] = src_rect->linesize[j]; + } + } + +success: + *dst = tmp; + + return 0; + +cleanup: + avsubtitle_free(&tmp); + + return ret; +} + +static int fix_sub_duration_heartbeat(InputStream *ist, int64_t signal_pts) +{ + int ret = AVERROR_BUG; + int got_output = 1; + AVSubtitle *prev_subtitle = &ist->prev_sub.subtitle; + AVSubtitle subtitle; + + if (!ist->fix_sub_duration || !prev_subtitle->num_rects || + signal_pts <= prev_subtitle->pts) + return 0; + + if ((ret = copy_av_subtitle(&subtitle, prev_subtitle)) < 0) + return ret; + + subtitle.pts = signal_pts; + + return encode_mux_subtitles(ist, &subtitle, &got_output); +} + +static int trigger_fix_sub_duration_heartbeat(OutputStream *ost, const AVPacket *pkt) +{ + int64_t signal_pts = av_rescale_q(pkt->pts, ost->mux_timebase, + AV_TIME_BASE_Q); + + if (!ost->fix_sub_duration_heartbeat || !(pkt->flags & AV_PKT_FLAG_KEY)) + // we are only interested in heartbeats on streams configured, and + // only on random access points. + return 0; + + for (int index = 0; index < nb_input_streams; index++) { + InputStream *subtitle_ist = input_streams[index]; + int ret = AVERROR_BUG; + + if (!subtitle_ist->decoding_needed || + subtitle_ist->dec_ctx->codec_type != AVMEDIA_TYPE_SUBTITLE) + continue; + + if ((ret = fix_sub_duration_heartbeat(subtitle_ist, signal_pts)) < 0) + return ret; + } + + return 0; +} + static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, int *decode_failed) { diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index 9b200b806a..b1e67a94e9 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -204,6 +204,8 @@ typedef struct OptionsContext { int nb_reinit_filters; SpecifierOpt *fix_sub_duration; int nb_fix_sub_duration; + SpecifierOpt *fix_sub_duration_heartbeat; + int nb_fix_sub_duration_heartbeat; SpecifierOpt *canvas_sizes; int nb_canvas_sizes; SpecifierOpt *pass; @@ -571,6 +573,12 @@ typedef struct OutputStream { /* frame encode sum of squared error values */ int64_t error[4]; + + /* + * bool on whether this stream should be utilized for splitting + * subtitles utilizing fix_sub_duration at random access points. + */ + unsigned int fix_sub_duration_heartbeat; } OutputStream; typedef struct OutputFile { diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index 9c820ab73f..64a08749ac 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -86,6 +86,7 @@ static const char *const opt_name_filters[] = {"filter", "af", static const char *const opt_name_filter_scripts[] = {"filter_script", NULL}; static const char *const opt_name_reinit_filters[] = {"reinit_filter", NULL}; static const char *const opt_name_fix_sub_duration[] = {"fix_sub_duration", NULL}; +static const char *const opt_name_fix_sub_duration_heartbeat[] = {"fix_sub_duration_heartbeat", NULL}; static const char *const opt_name_canvas_sizes[] = {"canvas_size", NULL}; static const char *const opt_name_pass[] = {"pass", NULL}; static const char *const opt_name_passlogfiles[] = {"passlogfile", NULL}; @@ -1600,6 +1601,9 @@ static OutputStream *new_output_stream(OptionsContext *o, AVFormatContext *oc, e MATCH_PER_STREAM_OPT(bits_per_raw_sample, i, ost->bits_per_raw_sample, oc, st); + MATCH_PER_STREAM_OPT(fix_sub_duration_heartbeat, i, ost->fix_sub_duration_heartbeat, + oc, st); + if (oc->oformat->flags & AVFMT_GLOBALHEADER) ost->enc_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER; @@ -3793,6 +3797,11 @@ const OptionDef options[] = { { "autoscale", HAS_ARG | OPT_BOOL | OPT_SPEC | OPT_EXPERT | OPT_OUTPUT, { .off = OFFSET(autoscale) }, "automatically insert a scale filter at the end of the filter graph" }, + { "fix_sub_duration_heartbeat", OPT_VIDEO | OPT_BOOL | OPT_EXPERT | + OPT_SPEC | OPT_OUTPUT, { .off = OFFSET(fix_sub_duration_heartbeat) }, + "set this video output stream to be a heartbeat stream for " + "fix_sub_duration, according to which subtitles should be split at " + "random access points" }, /* audio options */ { "aframes", OPT_AUDIO | HAS_ARG | OPT_PERFILE | OPT_OUTPUT, { .func_arg = opt_audio_frames },