From patchwork Mon Oct 10 12:44:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 38669 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4a86:b0:9d:28a3:170e with SMTP id fn6csp348500pzb; Mon, 10 Oct 2022 05:45:16 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7HulAdNfuwwBM7Pl75q9LmrmMT4ogJruytWvmBcm1rj1kFoEERpYBTyGOCCrEvQuGSpIkG X-Received: by 2002:a05:6402:11ce:b0:45b:f851:2941 with SMTP id j14-20020a05640211ce00b0045bf8512941mr7451647edw.177.1665405915689; Mon, 10 Oct 2022 05:45:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665405915; cv=none; d=google.com; s=arc-20160816; b=fExNhbUvvT3XaBmFE4gFGZkZkzcoJttUmSHgBup/30zrk7A+MIH+Qr+96Momfbdj/j NKRgDDXM7/zeDZa6H3JPZxpUEk32RbcNqh8xBLBvWvw5NMZYW6o8N34PtS22Hyd17ywc oIPWE0KT03W/D7xvzj52wHcI+rIbORbWcH2+YMkpk2B/+SST1z4eisN2wK7g/uLQglb4 6JRCAz27t1JasK6auBZDt4d7AdXwB/FJtD1jDW7tM57nsWAOMdM140LbsIAUZkKZ1n2t sPIWu6x5t+Ooih8/VzR7SyGTjBvhuLPw5RD082E1nYYkUNo3SvdFM4xRbr4TbyaS9zRm AdIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=WVYRsE/13JE4xv0xMAC4CLD24kA7VV7/p+PQQ2yBNaI=; b=B1XhDXWJLR0V8Y/mXWbFS47U1RsjpBmoXmpa4++h+EFtuMRJfA2EGZCQypVo2RwTYx /kcmKRQEpBfS4vVsw8NOyd9bL1KGZik/SqcvmmX2BdSUjJutku9WzxGWeQYJlcu8uT1l DNRiyRZh1IpgGaMekb33uGIeamMai03P/XwocWelaVTxuiR4fDEQFLFx9RXsWl4/Oqwe kYeUkgLRkOO5EBTv8xFlQlHpC8ukZERAh9wIJ9WBnvxZV62MFGMgLDyGVaTquBlFSwSu 3aOlXgYOvhk1eVjPEJolcvex90L8dYz39c1/jh81sj49N9J6Mj5I3KQ3cPi0CagMX/8v jg/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=oYa2irhl; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d9-20020aa7d689000000b004594e0c0fd8si8844614edr.473.2022.10.10.05.45.14; Mon, 10 Oct 2022 05:45:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=oYa2irhl; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BA52E68BCF0; Mon, 10 Oct 2022 15:45:10 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5711968BC82 for ; Mon, 10 Oct 2022 15:45:04 +0300 (EEST) Received: by mail-lj1-f177.google.com with SMTP id h8so1106822lja.11 for ; Mon, 10 Oct 2022 05:45:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=J73wwsUeNn41xwgMtd00inIX6ks7V+ANDHGVMLmxQXI=; b=oYa2irhl3+a0miwgPLanMuDqCmS5gt17jNBokzkn5vaIViB9ZOuPBtZ5N3jMsW53Fz 48L6lJCXeOfNntTs7J1EAWrAB5Jr+YY2F2gQW6woXq4xOq4a47lDoZ1Qux7vu0L7pfxL 3VLEsC6LXT3eWKTKvdYFaT5o9JJeKP1+YlcVOM1DtM0vJXmRs4wAZVsm1rLfbEGr7UoM R1TTCLFSoxd40kZeW8DNk7CQP/9QxH/TxIpJWeD1YHiP4jqfTsUuAZmvsTA0ae5MtAUo /o2bff5ehoFOL01X3GYGeJbWMlSj7yFrvD/aFkNAQuVRxwNrdmpi7VSPFMGT/BWTUy/p QHbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=J73wwsUeNn41xwgMtd00inIX6ks7V+ANDHGVMLmxQXI=; b=OHkMe51G5A+kbMNItLKjAuVh+icz5LIK6J/r1dazPjmQHS0UFusLzBae6/39X2zfh2 Fawrsbcy626AzWunndz9ki0qR8TsWtLOtiUXg1APx14KmkfYbmud2YZWk64SyAl7mp9L 5mhOLixqX0Ae8cgM8ki0fN8ycpuH5bXOpDNydHrB+gmQ2jgWqz7PXW31jgghv4KbQYD5 m+3qaxmvmUk+S+LpyRni16/tJIyKtHv7L/3+veTBFefx+akpqs2q9xa4QUrvIIU1rI7k lg537ophUB4qyB700qPLZJXTFsJIxF4lWm4fGQ2RH6xIlG9GCdfOagnAEnm4aorRqzcQ Y65Q== X-Gm-Message-State: ACrzQf28hyZnkzbIrRSLNoFGIHwsSp4iXlbZCxOHRspUBS19XMoDZTi+ rD82I+3M7cTZ0Ry3tQspnxioyNSHpqQ= X-Received: by 2002:a05:651c:107a:b0:26e:8998:3f63 with SMTP id y26-20020a05651c107a00b0026e89983f63mr4836036ljm.23.1665405903370; Mon, 10 Oct 2022 05:45:03 -0700 (PDT) Received: from localhost.localdomain (91-153-198-187.elisa-laajakaista.fi. [91.153.198.187]) by smtp.gmail.com with ESMTPSA id 14-20020a2e154e000000b0026dd1968533sm1657722ljv.86.2022.10.10.05.45.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Oct 2022 05:45:02 -0700 (PDT) From: =?utf-8?q?Jan_Ekstr=C3=B6m?= To: ffmpeg-devel@ffmpeg.org Date: Mon, 10 Oct 2022 15:44:55 +0300 Message-Id: <20221010124457.9213-1-jeebjp@gmail.com> X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v5 1/3] ffmpeg: refactor post-decoding steps for subtitles into a function X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Et2e5czWsPIk From: Jan Ekström This enables us to later call this when generating additional subtitles for splitting purposes. Co-authored-by: Andrzej Nadachowski Signed-off-by: Jan Ekström --- fftools/ffmpeg.c | 51 +++++++++++++++++++++++++++--------------------- 1 file changed, 29 insertions(+), 22 deletions(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index 754172e568..fe350fe2bb 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -2304,27 +2304,15 @@ fail: return err < 0 ? err : ret; } -static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, - int *decode_failed) +static int process_subtitle(InputStream *ist, AVSubtitle *subtitle, int *got_output) { - AVSubtitle subtitle; + int ret = 0; int free_sub = 1; - int i, ret = avcodec_decode_subtitle2(ist->dec_ctx, - &subtitle, got_output, pkt); - - check_decode_result(NULL, got_output, ret); - - if (ret < 0 || !*got_output) { - *decode_failed = 1; - if (!pkt->size) - sub2video_flush(ist); - return ret; - } if (ist->fix_sub_duration) { int end = 1; if (ist->prev_sub.got_output) { - end = av_rescale(subtitle.pts - ist->prev_sub.subtitle.pts, + end = av_rescale(subtitle->pts - ist->prev_sub.subtitle.pts, 1000, AV_TIME_BASE); if (end < ist->prev_sub.subtitle.end_display_time) { av_log(NULL, AV_LOG_DEBUG, @@ -2336,7 +2324,7 @@ static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, } FFSWAP(int, *got_output, ist->prev_sub.got_output); FFSWAP(int, ret, ist->prev_sub.ret); - FFSWAP(AVSubtitle, subtitle, ist->prev_sub.subtitle); + FFSWAP(AVSubtitle, *subtitle, ist->prev_sub.subtitle); if (end <= 0) goto out; } @@ -2345,40 +2333,59 @@ static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, return ret; if (ist->sub2video.frame) { - sub2video_update(ist, INT64_MIN, &subtitle); + sub2video_update(ist, INT64_MIN, subtitle); } else if (ist->nb_filters) { if (!ist->sub2video.sub_queue) ist->sub2video.sub_queue = av_fifo_alloc2(8, sizeof(AVSubtitle), AV_FIFO_FLAG_AUTO_GROW); if (!ist->sub2video.sub_queue) report_and_exit(AVERROR(ENOMEM)); - ret = av_fifo_write(ist->sub2video.sub_queue, &subtitle, 1); + ret = av_fifo_write(ist->sub2video.sub_queue, subtitle, 1); if (ret < 0) exit_program(1); free_sub = 0; } - if (!subtitle.num_rects) + if (!subtitle->num_rects) goto out; ist->frames_decoded++; - for (i = 0; i < nb_output_streams; i++) { + for (int i = 0; i < nb_output_streams; i++) { OutputStream *ost = output_streams[i]; if (!check_output_constraints(ist, ost) || !ost->enc_ctx || ost->enc_ctx->codec_type != AVMEDIA_TYPE_SUBTITLE) continue; - do_subtitle_out(output_files[ost->file_index], ost, &subtitle); + do_subtitle_out(output_files[ost->file_index], ost, subtitle); } out: if (free_sub) - avsubtitle_free(&subtitle); + avsubtitle_free(subtitle); return ret; } +static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, + int *decode_failed) +{ + AVSubtitle subtitle; + int ret = avcodec_decode_subtitle2(ist->dec_ctx, + &subtitle, got_output, pkt); + + check_decode_result(NULL, got_output, ret); + + if (ret < 0 || !*got_output) { + *decode_failed = 1; + if (!pkt->size) + sub2video_flush(ist); + return ret; + } + + return process_subtitle(ist, &subtitle, got_output); +} + static int send_filter_eof(InputStream *ist) { int i, ret; From patchwork Mon Oct 10 12:44:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 38670 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4a86:b0:9d:28a3:170e with SMTP id fn6csp348591pzb; Mon, 10 Oct 2022 05:45:25 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4UzB1fnJdd1B8zOoZLYXgksOLK4q0CwxjZq0sisdX/UItWxMBpQRcdxGBt8L2F/coRnjzV X-Received: by 2002:a05:6402:11d1:b0:45c:1fef:ee43 with SMTP id j17-20020a05640211d100b0045c1fefee43mr4449814edw.36.1665405924443; Mon, 10 Oct 2022 05:45:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665405924; cv=none; d=google.com; s=arc-20160816; b=SgeUiFxnWtk/Nt0HrISGZzHT0taNRNHsoygaXfqtlrSBtSZ+7cHuGd4XVG+hGlY4Um UwoERr/8PVocrh+iwG2ltPf8ig6JpdtSNgVjr/5a7fYYR7zajmRlJUq6Geve/pa7Q7N4 TX9f83/7p2CaJzoSxoJDn/NJKPdGu0rhDLWLq21jOHk71vtbsLnjYDTAWPRd32LfPTjw uOmYDc/fqDb41jmgyIHt0hV2+emCigth3wr5ygbrghzH7j+ZIxlgMbZzAn3MhjdU0iaS Vf+ukjfyoboRlyWNldsu06tEA0z0ppbnn9MeUAEXabOce1UKLXJq80xZB13vKP7axuaR wEnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=Nsn6Jk4qprytNLi1LYbrziiQnPir1UPmVWFh80fdIDM=; b=Pjbe7frMoBOIcGqWR7BofA5u5anxiJALTmzlVCLZ+Vss8E+UcrXq8V8t342QgS7yoX 681Fbzdh4apNgQI882+2d+c9pCZwhaARaeAdWGyxGOmzA+L9Q8rvTZqSUv4g4hCAVJUe od1+YrkxWb9OdU2hd6x55yuz9+h6d02vN0Qu0sqFSUACkp+HciCFiO0G7EaBKzNji7nW sEHbFxPO9G2dRBWCAr968971fYhQE4AEVhFzcQuRyeElY4CUpbJQ0KF9qugQgsRmOJyZ zHorJbPmGoLKkJeggFVIC6RddWgkx/uYR5/pXARL0LhhuwPAQIdWh1svvnbJvqwY8YuE yA1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=DFrNqQa1; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id he17-20020a1709073d9100b00776d4009294si12054576ejc.651.2022.10.10.05.45.23; Mon, 10 Oct 2022 05:45:24 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=DFrNqQa1; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 01B0668BC82; Mon, 10 Oct 2022 15:45:12 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com [209.85.208.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C78E568BC82 for ; Mon, 10 Oct 2022 15:45:04 +0300 (EEST) Received: by mail-lj1-f175.google.com with SMTP id by36so13120741ljb.4 for ; Mon, 10 Oct 2022 05:45:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=paQNRitfTZqGrcbmqiz4rcYjOGNl0Nv/qXv1E4bdQso=; b=DFrNqQa1wonEnL1VhcBOp4tx7mtXx1vvITY7VZIQL59Bkaybx/NRRQOvswN6JYgZjL h+wYy8nO+QbZ5dVbFnJBSKaLC+qnPW0WWIrNCbsZ5ZlFc+f4lkY5+/my+S6rv4xZyRtO hHvOne2Y4Og97WaIVuOO0QuvHidaybOYqo+EasWNsJbCs15TMEZyVZBp5I3UgsUb5QyH X1LIvALwb3corBrjjtZli02rSV2jD0DS+gLWCUD2EbqncrIi8dLKFXLe7zx0hF1InXMH uFM5WYSbr2/wP8w/LfGm7zFOQ3Yxv04vLWeptVHnlDD2h2pXiVz5gDknmJS2rSBn2b1P S+wQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=paQNRitfTZqGrcbmqiz4rcYjOGNl0Nv/qXv1E4bdQso=; b=OyGmycqcQ4L+a/pUfSuaIto5VI0bpJXCdokF7eEr7rcXnWfj3qLPgGT4+fL+DoVE0H zcx6NNG4JYSPVEeDcWP3PJ+Oqv8xSBGbecXE8JuosI4q8ZHe8adi2xH0YUlZPRnahse4 kGDdMunuzQkssigzW0x5uMIjIb6k/EwK6kfnS7j7baFmumdQMoPME9RGSwZnN5w+rs8H PiTJnCI3zEx/2jfWEd0p0W3H7abOy5PRg5X8EUFhzWkJ40Ep26C6aBuU8zYRc1s8WYmZ G9fjSE3DGkQqb1qoUAPU7b2/DWwCgxs9tDUjdkO/6msWxsmOwafgj8hDRs3jKQFy8oE5 6GQg== X-Gm-Message-State: ACrzQf0Y5BtPfBqjHN4DNg6Rghum5/BGVnjEkk6deY/9D8g8JN89DzEF aHD7elrUg5FcRjakD/cLHSSnQhMI5zc= X-Received: by 2002:a2e:a7cf:0:b0:26e:8d31:cadc with SMTP id x15-20020a2ea7cf000000b0026e8d31cadcmr4150762ljp.344.1665405904109; Mon, 10 Oct 2022 05:45:04 -0700 (PDT) Received: from localhost.localdomain (91-153-198-187.elisa-laajakaista.fi. [91.153.198.187]) by smtp.gmail.com with ESMTPSA id 14-20020a2e154e000000b0026dd1968533sm1657722ljv.86.2022.10.10.05.45.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Oct 2022 05:45:03 -0700 (PDT) From: =?utf-8?q?Jan_Ekstr=C3=B6m?= To: ffmpeg-devel@ffmpeg.org Date: Mon, 10 Oct 2022 15:44:56 +0300 Message-Id: <20221010124457.9213-2-jeebjp@gmail.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221010124457.9213-1-jeebjp@gmail.com> References: <20221010124457.9213-1-jeebjp@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v5 2/3] ffmpeg: move decoded frame counter from after post-processing to decode X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: C4IPxlHR1bMX From: Jan Ekström This way we can call process_subtitles without causing the decoded frame counter to get bumped. Additionally, this now takes into mention all of the decoded subtitle frames without fix_sub_duration latency/buffering, or filtering out decoded reset/end subtitles without any rendered rectangles, which matches the original intent in 4754345027eb85cfa51aeb88beec68d7b036c11e . Signed-off-by: Jan Ekström --- fftools/ffmpeg.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index fe350fe2bb..a7e6c0f6e0 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -2349,8 +2349,6 @@ static int process_subtitle(InputStream *ist, AVSubtitle *subtitle, int *got_out if (!subtitle->num_rects) goto out; - ist->frames_decoded++; - for (int i = 0; i < nb_output_streams; i++) { OutputStream *ost = output_streams[i]; @@ -2383,6 +2381,8 @@ static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, return ret; } + ist->frames_decoded++; + return process_subtitle(ist, &subtitle, got_output); } From patchwork Mon Oct 10 12:44:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 38671 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4a86:b0:9d:28a3:170e with SMTP id fn6csp348656pzb; Mon, 10 Oct 2022 05:45:33 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5uhZqX7VvEnf8BLQA9AekaaonYmSwB8WOccqfTt3eNZdv/9xfNYHaMAsrvZkqyielVVXlT X-Received: by 2002:a17:907:960e:b0:782:68e3:620f with SMTP id gb14-20020a170907960e00b0078268e3620fmr14417597ejc.663.1665405932616; Mon, 10 Oct 2022 05:45:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665405932; cv=none; d=google.com; s=arc-20160816; b=mulFvtZPZbenqdLF7XLR63P0pg5JjYAhem1R9RCVL2YWp+DslDlBNHnWslrLrJhxdB AzDeWinLxKXVdtDp8rQqz56QAebNZ6fTzMIPi9kRDE1+YXCmHHD30SvmfnDRh4PJg7VZ /rOw/UU4YB8hIJmvZ/UtMtItgnKtZ9B4nnzF0vjbvIlYN/bwsfIxjn85mGvG1YDxM1E5 HvHj5w7dmDlql2tn5loJPoGepoKQ/qlrxS5BtQJEBbHO+gl6GH0yOBDF0q5waS+0y7/t yX3wo0LZvmZTJCySztPYiZRhSzOX3d4eQZxfhJzGuxabFYRGw/KKPVUizhP2RGEYG79/ VssA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=ekky1OdOGSijEPZZXBMIjuHjGrTCKoJ6llPyhrwiAAc=; b=tV9/e+duWAYO921/bIT6vcm+mK4TR3cCodn1EsmVAdxuyTpQqbxfGoJqnH9IRPK3X3 D9ema/Aul5NMgJLLT9q+5Rv3Y2BUx5TfcZk44CIWjdPN6tJmbh2ZAOki+aG6bYYGBL8n aafzYxxwZQqENyWqm+FiO51MHF+CnPFIxwoZ3H2gfFcyL9gcOkvPT85A3jNe4I7iQJ6W pvNgv9Z89fRMT3W9NAzID/4tbKU12rfrSdY2bsUJvpsuE6zy2U5Sf2X63o/IGjoeDSTq xseluIPX6Q+xsz/Ms0wjA3ytA+vHlWdZzKXCzi6vhTDSGRaF+qGRaNyZ61z0ksxfoIRX FezA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=Pjf0w8HG; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hg7-20020a1709072cc700b00781599eb7d9si10332160ejc.542.2022.10.10.05.45.32; Mon, 10 Oct 2022 05:45:32 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=Pjf0w8HG; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0A93768BD05; Mon, 10 Oct 2022 15:45:13 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9E79A68BCEF for ; Mon, 10 Oct 2022 15:45:05 +0300 (EEST) Received: by mail-lj1-f177.google.com with SMTP id c20so641525ljj.7 for ; Mon, 10 Oct 2022 05:45:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=EZy8fF6ElKizOOF79NM5L74BRlDC12EPI8wDR3hFH7E=; b=Pjf0w8HG6ZfBEq9JSlIvN9kh27Pqend0cK2CDnHmT0QrYpqgKgwSD5xEjujd1mesWX bY8KLTtEyTwp6S/GgZ9wGmh2Y9E94DXOcMdMcov+wYp13pWa5PiaN0VnKhWgEqBiCPJk EVJIoIaB6nYZMbNLw6aAyhKZBUok8f+kyTinTCIqYU+s9OBH5oHCixf5oRO/FVnGgXnh xabcaCEIwZrwvzQOHu0aehQdoCSZ4wKD75kjS57BBO+0Vjnr6/AK+0UmBLgQxnWlKx42 KGK15DmQH85cFjdrJf0t/AwTsPylNTdq2xZB54J4H4kPwy54P7FsmXf65EyjeqorOFfJ jyyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EZy8fF6ElKizOOF79NM5L74BRlDC12EPI8wDR3hFH7E=; b=uHHfUErQgcQqX6zUBPZ0j3WevzKt+xJnxjLZtX6z6hOx5B3FntYkD5Gi3xc82jTZvh Bu+cV0YJPoBFr+IcwGMOVQyoaEHx/EfskeHweYsZdSA2RJ7OSFM1AOHOY6nlsdO/Frip +qfZYkOTbHGlUg0zT91en1YSjksp6FYsyGVLQNButwO+BKT199PA7UCtiOgk9Ti1yfkr 57VoxUfbB17nizGAB3Sl5hmXy+hYPXQkyaEjXcwVJKM4hBS6i2cr3vRN/c8LKRlcJLaT ne0kaQGFUWsUJYDt/ev6nBxkS8DWl5G7w+sqLo53oxitbC2N4jQO2TjiWkmIAL1I4hbB qE5A== X-Gm-Message-State: ACrzQf1NA6eYT1fkC9B1gX88svWpWex6KTHsJWFIbcxGzwtByxzX+05n tCLlEqSQ3BGWzEtiqMpDenZf4LWW1oE= X-Received: by 2002:a2e:9b91:0:b0:26a:ce59:c517 with SMTP id z17-20020a2e9b91000000b0026ace59c517mr6512973lji.181.1665405904932; Mon, 10 Oct 2022 05:45:04 -0700 (PDT) Received: from localhost.localdomain (91-153-198-187.elisa-laajakaista.fi. [91.153.198.187]) by smtp.gmail.com with ESMTPSA id 14-20020a2e154e000000b0026dd1968533sm1657722ljv.86.2022.10.10.05.45.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Oct 2022 05:45:04 -0700 (PDT) From: =?utf-8?q?Jan_Ekstr=C3=B6m?= To: ffmpeg-devel@ffmpeg.org Date: Mon, 10 Oct 2022 15:44:57 +0300 Message-Id: <20221010124457.9213-3-jeebjp@gmail.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221010124457.9213-1-jeebjp@gmail.com> References: <20221010124457.9213-1-jeebjp@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v5 3/3] ffmpeg: add video heartbeat capability to fix_sub_duration X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: K+SnOd07oEt0 From: Jan Ekström Splits the currently handled subtitle at random access point packets that can be configured to follow a specific output stream. This way the subtitle - which is known to be shown at this time can be split and passed to muxer before its full duration is yet known. Co-authored-by: Andrzej Nadachowski Co-authored-by: Bernard Boulay Signed-off-by: Jan Ekström --- doc/ffmpeg.texi | 11 ++ fftools/ffmpeg.c | 142 ++++++++++++++++++ fftools/ffmpeg.h | 8 + fftools/ffmpeg_opt.c | 9 ++ tests/fate/ffmpeg.mak | 14 ++ .../fate/ffmpeg-fix_sub_duration_heartbeat | 48 ++++++ 6 files changed, 232 insertions(+) create mode 100644 tests/ref/fate/ffmpeg-fix_sub_duration_heartbeat diff --git a/doc/ffmpeg.texi b/doc/ffmpeg.texi index e9020b30d5..cd957fa826 100644 --- a/doc/ffmpeg.texi +++ b/doc/ffmpeg.texi @@ -1313,6 +1313,17 @@ List all hardware acceleration components enabled in this build of ffmpeg. Actual runtime availability depends on the hardware and its suitable driver being installed. +@item -fix_sub_duration_heartbeat[:@var{stream_specifier}] +Set a specific output video stream as the heartbeat stream according to which +to split and push through currently in-progress subtitle upon receipt of a +random access packet. + +This lowers the latency of subtitles for which the end packet or the following +subtitle has not yet been received. + +Requires @option{-fix_sub_duration} to be set for the relevant input subtitle +stream for this to have any effect. + @end table @section Audio Options diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index a7e6c0f6e0..56fcb994e3 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -128,6 +128,7 @@ typedef struct BenchmarkTimeStamps { int64_t sys_usec; } BenchmarkTimeStamps; +static int trigger_fix_sub_duration_heartbeat(OutputStream *ost, const AVPacket *pkt); static BenchmarkTimeStamps get_benchmark_time_stamps(void); static int64_t getmaxrss(void); static int ifilter_has_all_input_formats(FilterGraph *fg); @@ -978,6 +979,13 @@ static int encode_frame(OutputFile *of, OutputStream *ost, AVFrame *frame) av_ts2str(pkt->duration), av_ts2timestr(pkt->duration, &enc->time_base)); } + if ((ret = trigger_fix_sub_duration_heartbeat(ost, pkt)) < 0) { + av_log(NULL, AV_LOG_ERROR, + "Subtitle heartbeat logic failed in %s! (%s)\n", + __func__, av_err2str(ret)); + exit_program(1); + } + ost->data_size_enc += pkt->size; if (enc->codec_type == AVMEDIA_TYPE_VIDEO) @@ -1934,6 +1942,16 @@ static void do_streamcopy(InputStream *ist, OutputStream *ost, const AVPacket *p opkt->duration = av_rescale_q(pkt->duration, ist->st->time_base, ost->mux_timebase); + { + int ret = trigger_fix_sub_duration_heartbeat(ost, pkt); + if (ret < 0) { + av_log(NULL, AV_LOG_ERROR, + "Subtitle heartbeat logic failed in %s! (%s)\n", + __func__, av_err2str(ret)); + exit_program(1); + } + } + output_packet(of, opkt, ost, 0); ost->streamcopy_started = 1; @@ -2365,6 +2383,130 @@ out: return ret; } +static int copy_av_subtitle(AVSubtitle *dst, AVSubtitle *src) +{ + int ret = AVERROR_BUG; + AVSubtitle tmp = { + .format = src->format, + .start_display_time = src->start_display_time, + .end_display_time = src->end_display_time, + .num_rects = 0, + .rects = NULL, + .pts = src->pts + }; + + if (!src->num_rects) + goto success; + + if (!(tmp.rects = av_calloc(src->num_rects, sizeof(*tmp.rects)))) + return AVERROR(ENOMEM); + + for (int i = 0; i < src->num_rects; i++) { + AVSubtitleRect *src_rect = src->rects[i]; + AVSubtitleRect *dst_rect; + + if (!(dst_rect = tmp.rects[i] = av_mallocz(sizeof(*tmp.rects[0])))) { + ret = AVERROR(ENOMEM); + goto cleanup; + } + + tmp.num_rects++; + + dst_rect->type = src_rect->type; + dst_rect->flags = src_rect->flags; + + dst_rect->x = src_rect->x; + dst_rect->y = src_rect->y; + dst_rect->w = src_rect->w; + dst_rect->h = src_rect->h; + dst_rect->nb_colors = src_rect->nb_colors; + + if (src_rect->text) + if (!(dst_rect->text = av_strdup(src_rect->text))) { + ret = AVERROR(ENOMEM); + goto cleanup; + } + + if (src_rect->ass) + if (!(dst_rect->ass = av_strdup(src_rect->ass))) { + ret = AVERROR(ENOMEM); + goto cleanup; + } + + for (int j = 0; j < 4; j++) { + // SUBTITLE_BITMAP images are special in the sense that they + // are like PAL8 images. first pointer to data, second to + // palette. This makes the size calculation match this. + size_t buf_size = src_rect->type == SUBTITLE_BITMAP && j == 1 ? + AVPALETTE_SIZE : + src_rect->h * src_rect->linesize[j]; + + if (!src_rect->data[j]) + continue; + + if (!(dst_rect->data[j] = av_memdup(src_rect->data[j], buf_size))) { + ret = AVERROR(ENOMEM); + goto cleanup; + } + dst_rect->linesize[j] = src_rect->linesize[j]; + } + } + +success: + *dst = tmp; + + return 0; + +cleanup: + avsubtitle_free(&tmp); + + return ret; +} + +static int fix_sub_duration_heartbeat(InputStream *ist, int64_t signal_pts) +{ + int ret = AVERROR_BUG; + int got_output = 1; + AVSubtitle *prev_subtitle = &ist->prev_sub.subtitle; + AVSubtitle subtitle; + + if (!ist->fix_sub_duration || !prev_subtitle->num_rects || + signal_pts <= prev_subtitle->pts) + return 0; + + if ((ret = copy_av_subtitle(&subtitle, prev_subtitle)) < 0) + return ret; + + subtitle.pts = signal_pts; + + return process_subtitle(ist, &subtitle, &got_output); +} + +static int trigger_fix_sub_duration_heartbeat(OutputStream *ost, const AVPacket *pkt) +{ + int64_t signal_pts = av_rescale_q(pkt->pts, ost->mux_timebase, + AV_TIME_BASE_Q); + + if (!ost->fix_sub_duration_heartbeat || !(pkt->flags & AV_PKT_FLAG_KEY)) + // we are only interested in heartbeats on streams configured, and + // only on random access points. + return 0; + + for (int index = 0; index < nb_input_streams; index++) { + InputStream *subtitle_ist = input_streams[index]; + int ret = AVERROR_BUG; + + if (!subtitle_ist->decoding_needed || + subtitle_ist->dec_ctx->codec_type != AVMEDIA_TYPE_SUBTITLE) + continue; + + if ((ret = fix_sub_duration_heartbeat(subtitle_ist, signal_pts)) < 0) + return ret; + } + + return 0; +} + static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, int *decode_failed) { diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index 8460391afa..91ac821cc2 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -219,6 +219,8 @@ typedef struct OptionsContext { int nb_reinit_filters; SpecifierOpt *fix_sub_duration; int nb_fix_sub_duration; + SpecifierOpt *fix_sub_duration_heartbeat; + int nb_fix_sub_duration_heartbeat; SpecifierOpt *canvas_sizes; int nb_canvas_sizes; SpecifierOpt *pass; @@ -610,6 +612,12 @@ typedef struct OutputStream { int sq_idx_encode; int sq_idx_mux; + + /* + * bool on whether this stream should be utilized for splitting + * subtitles utilizing fix_sub_duration at random access points. + */ + unsigned int fix_sub_duration_heartbeat; } OutputStream; typedef struct Muxer Muxer; diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index 8f57b699f1..023706c2d7 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -98,6 +98,7 @@ static const char *const opt_name_filters[] = {"filter", "af", static const char *const opt_name_filter_scripts[] = {"filter_script", NULL}; static const char *const opt_name_reinit_filters[] = {"reinit_filter", NULL}; static const char *const opt_name_fix_sub_duration[] = {"fix_sub_duration", NULL}; +static const char *const opt_name_fix_sub_duration_heartbeat[] = {"fix_sub_duration_heartbeat", NULL}; static const char *const opt_name_canvas_sizes[] = {"canvas_size", NULL}; static const char *const opt_name_pass[] = {"pass", NULL}; static const char *const opt_name_passlogfiles[] = {"passlogfile", NULL}; @@ -1718,6 +1719,9 @@ static OutputStream *new_output_stream(OptionsContext *o, AVFormatContext *oc, e MATCH_PER_STREAM_OPT(bits_per_raw_sample, i, ost->bits_per_raw_sample, oc, st); + MATCH_PER_STREAM_OPT(fix_sub_duration_heartbeat, i, ost->fix_sub_duration_heartbeat, + oc, st); + if (oc->oformat->flags & AVFMT_GLOBALHEADER && ost->enc_ctx) ost->enc_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER; @@ -4098,6 +4102,11 @@ const OptionDef options[] = { { "autoscale", HAS_ARG | OPT_BOOL | OPT_SPEC | OPT_EXPERT | OPT_OUTPUT, { .off = OFFSET(autoscale) }, "automatically insert a scale filter at the end of the filter graph" }, + { "fix_sub_duration_heartbeat", OPT_VIDEO | OPT_BOOL | OPT_EXPERT | + OPT_SPEC | OPT_OUTPUT, { .off = OFFSET(fix_sub_duration_heartbeat) }, + "set this video output stream to be a heartbeat stream for " + "fix_sub_duration, according to which subtitles should be split at " + "random access points" }, /* audio options */ { "aframes", OPT_AUDIO | HAS_ARG | OPT_PERFILE | OPT_OUTPUT, { .func_arg = opt_audio_frames }, diff --git a/tests/fate/ffmpeg.mak b/tests/fate/ffmpeg.mak index d87639c596..a983ace3bd 100644 --- a/tests/fate/ffmpeg.mak +++ b/tests/fate/ffmpeg.mak @@ -117,6 +117,20 @@ fate-ffmpeg-fix_sub_duration: CMD = fmtstdout srt -fix_sub_duration \ -real_time 1 -f lavfi \ -i "movie=$(TARGET_SAMPLES)/sub/Closedcaption_rollup.m2v[out0+subcc]" +# Basic test for fix_sub_duration_heartbeat, which causes a buffered subtitle +# to be pushed out when a video keyframe is received from an encoder. +FATE_SAMPLES_FFMPEG-$(call FILTERDEMDECENCMUX, MOVIE, MPEGVIDEO, \ + MPEG2VIDEO, SUBRIP, SRT, LAVFI_INDEV \ + MPEGVIDEO_PARSER CCAPTION_DECODER \ + MPEG2VIDEO_ENCODER NULL_MUXER PIPE_PROTOCOL) \ + += fate-ffmpeg-fix_sub_duration_heartbeat +fate-ffmpeg-fix_sub_duration_heartbeat: CMD = fmtstdout srt -fix_sub_duration \ + -real_time 1 -f lavfi \ + -i "movie=$(TARGET_SAMPLES)/sub/Closedcaption_rollup.m2v[out0+subcc]" \ + -map 0:v -fix_sub_duration_heartbeat:v:0 \ + -c mpeg2video -b:v 2M -g 30 -sc_threshold 1000000000 \ + -f null - + FATE_STREAMCOPY-$(call REMUX, MP4 MOV, EAC3_DEMUXER) += fate-copy-trac3074 fate-copy-trac3074: CMD = transcode eac3 $(TARGET_SAMPLES)/eac3/csi_miami_stereo_128_spx.eac3\ mp4 "-codec copy -map 0" "-codec copy" diff --git a/tests/ref/fate/ffmpeg-fix_sub_duration_heartbeat b/tests/ref/fate/ffmpeg-fix_sub_duration_heartbeat new file mode 100644 index 0000000000..957a410921 --- /dev/null +++ b/tests/ref/fate/ffmpeg-fix_sub_duration_heartbeat @@ -0,0 +1,48 @@ +1 +00:00:00,968 --> 00:00:01,001 +{\an7}( + +2 +00:00:01,001 --> 00:00:01,168 +{\an7}( + +3 +00:00:01,168 --> 00:00:01,368 +{\an7}( inaudibl + +4 +00:00:01,368 --> 00:00:01,568 +{\an7}( inaudible radio chat + +5 +00:00:01,568 --> 00:00:02,002 +{\an7}( inaudible radio chatter ) + +6 +00:00:02,002 --> 00:00:03,003 +{\an7}( inaudible radio chatter ) + +7 +00:00:03,003 --> 00:00:03,103 +{\an7}( inaudible radio chatter ) + +8 +00:00:03,103 --> 00:00:03,303 +{\an7}( inaudible radio chatter ) +>> + +9 +00:00:03,303 --> 00:00:03,503 +{\an7}( inaudible radio chatter ) +>> Safety rema + +10 +00:00:03,504 --> 00:00:03,704 +{\an7}( inaudible radio chatter ) +>> Safety remains our numb + +11 +00:00:03,704 --> 00:00:04,004 +{\an7}( inaudible radio chatter ) +>> Safety remains our number one +