From patchwork Fri Aug 31 12:14:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 10191 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:12c4:0:0:0:0:0 with SMTP id 65-v6csp692822jap; Fri, 31 Aug 2018 05:20:51 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZAtHhlOYxrEiyRWolZO1grviCuvK1gsURsH71vsLMAvnco6HXr/q9W5k6jFIm0xrlyZ7qx X-Received: by 2002:a1c:ed1a:: with SMTP id l26-v6mr4715658wmh.61.1535718050931; Fri, 31 Aug 2018 05:20:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535718050; cv=none; d=google.com; s=arc-20160816; b=Fjk3P1lng1gQ0gkmCB5g6dMt4nksX3qx0bZ971DhgYBhmmPHzKfN6zVFOryR9ERaTR m/lnUOrOaEtchDfwQQSHtzVB+DJcSgQAB0+6yFTOw44Bj6CmVxBQ30Hj73BOSaPfdRho G59LNBSPj3sX2dtvNK45p4iCCm+52K7A2wFVzTVdyvLxUU52eXwlx4sbr+ODfBMP/C74 VT1zxx+KGmpYdaUMEjKE3PLkEFj4uBCDRi+g+q4IuPnYURujat9/gEnWgI4mxP3Vx1cy aHGYLyU0jLuW0ukz5XfaV0gESoNnAoTtFnbrZ0Gl0vlvkaEPduOk/m/WcB27qv9QJZd5 bhzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject:to :message-id:date:from:mime-version:dkim-signature:delivered-to :arc-authentication-results; bh=UqcZbkLHoyzLI0Hsc22l9f1p8zg0YFWOy4kQD3VqD8E=; b=VIFaOCW2mE7TGbJ/ewh1wIRz29yquT8HEQh74b8GpLcOOFx3md0s4Xr01vaHGEAT6Y pmkkXi1Hc5duyuRPybvpGww/4S418pGA/C5N3k8Ue8ug+T+QDLQEVV8YQlzAjzF1bUE0 RaAODGcJ5QUeKBrghwmk4V9aqjKhLAxAfe6WC56Y6YlsJSSN+veyrOrR/o6rHHnwiMER huzwmg46U2xftTyLYZsO0p7KwTIrct1mYnbaJmKjMIl0XfPeTb+rlOvEiMC1+1grtLui dEWhh9Z5sNptarTbReoqvV6NxAk2FkeRdiECC6t3CizS3G72laAJwf/DJh9/ycwdaG2B bB2Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=B1brxXvA; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id w100-v6si8970880wrc.198.2018.08.31.05.20.50; Fri, 31 Aug 2018 05:20:50 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=B1brxXvA; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 318E9689D47; Fri, 31 Aug 2018 15:20:44 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BF1D7689BA7 for ; Fri, 31 Aug 2018 15:20:37 +0300 (EEST) Received: by mail-lf1-f41.google.com with SMTP id r4-v6so9803808lff.12 for ; Fri, 31 Aug 2018 05:20:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=BhDFliOCvzaLfOpydXhsAS0ongALsll0eoYdCxIAncc=; b=B1brxXvAf4SA/QX3zMafAc1x+LMrhYArstALonHeSlo8+/F1aELn2oTR+g3L1wvQet 803BlFQLKWMj7KfPtNzWPmmqt4Q9GN4pAnHfwApJAiiclyPgpUGB4WCccCJK6GaMv7Gc SNrfMtzqi3aogBdl+NCHjAylJRKZ4AbB6f3v+hVbv6nLGEA5YAdOfHFOxtEFp2v/GKfs wFYAFFoeG/NLYLhFUm5IJ5uk23WZZU1hQVstt/rRMUjUWworTXS4hPlXam9Ky9YzgglE xvaihSOO3zr1CK3hq11NDqBlrlFVGBMejVDcAL13jk4RQ5mlzE7bWY+VxIMjm0egQgmm OhuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=BhDFliOCvzaLfOpydXhsAS0ongALsll0eoYdCxIAncc=; b=YOiVZf1ZyVqbh6gPuDB9TiGO7T/ddGzZpdJM+rgHlo+MIEQgumSiWwFdLPm5BwZo6k xtLNi3C3N6/V/FPSrV0RwIgE7ArG41rZTJe3gBG42i4M6y4n50bm11U+51oGz59zrXhP Z+Ri8o2dgkw5oXV8FNHMx52Lqy8+BnCErDgYvGQFjTFBjc4e+5flGiE/6rw0Jr0X/+pk nBmhjiRtYhGoPM50kiq4I0oeht/hP0focO/xPaMbsR2uJUcsOGNG1w50pEEeCGUyUS/n i6UHWx1rwGnnnlUYlarofVZiN8fTmtivbY6oAV7jRVjmNy6q8LyD6DkCzsE9H4Og0U0r 8ekw== X-Gm-Message-State: APzg51BTZVjPT7LOuht5tho5BwuVV7XsPuwKxyD6zCV4A03QviF+MOzd eB57owUHaychjfY5UkwChwXbXRyxhiT5SRn/86PS+j/E X-Received: by 2002:a19:ead4:: with SMTP id y81-v6mr10104633lfi.110.1535717671187; Fri, 31 Aug 2018 05:14:31 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a2e:9914:0:0:0:0:0 with HTTP; Fri, 31 Aug 2018 05:14:30 -0700 (PDT) From: =?UTF-8?B?SmFuIEVrc3Ryw7Zt?= Date: Fri, 31 Aug 2018 15:14:30 +0300 Message-ID: To: FFmpeg development discussions and patches Subject: [FFmpeg-devel] [PATCH 1/2] ffmpeg: handle flushing and non-flushing separately in sub2video X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" The initial fix for sub2video flushing during filter chain restarts seemed to add additional buffering into the filter chain, which in case of files did work, but when handling live streams would cause premature failure. This fix reverts the timestamp filtering, and separates flushing from normal heartbeat functionality by adding an explicit parameter. From 9ed9d80535a1e6a2a70faef296fd8d3cbfde8e55 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan=20Ekstr=C3=B6m?= Date: Thu, 26 Jul 2018 19:07:38 +0300 Subject: [PATCH 1/2] ffmpeg: handle flushing and non-flushing separately in sub2video This stops accidental flushes from happening in sub2video in case the filter chain gets a re-configuration/re-init. Heartbeats should not cause a flush and additional buffering in the filter chain. Properly fixes filter chain flush handling, as previously the filter chain would keep buffering frames. Additionally, document the various stages of sub2video. --- fftools/ffmpeg.c | 57 +++++++++++++++++++++++++++++++++++------ fftools/ffmpeg.h | 2 +- fftools/ffmpeg_filter.c | 2 +- 3 files changed, 51 insertions(+), 10 deletions(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index 934dc71a74..758e3839a3 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -237,7 +237,7 @@ static void sub2video_push_ref(InputStream *ist, int64_t pts) } } -void sub2video_update(InputStream *ist, AVSubtitle *sub) +void sub2video_update(InputStream *ist, AVSubtitle *sub, int flush) { AVFrame *frame = ist->sub2video.frame; int8_t *dst; @@ -254,9 +254,47 @@ void sub2video_update(InputStream *ist, AVSubtitle *sub) AV_TIME_BASE_Q, ist->st->time_base); num_rects = sub->num_rects; } else { - pts = ist->sub2video.end_pts; - end_pts = INT64_MAX; num_rects = 0; + + if (flush) { + /* Flush of the sub2video filter chain was requested. + Utilize previous end_pts as the pts, and set end_pts to + the filter chain flush value (INT64_MAX). */ + pts = ist->sub2video.end_pts; + end_pts = INT64_MAX; + + av_log(ist->dec_ctx, AV_LOG_DEBUG, + "nullptr AVSubtitle (flush): utilized values: " + "pts=%"PRId64", end_pts=%"PRId64"\n", + pts, end_pts); + } else { + /* We do not have an active sub-picture and this was not a flush. + We will be sending a blank heartbeat frame to the filter + chain. */ + + if (ist->sub2video.last_pts != AV_NOPTS_VALUE && + ist->sub2video.last_pts < INT64_MAX) { + /* if the last PTS is not invalid (it's not unknown/unset or + the flush value), use the last PTS as the start point + for this filter chain heartbeat frame. */ + pts = ist->sub2video.last_pts; + } else { + /* If the last sub2video PTS is invalid, use the provided + sub2video frame PTS (set to the current input + timestamp by sub2video_heartbeat) */ + pts = frame->pts; + } + + /* FIXME: decide if pts+1 is good enough for this, currently set + for .2 seconds */ + end_pts = pts + av_rescale_q(2*1000LL, AV_TIME_BASE_Q, + ist->st->time_base); + + av_log(ist->dec_ctx, AV_LOG_DEBUG, + "nullptr AVSubtitle (no flush): utilized values: " + "pts=%"PRId64", end_pts=%"PRId64"\n", + pts, end_pts); + } } if (sub2video_get_blank_frame(ist) < 0) { av_log(ist->dec_ctx, AV_LOG_ERROR, @@ -291,9 +329,12 @@ static void sub2video_heartbeat(InputStream *ist, int64_t pts) /* do not send the heartbeat frame if the subtitle is already ahead */ if (pts2 <= ist2->sub2video.last_pts) continue; - if (pts2 >= ist2->sub2video.end_pts || - (!ist2->sub2video.frame->data[0] && ist2->sub2video.end_pts < INT64_MAX)) - sub2video_update(ist2, NULL); + if (pts2 >= ist2->sub2video.end_pts || !ist2->sub2video.frame->data[0]) { + /* if we do not have a subtitle to show for a specific time, + add the current stream time as the fall-back PTS */ + ist2->sub2video.frame->pts = pts2; + sub2video_update(ist2, NULL, 0); + } for (j = 0, nb_reqs = 0; j < ist2->nb_filters; j++) nb_reqs += av_buffersrc_get_nb_failed_requests(ist2->filters[j]->filter); if (nb_reqs) @@ -307,7 +348,7 @@ static void sub2video_flush(InputStream *ist) int ret; if (ist->sub2video.end_pts < INT64_MAX) - sub2video_update(ist, NULL); + sub2video_update(ist, NULL, 1); for (i = 0; i < ist->nb_filters; i++) { ret = av_buffersrc_add_frame(ist->filters[i]->filter, NULL); if (ret != AVERROR_EOF && ret < 0) @@ -2517,7 +2558,7 @@ static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output, return ret; if (ist->sub2video.frame) { - sub2video_update(ist, &subtitle); + sub2video_update(ist, &subtitle, 0); } else if (ist->nb_filters) { if (!ist->sub2video.sub_queue) ist->sub2video.sub_queue = av_fifo_alloc(8 * sizeof(AVSubtitle)); diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index eb1eaf6363..d2aba21ed1 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -646,7 +646,7 @@ int filtergraph_is_simple(FilterGraph *fg); int init_simple_filtergraph(InputStream *ist, OutputStream *ost); int init_complex_filtergraph(FilterGraph *fg); -void sub2video_update(InputStream *ist, AVSubtitle *sub); +void sub2video_update(InputStream *ist, AVSubtitle *sub, int flush); int ifilter_parameters_from_frame(InputFilter *ifilter, const AVFrame *frame); diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c index 6518d50870..4cb81ea41f 100644 --- a/fftools/ffmpeg_filter.c +++ b/fftools/ffmpeg_filter.c @@ -1161,7 +1161,7 @@ int configure_filtergraph(FilterGraph *fg) while (av_fifo_size(ist->sub2video.sub_queue)) { AVSubtitle tmp; av_fifo_generic_read(ist->sub2video.sub_queue, &tmp, sizeof(tmp), NULL); - sub2video_update(ist, &tmp); + sub2video_update(ist, &tmp, 0); avsubtitle_free(&tmp); } } -- 2.17.1