From patchwork Fri Jun 2 20:12:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marvin Scholz X-Patchwork-Id: 41961 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:c51c:b0:10c:5e6f:955f with SMTP id gm28csp1505075pzb; Fri, 2 Jun 2023 13:12:46 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4xWhSPXxuLnFMUB6Ym8ifMvvw4inihMRW3T0u1g0lAzR7W8Ec0P8Gw9xeQHvbqk2mJW2pF X-Received: by 2002:a17:907:d1b:b0:966:61b3:f630 with SMTP id gn27-20020a1709070d1b00b0096661b3f630mr12154652ejc.9.1685736765735; Fri, 02 Jun 2023 13:12:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685736765; cv=none; d=google.com; s=arc-20160816; b=sEORZVROZtWezk+V0RkFfoxSTUlCrT+XUqn/YoioI6SJ5ofJWRVIG6KIo70nAS9MsI 0e6P0a+R9lzDDxrMTuRqExADAn7AEtdrgcRiBlR5ZtmWldLB7Fb9RBOS0uTpN20C3WBm JP+thfkG001tKtprs0apCGc0qtmDF0w3dOnVYAMj/v8Pc2CQpyjVMHu7L+IQpAWmpSL8 9i+PdJs9fBqt/Y6UntD84Gj14lDObJSVCO2VODroM5xkhcktR5pYTc4DXZInmhi54VlP oXlJfKU2iQ75HvQV+rS7A+eU3CMbBydqKaIwTgWLl4AsZspse+SlBW99J0Tf0sRZXttR K5Zg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=nEfOjslTk0gPSNLwTcUEXevNp/81w+b5CotZDdOPQQE=; b=dG0AiysYSYG3iaUTN7PX+apQSQOXnBRyMzepfr1dqN2Nq/m0/CwSVmkGXaJgdhuBHJ q2CjD49ZCfUOaFatFBOuZQ/yO2pwMd4PLMKWmWIryLkfqrlc0MzqhXeQPyQObD4fU+Kd Q6x+lChY0OFZSgqzGQf9G+nncqYEAacoDyGzJWUnC3G6K4KvS+X8+HC2uYAxDTgfwQ4p eHvIUoCJvwt8kCTh7Fwi4Yhdl/n7bnLakF55FcvYGubNqVj2uXJ66fg436N9COXscRKH PUYFKFPvWTaB9e0zhxTtMvsnbiBoMjgw6GHr38w+AdZ9XCnh8v3qvhD9P4qqB1gdhUM1 GbFg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=ORfTjMdX; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d7-20020a170906304700b0096b12175757si1230378ejd.954.2023.06.02.13.12.44; Fri, 02 Jun 2023 13:12:45 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=ORfTjMdX; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4D33668C35C; Fri, 2 Jun 2023 23:12:41 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id DF60E68C31F for ; Fri, 2 Jun 2023 23:12:34 +0300 (EEST) Received: by mail-lf1-f47.google.com with SMTP id 2adb3069b0e04-4f3bb395e69so3440269e87.2 for ; Fri, 02 Jun 2023 13:12:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685736754; x=1688328754; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JOAj27pybTRmHEedVndvtx4O9aLNpS7/7dM3WS/7ykY=; b=ORfTjMdXfw7hpqVjAZPJGSH9/27ys/ctjJVM4uP/ep2G748QbXH//c4MI7mVIVKUCQ YAObz96cGGh9seEUndQfbkaWg4OCX27E2CQLHOLPEnp7Pe7z5DK2gOKKRZNb4viY7dU+ VOKS/p4EBeDTIuud6SpK4e7zdANZqt6CONnKYQEKfkNwhnr5SO1e93LvHEqIgd1jpQri cX2QtYeRwEDvdZlkNtloGKhWEC7AZn//ABUlOy53vQUFEoHAV9jr3nMlTtYsFKgEH/0x Zf69146WuqjzGhxMJmLx8f54dKnR8tEmusUg9bWBVbUt3RnmalkWey0FixqEFwlOPowf 28eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685736754; x=1688328754; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JOAj27pybTRmHEedVndvtx4O9aLNpS7/7dM3WS/7ykY=; b=GPA6Cf2ssINQTC/1IktWe/5/rJLZjoYItNpujEoqlEjKzce27e+k4P7FbdxqwNtMkB RthbRIDyUKtDSctKIAGF9qbNZ+9WTcSvqtNI1Fsi5F1A+h58HtRS7DzdmBz+4vKMP0An gIVdCiF+/3Nnwi7L/RjJA9ddby3AlZ4LcfemrnEWVlid2qgonUyeyFhnyCTyXFgGF/sD +7ZVumYahBzMGfy0q+D7otbMbMYQqCmv5+zsAafaxDDy2PahGD2W/9tLtxYKqpoxLofC 3TTyhbtvHrvltIlCBW5XkNpoMucz1cypphyk+rV888/Sbo70PxP6hBE2TX3Uh9a6BOSz FrtQ== X-Gm-Message-State: AC+VfDyLzP760PGtw/pyQSrUrbJMqHWEz6TuMEsbV+1oTqQLFC5CI9kF UBXTSrfbEZaIZsxfhmyeZDYgp6Y1ukM= X-Received: by 2002:a2e:834b:0:b0:2a2:47a8:728b with SMTP id l11-20020a2e834b000000b002a247a8728bmr661987ljh.13.1685736753635; Fri, 02 Jun 2023 13:12:33 -0700 (PDT) Received: from MBP-von-Marvin.fritz.box (dynamic-2a01-0c22-ac94-0c00-a061-c2d2-c3c3-fb48.c22.pool.telefonica.de. [2a01:c22:ac94:c00:a061:c2d2:c3c3:fb48]) by smtp.gmail.com with ESMTPSA id e10-20020a170906374a00b0096f803afbe3sm1141308ejc.66.2023.06.02.13.12.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 13:12:32 -0700 (PDT) From: Marvin Scholz To: ffmpeg-devel@ffmpeg.org Date: Fri, 2 Jun 2023 22:12:26 +0200 Message-Id: <20230602201226.25565-1-epirat07@gmail.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) In-Reply-To: <20230602161042.18365-1-epirat07@gmail.com> References: <20230602161042.18365-1-epirat07@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2] lavfi/vf_xfade: rewrite activate inputs handling X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marvin Scholz Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: IWYDJWJWAnq2 The old code was not properly handling a bunch of edge-cases with streams terminating earlier and also did not properly report back EOF to the first input. This fixes at least one case where the filter could stop doing anything: ffmpeg -f lavfi -i "color=blue:d=10" -f lavfi -i "color=aqua:d=0" -filter_complex "[0][1]xfade=duration=2:offset=0:transition=wiperight" -f null - --- Changes to v1: - Fix an issue discovered by Paul where the timestmap offset was incorrectly calculated leading to one frame with identical PTS as the previous one. - Removed accidentally introduced trailing spaces libavfilter/vf_xfade.c | 212 ++++++++++++++++++++++++----------------- 1 file changed, 124 insertions(+), 88 deletions(-) diff --git a/libavfilter/vf_xfade.c b/libavfilter/vf_xfade.c index a13a7db627..92f5139ec5 100644 --- a/libavfilter/vf_xfade.c +++ b/libavfilter/vf_xfade.c @@ -95,14 +95,23 @@ typedef struct XFadeContext { int depth; int is_rgb; + // PTS when the fade should start (in first inputs timebase) + int64_t start_pts; + + // PTS offset between first and second input + int64_t inputs_offset_pts; + + // Duration of the transition int64_t duration_pts; - int64_t offset_pts; - int64_t first_pts; - int64_t last_pts; + + // Current PTS of the first input int64_t pts; - int xfade_is_over; - int need_second; - int eof[2]; + + // If frames are currently just passed through unmodified, + // like before and after the actual transition. + int passthrough; + + int status[2]; AVFrame *xf[2]; int max_value; uint16_t black[4]; @@ -1935,12 +1944,10 @@ static int config_output(AVFilterLink *outlink) s->white[0] = s->white[3] = s->max_value; s->white[1] = s->white[2] = s->is_rgb ? s->max_value : s->max_value / 2; - s->first_pts = s->last_pts = s->pts = AV_NOPTS_VALUE; + s->start_pts = s->inputs_offset_pts = AV_NOPTS_VALUE; if (s->duration) s->duration_pts = av_rescale_q(s->duration, AV_TIME_BASE_Q, outlink->time_base); - if (s->offset) - s->offset_pts = av_rescale_q(s->offset, AV_TIME_BASE_Q, outlink->time_base); switch (s->transition) { case CUSTOM: s->transitionf = s->depth <= 8 ? custom8_transition : custom16_transition; break; @@ -2037,7 +2044,7 @@ static int xfade_frame(AVFilterContext *ctx, AVFrame *a, AVFrame *b) { XFadeContext *s = ctx->priv; AVFilterLink *outlink = ctx->outputs[0]; - float progress = av_clipf(1.f - ((float)(s->pts - s->first_pts - s->offset_pts) / s->duration_pts), 0.f, 1.f); + float progress = av_clipf(1.f - ((float)(s->pts - s->start_pts) / s->duration_pts), 0.f, 1.f); ThreadData td; AVFrame *out; @@ -2055,99 +2062,128 @@ static int xfade_frame(AVFilterContext *ctx, AVFrame *a, AVFrame *b) return ff_filter_frame(outlink, out); } -static int xfade_activate(AVFilterContext *ctx) +static int forward_frame(XFadeContext *s, + AVFilterLink *inlink, AVFilterLink *outlink) { - XFadeContext *s = ctx->priv; - AVFilterLink *outlink = ctx->outputs[0]; - AVFrame *in = NULL; + int64_t status_pts; int ret = 0, status; - int64_t pts; + AVFrame *frame = NULL; - FF_FILTER_FORWARD_STATUS_BACK_ALL(outlink, ctx); + ret = ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; - if (s->xfade_is_over) { - if (!s->eof[0]) { - if (ff_inlink_queued_frames(ctx->inputs[0]) > 0) { - ret = ff_inlink_consume_frame(ctx->inputs[0], &in); - if (ret > 0) - av_frame_free(&in); - } - ff_inlink_set_status(ctx->inputs[0], AVERROR_EOF); - s->eof[0] = 1; - } - ret = ff_inlink_consume_frame(ctx->inputs[1], &in); - if (ret < 0) { - return ret; - } else if (ret > 0) { - in->pts = (in->pts - s->last_pts) + s->pts; - return ff_filter_frame(outlink, in); - } else if (ff_inlink_acknowledge_status(ctx->inputs[1], &status, &pts)) { - ff_outlink_set_status(outlink, status, s->pts); - return 0; - } else if (!ret) { - if (ff_outlink_frame_wanted(outlink)) - ff_inlink_request_frame(ctx->inputs[1]); - return 0; - } + if (ret > 0) { + // If we do not have an offset yet, it's because we + // never got a first input. Just offset to 0 + if (s->inputs_offset_pts == AV_NOPTS_VALUE) + s->inputs_offset_pts = -frame->pts; + + // We got a frame, nothing to do other than adjusting the timestamp + frame->pts += s->inputs_offset_pts; + return ff_filter_frame(outlink, frame); } - if (ff_inlink_queued_frames(ctx->inputs[0]) > 0) { - s->xf[0] = ff_inlink_peek_frame(ctx->inputs[0], 0); - if (s->xf[0]) { - if (s->first_pts == AV_NOPTS_VALUE) { - s->first_pts = s->xf[0]->pts; - } - s->pts = s->xf[0]->pts; - if (s->first_pts + s->offset_pts > s->xf[0]->pts) { - s->xf[0] = NULL; - s->need_second = 0; - ff_inlink_consume_frame(ctx->inputs[0], &in); - return ff_filter_frame(outlink, in); + // Forward status with our timestamp + if (ff_inlink_acknowledge_status(inlink, &status, &status_pts)) { + if (s->inputs_offset_pts == AV_NOPTS_VALUE) + s->inputs_offset_pts = -status_pts; + + ff_outlink_set_status(outlink, status, status_pts + s->inputs_offset_pts); + return 0; + } + + // No frame available, request one if needed + if (ff_outlink_frame_wanted(outlink)) + ff_inlink_request_frame(inlink); + + return 0; +} + +static int xfade_activate(AVFilterContext *avctx) +{ + XFadeContext *s = avctx->priv; + AVFilterLink *in_a = avctx->inputs[0]; + AVFilterLink *in_b = avctx->inputs[1]; + AVFilterLink *outlink = avctx->outputs[0]; + int64_t status_pts; + + FF_FILTER_FORWARD_STATUS_BACK_ALL(outlink, avctx); + + // Check if we already transitioned or first input ended prematurely, + // in which case just forward the frames from second input with adjusted + // timestamps until EOF. + if (s->status[0] && !s->status[1]) + return forward_frame(s, in_b, outlink); + + // We did not finish transitioning yet and the first stream + // did not end either, so check if there are more frames to consume. + if (ff_inlink_check_available_frame(in_a)) { + AVFrame *peeked_frame = ff_inlink_peek_frame(in_a, 0); + s->pts = peeked_frame->pts; + + if (s->start_pts == AV_NOPTS_VALUE) + s->start_pts = + s->pts + av_rescale_q(s->offset, AV_TIME_BASE_Q, in_a->time_base); + + // Check if we are not yet transitioning, in which case + // just request and forward the input frame. + if (s->start_pts > s->pts) { + s->passthrough = 1; + ff_inlink_consume_frame(in_a, &s->xf[0]); + return ff_filter_frame(outlink, s->xf[0]); + } + s->passthrough = 0; + + // We are transitioning, so we need a frame from second input + if (ff_inlink_check_available_frame(in_b)) { + int ret; + ff_inlink_consume_frame(avctx->inputs[0], &s->xf[0]); + ff_inlink_consume_frame(avctx->inputs[1], &s->xf[1]); + + // Calculate PTS offset to first input + if (s->inputs_offset_pts == AV_NOPTS_VALUE) + s->inputs_offset_pts = s->pts - s->xf[1]->pts; + + // Check if we finished transitioning, in which case we + // report back EOF to first input as it is no longer needed. + if (s->pts - s->start_pts > s->duration_pts) { + s->status[0] = AVERROR_EOF; + ff_inlink_set_status(in_a, AVERROR_EOF); + s->passthrough = 1; } + ret = xfade_frame(avctx, s->xf[0], s->xf[1]); + av_frame_free(&s->xf[0]); + av_frame_free(&s->xf[1]); + return ret; + } - s->need_second = 1; + // We did not get a frame from second input, check its status. + if (ff_inlink_acknowledge_status(in_b, &s->status[1], &status_pts)) { + // We should transition, but second input is EOF so just report EOF output now. + ff_outlink_set_status(outlink, s->status[1], s->pts); + return 0; } - } - if (s->xf[0] && ff_inlink_queued_frames(ctx->inputs[1]) > 0) { - ff_inlink_consume_frame(ctx->inputs[0], &s->xf[0]); - ff_inlink_consume_frame(ctx->inputs[1], &s->xf[1]); - - s->last_pts = s->xf[1]->pts; - s->pts = s->xf[0]->pts; - if (s->xf[0]->pts - (s->first_pts + s->offset_pts) > s->duration_pts) - s->xfade_is_over = 1; - ret = xfade_frame(ctx, s->xf[0], s->xf[1]); - av_frame_free(&s->xf[0]); - av_frame_free(&s->xf[1]); - return ret; + // We did not get a frame for second input but no EOF either, so just request more. + if (ff_outlink_frame_wanted(outlink)) { + ff_inlink_request_frame(in_b); + return 0; + } } - if (ff_inlink_queued_frames(ctx->inputs[0]) > 0 && - ff_inlink_queued_frames(ctx->inputs[1]) > 0) { - ff_filter_set_ready(ctx, 100); + // We did not get a frame from first input, check its status. + if (ff_inlink_acknowledge_status(in_a, &s->status[0], &status_pts)) { + // No more frames from first input, do not report EOF though, we will just + // forward the second input frames in the next activate calls. + s->passthrough = 1; + ff_filter_set_ready(avctx, 100); return 0; } + // We have no frames yet from first input and no EOF, so request some. if (ff_outlink_frame_wanted(outlink)) { - if (!s->eof[0] && ff_outlink_get_status(ctx->inputs[0])) { - s->eof[0] = 1; - s->xfade_is_over = 1; - } - if (!s->eof[1] && ff_outlink_get_status(ctx->inputs[1])) { - s->eof[1] = 1; - } - if (!s->eof[0] && !s->xf[0] && ff_inlink_queued_frames(ctx->inputs[0]) == 0) - ff_inlink_request_frame(ctx->inputs[0]); - if (!s->eof[1] && (s->need_second || s->eof[0]) && ff_inlink_queued_frames(ctx->inputs[1]) == 0) - ff_inlink_request_frame(ctx->inputs[1]); - if (s->eof[0] && s->eof[1] && ( - ff_inlink_queued_frames(ctx->inputs[0]) <= 0 && - ff_inlink_queued_frames(ctx->inputs[1]) <= 0)) { - ff_outlink_set_status(outlink, AVERROR_EOF, AV_NOPTS_VALUE); - } else if (s->xfade_is_over) { - ff_filter_set_ready(ctx, 100); - } + ff_inlink_request_frame(in_a); return 0; } @@ -2158,7 +2194,7 @@ static AVFrame *get_video_buffer(AVFilterLink *inlink, int w, int h) { XFadeContext *s = inlink->dst->priv; - return s->xfade_is_over || !s->need_second ? + return s->passthrough ? ff_null_get_video_buffer (inlink, w, h) : ff_default_get_video_buffer(inlink, w, h); }