From patchwork Fri Jun 2 16:10:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marvin Scholz X-Patchwork-Id: 41960 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:c51c:b0:10c:5e6f:955f with SMTP id gm28csp1364097pzb; Fri, 2 Jun 2023 09:11:00 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7cz8Q1sI9yyeTJA16fhw32vYtxS8DXjOCxYypqzrRS9UVppn3Hgx/QAA7YCMB8JjBo3qsd X-Received: by 2002:a17:907:7207:b0:965:ffda:b9d2 with SMTP id dr7-20020a170907720700b00965ffdab9d2mr11481622ejc.11.1685722260182; Fri, 02 Jun 2023 09:11:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685722260; cv=none; d=google.com; s=arc-20160816; b=EfOgUuCcoFg4Sm3i9aSvb1s478KYZC03FhOhyhR/CvlLBV1ohozwh2Vn9v42sbUka0 UfU9LOxRYN546PEHIKq7Er/vJG7ErNRcdjlKxO2W0rAi8otDytvMPlVTtHOWfHK2jQj2 KSDSKz20pR3E/IHMGcUeOPSxBzlpXvTvmvq30T2GaicWwR1hAWCZTAzYte7qFTd5ozQP jvKWFtj8FDaz5nol4ZMLSEnmo8nv6aBryp5/BZx5C+WqQz8d3Vdd0+hKTllZ9KwPa3pm bUYyq6yYr4KxPElUXGqr8e8QocFQDa+PEJWnLcawMXVVksF6z4mbxTJIKqZ01JoewsiT rh+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=PaXtOAwsFmGHb72QMVjsNsfh9U6fJB0t0LjiDTXy0Lg=; b=Ag66kzolwX+w0rGTJMnlbjmjq+t81txI2W7ltWUjg5sIyKTdW6S0Y2pN4B8jY57eZf xHEPTqu5xfV1C4XECTDtZB2x6Mxd/XRZcChdm1CJm39nq6wstloScrhRuditK6OTRn/c gxhrH2QE1bl2kwCpphgcfVdKDsI/k7HAbMruuQIaIhqLygbXb/gW2CIHEcpNN11DI9AV t/pYVwIH+4VPFR+pSbQBt/BugbQ1oUeVi7vrR7R0SzuqBlB7uJEf5w55rzCSDMG/J+WG LA1G1uMnvhRdeKopPTn98rK7UAVAZzQNmtwInybflDY0NA1KmvYrM/yGROrvnPWVgI1y 1Uag== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=lMALBeu0; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id su24-20020a17090703d800b0096996a5ea2asi1010542ejb.397.2023.06.02.09.10.59; Fri, 02 Jun 2023 09:11:00 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=lMALBeu0; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 73E8368C31E; Fri, 2 Jun 2023 19:10:56 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 56D1868C277 for ; Fri, 2 Jun 2023 19:10:50 +0300 (EEST) Received: by mail-ed1-f52.google.com with SMTP id 4fb4d7f45d1cf-5149c76f4dbso3247434a12.1 for ; Fri, 02 Jun 2023 09:10:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685722249; x=1688314249; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=aiCQ3NHKpzJluT038Wmfq1JaqHc8do0fLWUTceyrdNA=; b=lMALBeu0InftRcUVCpnmJ4OfhLmK6Tf7PqJQfrGlb2j+z1JHjdQ+90SgBeScg5TSmQ OVCaW4JKORTyKGDsLwSMBrwAn5lP4xUAEskvsjhzo2QuQpkfwjym84nJmXm0LqCwdza5 kp5Yqz8HWBlDM9wndjDwYxvUrYtm4ifKu+AHWlYITYOjkS6sFxB+bMgAw9UONqQDXuvH 33ZpQFjocdm7VcRk53k3rw0f+4DM9TStssAZRXaOFgdieQuuZJ1t+VI6XKv4WD3NqElC cMavO/BgFT3Npr8Qh3zINmZW6LCzjMLDcG/jjsEZ6rNxZlUmG37npoa+xg7KWl/aoDRJ 99iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685722249; x=1688314249; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=aiCQ3NHKpzJluT038Wmfq1JaqHc8do0fLWUTceyrdNA=; b=QR3jFE4HWsWrN2hRHAnvxbmfTeJXH7EbRJDwONUWpuZzfz4cYJI10xP8Jnk5z3h68P XbjqWvBCHrBIXafR1eWe19AhO9X21wtFm6bvXmY8dA5NVFnWueBlnUy9fuh/DcnhasKD 3edJJsyqolo0tVMnMmzOlvsC8CKvzbvp2tTA8PyEae3ksBCtf41r1KrE34cw1DKma+xu 6T3J2PvSkH7ALuiveLY66ndcz7dhyBrvipK4V7vVlOBCFHu1FmahBVZlBk71eMMB68Dn K6wd9Tyc1vxxxJUk40oYpXU0DGZxigZncPN5A1NMbnOkp9OUh09K3gijUemsYLJdu3P5 PDwg== X-Gm-Message-State: AC+VfDye7+3UqSLtahmIMVr2C+RCEjuP6IMAx0nmKjjuTfby8Lf98gt2 9yWh2VojoXTzeLbIztzaD10+rz+v0oo= X-Received: by 2002:a17:907:7da5:b0:966:4d75:4a44 with SMTP id oz37-20020a1709077da500b009664d754a44mr10587782ejc.24.1685722249330; Fri, 02 Jun 2023 09:10:49 -0700 (PDT) Received: from MBP-von-Marvin.fritz.box (dynamic-2a01-0c22-ac94-0c00-a061-c2d2-c3c3-fb48.c22.pool.telefonica.de. [2a01:c22:ac94:c00:a061:c2d2:c3c3:fb48]) by smtp.gmail.com with ESMTPSA id y22-20020a170906071600b0096f603dc8ddsm910861ejb.142.2023.06.02.09.10.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 09:10:48 -0700 (PDT) From: Marvin Scholz To: ffmpeg-devel@ffmpeg.org Date: Fri, 2 Jun 2023 18:10:42 +0200 Message-Id: <20230602161042.18365-1-epirat07@gmail.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] lavfi/vf_xfade: rewrite activate inputs handling X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marvin Scholz Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: /VGCaqmJZoh5 The old code was not properly handling a bunch of edge-cases with streams terminating earlier and also did not properly report back EOF to the first input. This fixes at least one case where the filter could stop doing anything: ffmpeg -f lavfi -i "color=blue:d=10" -f lavfi -i "color=aqua:d=0" -filter_complex "[0][1]xfade=duration=2:offset=0:transition=wiperight" -f null - --- libavfilter/vf_xfade.c | 208 ++++++++++++++++++++++++----------------- 1 file changed, 120 insertions(+), 88 deletions(-) diff --git a/libavfilter/vf_xfade.c b/libavfilter/vf_xfade.c index a13a7db627..f98ff81918 100644 --- a/libavfilter/vf_xfade.c +++ b/libavfilter/vf_xfade.c @@ -95,14 +95,23 @@ typedef struct XFadeContext { int depth; int is_rgb; + // PTS when the fade should start (in first inputs timebase) + int64_t start_pts; + + // PTS offset between first and second input + int64_t inputs_offset_pts; + + // Duration of the transition int64_t duration_pts; - int64_t offset_pts; - int64_t first_pts; - int64_t last_pts; + + // Current PTS of the first input int64_t pts; - int xfade_is_over; - int need_second; - int eof[2]; + + // If frames are currently just passed through unmodified, + // like before and after the actual transition. + int passthrough; + + int status[2]; AVFrame *xf[2]; int max_value; uint16_t black[4]; @@ -1935,12 +1944,10 @@ static int config_output(AVFilterLink *outlink) s->white[0] = s->white[3] = s->max_value; s->white[1] = s->white[2] = s->is_rgb ? s->max_value : s->max_value / 2; - s->first_pts = s->last_pts = s->pts = AV_NOPTS_VALUE; + s->start_pts = s->inputs_offset_pts = AV_NOPTS_VALUE; if (s->duration) s->duration_pts = av_rescale_q(s->duration, AV_TIME_BASE_Q, outlink->time_base); - if (s->offset) - s->offset_pts = av_rescale_q(s->offset, AV_TIME_BASE_Q, outlink->time_base); switch (s->transition) { case CUSTOM: s->transitionf = s->depth <= 8 ? custom8_transition : custom16_transition; break; @@ -2037,7 +2044,7 @@ static int xfade_frame(AVFilterContext *ctx, AVFrame *a, AVFrame *b) { XFadeContext *s = ctx->priv; AVFilterLink *outlink = ctx->outputs[0]; - float progress = av_clipf(1.f - ((float)(s->pts - s->first_pts - s->offset_pts) / s->duration_pts), 0.f, 1.f); + float progress = av_clipf(1.f - ((float)(s->pts - s->start_pts) / s->duration_pts), 0.f, 1.f); ThreadData td; AVFrame *out; @@ -2055,99 +2062,124 @@ static int xfade_frame(AVFilterContext *ctx, AVFrame *a, AVFrame *b) return ff_filter_frame(outlink, out); } -static int xfade_activate(AVFilterContext *ctx) +static int forward_frame(XFadeContext *s, + AVFilterLink *inlink, AVFilterLink *outlink) { - XFadeContext *s = ctx->priv; - AVFilterLink *outlink = ctx->outputs[0]; - AVFrame *in = NULL; + int64_t status_pts; int ret = 0, status; - int64_t pts; + AVFrame *frame = NULL; - FF_FILTER_FORWARD_STATUS_BACK_ALL(outlink, ctx); + ret = ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; - if (s->xfade_is_over) { - if (!s->eof[0]) { - if (ff_inlink_queued_frames(ctx->inputs[0]) > 0) { - ret = ff_inlink_consume_frame(ctx->inputs[0], &in); - if (ret > 0) - av_frame_free(&in); - } - ff_inlink_set_status(ctx->inputs[0], AVERROR_EOF); - s->eof[0] = 1; - } - ret = ff_inlink_consume_frame(ctx->inputs[1], &in); - if (ret < 0) { - return ret; - } else if (ret > 0) { - in->pts = (in->pts - s->last_pts) + s->pts; - return ff_filter_frame(outlink, in); - } else if (ff_inlink_acknowledge_status(ctx->inputs[1], &status, &pts)) { - ff_outlink_set_status(outlink, status, s->pts); - return 0; - } else if (!ret) { - if (ff_outlink_frame_wanted(outlink)) - ff_inlink_request_frame(ctx->inputs[1]); - return 0; - } + if (ret > 0) { + // Calculate PTS offset to first input + if (s->inputs_offset_pts == AV_NOPTS_VALUE) + s->inputs_offset_pts = s->pts - frame->pts; + + // We got a frame, nothing to do other than adjusting the timestamp + frame->pts += s->inputs_offset_pts; + return ff_filter_frame(outlink, frame); } - if (ff_inlink_queued_frames(ctx->inputs[0]) > 0) { - s->xf[0] = ff_inlink_peek_frame(ctx->inputs[0], 0); - if (s->xf[0]) { - if (s->first_pts == AV_NOPTS_VALUE) { - s->first_pts = s->xf[0]->pts; - } - s->pts = s->xf[0]->pts; - if (s->first_pts + s->offset_pts > s->xf[0]->pts) { - s->xf[0] = NULL; - s->need_second = 0; - ff_inlink_consume_frame(ctx->inputs[0], &in); - return ff_filter_frame(outlink, in); + // Forward status with our timestamp + if (ff_inlink_acknowledge_status(inlink, &status, &status_pts)) { + // Calculate PTS offset to first input + if (s->inputs_offset_pts == AV_NOPTS_VALUE) + s->inputs_offset_pts = s->pts - status_pts; + + ff_outlink_set_status(outlink, status, status_pts + s->inputs_offset_pts); + return 0; + } + + // No frame available, request one if needed + if (ff_outlink_frame_wanted(outlink)) + ff_inlink_request_frame(inlink); + + return 0; +} + +static int xfade_activate(AVFilterContext *avctx) +{ + XFadeContext *s = avctx->priv; + AVFilterLink *in_a = avctx->inputs[0]; + AVFilterLink *in_b = avctx->inputs[1]; + AVFilterLink *outlink = avctx->outputs[0]; + int64_t status_pts; + + FF_FILTER_FORWARD_STATUS_BACK_ALL(outlink, avctx); + + // Check if we already transitioned or first input ended prematurely, + // in which case just forward the frames from second input with adjusted + // timestamps until EOF. + if (s->status[0] && !s->status[1]) + return forward_frame(s, in_b, outlink); + + // We did not finish transitioning yet and the first stream + // did not end either, so check if there are more frames to consume. + if (ff_inlink_check_available_frame(in_a)) { + AVFrame *peeked_frame = ff_inlink_peek_frame(in_a, 0); + s->pts = peeked_frame->pts; + + if (s->start_pts == AV_NOPTS_VALUE) + s->start_pts = + s->pts + av_rescale_q(s->offset, AV_TIME_BASE_Q, in_a->time_base); + + // Check if we are not yet transitioning, in which case + // just request and forward the input frame. + if (s->start_pts > s->pts) { + s->passthrough = 1; + ff_inlink_consume_frame(in_a, &s->xf[0]); + return ff_filter_frame(outlink, s->xf[0]); + } + s->passthrough = 0; + + // We are transitioning, so we need a frame from second input + if (ff_inlink_check_available_frame(in_b)) { + int ret; + ff_inlink_consume_frame(avctx->inputs[0], &s->xf[0]); + ff_inlink_consume_frame(avctx->inputs[1], &s->xf[1]); + + // Check if we finished transitioning, in which case we + // report back EOF to first input as it is no longer needed. + if (s->pts - s->start_pts > s->duration_pts) { + s->status[0] = AVERROR_EOF; + ff_inlink_set_status(in_a, AVERROR_EOF); + s->passthrough = 1; } + ret = xfade_frame(avctx, s->xf[0], s->xf[1]); + av_frame_free(&s->xf[0]); + av_frame_free(&s->xf[1]); + return ret; + } - s->need_second = 1; + // We did not get a frame from second input, check its status. + if (ff_inlink_acknowledge_status(in_b, &s->status[1], &status_pts)) { + // We should transition, but second input is EOF so just report EOF output now. + ff_outlink_set_status(outlink, s->status[1], s->pts); + return 0; } - } - if (s->xf[0] && ff_inlink_queued_frames(ctx->inputs[1]) > 0) { - ff_inlink_consume_frame(ctx->inputs[0], &s->xf[0]); - ff_inlink_consume_frame(ctx->inputs[1], &s->xf[1]); - - s->last_pts = s->xf[1]->pts; - s->pts = s->xf[0]->pts; - if (s->xf[0]->pts - (s->first_pts + s->offset_pts) > s->duration_pts) - s->xfade_is_over = 1; - ret = xfade_frame(ctx, s->xf[0], s->xf[1]); - av_frame_free(&s->xf[0]); - av_frame_free(&s->xf[1]); - return ret; + // We did not get a frame for second input but no EOF either, so just request more. + if (ff_outlink_frame_wanted(outlink)) { + ff_inlink_request_frame(in_b); + return 0; + } } - if (ff_inlink_queued_frames(ctx->inputs[0]) > 0 && - ff_inlink_queued_frames(ctx->inputs[1]) > 0) { - ff_filter_set_ready(ctx, 100); + // We did not get a frame from first input, check its status. + if (ff_inlink_acknowledge_status(in_a, &s->status[0], &status_pts)) { + // No more frames from first input, do not report EOF though, we will just + // forward the second input frames in the next activate calls. + s->passthrough = 1; + ff_filter_set_ready(avctx, 100); return 0; } + // We have no frames yet from first input and no EOF, so request some. if (ff_outlink_frame_wanted(outlink)) { - if (!s->eof[0] && ff_outlink_get_status(ctx->inputs[0])) { - s->eof[0] = 1; - s->xfade_is_over = 1; - } - if (!s->eof[1] && ff_outlink_get_status(ctx->inputs[1])) { - s->eof[1] = 1; - } - if (!s->eof[0] && !s->xf[0] && ff_inlink_queued_frames(ctx->inputs[0]) == 0) - ff_inlink_request_frame(ctx->inputs[0]); - if (!s->eof[1] && (s->need_second || s->eof[0]) && ff_inlink_queued_frames(ctx->inputs[1]) == 0) - ff_inlink_request_frame(ctx->inputs[1]); - if (s->eof[0] && s->eof[1] && ( - ff_inlink_queued_frames(ctx->inputs[0]) <= 0 && - ff_inlink_queued_frames(ctx->inputs[1]) <= 0)) { - ff_outlink_set_status(outlink, AVERROR_EOF, AV_NOPTS_VALUE); - } else if (s->xfade_is_over) { - ff_filter_set_ready(ctx, 100); - } + ff_inlink_request_frame(in_a); return 0; } @@ -2158,7 +2190,7 @@ static AVFrame *get_video_buffer(AVFilterLink *inlink, int w, int h) { XFadeContext *s = inlink->dst->priv; - return s->xfade_is_over || !s->need_second ? + return s->passthrough ? ff_null_get_video_buffer (inlink, w, h) : ff_default_get_video_buffer(inlink, w, h); }