From patchwork Tue Jun 6 22:22:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marvin Scholz X-Patchwork-Id: 41982 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:c526:b0:117:ac03:c9de with SMTP id gm38csp689971pzb; Tue, 6 Jun 2023 15:24:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7RyM7C1FiE0Svj12e8ED+XiYkfhfHLxK4NJHB9tjXEQOndiVfcizAamEt1ixaUajN22PVK X-Received: by 2002:a17:907:a424:b0:960:ce5:20c0 with SMTP id sg36-20020a170907a42400b009600ce520c0mr3624581ejc.20.1686090253890; Tue, 06 Jun 2023 15:24:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686090253; cv=none; d=google.com; s=arc-20160816; b=XFUrdhNcesj99prAcZHuz6Ft/POuCnc46SP1qweOVPbXDrCASfKbAYc6dloHGr0HWW 11cox/ghtthTZyNYlTESj2TMP2V4cjvTcmp6dxPHwzP3WtAHGH/AQW3Zq+MhIOKfljJR SiqO/W+eCYYKejpjkmgQ2wMEejjFYiUMTg8BQgzSUQXYXK4du6DynyNEfi5TlsD8AQ/b NW2LSbfzdMBXFAIyOjAgaQaqxAy2J6Iv0cFLIXrjo9wbSa0EmglH9jbyJEecff3h8Kaq 8eB7eO8SmgLMaEKYwo4NADte4zyxzcBIve4sGoGpGuLHVyh2qTEgU2dK52ie6irbENPd Re1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=oRkELjvFWVUHIgZS9ifA4J5E5lK8HfRI5OwTz8h70Tw=; b=Urxh3/GKERAhuwumPpZ5vFW4f+g9Opy2GTPRb2W9xcFh8OEAxyhzo+1MIBxqLwd9OO k5H4bu4XOFCApGcFLEDNdrkL5+Eg/k5kFu1/h7gCmmynWGcC/cIdeqjYABHRhi3NQUdO VeEP7C60AC2oGDJCzmfIFwtZmmoXzoQamg6f7XGT0RWtx04P7l2Ih+5CIPlkSb3Pa/yo MKfOzdfCG2a9wrZkZgGiVYC99/FZezA4rD0RCAa6hGcQYq+ed6ZNsorT5qhbu6njGSkn ZE+H/oDFIH0uhadg7ZEbaBOONmC+HtFiXN0VbgE/vzy8T6zB+2k4Czyn0rDO/EGYR5H7 EpQg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=QfIBwUfe; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 17-20020a170906059100b0097871336538si1264989ejn.87.2023.06.06.15.23.48; Tue, 06 Jun 2023 15:24:13 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=QfIBwUfe; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 03EEC68C31C; Wed, 7 Jun 2023 01:23:44 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6B58A68B9D5 for ; Wed, 7 Jun 2023 01:23:37 +0300 (EEST) Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-9745ba45cd1so867611666b.1 for ; Tue, 06 Jun 2023 15:23:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686090216; x=1688682216; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hxux47Kmr5YTDjUfJ6gcFHyMfgkz/NvVvYEF4ZljNAg=; b=QfIBwUfeQgNGJ6le3KtEe7LBBJXIEMKjiQv4n6hFNE4GmQFhLlbirTQMs8PrCwurOC yduKuapt6B7togxywEZw7PwugsRJdTKwCa5H2urElUwhGfyUkuBID/tum8vqoz58ymKQ gN3tKlR/kGgukyIZlLmkR7IMBP2Qrq5JXm1Ri5LMsNVGmyuJsScBXR6mIZPRti/Nr7v0 VbEYNuVwCsCwiTr8Y3CMkMNccKAADTS3XLIAQG/CgYNMILu+NIUKjP6FuQg8QnyBeg2p XHbWFt/cOk3PJsMjGkSJvFxEGryetsC2e04sGF1t23BGfIkGS7veBOEikHHLIR12R286 gGrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686090216; x=1688682216; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hxux47Kmr5YTDjUfJ6gcFHyMfgkz/NvVvYEF4ZljNAg=; b=BC3+GViFQfZPdHMAUwN1Gq0jDfLOzHbYWa8f/E+hTDqpOV0YVFWZ7HR37YRy8yoCli D+fZ6Bf3z4C+DTXZ+ZaU5Qk0EL8U50jw+lYae12v/3wV3iuIbE3Bv7nXmBmgToAHNtz+ qnRLal9xYDOKq9GzR/YZ/JcVc6YQZDg2wwsd6s0a741H5ORimlZJSmMV5gCeVZkblaYd P7h95UsIM5Iljk9PIBmhFjcU3T/Z74jRFeMKQ5VOCQoJyKBxPxks2SDovvtpKYTWlLOu RCwkDHhHwOx9eSr5xk38/aKvpE5cNK6h9f7cM3f3e8u/tuqjQE4e/NsB4cCt3VcaP7X6 4M2Q== X-Gm-Message-State: AC+VfDxVJidAtj/r6cebDL4XIo3BcXa++idtZgpKHj19U8T/L9QkrEIU xCl2kWuxe+mDAU7Q1Gzc7yegyluVD7c= X-Received: by 2002:a17:906:6a0e:b0:973:91a5:bff1 with SMTP id qw14-20020a1709066a0e00b0097391a5bff1mr4634391ejc.68.1686090216259; Tue, 06 Jun 2023 15:23:36 -0700 (PDT) Received: from MBP-von-Marvin.fritz.box (dynamic-2a01-0c22-a967-d100-5963-60a3-e682-252f.c22.pool.telefonica.de. [2a01:c22:a967:d100:5963:60a3:e682:252f]) by smtp.gmail.com with ESMTPSA id f18-20020a170906825200b00965ac1510f8sm6087541ejx.185.2023.06.06.15.23.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jun 2023 15:23:35 -0700 (PDT) From: Marvin Scholz To: ffmpeg-devel@ffmpeg.org Date: Wed, 7 Jun 2023 00:22:59 +0200 Message-Id: <20230606222302.21495-1-epirat07@gmail.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) In-Reply-To: <20230530015854.31900-1-epirat07@gmail.com> References: <20230530015854.31900-1-epirat07@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 1/4] libavfilter: add vf_xfade_vulkan X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marvin Scholz Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: GedfyFYEQhf5 This is an initial version of vf_xfade_vulkan based on vf_xfade_opencl, for now only a subset of transitions are supported. --- Changes to v2: - Fixed activate handling, same as in my patch for the xfade filter - Added all remaining transitions the OpenCL filter supports configure | 1 + libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_xfade_vulkan.c | 499 ++++++++++++++++++++++++++++++++++ 4 files changed, 502 insertions(+) create mode 100644 libavfilter/vf_xfade_vulkan.c diff --git a/configure b/configure index 2992dae283..378cab1f3d 100755 --- a/configure +++ b/configure @@ -3824,6 +3824,7 @@ scale_vulkan_filter_deps="vulkan spirv_compiler" vpp_qsv_filter_deps="libmfx" vpp_qsv_filter_select="qsvvpp" xfade_opencl_filter_deps="opencl" +xfade_vulkan_filter_deps="vulkan spirv_compiler" yadif_cuda_filter_deps="ffnvcodec" yadif_cuda_filter_deps_any="cuda_nvcc cuda_llvm" yadif_videotoolbox_filter_deps="metal corevideo videotoolbox" diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 18935b1616..ff149a3733 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -552,6 +552,7 @@ OBJS-$(CONFIG_XBR_FILTER) += vf_xbr.o OBJS-$(CONFIG_XCORRELATE_FILTER) += vf_convolve.o framesync.o OBJS-$(CONFIG_XFADE_FILTER) += vf_xfade.o OBJS-$(CONFIG_XFADE_OPENCL_FILTER) += vf_xfade_opencl.o opencl.o opencl/xfade.o +OBJS-$(CONFIG_XFADE_VULKAN_FILTER) += vf_xfade_vulkan.o vulkan.o vulkan_filter.o OBJS-$(CONFIG_XMEDIAN_FILTER) += vf_xmedian.o framesync.o OBJS-$(CONFIG_XSTACK_FILTER) += vf_stack.o framesync.o OBJS-$(CONFIG_YADIF_FILTER) += vf_yadif.o yadif_common.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index f1f781101b..6593e4eb83 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -519,6 +519,7 @@ extern const AVFilter ff_vf_xbr; extern const AVFilter ff_vf_xcorrelate; extern const AVFilter ff_vf_xfade; extern const AVFilter ff_vf_xfade_opencl; +extern const AVFilter ff_vf_xfade_vulkan; extern const AVFilter ff_vf_xmedian; extern const AVFilter ff_vf_xstack; extern const AVFilter ff_vf_yadif; diff --git a/libavfilter/vf_xfade_vulkan.c b/libavfilter/vf_xfade_vulkan.c new file mode 100644 index 0000000000..f18a7b33aa --- /dev/null +++ b/libavfilter/vf_xfade_vulkan.c @@ -0,0 +1,499 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/avassert.h" +#include "libavutil/random_seed.h" +#include "libavutil/opt.h" +#include "vulkan_filter.h" +#include "vulkan_spirv.h" +#include "filters.h" +#include "internal.h" + +#define IN_A 0 +#define IN_B 1 +#define IN_NB 2 + +typedef struct XFadeParameters { + float progress; +} XFadeParameters; + +typedef struct XFadeVulkanContext { + FFVulkanContext vkctx; + + int transition; + int64_t duration; + int64_t offset; + + int initialized; + FFVulkanPipeline pl; + FFVkExecPool e; + FFVkQueueFamilyCtx qf; + FFVkSPIRVShader shd; + VkSampler sampler; + + // PTS when the fade should start (in IN_A timebase) + int64_t start_pts; + + // PTS offset between IN_A and IN_B + int64_t inputs_offset_pts; + + // Duration of the transition + int64_t duration_pts; + + // Current PTS of the first input (IN_A) + int64_t pts; + + // If frames are currently just passed through + // unmodified, like before and after the actual + // transition. + int passthrough; + + int status[IN_NB]; +} XFadeVulkanContext; + +enum XFadeTransitions { + FADE, + WIPELEFT, + WIPERIGHT, + NB_TRANSITIONS, +}; + +static const char transition_fade[] = { + C(0, void transition(int idx, ivec2 pos, float progress) ) + C(0, { ) + C(1, vec4 a = texture(a_images[idx], pos); ) + C(1, vec4 b = texture(b_images[idx], pos); ) + C(1, imageStore(output_images[idx], pos, mix(a, b, progress)); ) + C(0, } ) +}; + +static const char transition_wipeleft[] = { + C(0, void transition(int idx, ivec2 pos, float progress) ) + C(0, { ) + C(1, ivec2 size = imageSize(output_images[idx]); ) + C(1, int s = int(size.x * (1.0 - progress)); ) + C(1, vec4 a = texture(a_images[idx], pos); ) + C(1, vec4 b = texture(b_images[idx], pos); ) + C(1, imageStore(output_images[idx], pos, pos.x > s ? b : a); ) + C(0, } ) +}; + +static const char transition_wiperight[] = { + C(0, void transition(int idx, ivec2 pos, float progress) ) + C(0, { ) + C(1, ivec2 size = imageSize(output_images[idx]); ) + C(1, int s = int(size.x * progress); ) + C(1, vec4 a = texture(a_images[idx], pos); ) + C(1, vec4 b = texture(b_images[idx], pos); ) + C(1, imageStore(output_images[idx], pos, pos.x > s ? a : b); ) + C(0, } ) +}; + +static const char* transitions_map[NB_TRANSITIONS] = { + [FADE] = transition_fade, + [WIPELEFT] = transition_wipeleft, + [WIPERIGHT] = transition_wiperight, +}; + +static av_cold int init_vulkan(AVFilterContext *avctx) +{ + int err = 0; + uint8_t *spv_data; + size_t spv_len; + void *spv_opaque = NULL; + XFadeVulkanContext *s = avctx->priv; + FFVulkanContext *vkctx = &s->vkctx; + const int planes = av_pix_fmt_count_planes(s->vkctx.output_format); + FFVkSPIRVShader *shd = &s->shd; + FFVkSPIRVCompiler *spv; + FFVulkanDescriptorSetBinding *desc; + + spv = ff_vk_spirv_init(); + if (!spv) { + av_log(avctx, AV_LOG_ERROR, "Unable to initialize SPIR-V compiler!\n"); + return AVERROR_EXTERNAL; + } + + ff_vk_qf_init(vkctx, &s->qf, VK_QUEUE_COMPUTE_BIT); + RET(ff_vk_exec_pool_init(vkctx, &s->qf, &s->e, s->qf.nb_queues*4, 0, 0, 0, NULL)); + RET(ff_vk_init_sampler(vkctx, &s->sampler, 1, VK_FILTER_NEAREST)); + RET(ff_vk_shader_init(&s->pl, &s->shd, "xfade_compute", + VK_SHADER_STAGE_COMPUTE_BIT, 0)); + + ff_vk_shader_set_compute_sizes(&s->shd, 32, 32, 1); + + desc = (FFVulkanDescriptorSetBinding []) { + { + .name = "a_images", + .type = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, + .dimensions = 2, + .elems = planes, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + .samplers = DUP_SAMPLER(s->sampler), + }, + { + .name = "b_images", + .type = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, + .dimensions = 2, + .elems = planes, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + .samplers = DUP_SAMPLER(s->sampler), + }, + { + .name = "output_images", + .type = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE, + .mem_layout = ff_vk_shader_rep_fmt(s->vkctx.output_format), + .mem_quali = "writeonly", + .dimensions = 2, + .elems = planes, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + }, + }; + + RET(ff_vk_pipeline_descriptor_set_add(vkctx, &s->pl, shd, desc, 3, 0, 0)); + + GLSLC(0, layout(push_constant, std430) uniform pushConstants { ); + GLSLC(1, float progress; ); + GLSLC(0, }; ); + + ff_vk_add_push_constant(&s->pl, 0, sizeof(XFadeParameters), + VK_SHADER_STAGE_COMPUTE_BIT); + + // Add the right transition type function to the shader + GLSLD(transitions_map[s->transition]); + + GLSLC(0, void main() ); + GLSLC(0, { ); + GLSLC(1, ivec2 pos = ivec2(gl_GlobalInvocationID.xy); ); + GLSLF(1, int planes = %i; ,planes); + GLSLC(1, for (int i = 0; i < planes; i++) { ); + GLSLC(2, transition(i, pos, progress); ); + GLSLC(1, } ); + GLSLC(0, } ); + + RET(spv->compile_shader(spv, avctx, shd, &spv_data, &spv_len, "main", + &spv_opaque)); + RET(ff_vk_shader_create(vkctx, shd, spv_data, spv_len, "main")); + + RET(ff_vk_init_compute_pipeline(vkctx, &s->pl, shd)); + RET(ff_vk_exec_pipeline_register(vkctx, &s->e, &s->pl)); + + s->initialized = 1; + +fail: + if (spv_opaque) + spv->free_shader(spv, &spv_opaque); + if (spv) + spv->uninit(&spv); + + return err; +} + +static int xfade_frame(AVFilterContext *avctx, AVFrame *frame_a, AVFrame *frame_b) +{ + int err; + AVFilterLink *outlink = avctx->outputs[0]; + XFadeVulkanContext *s = avctx->priv; + float progress; + + AVFrame *output = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!output) { + err = AVERROR(ENOMEM); + goto fail; + } + + if (!s->initialized) { + AVHWFramesContext *a_fc = (AVHWFramesContext*)frame_a->hw_frames_ctx->data; + AVHWFramesContext *b_fc = (AVHWFramesContext*)frame_b->hw_frames_ctx->data; + if (a_fc->sw_format != b_fc->sw_format) { + av_log(avctx, AV_LOG_ERROR, + "Currently the sw format of the first input needs to match the second!\n"); + return AVERROR(EINVAL); + } + RET(init_vulkan(avctx)); + } + + RET(av_frame_copy_props(output, frame_a)); + output->pts = s->pts; + + progress = av_clipf( + (float)(s->pts - s->start_pts) / s->duration_pts, + 0.f, 1.f); + + RET(ff_vk_filter_process_Nin(&s->vkctx, &s->e, &s->pl, output, + (AVFrame *[]){ frame_a, frame_b }, 2, s->sampler, + &(XFadeParameters){ progress }, sizeof(XFadeParameters))); + + return ff_filter_frame(outlink, output); + +fail: + av_frame_free(&output); + return err; +} + +static int config_props_output(AVFilterLink *outlink) +{ + int err; + AVFilterContext *avctx = outlink->src; + XFadeVulkanContext *s = avctx->priv; + AVFilterLink *inlink_a = avctx->inputs[IN_A]; + AVFilterLink *inlink_b = avctx->inputs[IN_B]; + + if (inlink_a->w != inlink_b->w || inlink_a->h != inlink_b->h) { + av_log(avctx, AV_LOG_ERROR, "First input link %s parameters " + "(size %dx%d) do not match the corresponding " + "second input link %s parameters (size %dx%d)\n", + avctx->input_pads[IN_A].name, inlink_a->w, inlink_a->h, + avctx->input_pads[IN_B].name, inlink_b->w, inlink_b->h); + return AVERROR(EINVAL); + } + + if (inlink_a->time_base.num != inlink_b->time_base.num || + inlink_a->time_base.den != inlink_b->time_base.den) { + av_log(avctx, AV_LOG_ERROR, "First input link %s timebase " + "(%d/%d) does not match the corresponding " + "second input link %s timebase (%d/%d)\n", + avctx->input_pads[IN_A].name, inlink_a->time_base.num, inlink_a->time_base.den, + avctx->input_pads[IN_B].name, inlink_b->time_base.num, inlink_b->time_base.den); + return AVERROR(EINVAL); + } + + s->start_pts = s->inputs_offset_pts = AV_NOPTS_VALUE; + + outlink->time_base = inlink_a->time_base; + outlink->frame_rate = inlink_a->frame_rate; + outlink->sample_aspect_ratio = inlink_a->sample_aspect_ratio; + + if (s->duration) + s->duration_pts = av_rescale_q(s->duration, AV_TIME_BASE_Q, inlink_a->time_base); + RET(ff_vk_filter_config_output(outlink)); + +fail: + return err; +} + +static int forward_frame(XFadeVulkanContext *s, + AVFilterLink *inlink, AVFilterLink *outlink) +{ + int64_t status_pts; + int ret = 0, status; + AVFrame *frame = NULL; + + ret = ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + + if (ret > 0) { + // If we do not have an offset yet, it's because we + // never got a first input. Just offset to 0 + if (s->inputs_offset_pts == AV_NOPTS_VALUE) + s->inputs_offset_pts = -frame->pts; + + // We got a frame, nothing to do other than adjusting the timestamp + frame->pts += s->inputs_offset_pts; + return ff_filter_frame(outlink, frame); + } + + // Forward status with our timestamp + if (ff_inlink_acknowledge_status(inlink, &status, &status_pts)) { + if (s->inputs_offset_pts == AV_NOPTS_VALUE) + s->inputs_offset_pts = -status_pts; + + ff_outlink_set_status(outlink, status, status_pts + s->inputs_offset_pts); + return 0; + } + + // No frame available, request one if needed + if (ff_outlink_frame_wanted(outlink)) + ff_inlink_request_frame(inlink); + + return 0; +} + +static int activate(AVFilterContext *avctx) +{ + XFadeVulkanContext *s = avctx->priv; + AVFilterLink *in_a = avctx->inputs[IN_A]; + AVFilterLink *in_b = avctx->inputs[IN_B]; + AVFilterLink *outlink = avctx->outputs[0]; + int64_t status_pts; + + FF_FILTER_FORWARD_STATUS_BACK_ALL(outlink, avctx); + + // Check if we already transitioned or IN_A ended prematurely, + // in which case just forward the frames from IN_B with adjusted + // timestamps until EOF. + if (s->status[IN_A] && !s->status[IN_B]) + return forward_frame(s, in_b, outlink); + + // We did not finish transitioning yet and the first stream + // did not end either, so check if there are more frames to consume. + if (ff_inlink_check_available_frame(in_a)) { + AVFrame *peeked_frame = ff_inlink_peek_frame(in_a, 0); + s->pts = peeked_frame->pts; + + if (s->start_pts == AV_NOPTS_VALUE) + s->start_pts = + s->pts + av_rescale_q(s->offset, AV_TIME_BASE_Q, in_a->time_base); + + // Check if we are not yet transitioning, in which case + // just request and forward the input frame. + if (s->start_pts > s->pts) { + AVFrame *frame_a = NULL; + s->passthrough = 1; + ff_inlink_consume_frame(in_a, &frame_a); + return ff_filter_frame(outlink, frame_a); + } + s->passthrough = 0; + + // We are transitioning, so we need a frame from IN_B + if (ff_inlink_check_available_frame(in_b)) { + int ret; + AVFrame *frame_a = NULL, *frame_b = NULL; + ff_inlink_consume_frame(avctx->inputs[IN_A], &frame_a); + ff_inlink_consume_frame(avctx->inputs[IN_B], &frame_b); + + // Calculate PTS offset to first input + if (s->inputs_offset_pts == AV_NOPTS_VALUE) + s->inputs_offset_pts = s->pts - frame_b->pts; + + // Check if we finished transitioning, in which case we + // report back EOF to IN_A as it is no longer needed. + if (s->pts - s->start_pts > s->duration_pts) { + s->status[IN_A] = AVERROR_EOF; + ff_inlink_set_status(in_a, AVERROR_EOF); + s->passthrough = 1; + } + ret = xfade_frame(avctx, frame_a, frame_b); + av_frame_free(&frame_a); + av_frame_free(&frame_b); + return ret; + } + + // We did not get a frame from IN_B, check its status. + if (ff_inlink_acknowledge_status(in_b, &s->status[IN_B], &status_pts)) { + // We should transition, but IN_B is EOF so just report EOF output now. + ff_outlink_set_status(outlink, s->status[IN_B], s->pts); + return 0; + } + + // We did not get a frame for IN_B but no EOF either, so just request more. + if (ff_outlink_frame_wanted(outlink)) { + ff_inlink_request_frame(in_b); + return 0; + } + } + + // We did not get a frame from IN_A, check its status. + if (ff_inlink_acknowledge_status(in_a, &s->status[IN_A], &status_pts)) { + // No more frames from IN_A, do not report EOF though, we will just + // forward the IN_B frames in the next activate calls. + s->passthrough = 1; + ff_filter_set_ready(avctx, 100); + return 0; + } + + // We have no frames yet from IN_A and no EOF, so request some. + if (ff_outlink_frame_wanted(outlink)) { + ff_inlink_request_frame(in_a); + return 0; + } + + return FFERROR_NOT_READY; +} + +static av_cold void uninit(AVFilterContext *avctx) +{ + XFadeVulkanContext *s = avctx->priv; + FFVulkanContext *vkctx = &s->vkctx; + FFVulkanFunctions *vk = &vkctx->vkfn; + + ff_vk_exec_pool_free(vkctx, &s->e); + ff_vk_pipeline_free(vkctx, &s->pl); + ff_vk_shader_free(vkctx, &s->shd); + + if (s->sampler) + vk->DestroySampler(vkctx->hwctx->act_dev, s->sampler, + vkctx->hwctx->alloc); + + ff_vk_uninit(&s->vkctx); + + s->initialized = 0; +} + +static AVFrame *get_video_buffer(AVFilterLink *inlink, int w, int h) +{ + XFadeVulkanContext *s = inlink->dst->priv; + + return s->passthrough ? + ff_null_get_video_buffer (inlink, w, h) : + ff_default_get_video_buffer(inlink, w, h); +} + +#define OFFSET(x) offsetof(XFadeVulkanContext, x) +#define FLAGS (AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM) + +static const AVOption xfade_vulkan_options[] = { + { "transition", "set cross fade transition", OFFSET(transition), AV_OPT_TYPE_INT, {.i64=FADE}, 0, NB_TRANSITIONS-1, FLAGS, "transition" }, + { "fade", "fade transition", 0, AV_OPT_TYPE_CONST, {.i64=FADE}, 0, 0, FLAGS, "transition" }, + { "wipeleft", "wipe left transition", 0, AV_OPT_TYPE_CONST, {.i64=WIPELEFT}, 0, 0, FLAGS, "transition" }, + { "wiperight", "wipe right transition", 0, AV_OPT_TYPE_CONST, {.i64=WIPERIGHT}, 0, 0, FLAGS, "transition" }, + { "duration", "set cross fade duration", OFFSET(duration), AV_OPT_TYPE_DURATION, {.i64=1000000}, 0, 60000000, FLAGS }, + { "offset", "set cross fade start relative to first input stream", OFFSET(offset), AV_OPT_TYPE_DURATION, {.i64=0}, INT64_MIN, INT64_MAX, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(xfade_vulkan); + +static const AVFilterPad xfade_vulkan_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .get_buffer.video = &get_video_buffer, + .config_props = &ff_vk_filter_config_input, + }, + { + .name = "xfade", + .type = AVMEDIA_TYPE_VIDEO, + .get_buffer.video = &get_video_buffer, + .config_props = &ff_vk_filter_config_input, + }, +}; + +static const AVFilterPad xfade_vulkan_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = &config_props_output, + }, +}; + +const AVFilter ff_vf_xfade_vulkan = { + .name = "xfade_vulkan", + .description = NULL_IF_CONFIG_SMALL("Cross fade one video with another video."), + .priv_size = sizeof(XFadeVulkanContext), + .init = &ff_vk_filter_init, + .uninit = &uninit, + .activate = &activate, + FILTER_INPUTS(xfade_vulkan_inputs), + FILTER_OUTPUTS(xfade_vulkan_outputs), + FILTER_SINGLE_PIXFMT(AV_PIX_FMT_VULKAN), + .priv_class = &xfade_vulkan_class, + .flags_internal = FF_FILTER_FLAG_HWFRAME_AWARE, + .flags = AVFILTER_FLAG_HWDEVICE, +};