From patchwork Fri Sep 20 06:23:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lynne X-Patchwork-Id: 51676 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:d154:0:b0:48e:c0f8:d0de with SMTP id bt20csp791192vqb; Fri, 20 Sep 2024 00:39:13 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWuegEY+UX1LKR5KcpD+zCDm6TsKzMHXUZipFAi/aGFrJpihVqQC1/rPM5bIUWDgZU2QW13i5EgJ4zDpS7suJgP@gmail.com X-Google-Smtp-Source: AGHT+IGe/9GYe0WL09KhyeNzSmBSX4Enoalseg6Hb0hPQJUoa2g3CuCRr5ny6Eb5rRnqZyJD3dMa X-Received: by 2002:a05:6512:6d2:b0:535:69ee:9717 with SMTP id 2adb3069b0e04-536ac2d675emr1192697e87.3.1726817953608; Fri, 20 Sep 2024 00:39:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1726817953; cv=none; d=google.com; s=arc-20240605; b=OD7KfL5BxISvUuHpNuVaGJmZ8SGxMXNxRIw+AiuFxuDlWMK6CxkSVNpjSWgfGxqTK2 mfKfgYW3NmtOWwoGjBtoSGQd1cS6gupL+rDA5WUQNgOad9sa6prUJdG5Fxkm0id7aCNj reMLpiIq1Yt8H4K1q6Ca3TDTFgAb8jz3UtpoiTFBFwwaC/9gqvNDz3n5UQJZztIZWGVm 0iHmz92QYzYOiesa96/2gu5Pp5fh0e1BkwP88CPz0/JPAUALm4eUVnYB4tdGZksAZCG3 6ihmk7klJD0cRH7x3A6BitRuPYHWTTk64i5mL986WzfNtQCa9lS9KaYJe8IH76lFhLrU tGrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:delivered-to; bh=SpY0YKg50gY2Tieh4stkzMDbkRBBzQJ74k5T14r0oyQ=; fh=nenT92/WZoU6unXd3J6UhGUdod4piddKfVtctNBOh6k=; b=D9Od4Zyjzf8PzmoZsGeDEaDKHWVuH6aU27kB7/AIlDLKKDq7xhJoqrltbeBoipDEj/ RuhQQbJKP5WO6yrdcsGjEXdRmaSQLcHmIiOO3KbSq+GORdIi/Sd6ylW2ApvU7+YpiEvs MlV+DaGIBFtiy6fW1AsZlMNwpfys0C/YwfndzAvUQtjD/l5ftxoeXhtJbRLtwZ3kG9fE gGvFeF78v/ElLph4jo1jmpQTrzg+8fzZAvbrmMMJvioPLHCvCXJUdxmmsL+hBSjgsBWc 5AF48Sm68H56kWsB6X8UO8tJyQFyGTeshcEfDoBNUR2AJW0ZIUsVxMk/F0WcWPgdhLte Q6JQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-5368708dd47si4909478e87.230.2024.09.20.00.39.13; Fri, 20 Sep 2024 00:39:13 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id F278968DCD0; Fri, 20 Sep 2024 09:23:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vidala.lynne.ee (vidala.pars.ee [116.203.72.101]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1A1A968D80D for ; Fri, 20 Sep 2024 09:23:39 +0300 (EEST) To: ffmpeg-devel@ffmpeg.org Date: Fri, 20 Sep 2024 08:23:32 +0200 Message-ID: <20240920062337.933413-3-dev@lynne.ee> X-Mailer: git-send-email 2.45.2.753.g447d99e1c3b In-Reply-To: <20240920062337.933413-1-dev@lynne.ee> References: <20240920062337.933413-1-dev@lynne.ee> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/3] hwcontext_vulkan: add support for implicit DRM synchronization X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Lynne via ffmpeg-devel From: Lynne Reply-To: FFmpeg development discussions and patches Cc: Lynne Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: H7sRCgydhQf5 More recent kernel versions allow for users to extract a sync_file handle from a DMA-BUF, which can then be imported into Vulkan as a binary semaphore. This finally allows for synchronization between Vulkan and DMA-BUF images, such as those from screen capture software, or VAAPI, avoiding any corruption artifacts. This is done fully asynchronously, where we use the kernel's given binary semaphores as a dependency to increment the image's usual VkSemaphores we allocate. The old imported binary semaphores are cleaned up after execution as usual. In the future, hwcontext_drm should receive support for explicitly synchronized images as well, which would make the synchronization more robust and portable. --- libavutil/hwcontext_vulkan.c | 145 ++++++++++++++++++++++++++++++++--- libavutil/vulkan_functions.h | 1 + 2 files changed, 135 insertions(+), 11 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 1de08c6181..aba686e7ca 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -61,6 +61,11 @@ #include "hwcontext_drm.h" #endif +#if HAVE_LINUX_DMA_BUF_H +#include +#include +#endif + #if CONFIG_CUDA #include "hwcontext_cuda_internal.h" #include "cuda_check.h" @@ -2842,7 +2847,6 @@ static int vulkan_map_from_drm_frame_desc(AVHWFramesContext *hwfc, AVVkFrame **f VulkanDevicePriv *p = ctx->hwctx; AVVulkanDeviceContext *hwctx = &p->p; FFVulkanFunctions *vk = &p->vkctx.vkfn; - VulkanFramesPriv *fp = hwfc->hwctx; const AVDRMFrameDescriptor *desc = (AVDRMFrameDescriptor *)src->data[0]; VkBindImageMemoryInfo bind_info[AV_DRM_MAX_PLANES]; VkBindImagePlaneMemoryInfo plane_info[AV_DRM_MAX_PLANES]; @@ -2990,12 +2994,7 @@ static int vulkan_map_from_drm_frame_desc(AVHWFramesContext *hwfc, AVVkFrame **f goto fail; } - /* We'd import a semaphore onto the one we created using - * vkImportSemaphoreFdKHR but unfortunately neither DRM nor VAAPI - * offer us anything we could import and sync with, so instead - * just signal the semaphore we created. */ - - f->queue_family[i] = p->nb_img_qfs > 1 ? VK_QUEUE_FAMILY_IGNORED : p->img_qfs[0]; + f->queue_family[i] = VK_QUEUE_FAMILY_EXTERNAL; f->layout[i] = create_info.initialLayout; f->access[i] = 0x0; f->sem_value[i] = 0; @@ -3097,10 +3096,6 @@ static int vulkan_map_from_drm_frame_desc(AVHWFramesContext *hwfc, AVVkFrame **f goto fail; } - err = prepare_frame(hwfc, &fp->compute_exec, f, PREP_MODE_EXTERNAL_IMPORT); - if (err) - goto fail; - *frame = f; return 0; @@ -3111,6 +3106,130 @@ fail: return err; } +static int vulkan_map_from_drm_frame_sync(AVHWFramesContext *hwfc, AVFrame *dst, + const AVFrame *src, int flags) +{ + int err; + VkResult ret; + AVHWDeviceContext *ctx = hwfc->device_ctx; + VulkanDevicePriv *p = ctx->hwctx; + VulkanFramesPriv *fp = hwfc->hwctx; + AVVulkanDeviceContext *hwctx = &p->p; + FFVulkanFunctions *vk = &p->vkctx.vkfn; + + const AVDRMFrameDescriptor *desc = (AVDRMFrameDescriptor *)src->data[0]; + +#ifdef DMA_BUF_IOCTL_EXPORT_SYNC_FILE + if (p->vkctx.extensions & FF_VK_EXT_EXTERNAL_FD_SEM) { + VkCommandBuffer cmd_buf; + FFVkExecContext *exec; + VkImageMemoryBarrier2 img_bar[AV_NUM_DATA_POINTERS]; + VkSemaphore drm_sync_sem[AV_DRM_MAX_PLANES] = { 0 }; + int nb_img_bar = 0; + + for (int i = 0; i < desc->nb_objects; i++) { + VkSemaphoreTypeCreateInfo sem_type_info = { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_TYPE_CREATE_INFO, + .semaphoreType = VK_SEMAPHORE_TYPE_BINARY, + }; + VkSemaphoreCreateInfo sem_spawn = { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_CREATE_INFO, + .pNext = &sem_type_info, + }; + VkImportSemaphoreFdInfoKHR import_info; + struct dma_buf_export_sync_file implicit_fd_info = { + .flags = DMA_BUF_SYNC_READ, + .fd = -1, + }; + + if (ioctl(desc->objects[i].fd, DMA_BUF_IOCTL_EXPORT_SYNC_FILE, + &implicit_fd_info)) { + err = AVERROR(errno); + av_log(hwctx, AV_LOG_ERROR, "Failed to retrieve implicit DRM sync file: %s\n", + av_err2str(err)); + for (; i >= 0; i--) + vk->DestroySemaphore(hwctx->act_dev, drm_sync_sem[i], hwctx->alloc); + return err; + } + + ret = vk->CreateSemaphore(hwctx->act_dev, &sem_spawn, + hwctx->alloc, &drm_sync_sem[i]); + if (ret != VK_SUCCESS) { + av_log(hwctx, AV_LOG_ERROR, "Failed to create semaphore: %s\n", + ff_vk_ret2str(ret)); + err = AVERROR_EXTERNAL; + for (; i >= 0; i--) + vk->DestroySemaphore(hwctx->act_dev, drm_sync_sem[i], hwctx->alloc); + return err; + } + + import_info = (VkImportSemaphoreFdInfoKHR) { + .sType = VK_STRUCTURE_TYPE_IMPORT_SEMAPHORE_FD_INFO_KHR, + .handleType = VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_SYNC_FD_BIT, + .flags = VK_SEMAPHORE_IMPORT_TEMPORARY_BIT, + .semaphore = drm_sync_sem[i], + .fd = implicit_fd_info.fd, + }; + + ret = vk->ImportSemaphoreFdKHR(hwctx->act_dev, &import_info); + if (ret != VK_SUCCESS) { + av_log(hwctx, AV_LOG_ERROR, "Failed to import semaphore: %s\n", + ff_vk_ret2str(ret)); + err = AVERROR_EXTERNAL; + for (; i >= 0; i--) + vk->DestroySemaphore(hwctx->act_dev, drm_sync_sem[i], hwctx->alloc); + return err; + } + } + + exec = ff_vk_exec_get(&fp->compute_exec); + cmd_buf = exec->buf; + + ff_vk_exec_start(&p->vkctx, exec); + + /* Ownership of semaphores is passed */ + err = ff_vk_exec_add_dep_bool_sem(&p->vkctx, exec, + drm_sync_sem, desc->nb_objects, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, 1); + if (err < 0) + return err; + + err = ff_vk_exec_add_dep_frame(&p->vkctx, exec, dst, + VK_PIPELINE_STAGE_2_NONE, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT); + if (err < 0) + return err; + + ff_vk_frame_barrier(&p->vkctx, exec, dst, img_bar, &nb_img_bar, + VK_PIPELINE_STAGE_2_NONE, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT, + VK_IMAGE_LAYOUT_GENERAL, + VK_QUEUE_FAMILY_IGNORED); + + vk->CmdPipelineBarrier2(cmd_buf, &(VkDependencyInfo) { + .sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, + .pImageMemoryBarriers = img_bar, + .imageMemoryBarrierCount = nb_img_bar, + }); + + err = ff_vk_exec_submit(&p->vkctx, exec); + if (err < 0) + return err; + } else +#endif + { + AVVkFrame *f = (AVVkFrame *)dst->data[0]; + av_log(hwctx, AV_LOG_WARNING, "No support for synchronization when importing DMA-BUFs, " + "image may be corrupted.\n"); + err = prepare_frame(hwfc, &fp->compute_exec, f, PREP_MODE_EXTERNAL_IMPORT); + if (err) + return err; + } + + return 0; +} + static int vulkan_map_from_drm(AVHWFramesContext *hwfc, AVFrame *dst, const AVFrame *src, int flags) { @@ -3130,6 +3249,10 @@ static int vulkan_map_from_drm(AVHWFramesContext *hwfc, AVFrame *dst, if (err < 0) goto fail; + err = vulkan_map_from_drm_frame_sync(hwfc, dst, src, flags); + if (err < 0) + return err; + av_log(hwfc, AV_LOG_DEBUG, "Mapped DRM object to Vulkan!\n"); return 0; diff --git a/libavutil/vulkan_functions.h b/libavutil/vulkan_functions.h index d4697ec34c..6aeaf4e79a 100644 --- a/libavutil/vulkan_functions.h +++ b/libavutil/vulkan_functions.h @@ -114,6 +114,7 @@ typedef enum FFVulkanExtensions { \ /* Semaphores */ \ MACRO(1, 1, FF_VK_EXT_EXTERNAL_FD_SEM, GetSemaphoreFdKHR) \ + MACRO(1, 1, FF_VK_EXT_EXTERNAL_FD_SEM, ImportSemaphoreFdKHR) \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, CreateSemaphore) \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, WaitSemaphores) \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, DestroySemaphore) \