From patchwork Wed Aug 7 21:33:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lynne X-Patchwork-Id: 50940 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:d7b2:0:b0:489:2eb3:e4c4 with SMTP id dc18csp654871vqb; Wed, 7 Aug 2024 14:51:14 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCV5RBiRSPavOALlwdF8JdCNKV/gjmS/FIs13BV4ufZ8Bkk8KuaQCfHVuQacXh/6hExq5z1PbMnWzsEr122Ilgy3v07hSXpEio0zzQ== X-Google-Smtp-Source: AGHT+IFYOF/YQG5frvKU4Y11SwJ0h0sHk9eX9b+j8KzOo0gf6C5/qX4cvBW3QcrS7RRzfHmlpLjl X-Received: by 2002:a05:6512:3c82:b0:52e:751a:a172 with SMTP id 2adb3069b0e04-530bb3bc040mr14378180e87.49.1723067474137; Wed, 07 Aug 2024 14:51:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1723067474; cv=none; d=google.com; s=arc-20160816; b=fW5/XfEW7evYJTAKkoWu06Bc3OiI/hhIgI0ySaKVnSKyVo8fXfWI/zXudLgkxMcu5s SHkn9zKoU6jTN37wdm14PIBZNIyyqq7P6vtTKQdaimZoSf/PlwwN6eOF7+9qIyCWFDkO dvJM515SNCcJMJfUn7fqhYpMowNq906WCYve/jzpd1ZYtjc4laOqiGy0AwS77zfGYZNt d/K/ReGcFh2uEI/C8akFON6Gwf0p7ljhDEVFB2Ys7QtArIKtozKcZh8FZ+PP3W0LslLk 0xvG/PFK96fKgddns6ozk43XrPCgXTGH0jYvKTae9NWAhG+2Bm/RsuR0uaBbbRAdUwd3 sjkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:delivered-to; bh=az4J7gyW8QuPSsbgonZIk3L1fJzXU88mScvitETf6uU=; fh=nenT92/WZoU6unXd3J6UhGUdod4piddKfVtctNBOh6k=; b=eLaPcV/m/jpTbJGtAHzVmW9Y4nizJJLj592cFA1j6Pk+NRHyJT+11EYSSw4+0FGuMT d9B7/VFQDAhV2wZtPfWzqI7PJUg1yLwAozyaZifKzZOUuHF7IKsZYUH7GcKe8gXELYhF bH6rRNVi0aDUTDIqZIyBHokC4rb2cEfaoX9V3XsndjVvpkswGqxoApRVLMGs6ec4cLUT Q4mUdlLpgDmMdhGEXw5xSSxSAV1lESHTKGb3dsmDlCDSpG6Q6vEwoGmshs4kx1BCJk81 I5zK61E/zLb4Y3lBAcOPyjxCUSmsyjtdyXwexusIgrRtSGWfm5DeBrGq89Jb9CZenJ/s pVOw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-530de48f00bsi591484e87.230.2024.08.07.14.51.13; Wed, 07 Aug 2024 14:51:14 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C977768DB8A; Thu, 8 Aug 2024 00:34:11 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vidala.lynne.ee (vidala.pars.ee [116.203.72.101]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 813B568DB7E for ; Thu, 8 Aug 2024 00:34:07 +0300 (EEST) To: ffmpeg-devel@ffmpeg.org Date: Wed, 7 Aug 2024 23:33:29 +0200 Message-ID: <20240807213347.917235-4-dev@lynne.ee> X-Mailer: git-send-email 2.45.2.753.g447d99e1c3b In-Reply-To: <20240807213347.917235-1-dev@lynne.ee> References: <20240807213347.917235-1-dev@lynne.ee> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 04/13] hwcontext_vulkan: initialize optical flow queues if available X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Lynne via ffmpeg-devel From: Lynne Reply-To: FFmpeg development discussions and patches Cc: Lynne Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: tEtStBGyhSh8 Lets us implement FPS conversion. --- libavutil/hwcontext_vulkan.c | 18 +++++++++++++++--- libavutil/vulkan.c | 6 +++++- libavutil/vulkan.h | 1 + libavutil/vulkan_functions.h | 8 ++++++++ libavutil/vulkan_loader.h | 1 + 5 files changed, 30 insertions(+), 4 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 5baf68660a..c81fc95af2 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -105,6 +105,7 @@ typedef struct VulkanDevicePriv { VkPhysicalDeviceDescriptorBufferFeaturesEXT desc_buf_features; VkPhysicalDeviceShaderAtomicFloatFeaturesEXT atomic_float_features; VkPhysicalDeviceCooperativeMatrixFeaturesKHR coop_matrix_features; + VkPhysicalDeviceOpticalFlowFeaturesNV optical_flow_features; /* Queues */ pthread_mutex_t **qf_mutex; @@ -429,6 +430,7 @@ static const VulkanOptExtension optional_device_exts[] = { { VK_EXT_PHYSICAL_DEVICE_DRM_EXTENSION_NAME, FF_VK_EXT_DEVICE_DRM }, { VK_EXT_SHADER_ATOMIC_FLOAT_EXTENSION_NAME, FF_VK_EXT_ATOMIC_FLOAT }, { VK_KHR_COOPERATIVE_MATRIX_EXTENSION_NAME, FF_VK_EXT_COOP_MATRIX }, + { VK_NV_OPTICAL_FLOW_EXTENSION_NAME, FF_VK_EXT_OPTICAL_FLOW }, /* Imports/exports */ { VK_KHR_EXTERNAL_MEMORY_FD_EXTENSION_NAME, FF_VK_EXT_EXTERNAL_FD_MEMORY }, @@ -1127,13 +1129,14 @@ static int setup_queue_families(AVHWDeviceContext *ctx, VkDeviceCreateInfo *cd) av_log(ctx, AV_LOG_VERBOSE, "Queue families:\n"); for (int i = 0; i < num; i++) { - av_log(ctx, AV_LOG_VERBOSE, " %i:%s%s%s%s%s%s%s (queues: %i)\n", i, + av_log(ctx, AV_LOG_VERBOSE, " %i:%s%s%s%s%s%s%s%s (queues: %i)\n", i, ((qf[i].queueFamilyProperties.queueFlags) & VK_QUEUE_GRAPHICS_BIT) ? " graphics" : "", ((qf[i].queueFamilyProperties.queueFlags) & VK_QUEUE_COMPUTE_BIT) ? " compute" : "", ((qf[i].queueFamilyProperties.queueFlags) & VK_QUEUE_TRANSFER_BIT) ? " transfer" : "", ((qf[i].queueFamilyProperties.queueFlags) & VK_QUEUE_VIDEO_ENCODE_BIT_KHR) ? " encode" : "", ((qf[i].queueFamilyProperties.queueFlags) & VK_QUEUE_VIDEO_DECODE_BIT_KHR) ? " decode" : "", ((qf[i].queueFamilyProperties.queueFlags) & VK_QUEUE_SPARSE_BINDING_BIT) ? " sparse" : "", + ((qf[i].queueFamilyProperties.queueFlags) & VK_QUEUE_OPTICAL_FLOW_BIT_NV) ? " optical_flow" : "", ((qf[i].queueFamilyProperties.queueFlags) & VK_QUEUE_PROTECTED_BIT) ? " protected" : "", qf[i].queueFamilyProperties.queueCount); @@ -1177,6 +1180,7 @@ static int setup_queue_families(AVHWDeviceContext *ctx, VkDeviceCreateInfo *cd) PICK_QF(VK_QUEUE_GRAPHICS_BIT, VK_VIDEO_CODEC_OPERATION_NONE_KHR); PICK_QF(VK_QUEUE_COMPUTE_BIT, VK_VIDEO_CODEC_OPERATION_NONE_KHR); PICK_QF(VK_QUEUE_TRANSFER_BIT, VK_VIDEO_CODEC_OPERATION_NONE_KHR); + PICK_QF(VK_QUEUE_OPTICAL_FLOW_BIT_NV, VK_VIDEO_CODEC_OPERATION_NONE_KHR); PICK_QF(VK_QUEUE_VIDEO_ENCODE_BIT_KHR, VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR); PICK_QF(VK_QUEUE_VIDEO_DECODE_BIT_KHR, VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR); @@ -1318,9 +1322,13 @@ static int vulkan_device_create_internal(AVHWDeviceContext *ctx, VkPhysicalDeviceTimelineSemaphoreFeatures timeline_features = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_TIMELINE_SEMAPHORE_FEATURES, }; + VkPhysicalDeviceOpticalFlowFeaturesNV optical_flow_features = { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_OPTICAL_FLOW_FEATURES_NV, + .pNext = &timeline_features, + }; VkPhysicalDeviceCooperativeMatrixFeaturesKHR coop_matrix_features = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_COOPERATIVE_MATRIX_FEATURES_KHR, - .pNext = &timeline_features, + .pNext = &optical_flow_features, }; VkPhysicalDeviceShaderAtomicFloatFeaturesEXT atomic_float_features = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_ATOMIC_FLOAT_FEATURES_EXT, @@ -1364,7 +1372,9 @@ static int vulkan_device_create_internal(AVHWDeviceContext *ctx, p->atomic_float_features.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_ATOMIC_FLOAT_FEATURES_EXT; p->atomic_float_features.pNext = &p->coop_matrix_features; p->coop_matrix_features.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_COOPERATIVE_MATRIX_FEATURES_KHR; - p->coop_matrix_features.pNext = NULL; + p->coop_matrix_features.pNext = &p->optical_flow_features; + p->optical_flow_features.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_OPTICAL_FLOW_FEATURES_NV; + p->optical_flow_features.pNext = NULL; ctx->free = vulkan_device_free; @@ -1428,6 +1438,8 @@ static int vulkan_device_create_internal(AVHWDeviceContext *ctx, p->coop_matrix_features.cooperativeMatrix = coop_matrix_features.cooperativeMatrix; + p->optical_flow_features.opticalFlow = optical_flow_features.opticalFlow; + dev_info.pNext = &hwctx->device_features; /* Setup queue family */ diff --git a/libavutil/vulkan.c b/libavutil/vulkan.c index cec8354ba6..ade8d482b9 100644 --- a/libavutil/vulkan.c +++ b/libavutil/vulkan.c @@ -90,9 +90,13 @@ int ff_vk_load_props(FFVulkanContext *s) s->hprops = (VkPhysicalDeviceExternalMemoryHostPropertiesEXT) { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_MEMORY_HOST_PROPERTIES_EXT, }; + s->optical_flow_props = (VkPhysicalDeviceOpticalFlowPropertiesNV) { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_OPTICAL_FLOW_PROPERTIES_NV, + .pNext = &s->hprops, + }; s->coop_matrix_props = (VkPhysicalDeviceCooperativeMatrixPropertiesKHR) { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_COOPERATIVE_MATRIX_PROPERTIES_KHR, - .pNext = &s->hprops, + .pNext = &s->optical_flow_props, }; s->subgroup_props = (VkPhysicalDeviceSubgroupSizeControlProperties) { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SUBGROUP_SIZE_CONTROL_PROPERTIES, diff --git a/libavutil/vulkan.h b/libavutil/vulkan.h index eaefc954ed..1764d0eb33 100644 --- a/libavutil/vulkan.h +++ b/libavutil/vulkan.h @@ -238,6 +238,7 @@ typedef struct FFVulkanContext { VkPhysicalDeviceDescriptorBufferPropertiesEXT desc_buf_props; VkPhysicalDeviceSubgroupSizeControlProperties subgroup_props; VkPhysicalDeviceCooperativeMatrixPropertiesKHR coop_matrix_props; + VkPhysicalDeviceOpticalFlowPropertiesNV optical_flow_props; VkQueueFamilyQueryResultStatusPropertiesKHR *query_props; VkQueueFamilyVideoPropertiesKHR *video_props; VkQueueFamilyProperties2 *qf_props; diff --git a/libavutil/vulkan_functions.h b/libavutil/vulkan_functions.h index 6b379acf93..3f0b96f77e 100644 --- a/libavutil/vulkan_functions.h +++ b/libavutil/vulkan_functions.h @@ -46,6 +46,7 @@ typedef enum FFVulkanExtensions { FF_VK_EXT_VIDEO_DECODE_AV1 = 1ULL << 14, /* VK_KHR_video_decode_av1 */ FF_VK_EXT_ATOMIC_FLOAT = 1ULL << 15, /* VK_EXT_shader_atomic_float */ FF_VK_EXT_COOP_MATRIX = 1ULL << 16, /* VK_KHR_cooperative_matrix */ + FF_VK_EXT_OPTICAL_FLOW = 1ULL << 17, /* VK_NV_optical_flow */ FF_VK_EXT_NO_FLAG = 1ULL << 31, } FFVulkanExtensions; @@ -208,6 +209,13 @@ typedef enum FFVulkanExtensions { MACRO(1, 1, FF_VK_EXT_NO_FLAG, DestroySamplerYcbcrConversion) \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, CreateSampler) \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, DestroySampler) \ + \ + /* Optical flow */ \ + MACRO(1, 1, FF_VK_EXT_OPTICAL_FLOW, BindOpticalFlowSessionImageNV) \ + MACRO(1, 1, FF_VK_EXT_OPTICAL_FLOW, CmdOpticalFlowExecuteNV) \ + MACRO(1, 1, FF_VK_EXT_OPTICAL_FLOW, CreateOpticalFlowSessionNV) \ + MACRO(1, 1, FF_VK_EXT_OPTICAL_FLOW, DestroyOpticalFlowSessionNV) \ + MACRO(1, 0, FF_VK_EXT_OPTICAL_FLOW, GetPhysicalDeviceOpticalFlowImageFormatsNV)\ \ /* Shaders */ \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, CreateShaderModule) \ diff --git a/libavutil/vulkan_loader.h b/libavutil/vulkan_loader.h index 73cf03935d..bb22591957 100644 --- a/libavutil/vulkan_loader.h +++ b/libavutil/vulkan_loader.h @@ -49,6 +49,7 @@ static inline uint64_t ff_vk_extensions_to_mask(const char * const *extensions, { VK_EXT_PHYSICAL_DEVICE_DRM_EXTENSION_NAME, FF_VK_EXT_DEVICE_DRM }, { VK_EXT_SHADER_ATOMIC_FLOAT_EXTENSION_NAME, FF_VK_EXT_ATOMIC_FLOAT }, { VK_KHR_COOPERATIVE_MATRIX_EXTENSION_NAME, FF_VK_EXT_COOP_MATRIX }, + { VK_NV_OPTICAL_FLOW_EXTENSION_NAME, FF_VK_EXT_OPTICAL_FLOW }, #ifdef _WIN32 { VK_KHR_EXTERNAL_MEMORY_WIN32_EXTENSION_NAME, FF_VK_EXT_EXTERNAL_WIN32_MEMORY }, { VK_KHR_EXTERNAL_SEMAPHORE_WIN32_EXTENSION_NAME, FF_VK_EXT_EXTERNAL_WIN32_SEM },