From patchwork Wed Aug 14 18:51:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lynne X-Patchwork-Id: 51026 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:a746:0:b0:489:2eb3:e4c4 with SMTP id f6csp1118462vqm; Wed, 14 Aug 2024 11:51:50 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWslJI5AwKZYEB473uXe0riX1TSHHMkZttRX2cv3Os+8/Dbi2L8D0GdQhbZoZOel0nxa9WZhoRhEjt+2MYV3y2y@gmail.com X-Google-Smtp-Source: AGHT+IFvvbKRxk1YXutwz41nryPhsBSQf3Inse/gj4pQlrJliAIWrRnSoTz8h1BJKcUOSNVkNHUu X-Received: by 2002:a17:907:60d1:b0:a80:ed7a:c114 with SMTP id a640c23a62f3a-a83801c75aemr4895966b.0.1723661510071; Wed, 14 Aug 2024 11:51:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1723661510; cv=none; d=google.com; s=arc-20160816; b=uxl+FGux8EUcFTHrUEI8KmkqQOoNLtbbK1W3Gkh4eOF8+aZLFZI3yMnIqN4it2vEqf Umr4+cKd40ZtDTmYlvfqkr8Vczn5JystbCp2x7MNNs5VbApx/qbKyGRI6qGkXO09VemR 9PPDtxKjS4LkqL0J8b7WQdCXzSHhFCgapvbfdQaEj9OXMxuo34MUDqQ5HdtCOoXdN4ez fvKpVHvJ88i/hp0Pg6hRh3G8jfdDg1qU/7IMCh8XYgBX5cogY067wxLoxgv7wtN4ARmR /Lb4bIavNgD9MbdiqGKDBx2Zk++2GiUVNURHSvlFVJSCrMtO/5xkYa16fzo0CtOC2aaO ZLiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to :delivered-to; bh=20jyiyaai0AzXFV8VQxAhKp6zOIE3Z6OKuiH/rfU8eU=; fh=nenT92/WZoU6unXd3J6UhGUdod4piddKfVtctNBOh6k=; b=kUFCVHSgeexk8KLgTwmSZAM+DGXJjfEoEH00sCnvHvnAZepJGsKEh9RuP4pgA0o5s5 OeyvvFUsQgumEp8x6UXZwrxQv167DHZ/LIJwtQQQR+dHfEYIFzRiBEzd5PZOMQZgvnid AEm3MSkYGDRueXF74TQG7z7yMDP9oBWbjCU9gTenW0xYxHivhBbxrnuHe37UW7340k8y UAQCg27+oMnlfFLIvjcxKj7LCv0nV6O6gXrMg/F6qbNyZGjgd6W4eslpygdDsOJTiR5Y 7o4gCv6b9naZuR8Vnl9nx00CuXoBnU1NjyTF3fulLHDU2RsX7vz+F/f/FQ38XqlNV8gG QzCA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a640c23a62f3a-a80f411bb91si230921366b.336.2024.08.14.11.51.49; Wed, 14 Aug 2024 11:51:50 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DDFC468DABF; Wed, 14 Aug 2024 21:51:45 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vidala.lynne.ee (vidala.pars.ee [116.203.72.101]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AC1FB68D8FF for ; Wed, 14 Aug 2024 21:51:38 +0300 (EEST) To: ffmpeg-devel@ffmpeg.org Date: Wed, 14 Aug 2024 20:51:26 +0200 Message-ID: <20240814185134.935816-1-dev@lynne.ee> X-Mailer: git-send-email 2.45.2.753.g447d99e1c3b MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/2] hwcontext_vulkan: enable VK_KHR_video_maintenance1 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Lynne via ffmpeg-devel From: Lynne Reply-To: FFmpeg development discussions and patches Cc: Lynne Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: tZjDh7ERlDRc We require it for encoding. --- libavutil/hwcontext_vulkan.c | 12 +++++++++++- libavutil/vulkan_functions.h | 1 + 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 3e562716d8..a82355e8bf 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -94,6 +94,7 @@ typedef struct VulkanDevicePriv { VkPhysicalDeviceCooperativeMatrixFeaturesKHR coop_matrix_features; VkPhysicalDeviceOpticalFlowFeaturesNV optical_flow_features; VkPhysicalDeviceShaderObjectFeaturesEXT shader_object_features; + VkPhysicalDeviceVideoMaintenance1FeaturesKHR video_maint_1_features; /* Queues */ pthread_mutex_t **qf_mutex; @@ -422,6 +423,7 @@ static const VulkanOptExtension optional_device_exts[] = { { VK_KHR_COOPERATIVE_MATRIX_EXTENSION_NAME, FF_VK_EXT_COOP_MATRIX }, { VK_NV_OPTICAL_FLOW_EXTENSION_NAME, FF_VK_EXT_OPTICAL_FLOW }, { VK_EXT_SHADER_OBJECT_EXTENSION_NAME, FF_VK_EXT_SHADER_OBJECT }, + { VK_KHR_VIDEO_MAINTENANCE_1_EXTENSION_NAME, FF_VK_EXT_VIDEO_MAINTENANCE_1 }, /* Imports/exports */ { VK_KHR_EXTERNAL_MEMORY_FD_EXTENSION_NAME, FF_VK_EXT_EXTERNAL_FD_MEMORY }, @@ -1399,9 +1401,13 @@ static int vulkan_device_create_internal(AVHWDeviceContext *ctx, VkPhysicalDeviceTimelineSemaphoreFeatures timeline_features = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_TIMELINE_SEMAPHORE_FEATURES, }; + VkPhysicalDeviceVideoMaintenance1FeaturesKHR video_maint_1_features = { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VIDEO_MAINTENANCE_1_FEATURES_KHR, + .pNext = &timeline_features, + }; VkPhysicalDeviceShaderObjectFeaturesEXT shader_object_features = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_OBJECT_FEATURES_EXT, - .pNext = &timeline_features, + .pNext = &video_maint_1_features, }; VkPhysicalDeviceOpticalFlowFeaturesNV optical_flow_features = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_OPTICAL_FLOW_FEATURES_NV, @@ -1496,6 +1502,8 @@ static int vulkan_device_create_internal(AVHWDeviceContext *ctx, p->device_features_1_3.shaderZeroInitializeWorkgroupMemory = dev_features_1_3.shaderZeroInitializeWorkgroupMemory; p->device_features_1_3.dynamicRendering = dev_features_1_3.dynamicRendering; + p->video_maint_1_features.videoMaintenance1 = video_maint_1_features.videoMaintenance1; + p->desc_buf_features.descriptorBuffer = desc_buf_features.descriptorBuffer; p->desc_buf_features.descriptorBufferPushDescriptors = desc_buf_features.descriptorBufferPushDescriptors; @@ -1545,6 +1553,8 @@ static int vulkan_device_create_internal(AVHWDeviceContext *ctx, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_OBJECT_FEATURES_EXT); OPT_CHAIN(FF_VK_EXT_OPTICAL_FLOW, &p->optical_flow_features, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_OPTICAL_FLOW_FEATURES_NV); + OPT_CHAIN(FF_VK_EXT_VIDEO_MAINTENANCE_1, &p->video_maint_1_features, + VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VIDEO_MAINTENANCE_1_FEATURES_KHR); #undef OPT_CHAIN /* Add the enabled features into the pnext chain of device creation */ diff --git a/libavutil/vulkan_functions.h b/libavutil/vulkan_functions.h index 5fbde96cfe..d4697ec34c 100644 --- a/libavutil/vulkan_functions.h +++ b/libavutil/vulkan_functions.h @@ -49,6 +49,7 @@ typedef enum FFVulkanExtensions { FF_VK_EXT_OPTICAL_FLOW = 1ULL << 17, /* VK_NV_optical_flow */ FF_VK_EXT_SHADER_OBJECT = 1ULL << 18, /* VK_EXT_shader_object */ + FF_VK_EXT_VIDEO_MAINTENANCE_1 = 1ULL << 27, /* VK_KHR_video_maintenance1 */ FF_VK_EXT_VIDEO_ENCODE_QUEUE = 1ULL << 28, /* VK_KHR_video_encode_queue */ FF_VK_EXT_VIDEO_ENCODE_H264 = 1ULL << 29, /* VK_KHR_video_encode_h264 */ FF_VK_EXT_VIDEO_ENCODE_H265 = 1ULL << 30, /* VK_KHR_video_encode_h265 */ From patchwork Wed Aug 14 18:51:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lynne X-Patchwork-Id: 51027 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:a746:0:b0:489:2eb3:e4c4 with SMTP id f6csp1118526vqm; Wed, 14 Aug 2024 11:52:01 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWPKIjJWonr56xdmPp2RfpKLk3Uov67+ZafRNGDZw+EOAc2Z23XMfkxIFqxHjA0jAqyrKz7kqF/+eRbjLajjEx/@gmail.com X-Google-Smtp-Source: AGHT+IFkWXKt+20NI4hDu4Cg6kR+xCN5ymUOh/S53cnObCY2kk3XDAa8Plb20Pa5mMa+AHrPMh7y X-Received: by 2002:a05:6512:ba1:b0:52f:c285:323d with SMTP id 2adb3069b0e04-53309ead340mr37165e87.5.1723661521571; Wed, 14 Aug 2024 11:52:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1723661521; cv=none; d=google.com; s=arc-20160816; b=oFpUstwnDQpATf8LG7jCWPMdRLyd0sW9FnI7thazWl7e6k8gKVt/XJoOjaeThDaneW uhw1UHENhJJgyozf7bVlXl47I6BLf0MaLrWnmImcmbnQFuQNDbpc7GP9g47xiTOyzK3a afp1Fcl0rEwUYVYFes5XZggHKRDioxsnCGuyH7EgXewESq5wDn5amHJdlQ0uGCvXhn1l cUgDknKHa3Lv4jp45mph62tqf3M/8VDWyTVzjkKLTgh1iE5MZdWqJflM0xQGUo8gMjVA iS9759fI0WIU5L6xs5AaHV27Qse/cnetmdmPssnnDXmKB0RoZvVY4XUqGT5/nwcbR6W0 4CLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:delivered-to; bh=BEegQYhuubUawDOH747PiVwjije/ndnK8i5okhJgspM=; fh=nenT92/WZoU6unXd3J6UhGUdod4piddKfVtctNBOh6k=; b=LycjcFo0QLQSMUFOo4fk++Rl1X27Od2UD7eqfnPlJS6jIHjxwliSKKL0umNFHspY10 mOZ4W22/BxQ8jXLTpb2kMGw+ci9XFcUJTPe3Tqs3CiyM/DOAE7YEsa6gy5QJhVFBnAke UtuzfER42LFIobsg2G4V170LQQiQ4lMiGllwkNYoVApwf7Vdw/EBsBHAEltLdmNaiooK VYAIzZgl4Zr5UNkGOlqdxAABo0Rj9Pn15BlFUx965RhOJnoqQ+yKuvNK4RTZ7ddLCgbc YwrePXl75/uN0CSwb2l8DCtwqTBub+WsCi+ZfPWRaozxZKjZ14Vm+TZoayQSqrsHQBX6 LGbA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-53200f0f108si2942031e87.356.2024.08.14.11.52.00; Wed, 14 Aug 2024 11:52:01 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0D44F68DADA; Wed, 14 Aug 2024 21:51:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vidala.lynne.ee (vidala.pars.ee [116.203.72.101]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9DDE968D9AF for ; Wed, 14 Aug 2024 21:51:41 +0300 (EEST) To: ffmpeg-devel@ffmpeg.org Date: Wed, 14 Aug 2024 20:51:27 +0200 Message-ID: <20240814185134.935816-2-dev@lynne.ee> X-Mailer: git-send-email 2.45.2.753.g447d99e1c3b In-Reply-To: <20240814185134.935816-1-dev@lynne.ee> References: <20240814185134.935816-1-dev@lynne.ee> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] hwcontext_vulkan: enable encoding of images if video_maintenance1 is enabled X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Lynne via ffmpeg-devel From: Lynne Reply-To: FFmpeg development discussions and patches Cc: Lynne Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ZzG6/svY95lB Vulkan encoding was designed in a very... consolidated way. You had to know the exact codec and profile that the image was going to eventually be encoded as at... image creation time. Unfortunately, as good as our code is, glimpsing into the exact future isn't what its capable of. video_maintenance1 removed that requirement, which only then made encoding images practically possible. --- libavcodec/vulkan_decode.c | 10 ++++++++++ libavfilter/vulkan_filter.c | 23 ++++++++++++----------- libavutil/hwcontext_vulkan.c | 13 +++++++++++++ 3 files changed, 35 insertions(+), 11 deletions(-) diff --git a/libavcodec/vulkan_decode.c b/libavcodec/vulkan_decode.c index c7a32cc439..e73d4f93c2 100644 --- a/libavcodec/vulkan_decode.c +++ b/libavcodec/vulkan_decode.c @@ -934,6 +934,10 @@ static int vulkan_decode_get_profile(AVCodecContext *avctx, AVBufferRef *frames_ VK_IMAGE_USAGE_VIDEO_DECODE_DST_BIT_KHR | VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_SAMPLED_BIT; + + if (ctx->s.extensions & (FF_VK_EXT_VIDEO_ENCODE_QUEUE | + FF_VK_EXT_VIDEO_MAINTENANCE_1)) + fmt_info.imageUsage |= VK_IMAGE_USAGE_VIDEO_ENCODE_SRC_BIT_KHR; } /* Get the format of the images necessary */ @@ -1023,6 +1027,7 @@ int ff_vk_frame_params(AVCodecContext *avctx, AVBufferRef *hw_frames_ctx) AVVulkanFramesContext *hwfc = frames_ctx->hwctx; FFVulkanDecodeContext *dec = avctx->internal->hwaccel_priv_data; FFVulkanDecodeProfileData *prof; + FFVulkanDecodeShared *ctx; frames_ctx->sw_format = AV_PIX_FMT_NONE; @@ -1059,6 +1064,11 @@ int ff_vk_frame_params(AVCodecContext *avctx, AVBufferRef *hw_frames_ctx) if (!dec->dedicated_dpb) hwfc->usage |= VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR; + ctx = dec->shared_ctx; + if (ctx->s.extensions & (FF_VK_EXT_VIDEO_ENCODE_QUEUE | + FF_VK_EXT_VIDEO_MAINTENANCE_1)) + hwfc->usage |= VK_IMAGE_USAGE_VIDEO_ENCODE_SRC_BIT_KHR; + return err; } diff --git a/libavfilter/vulkan_filter.c b/libavfilter/vulkan_filter.c index cef42eeb4d..fd70d90b1a 100644 --- a/libavfilter/vulkan_filter.c +++ b/libavfilter/vulkan_filter.c @@ -36,6 +36,7 @@ int ff_vk_filter_init_context(AVFilterContext *avctx, FFVulkanContext *s, if (frames_ref) { int no_storage = 0; FFVulkanFunctions *vk; + VkImageUsageFlagBits usage_req; const VkFormat *sub = av_vkfmt_from_pixfmt(sw_format); frames_ctx = (AVHWFramesContext *)frames_ref->data; @@ -56,13 +57,20 @@ int ff_vk_filter_init_context(AVFilterContext *avctx, FFVulkanContext *s, if (vk_frames->tiling != VK_IMAGE_TILING_OPTIMAL) goto skip; + s->extensions = ff_vk_extensions_to_mask(vk_dev->enabled_dev_extensions, + vk_dev->nb_enabled_dev_extensions); + /* Usage mismatch */ - if ((vk_frames->usage & (VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_STORAGE_BIT)) != - (VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_STORAGE_BIT)) + usage_req = VK_IMAGE_USAGE_SAMPLED_BIT | + VK_IMAGE_USAGE_STORAGE_BIT; + if (s->extensions & (FF_VK_EXT_VIDEO_ENCODE_QUEUE | + FF_VK_EXT_VIDEO_MAINTENANCE_1)) + usage_req |= VK_IMAGE_USAGE_VIDEO_ENCODE_SRC_BIT_KHR; + + if ((vk_frames->usage & usage_req) != usage_req) goto skip; - s->extensions = ff_vk_extensions_to_mask(vk_dev->enabled_dev_extensions, - vk_dev->nb_enabled_dev_extensions); + /* More advanced format checks */ err = ff_vk_load_functions(device_ctx, &s->vkfn, s->extensions, 1, 1); if (err < 0) return err; @@ -112,13 +120,6 @@ skip: frames_ctx->width = width; frames_ctx->height = height; - vk_frames = frames_ctx->hwctx; - vk_frames->tiling = VK_IMAGE_TILING_OPTIMAL; - vk_frames->usage = VK_IMAGE_USAGE_SAMPLED_BIT | - VK_IMAGE_USAGE_STORAGE_BIT | - VK_IMAGE_USAGE_TRANSFER_SRC_BIT | - VK_IMAGE_USAGE_TRANSFER_DST_BIT; - err = av_hwframe_ctx_init(frames_ref); if (err < 0) { av_buffer_unref(&frames_ref); diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index a82355e8bf..3ccefa9525 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -2630,6 +2630,11 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_STORAGE_BIT | VK_IMAGE_USAGE_SAMPLED_BIT); + + /* Enables encoding of images */ + if (p->vkctx.extensions & (FF_VK_EXT_VIDEO_ENCODE_QUEUE | + FF_VK_EXT_VIDEO_MAINTENANCE_1)) + hwctx->usage |= VK_IMAGE_USAGE_VIDEO_ENCODE_SRC_BIT_KHR; } /* Image creation flags. @@ -2648,6 +2653,14 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) } } + /* If image has an encode usage, yet there's no profile attached, + * and maintenance1 is supported, enable the profile independent + * creation flag. */ + if ((hwctx->usage & VK_IMAGE_USAGE_VIDEO_ENCODE_SRC_BIT_KHR) && + !(ff_vk_find_struct(hwctx->create_pnext, VK_STRUCTURE_TYPE_VIDEO_PROFILE_LIST_INFO_KHR)) && + p->video_maint_1_features.videoMaintenance1) + hwctx->img_flags |= VK_IMAGE_CREATE_VIDEO_PROFILE_INDEPENDENT_BIT_KHR; + if (!hwctx->lock_frame) hwctx->lock_frame = lock_frame;