From patchwork Tue Aug 31 01:43:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 29885 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp4497925iov; Mon, 30 Aug 2021 18:47:21 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwdjJy8JUUNUiWfNlKwizP5dF+2UMysa1kB/rY7fp6dLlZrcJbnpDb3ki1RVRk+UDkGnp8x X-Received: by 2002:a17:906:1615:: with SMTP id m21mr28375180ejd.279.1630374441695; Mon, 30 Aug 2021 18:47:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630374441; cv=none; d=google.com; s=arc-20160816; b=V2ITnfF6Lfp3djQwJuGXnG+72GmqSN3wUQgyQV4L3ZYZNxW5HGtabvS6ita8AuXbU5 i/4b5FLuZNmN5C12Zd6Z0EyOVYVh5eUyi9JHM2WkagHeR8IOCU1n3vGMaw3A5Xrg349h HY+pY3Uo3yGf5EsmvV9Q2L902hUV7mFbK1NLq949LP8iSOb+4096NNDpEIKWRx+VbMrg RqUevObEogq/P6as7R+0U+WYtfIO6IIanseCeABIE+k6XXlXpHmwNEmltn0KXEZTiGFI L/m7SfB8YEmlp+x0bnJhJrby5Kp2WOWbjb0AOxDkKASUSYWuBDT/Mdkqh7MmzFe/D7lb ZwWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=vzy98ilcdP3qXjGbi5aZOsXej3unZOL3hVDilUdSNLk=; b=vzcD5QqEGs/JbYCtDYCY5CeQ04UnnAVyZ6FHoOhJikL80NQlDaNz2+LBuSmlD/a+rV MxVeCYqxkBDdx4lIhp1X/DMLaQigboBU8KxSZFYWULXpL38lavpTcuCj56dkWkr/CELZ YXGs2B3qj9wSMRg4d91j8lImQWl7767nNIbgTql8JBMj/FkGMsh+QOHtfl2SoZrtigpY Y9E7wWrYpUoKG6SdCJPNyRL6mxWnH4nk140z3uBuW2mPP09c0IWPSX4SorFoTvY4b/Tr y2X3X7iDgRk1acgUWvE7gnUMW73tSQWruw/w8QHlVbUbz/c6bNwLOul3JGxuHQ+x0vmy RO9Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id la20si19282799ejc.3.2021.08.30.18.47.21; Mon, 30 Aug 2021 18:47:21 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 81B5468A4E5; Tue, 31 Aug 2021 04:46:09 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3A86368A41C for ; Tue, 31 Aug 2021 04:46:05 +0300 (EEST) X-IronPort-AV: E=McAfee;i="6200,9189,10092"; a="216531746" X-IronPort-AV: E=Sophos;i="5.84,365,1620716400"; d="scan'208";a="216531746" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Aug 2021 18:45:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,365,1620716400"; d="scan'208";a="530827101" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.5]) by FMSMGA003.fm.intel.com with ESMTP; 30 Aug 2021 18:45:54 -0700 From: wenbin.chen@intel.com To: ffmpeg-devel@ffmpeg.org Date: Tue, 31 Aug 2021 09:43:37 +0800 Message-Id: <20210831014338.134086-9-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210831014338.134086-1-wenbin.chen@intel.com> References: <20210831014338.134086-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 09/10] libavutil/hwcontext_vulkan: specify the modifier to create VKImage X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Wenbin Chen Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Eni+pDJEwtvO From: Wenbin Chen On the lastset intel-vulkan-driver the VK_EXT_image_drm_format_modifier flags is enabled. As what driver log recommand, we need to use VK_IMAGE_TILING_LINEAR or VK_IMAGE_DRM_FORMAT_MODIFIER_EXT to create VKImage. Add code to get supported modifier for sw_format and use these modifier to create VKImage. Now the following command line works: ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input_1080p.264 -vf "hwmap=derive_device=vulkan,format=vulkan, scale_vulkan=1920:1080,hwmap=derive_device=vaapi,format=vaapi" -c:v h264_vaapi output.264 Signed-off-by: Wenbin Chen --- libavutil/hwcontext_vulkan.c | 77 +++++++++++++++++++++++++++++++++--- libavutil/hwcontext_vulkan.h | 5 +++ 2 files changed, 76 insertions(+), 6 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 3a639c997b..99b2190dc3 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -1967,6 +1967,8 @@ static void try_export_flags(AVHWFramesContext *hwfc, AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; VulkanFunctions *vk = &p->vkfn; + const int has_modifiers = hwctx->tiling == VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT; + VkExternalImageFormatProperties eprops = { .sType = VK_STRUCTURE_TYPE_EXTERNAL_IMAGE_FORMAT_PROPERTIES_KHR, }; @@ -1974,9 +1976,18 @@ static void try_export_flags(AVHWFramesContext *hwfc, .sType = VK_STRUCTURE_TYPE_IMAGE_FORMAT_PROPERTIES_2, .pNext = &eprops, }; + VkPhysicalDeviceImageDrmFormatModifierInfoEXT phy_dev_mod_info = { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_DRM_FORMAT_MODIFIER_INFO_EXT, + .pNext = NULL, + .pQueueFamilyIndices = p->qfs, + .queueFamilyIndexCount = p->num_qfs, + .sharingMode = p->num_qfs > 1 ? VK_SHARING_MODE_CONCURRENT : + VK_SHARING_MODE_EXCLUSIVE, + }; VkPhysicalDeviceExternalImageFormatInfo enext = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_IMAGE_FORMAT_INFO, .handleType = exp, + .pNext = has_modifiers ? &phy_dev_mod_info : NULL, }; VkPhysicalDeviceImageFormatInfo2 pinfo = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_FORMAT_INFO_2, @@ -1988,11 +1999,15 @@ static void try_export_flags(AVHWFramesContext *hwfc, .flags = VK_IMAGE_CREATE_ALIAS_BIT, }; - ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, - &pinfo, &props); - if (ret == VK_SUCCESS) { - *iexp |= exp; - *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + for (int i = 0; i < (has_modifiers ? hwctx->modifier_count : 1); i++) { + if (has_modifiers && hwctx->modifier_count) + phy_dev_mod_info.drmFormatModifier = hwctx->modifiers[i]; + ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, + &pinfo, &props); + if (ret == VK_SUCCESS) { + *iexp |= exp; + *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + } } } @@ -2055,6 +2070,7 @@ fail: static void vulkan_frames_uninit(AVHWFramesContext *hwfc) { VulkanFramesPriv *fp = hwfc->internal->priv; + AVVulkanFramesContext *hwctx = hwfc->hwctx; free_exec_ctx(hwfc, &fp->conv_ctx); free_exec_ctx(hwfc, &fp->upload_ctx); @@ -2069,11 +2085,60 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) VulkanFramesPriv *fp = hwfc->internal->priv; AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; + const int has_modifiers = !!(p->extensions & EXT_DRM_MODIFIER_FLAGS); /* Default pool flags */ - hwctx->tiling = hwctx->tiling ? hwctx->tiling : p->use_linear_images ? + hwctx->tiling = hwctx->tiling ? hwctx->tiling : has_modifiers ? + VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT : p->use_linear_images ? VK_IMAGE_TILING_LINEAR : VK_IMAGE_TILING_OPTIMAL; + /* get the supported modifier */ + if (has_modifiers) { + const VkFormat *fmt = av_vkfmt_from_pixfmt(hwfc->sw_format); + VulkanFunctions *vk = &p->vkfn; + VkDrmFormatModifierPropertiesEXT mod_props[MAX_VULKAN_MODIFIERS]; + + VkDrmFormatModifierPropertiesListEXT mod_props_list = { + .sType = VK_STRUCTURE_TYPE_DRM_FORMAT_MODIFIER_PROPERTIES_LIST_EXT, + .pNext = NULL, + .drmFormatModifierCount = 0, + .pDrmFormatModifierProperties = NULL, + }; + VkFormatProperties2 prop = { + .sType = VK_STRUCTURE_TYPE_FORMAT_PROPERTIES_2, + .pNext = &mod_props_list, + }; + + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + if (!mod_props_list.drmFormatModifierCount) { + av_log(hwfc, AV_LOG_ERROR, "There are not supported modifiers for sw_format\n"); + return AVERROR(EINVAL); + } + mod_props_list.drmFormatModifierCount = + FFMIN(mod_props_list.drmFormatModifierCount, MAX_VULKAN_MODIFIERS); + + mod_props_list.pDrmFormatModifierProperties = mod_props; + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + + hwctx->modifier_count = 0; + for (int i = 0; i < mod_props_list.drmFormatModifierCount; i++) { + if (!(mod_props[i].drmFormatModifierTilingFeatures & DEFAULT_USAGE_FLAGS)) + continue; + hwctx->modifiers[hwctx->modifier_count++] = mod_props[i].drmFormatModifier; + } + if (!hwctx->modifier_count) { + av_log(hwfc, AV_LOG_ERROR, "The supported modifiers doesn't support" + "default usage\n"); + return AVERROR(EINVAL); + } + + hwctx->create_pnext = &hwctx->modifier_info; + hwctx->modifier_info.pNext = NULL; + hwctx->modifier_info.sType = VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT; + hwctx->modifier_info.drmFormatModifierCount = hwctx->modifier_count; + hwctx->modifier_info.pDrmFormatModifiers = hwctx->modifiers; + } + if (!hwctx->usage) hwctx->usage = DEFAULT_USAGE_FLAGS; diff --git a/libavutil/hwcontext_vulkan.h b/libavutil/hwcontext_vulkan.h index 8fb25d1485..95fd767d1b 100644 --- a/libavutil/hwcontext_vulkan.h +++ b/libavutil/hwcontext_vulkan.h @@ -32,6 +32,7 @@ * with the data pointer set to an AVVkFrame. */ +#define MAX_VULKAN_MODIFIERS 16 /** * Main Vulkan context, allocated as AVHWDeviceContext.hwctx. * All of these can be set before init to change what the context uses @@ -142,6 +143,10 @@ typedef struct AVVulkanFramesContext { */ void *create_pnext; + VkImageDrmFormatModifierListCreateInfoEXT modifier_info; + uint64_t modifiers[MAX_VULKAN_MODIFIERS]; + int modifier_count; + /** * Extension data for memory allocation. Must have as many entries as * the number of planes of the sw_format.