From patchwork Tue Nov 9 09:18:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 31332 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:d206:0:0:0:0:0 with SMTP id q6csp886404iob; Tue, 9 Nov 2021 01:24:09 -0800 (PST) X-Google-Smtp-Source: ABdhPJzjORvC4UrdG48zksCAudgH9VmOEDHRcixSUCBslybHb7VUuWLKBfDpdFzs7rHOmo2kY5PE X-Received: by 2002:a50:9993:: with SMTP id m19mr7975350edb.72.1636449846236; Tue, 09 Nov 2021 01:24:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1636449846; cv=none; d=google.com; s=arc-20160816; b=L3WWpQT2e9ag458MNgbiQrYjHpHaaWi8ZJI/M/wVKXRgRqtYWJyhxR2LCi/R1comfp B9ZItuTgdIzqf/XDwKGuGjPrx6EEZlOCZWxj6jGvS5zyx6sHoPm7ebSzeB0WV7/5CJ12 9qmcJJkjxD6gR4rY1heI6uzsO0joBDLJMprRLjCeBQNkmziTGO58yKXzoScPl4XLyd7X 9HlY1Q7nCl/OgMgBt+LtAeXhupi1DuqZmo0oGjTW2GNixXUfWzWBiEKwCXmvp6IBHMsN HAZtPvRTMgbjQpuCpimigtZpD8pI9HBwQEYu9PpFJyL/DURmSsm/5/C0e4YbSRxPvGl8 KXBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=9GGp5m8g4UcciloSxU6oSK33DpBpFXM83nDprQbjB90=; b=zGTd1zy1iLde+irxHlPJvjE5FpUS0Aq1yNrrQpcXl5kqcL5GTEYPqwHign+l3gW3u1 sWHlNCQB7hoHwnNfL3M2LNE3SruH5X6O4U/tSK46sjR0mELDwMN00zH3ZZaYJ1ZWldSg J5UCeUp0cfhlLU60iSb78tah+/w1wMKwA1aA5OgM2UsQ0JmgP2O0QCK7SG+VUglVjVcf Z92BZvfaQFSEEnH113Ila7ztq4XuRPhD1QJOx/zOyYgShRong56XOXIC6y+e+FJojkwZ zwHRSSMV1s0PhYzQtB0/ExmwG7Dj4y1SRNIBn07izaFva6u4BOn7+AaOz6NHcocbJgdX e2nA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t13si17212502ejf.616.2021.11.09.01.24.05; Tue, 09 Nov 2021 01:24:06 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D51A768AC4D; Tue, 9 Nov 2021 11:23:04 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AD15168AB7A for ; Tue, 9 Nov 2021 11:22:56 +0200 (EET) X-IronPort-AV: E=McAfee;i="6200,9189,10162"; a="295849401" X-IronPort-AV: E=Sophos;i="5.87,219,1631602800"; d="scan'208";a="295849401" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2021 01:22:54 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,219,1631602800"; d="scan'208";a="491593978" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by orsmga007.jf.intel.com with ESMTP; 09 Nov 2021 01:22:53 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Tue, 9 Nov 2021 17:18:43 +0800 Message-Id: <20211109091843.2024927-7-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211109091843.2024927-1-wenbin.chen@intel.com> References: <20211109091843.2024927-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 7/7] libavutil/hwcontext_vulkan: specify the modifier to create VKImage X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Wenbin Chen Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: uO/sc//7mqJG When vulkan image exports to drm, the tilling need to be VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. Now add code to create vulkan image using this format. Now the following command line works: ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format \ vaapi -i input_1080p.264 -vf "hwmap=derive_device=vulkan,format=vulkan, \ scale_vulkan=1920:1080,hwmap=derive_device=vaapi,format=vaapi" -c:v h264_vaapi output.264 Signed-off-by: Wenbin Chen --- libavutil/hwcontext_vulkan.c | 76 +++++++++++++++++++++++++++++++++--- libavutil/hwcontext_vulkan.h | 5 +++ 2 files changed, 75 insertions(+), 6 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 29ade94b7f..e252c2177e 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -1919,6 +1919,7 @@ static void try_export_flags(AVHWFramesContext *hwfc, AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; FFVulkanFunctions *vk = &p->vkfn; + const int has_modifiers = hwctx->tiling == VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT; VkExternalImageFormatProperties eprops = { .sType = VK_STRUCTURE_TYPE_EXTERNAL_IMAGE_FORMAT_PROPERTIES_KHR, }; @@ -1926,9 +1927,18 @@ static void try_export_flags(AVHWFramesContext *hwfc, .sType = VK_STRUCTURE_TYPE_IMAGE_FORMAT_PROPERTIES_2, .pNext = &eprops, }; + VkPhysicalDeviceImageDrmFormatModifierInfoEXT phy_dev_mod_info = { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_DRM_FORMAT_MODIFIER_INFO_EXT, + .pNext = NULL, + .pQueueFamilyIndices = p->qfs, + .queueFamilyIndexCount = p->num_qfs, + .sharingMode = p->num_qfs > 1 ? VK_SHARING_MODE_CONCURRENT : + VK_SHARING_MODE_EXCLUSIVE, + }; VkPhysicalDeviceExternalImageFormatInfo enext = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_IMAGE_FORMAT_INFO, .handleType = exp, + .pNext = has_modifiers ? &phy_dev_mod_info : NULL, }; VkPhysicalDeviceImageFormatInfo2 pinfo = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_FORMAT_INFO_2, @@ -1940,11 +1950,15 @@ static void try_export_flags(AVHWFramesContext *hwfc, .flags = VK_IMAGE_CREATE_ALIAS_BIT, }; - ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, - &pinfo, &props); - if (ret == VK_SUCCESS) { - *iexp |= exp; - *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + for (int i = 0; i < (has_modifiers ? hwctx->modifier_count : 1); i++) { + if (has_modifiers && hwctx->modifier_count) + phy_dev_mod_info.drmFormatModifier = hwctx->modifiers[i]; + ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, + &pinfo, &props); + if (ret == VK_SUCCESS) { + *iexp |= exp; + *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + } } } @@ -2007,6 +2021,7 @@ fail: static void vulkan_frames_uninit(AVHWFramesContext *hwfc) { VulkanFramesPriv *fp = hwfc->internal->priv; + AVVulkanFramesContext *hwctx = hwfc->hwctx; free_exec_ctx(hwfc, &fp->conv_ctx); free_exec_ctx(hwfc, &fp->upload_ctx); @@ -2021,11 +2036,60 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) VulkanFramesPriv *fp = hwfc->internal->priv; AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; + const int has_modifiers = !!(p->extensions & FF_VK_EXT_DRM_MODIFIER_FLAGS); /* Default pool flags */ - hwctx->tiling = hwctx->tiling ? hwctx->tiling : p->use_linear_images ? + hwctx->tiling = hwctx->tiling ? hwctx->tiling : has_modifiers ? + VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT : p->use_linear_images ? VK_IMAGE_TILING_LINEAR : VK_IMAGE_TILING_OPTIMAL; + /* get the supported modifier */ + if (has_modifiers) { + const VkFormat *fmt = av_vkfmt_from_pixfmt(hwfc->sw_format); + FFVulkanFunctions *vk = &p->vkfn; + VkDrmFormatModifierPropertiesEXT mod_props[MAX_VULKAN_MODIFIERS]; + + VkDrmFormatModifierPropertiesListEXT mod_props_list = { + .sType = VK_STRUCTURE_TYPE_DRM_FORMAT_MODIFIER_PROPERTIES_LIST_EXT, + .pNext = NULL, + .drmFormatModifierCount = 0, + .pDrmFormatModifierProperties = NULL, + }; + VkFormatProperties2 prop = { + .sType = VK_STRUCTURE_TYPE_FORMAT_PROPERTIES_2, + .pNext = &mod_props_list, + }; + + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + if (!mod_props_list.drmFormatModifierCount) { + av_log(hwfc, AV_LOG_ERROR, "There are not supported modifiers for sw_format\n"); + return AVERROR(EINVAL); + } + mod_props_list.drmFormatModifierCount = + FFMIN(mod_props_list.drmFormatModifierCount, MAX_VULKAN_MODIFIERS); + + mod_props_list.pDrmFormatModifierProperties = mod_props; + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + + hwctx->modifier_count = 0; + for (int i = 0; i < mod_props_list.drmFormatModifierCount; i++) { + if (!(mod_props[i].drmFormatModifierTilingFeatures & DEFAULT_USAGE_FLAGS)) + continue; + hwctx->modifiers[hwctx->modifier_count++] = mod_props[i].drmFormatModifier; + } + if (!hwctx->modifier_count) { + av_log(hwfc, AV_LOG_ERROR, "The supported modifiers doesn't support" + "default usage\n"); + return AVERROR(EINVAL); + } + + hwctx->create_pnext = &hwctx->modifier_info; + hwctx->modifier_info.pNext = NULL; + hwctx->modifier_info.sType = VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT; + hwctx->modifier_info.drmFormatModifierCount = hwctx->modifier_count; + hwctx->modifier_info.pDrmFormatModifiers = hwctx->modifiers; + } + if (!hwctx->usage) hwctx->usage = DEFAULT_USAGE_FLAGS; diff --git a/libavutil/hwcontext_vulkan.h b/libavutil/hwcontext_vulkan.h index efb602ef27..113448bb51 100644 --- a/libavutil/hwcontext_vulkan.h +++ b/libavutil/hwcontext_vulkan.h @@ -32,6 +32,7 @@ * with the data pointer set to an AVVkFrame. */ +#define MAX_VULKAN_MODIFIERS 16 /** * Main Vulkan context, allocated as AVHWDeviceContext.hwctx. * All of these can be set before init to change what the context uses @@ -149,6 +150,10 @@ typedef struct AVVulkanFramesContext { */ void *create_pnext; + VkImageDrmFormatModifierListCreateInfoEXT modifier_info; + uint64_t modifiers[MAX_VULKAN_MODIFIERS]; + int modifier_count; + /** * Extension data for memory allocation. Must have as many entries as * the number of planes of the sw_format.