From patchwork Thu Dec 2 01:43:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 31870 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:d206:0:0:0:0:0 with SMTP id q6csp10108671iob; Wed, 1 Dec 2021 17:48:18 -0800 (PST) X-Google-Smtp-Source: ABdhPJwRaH+W1+n6a29xay0wtf8O/SPHNeM7fvOO3ni2zm7KklEadyt/AYvg6OgJVlesnMJLq+W2 X-Received: by 2002:a17:906:bc46:: with SMTP id s6mr11668341ejv.467.1638409698031; Wed, 01 Dec 2021 17:48:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638409698; cv=none; d=google.com; s=arc-20160816; b=GeoHOGK2EADjoZtN0zg6CfxvjIgymunhRGiAQUd/1nCgDz+iSJW8gFxs3myjGNKAbx pqETN3ny8U3ehyfuQb3x7+td6IwGb9zhPDJuSr3afANmvelDpCPAmVPQtg0NiPv2Z4/5 kh6re1N5EHzIPPbMHpDktMhQ3YrHJ+UWMtYs60hn0R1sy3kbWeLupexrTGCH1hQjshGf qGx80nXEUep9NoheK8xq2lvAD7d2cke6IYP+lLhqnmPVrbnUCkTddbvmHymSlbtDJ9Uq fQ1supv7sLAESjh2r5puiO1u00vt+6AdwRNDO3ErWs8wfZ0XnPsxTaqd7aE7msjUwx2z yiJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=jhAhhb3uZ8sp7zlFN8GPuZQwoOhod4KQgLA4GZbGMY0=; b=AxpHUqbu/CpTqMcXot+Nedf9g/+7wHcjxlO3p+izqkKY0U3KhSNtx7HTYHRgnoXsEi aODJUvNCcnolmsDRGFZsUj99oHbuC6Q4tVFaiItIST3Cfufrw+QgD13XEqAX5HGS/JbG CYpidXoXIaelpJ3FQ2B/lUskxDzL3PHOkEBQsCIDsCXIoaP5M2tZAffKdDZFoB5tFolG v6ua0RksHxXgTt1WRUeJ0VlGnI5VRXYnb7rxm6ZhuDfRQ8+h2xBgJylw3Mag4WOuXG2C QVlgfGMW+opHHGyVb/VGD6lAdSFlyLlsgcLt94TviAd/oh8cd/I6486I1gd5F8FRwP8z xaSA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y15si1853548ejp.478.2021.12.01.17.48.17; Wed, 01 Dec 2021 17:48:18 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9DD6C68AEC9; Thu, 2 Dec 2021 03:47:41 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A418468AEE2 for ; Thu, 2 Dec 2021 03:47:33 +0200 (EET) X-IronPort-AV: E=McAfee;i="6200,9189,10185"; a="236414971" X-IronPort-AV: E=Sophos;i="5.87,280,1631602800"; d="scan'208";a="236414971" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Dec 2021 17:47:26 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,280,1631602800"; d="scan'208";a="512964521" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by orsmga008.jf.intel.com with ESMTP; 01 Dec 2021 17:47:25 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Thu, 2 Dec 2021 09:43:20 +0800 Message-Id: <20211202014320.1066212-5-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211202014320.1066212-1-wenbin.chen@intel.com> References: <20211202014320.1066212-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V5 5/5] libavutil/hwcontext_vulkan: specify the modifier to create VKImage X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 9jq6HtSKKV8Y When vulkan image exports to drm, the tilling need to be VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. Now add code to create vulkan image using this format. Now the following command line works: ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format \ vaapi -i input_1080p.264 -vf "hwmap=derive_device=vulkan,format=vulkan, \ scale_vulkan=1920:1080,hwmap=derive_device=vaapi,format=vaapi" -c:v h264_vaapi output.264 Signed-off-by: Wenbin Chen --- libavutil/hwcontext_vulkan.c | 127 +++++++++++++++++++++++++++++++++-- 1 file changed, 121 insertions(+), 6 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index f980b72720..8224c0d4e4 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -120,6 +120,9 @@ typedef struct VulkanFramesPriv { /* Image transfers */ VulkanExecCtx upload_ctx; VulkanExecCtx download_ctx; + + /*modifier info*/ + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info; } VulkanFramesPriv; typedef struct AVVkFrameInternal { @@ -242,6 +245,28 @@ const VkFormat *av_vkfmt_from_pixfmt(enum AVPixelFormat p) return NULL; } +static void *find_in_structure_list(VkBaseOutStructure *stru_list, VkStructureType sType) { + if (!stru_list) + return NULL; + + for(;stru_list;stru_list = stru_list->pNext) + if (stru_list->sType == sType) + return stru_list; + + return NULL; +} + +static void append_to_structure_list(VkBaseOutStructure **stru_list, VkBaseOutStructure *added_stru) { + VkBaseOutStructure *p; + if (!*stru_list) { + *stru_list = added_stru; + return; + } + for(p = *stru_list; p->pNext; p = p->pNext); + p->pNext = added_stru; + return; +} + static int pixfmt_is_supported(AVHWDeviceContext *dev_ctx, enum AVPixelFormat p, int linear) { @@ -2094,6 +2119,10 @@ static void try_export_flags(AVHWFramesContext *hwfc, AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; FFVulkanFunctions *vk = &p->vkfn; + const int has_modifiers = hwctx->tiling == VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT; + int loop_count; + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info = find_in_structure_list(hwctx->create_pnext, + VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT); VkExternalImageFormatProperties eprops = { .sType = VK_STRUCTURE_TYPE_EXTERNAL_IMAGE_FORMAT_PROPERTIES_KHR, }; @@ -2101,9 +2130,18 @@ static void try_export_flags(AVHWFramesContext *hwfc, .sType = VK_STRUCTURE_TYPE_IMAGE_FORMAT_PROPERTIES_2, .pNext = &eprops, }; + VkPhysicalDeviceImageDrmFormatModifierInfoEXT phy_dev_mod_info = { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_DRM_FORMAT_MODIFIER_INFO_EXT, + .pNext = NULL, + .pQueueFamilyIndices = p->qfs, + .queueFamilyIndexCount = p->num_qfs, + .sharingMode = p->num_qfs > 1 ? VK_SHARING_MODE_CONCURRENT : + VK_SHARING_MODE_EXCLUSIVE, + }; VkPhysicalDeviceExternalImageFormatInfo enext = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_IMAGE_FORMAT_INFO, .handleType = exp, + .pNext = has_modifiers && modifier_info ? &phy_dev_mod_info : NULL, }; VkPhysicalDeviceImageFormatInfo2 pinfo = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_FORMAT_INFO_2, @@ -2115,11 +2153,16 @@ static void try_export_flags(AVHWFramesContext *hwfc, .flags = VK_IMAGE_CREATE_ALIAS_BIT, }; - ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, - &pinfo, &props); - if (ret == VK_SUCCESS) { - *iexp |= exp; - *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + loop_count = has_modifiers && modifier_info ? modifier_info->drmFormatModifierCount : 1; + for (int i = 0; i < loop_count; i++) { + if (has_modifiers && modifier_info) + phy_dev_mod_info.drmFormatModifier = modifier_info->pDrmFormatModifiers[i]; + ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, + &pinfo, &props); + if (ret == VK_SUCCESS) { + *iexp |= exp; + *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + } } } @@ -2190,6 +2233,12 @@ static void vulkan_frames_uninit(AVHWFramesContext *hwfc) { VulkanFramesPriv *fp = hwfc->internal->priv; + if (fp->modifier_info) { + if (fp->modifier_info->pDrmFormatModifiers) + av_freep(&fp->modifier_info->pDrmFormatModifiers); + av_freep(&fp->modifier_info); + } + free_exec_ctx(hwfc, &fp->conv_ctx); free_exec_ctx(hwfc, &fp->upload_ctx); free_exec_ctx(hwfc, &fp->download_ctx); @@ -2203,11 +2252,77 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) VulkanFramesPriv *fp = hwfc->internal->priv; AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info; + const int has_modifiers = !!(p->extensions & FF_VK_EXT_DRM_MODIFIER_FLAGS); /* Default pool flags */ - hwctx->tiling = hwctx->tiling ? hwctx->tiling : p->use_linear_images ? + hwctx->tiling = hwctx->tiling ? hwctx->tiling : has_modifiers ? + VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT : p->use_linear_images ? VK_IMAGE_TILING_LINEAR : VK_IMAGE_TILING_OPTIMAL; + modifier_info = find_in_structure_list(hwctx->create_pnext, + VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT); + /* get the supported modifier */ + if (has_modifiers && !modifier_info) { + const VkFormat *fmt = av_vkfmt_from_pixfmt(hwfc->sw_format); + FFVulkanFunctions *vk = &p->vkfn; + VkDrmFormatModifierPropertiesEXT *mod_props; + uint64_t *modifiers; + int modifier_count = 0; + + VkDrmFormatModifierPropertiesListEXT mod_props_list = { + .sType = VK_STRUCTURE_TYPE_DRM_FORMAT_MODIFIER_PROPERTIES_LIST_EXT, + .pNext = NULL, + .drmFormatModifierCount = 0, + .pDrmFormatModifierProperties = NULL, + }; + VkFormatProperties2 prop = { + .sType = VK_STRUCTURE_TYPE_FORMAT_PROPERTIES_2, + .pNext = &mod_props_list, + }; + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + if (!mod_props_list.drmFormatModifierCount) { + av_log(hwfc, AV_LOG_ERROR, "There are not supported modifiers for sw_format\n"); + return AVERROR(EINVAL); + } + + modifier_info = av_malloc(sizeof(*modifier_info)); + if (!modifier_info) + return AVERROR(ENOMEM); + append_to_structure_list((VkBaseOutStructure **)&hwctx->create_pnext, + (VkBaseOutStructure *)modifier_info); + fp->modifier_info = modifier_info; + modifier_info->pNext = NULL; + modifier_info->sType = VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT; + + modifiers = av_malloc(mod_props_list.drmFormatModifierCount * + sizeof(*modifiers)); + if (!modifiers) + return AVERROR(ENOMEM); + modifier_info->pDrmFormatModifiers = modifiers; + + mod_props = av_malloc(mod_props_list.drmFormatModifierCount * + sizeof(*mod_props)); + if (!mod_props) + return AVERROR(ENOMEM); + mod_props_list.pDrmFormatModifierProperties = mod_props; + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + for (int i = 0; i < mod_props_list.drmFormatModifierCount; i++) { + if (!(mod_props[i].drmFormatModifierTilingFeatures & FF_VK_DEFAULT_USAGE_FLAGS)) + continue; + modifiers[modifier_count++] = mod_props[i].drmFormatModifier; + } + if (!modifier_count) { + av_log(hwfc, AV_LOG_ERROR, "The supported modifiers doesn't support " + "default usage\n"); + av_freep(&mod_props); + return AVERROR(EINVAL); + } + modifier_info->drmFormatModifierCount = modifier_count; + av_freep(&mod_props); + } + + if (!hwctx->usage) hwctx->usage = FF_VK_DEFAULT_USAGE_FLAGS;