From patchwork Tue Nov 30 06:28:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 31829 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:d206:0:0:0:0:0 with SMTP id q6csp7254778iob; Mon, 29 Nov 2021 22:33:10 -0800 (PST) X-Google-Smtp-Source: ABdhPJyGneY7xTuvAbEW84NfMnlJ0+mHI1U2hXJNqfRrQS3Qo2BDnaEHkjYT8reENl4RxOcvgkGN X-Received: by 2002:a05:6402:2751:: with SMTP id z17mr79594078edd.296.1638253989958; Mon, 29 Nov 2021 22:33:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638253989; cv=none; d=google.com; s=arc-20160816; b=ZT6VjZTeSK11Lhd2MR3AfEeAANdZlvJ4pC07xcL/AFbaAGfl18nN/GdI9KlGREVern hbYzGZcElQ1WLjJTaAlCuGFEpATg3bCYYYtIXGPUkuri1277ymnWackB9AHCnLu3a/xe x3Hd53ePDa8hgqQpWnUAqicSdM5/XKgSVgvUEI7GG+95VxfeOz2Kb0fyOVkztuRFTS+A Bgwssa8OH5afyiiOBQFbwO/rC54jvtThg4hc/rYHMDfMSo8a3qSlf8SnBuab55m42OHE 5GDKJc6DoENjUFjAlJlJRwARKBF43WTu11TfesL8zSKEGjE5gKb3WOA9EW30+C2CX8fg Decw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=jhAhhb3uZ8sp7zlFN8GPuZQwoOhod4KQgLA4GZbGMY0=; b=TibQ3ohWNgWy6S5OF0E8vV5aQDhzizeIyHhjITjg/F2Zx3pf6W/iveONx75hlSvwTr dr3qzZ60JCWCojZv3GW6PuKtCl2nNOuL9MmAvR7DA0LXqrlygIjZlaBNJfmro0QvGBdB 2K5ItSl9ZyxzYVhy5oE2E/dOXt+CiH9vRchs1tKD0bBRvReoyfqwknQVECC3BXvOXWnu fcDJacFQ/u+H/Usg8hDu9MhtocmuUgeX4bs/Hb+PZ0osv06ng4x5CM3B3OUtZdw6sopf MvLOL1xhYc8X573IOsnvg/7CkYpVTcUmnuf0O1vo//NnYK6NJNrb2jxQcmFwNgo6ZKfD HbUw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t13si27394399ejf.616.2021.11.29.22.33.09; Mon, 29 Nov 2021 22:33:09 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3D9FC68AE6B; Tue, 30 Nov 2021 08:32:34 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id F2B3068A7DE for ; Tue, 30 Nov 2021 08:32:25 +0200 (EET) X-IronPort-AV: E=McAfee;i="6200,9189,10183"; a="296958620" X-IronPort-AV: E=Sophos;i="5.87,275,1631602800"; d="scan'208";a="296958620" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Nov 2021 22:32:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,275,1631602800"; d="scan'208";a="653962408" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by fmsmga001.fm.intel.com with ESMTP; 29 Nov 2021 22:32:18 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Nov 2021 14:28:15 +0800 Message-Id: <20211130062815.708915-5-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211130062815.708915-1-wenbin.chen@intel.com> References: <20211130062815.708915-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V4 5/5] libavutil/hwcontext_vulkan: specify the modifier to create VKImage X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: mjPVi/PStIVU When vulkan image exports to drm, the tilling need to be VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. Now add code to create vulkan image using this format. Now the following command line works: ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format \ vaapi -i input_1080p.264 -vf "hwmap=derive_device=vulkan,format=vulkan, \ scale_vulkan=1920:1080,hwmap=derive_device=vaapi,format=vaapi" -c:v h264_vaapi output.264 Signed-off-by: Wenbin Chen --- libavutil/hwcontext_vulkan.c | 127 +++++++++++++++++++++++++++++++++-- 1 file changed, 121 insertions(+), 6 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index f980b72720..8224c0d4e4 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -120,6 +120,9 @@ typedef struct VulkanFramesPriv { /* Image transfers */ VulkanExecCtx upload_ctx; VulkanExecCtx download_ctx; + + /*modifier info*/ + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info; } VulkanFramesPriv; typedef struct AVVkFrameInternal { @@ -242,6 +245,28 @@ const VkFormat *av_vkfmt_from_pixfmt(enum AVPixelFormat p) return NULL; } +static void *find_in_structure_list(VkBaseOutStructure *stru_list, VkStructureType sType) { + if (!stru_list) + return NULL; + + for(;stru_list;stru_list = stru_list->pNext) + if (stru_list->sType == sType) + return stru_list; + + return NULL; +} + +static void append_to_structure_list(VkBaseOutStructure **stru_list, VkBaseOutStructure *added_stru) { + VkBaseOutStructure *p; + if (!*stru_list) { + *stru_list = added_stru; + return; + } + for(p = *stru_list; p->pNext; p = p->pNext); + p->pNext = added_stru; + return; +} + static int pixfmt_is_supported(AVHWDeviceContext *dev_ctx, enum AVPixelFormat p, int linear) { @@ -2094,6 +2119,10 @@ static void try_export_flags(AVHWFramesContext *hwfc, AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; FFVulkanFunctions *vk = &p->vkfn; + const int has_modifiers = hwctx->tiling == VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT; + int loop_count; + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info = find_in_structure_list(hwctx->create_pnext, + VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT); VkExternalImageFormatProperties eprops = { .sType = VK_STRUCTURE_TYPE_EXTERNAL_IMAGE_FORMAT_PROPERTIES_KHR, }; @@ -2101,9 +2130,18 @@ static void try_export_flags(AVHWFramesContext *hwfc, .sType = VK_STRUCTURE_TYPE_IMAGE_FORMAT_PROPERTIES_2, .pNext = &eprops, }; + VkPhysicalDeviceImageDrmFormatModifierInfoEXT phy_dev_mod_info = { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_DRM_FORMAT_MODIFIER_INFO_EXT, + .pNext = NULL, + .pQueueFamilyIndices = p->qfs, + .queueFamilyIndexCount = p->num_qfs, + .sharingMode = p->num_qfs > 1 ? VK_SHARING_MODE_CONCURRENT : + VK_SHARING_MODE_EXCLUSIVE, + }; VkPhysicalDeviceExternalImageFormatInfo enext = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_IMAGE_FORMAT_INFO, .handleType = exp, + .pNext = has_modifiers && modifier_info ? &phy_dev_mod_info : NULL, }; VkPhysicalDeviceImageFormatInfo2 pinfo = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_FORMAT_INFO_2, @@ -2115,11 +2153,16 @@ static void try_export_flags(AVHWFramesContext *hwfc, .flags = VK_IMAGE_CREATE_ALIAS_BIT, }; - ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, - &pinfo, &props); - if (ret == VK_SUCCESS) { - *iexp |= exp; - *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + loop_count = has_modifiers && modifier_info ? modifier_info->drmFormatModifierCount : 1; + for (int i = 0; i < loop_count; i++) { + if (has_modifiers && modifier_info) + phy_dev_mod_info.drmFormatModifier = modifier_info->pDrmFormatModifiers[i]; + ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, + &pinfo, &props); + if (ret == VK_SUCCESS) { + *iexp |= exp; + *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + } } } @@ -2190,6 +2233,12 @@ static void vulkan_frames_uninit(AVHWFramesContext *hwfc) { VulkanFramesPriv *fp = hwfc->internal->priv; + if (fp->modifier_info) { + if (fp->modifier_info->pDrmFormatModifiers) + av_freep(&fp->modifier_info->pDrmFormatModifiers); + av_freep(&fp->modifier_info); + } + free_exec_ctx(hwfc, &fp->conv_ctx); free_exec_ctx(hwfc, &fp->upload_ctx); free_exec_ctx(hwfc, &fp->download_ctx); @@ -2203,11 +2252,77 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) VulkanFramesPriv *fp = hwfc->internal->priv; AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info; + const int has_modifiers = !!(p->extensions & FF_VK_EXT_DRM_MODIFIER_FLAGS); /* Default pool flags */ - hwctx->tiling = hwctx->tiling ? hwctx->tiling : p->use_linear_images ? + hwctx->tiling = hwctx->tiling ? hwctx->tiling : has_modifiers ? + VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT : p->use_linear_images ? VK_IMAGE_TILING_LINEAR : VK_IMAGE_TILING_OPTIMAL; + modifier_info = find_in_structure_list(hwctx->create_pnext, + VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT); + /* get the supported modifier */ + if (has_modifiers && !modifier_info) { + const VkFormat *fmt = av_vkfmt_from_pixfmt(hwfc->sw_format); + FFVulkanFunctions *vk = &p->vkfn; + VkDrmFormatModifierPropertiesEXT *mod_props; + uint64_t *modifiers; + int modifier_count = 0; + + VkDrmFormatModifierPropertiesListEXT mod_props_list = { + .sType = VK_STRUCTURE_TYPE_DRM_FORMAT_MODIFIER_PROPERTIES_LIST_EXT, + .pNext = NULL, + .drmFormatModifierCount = 0, + .pDrmFormatModifierProperties = NULL, + }; + VkFormatProperties2 prop = { + .sType = VK_STRUCTURE_TYPE_FORMAT_PROPERTIES_2, + .pNext = &mod_props_list, + }; + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + if (!mod_props_list.drmFormatModifierCount) { + av_log(hwfc, AV_LOG_ERROR, "There are not supported modifiers for sw_format\n"); + return AVERROR(EINVAL); + } + + modifier_info = av_malloc(sizeof(*modifier_info)); + if (!modifier_info) + return AVERROR(ENOMEM); + append_to_structure_list((VkBaseOutStructure **)&hwctx->create_pnext, + (VkBaseOutStructure *)modifier_info); + fp->modifier_info = modifier_info; + modifier_info->pNext = NULL; + modifier_info->sType = VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT; + + modifiers = av_malloc(mod_props_list.drmFormatModifierCount * + sizeof(*modifiers)); + if (!modifiers) + return AVERROR(ENOMEM); + modifier_info->pDrmFormatModifiers = modifiers; + + mod_props = av_malloc(mod_props_list.drmFormatModifierCount * + sizeof(*mod_props)); + if (!mod_props) + return AVERROR(ENOMEM); + mod_props_list.pDrmFormatModifierProperties = mod_props; + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + for (int i = 0; i < mod_props_list.drmFormatModifierCount; i++) { + if (!(mod_props[i].drmFormatModifierTilingFeatures & FF_VK_DEFAULT_USAGE_FLAGS)) + continue; + modifiers[modifier_count++] = mod_props[i].drmFormatModifier; + } + if (!modifier_count) { + av_log(hwfc, AV_LOG_ERROR, "The supported modifiers doesn't support " + "default usage\n"); + av_freep(&mod_props); + return AVERROR(EINVAL); + } + modifier_info->drmFormatModifierCount = modifier_count; + av_freep(&mod_props); + } + + if (!hwctx->usage) hwctx->usage = FF_VK_DEFAULT_USAGE_FLAGS;