From patchwork Wed Nov 24 05:28:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 31618 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:d206:0:0:0:0:0 with SMTP id q6csp9859631iob; Tue, 23 Nov 2021 21:33:42 -0800 (PST) X-Google-Smtp-Source: ABdhPJx1L02nx5zYAPPH9Y8/pHhWEFJJukc3iIEdawvOAoG63thMLU1Ru/oYFiR1HRYf2gF9dMQd X-Received: by 2002:a50:ee96:: with SMTP id f22mr19776233edr.77.1637732022616; Tue, 23 Nov 2021 21:33:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1637732022; cv=none; d=google.com; s=arc-20160816; b=WQGyXVDCDg2AFAwSJpV4fWrrQdKD+LwshSE1d4sH0WZlCgNioPvUGuRPjJ+FT8/DEy RjUky2anSfXEu1QENWGbM/ISEkX9JoD3MmxrWISR00Lixbb27FI9eQHWZHHZok96g8Fe PCPGtbeuwNMgXWuHYsfo+lltI4gnB63WpvFNE1OGxCZDh/8iaEb5BH9nBvLRWjfSGwTQ 5QIb4m7GKbMzwEiyekgR5coiFv6p216oMbN1t1FY19Vf8R8tC94k4uwTipBbXcGPBPLB xGpLT84BFJv0GefModgRNfa6JFt2sGA3LfDgBbidshEHR29yeLEZgBDJE63ecn/pf8qn BVrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=wWwE0C8XfZBMDatjljx9o97br6Yruz1V2Wy12mepmuQ=; b=gwE2ekkmbgUgdZFWvWODSigkBTXnbAdiN/MwzN1uoWQyERNJWrwun2HX2BZ8wfb9gM 5RWnycEIyqeHUwddGuEBtWUVvaUbkn+RHPXd2h8aw16FgsTnmnwQe8djNdRa01uhHveQ lZym7NIMUPb7oakC4jTH05KUQJFo+u/mKurAxY/yk9klI2dcZDSfikA2tucivDRiCG0O HDjgO34KAXpnVsHZoKCClrYv6Np9BvbdYXB3jIlBqc84//pvCy6GptHZFLrtzKRY55jR 7L+cx5S8ER0UvvEyuVW8QToGEOcqBGCpCWv01JfSESZHBBpUz8ls9vyYTq3RjT8p7RzB IzQg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z9si51770706edd.108.2021.11.23.21.33.41; Tue, 23 Nov 2021 21:33:42 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CB6EF68AF21; Wed, 24 Nov 2021 07:33:11 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 42AA468AF1A for ; Wed, 24 Nov 2021 07:33:04 +0200 (EET) X-IronPort-AV: E=McAfee;i="6200,9189,10177"; a="296011587" X-IronPort-AV: E=Sophos;i="5.87,258,1631602800"; d="scan'208";a="296011587" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2021 21:32:49 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,258,1631602800"; d="scan'208";a="457334331" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by orsmga006.jf.intel.com with ESMTP; 23 Nov 2021 21:32:48 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Wed, 24 Nov 2021 13:28:48 +0800 Message-Id: <20211124052848.1122682-5-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211124052848.1122682-1-wenbin.chen@intel.com> References: <20211124052848.1122682-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 5/5] libavutil/hwcontext_vulkan: specify the modifier to create VKImage X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 6vCV69DHMCjS When vulkan image exports to drm, the tilling need to be VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. Now add code to create vulkan image using this format. Now the following command line works: ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format \ vaapi -i input_1080p.264 -vf "hwmap=derive_device=vulkan,format=vulkan, \ scale_vulkan=1920:1080,hwmap=derive_device=vaapi,format=vaapi" -c:v h264_vaapi output.264 Signed-off-by: Wenbin Chen --- libavutil/hwcontext_vulkan.c | 130 +++++++++++++++++++++++++++++++++-- 1 file changed, 124 insertions(+), 6 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 6421115385..4b951fb202 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -230,6 +230,28 @@ const VkFormat *av_vkfmt_from_pixfmt(enum AVPixelFormat p) return NULL; } +static void *find_in_structure_list(VkBaseOutStructure *stru_list, VkStructureType sType) { + if (!stru_list) + return NULL; + + for(;stru_list;stru_list = stru_list->pNext) + if (stru_list->sType == sType) + return stru_list; + + return NULL; +} + +static void append_to_structure_list(VkBaseOutStructure **stru_list, VkBaseOutStructure *added_stru) { + VkBaseOutStructure *p; + if (!*stru_list) { + *stru_list = added_stru; + return; + } + for(p = *stru_list; p->pNext; p = p->pNext); + p->pNext = added_stru; + return; +} + static int pixfmt_is_supported(AVHWDeviceContext *dev_ctx, enum AVPixelFormat p, int linear) { @@ -1979,6 +2001,10 @@ static void try_export_flags(AVHWFramesContext *hwfc, AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; FFVulkanFunctions *vk = &p->vkfn; + const int has_modifiers = hwctx->tiling == VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT; + int loop_count; + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info = find_in_structure_list(hwctx->create_pnext, + VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT); VkExternalImageFormatProperties eprops = { .sType = VK_STRUCTURE_TYPE_EXTERNAL_IMAGE_FORMAT_PROPERTIES_KHR, }; @@ -1986,9 +2012,18 @@ static void try_export_flags(AVHWFramesContext *hwfc, .sType = VK_STRUCTURE_TYPE_IMAGE_FORMAT_PROPERTIES_2, .pNext = &eprops, }; + VkPhysicalDeviceImageDrmFormatModifierInfoEXT phy_dev_mod_info = { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_DRM_FORMAT_MODIFIER_INFO_EXT, + .pNext = NULL, + .pQueueFamilyIndices = p->qfs, + .queueFamilyIndexCount = p->num_qfs, + .sharingMode = p->num_qfs > 1 ? VK_SHARING_MODE_CONCURRENT : + VK_SHARING_MODE_EXCLUSIVE, + }; VkPhysicalDeviceExternalImageFormatInfo enext = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_IMAGE_FORMAT_INFO, .handleType = exp, + .pNext = has_modifiers && modifier_info ? &phy_dev_mod_info : NULL, }; VkPhysicalDeviceImageFormatInfo2 pinfo = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_FORMAT_INFO_2, @@ -2000,11 +2035,16 @@ static void try_export_flags(AVHWFramesContext *hwfc, .flags = VK_IMAGE_CREATE_ALIAS_BIT, }; - ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, - &pinfo, &props); - if (ret == VK_SUCCESS) { - *iexp |= exp; - *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + loop_count = has_modifiers && modifier_info ? modifier_info->drmFormatModifierCount : 1; + for (int i = 0; i < loop_count; i++) { + if (has_modifiers && modifier_info) + phy_dev_mod_info.drmFormatModifier = modifier_info->pDrmFormatModifiers[i]; + ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, + &pinfo, &props); + if (ret == VK_SUCCESS) { + *iexp |= exp; + *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + } } } @@ -2074,6 +2114,20 @@ fail: static void vulkan_frames_uninit(AVHWFramesContext *hwfc) { VulkanFramesPriv *fp = hwfc->internal->priv; + AVVulkanFramesContext *hwctx = hwfc->hwctx; + VkBaseOutStructure *structure_p_next,*structure_p = hwctx->create_pnext; + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info; + while (structure_p) { + structure_p_next = structure_p->pNext; + switch (structure_p->sType) { + case VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT: + modifier_info = (VkImageDrmFormatModifierListCreateInfoEXT *)structure_p; + av_freep(&modifier_info->pDrmFormatModifiers); + av_freep(&modifier_info); + break; + } + structure_p = structure_p_next; + } free_exec_ctx(hwfc, &fp->conv_ctx); free_exec_ctx(hwfc, &fp->upload_ctx); @@ -2088,11 +2142,75 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) VulkanFramesPriv *fp = hwfc->internal->priv; AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; + const int has_modifiers = !!(p->extensions & FF_VK_EXT_DRM_MODIFIER_FLAGS); /* Default pool flags */ - hwctx->tiling = hwctx->tiling ? hwctx->tiling : p->use_linear_images ? + hwctx->tiling = hwctx->tiling ? hwctx->tiling : has_modifiers ? + VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT : p->use_linear_images ? VK_IMAGE_TILING_LINEAR : VK_IMAGE_TILING_OPTIMAL; + /* get the supported modifier */ + if (has_modifiers) { + const VkFormat *fmt = av_vkfmt_from_pixfmt(hwfc->sw_format); + FFVulkanFunctions *vk = &p->vkfn; + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info; + VkDrmFormatModifierPropertiesEXT *mod_props; + uint64_t *modifiers; + int modifier_count = 0; + + + VkDrmFormatModifierPropertiesListEXT mod_props_list = { + .sType = VK_STRUCTURE_TYPE_DRM_FORMAT_MODIFIER_PROPERTIES_LIST_EXT, + .pNext = NULL, + .drmFormatModifierCount = 0, + .pDrmFormatModifierProperties = NULL, + }; + VkFormatProperties2 prop = { + .sType = VK_STRUCTURE_TYPE_FORMAT_PROPERTIES_2, + .pNext = &mod_props_list, + }; + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + if (!mod_props_list.drmFormatModifierCount) { + av_log(hwfc, AV_LOG_ERROR, "There are not supported modifiers for sw_format\n"); + return AVERROR(EINVAL); + } + + modifier_info = av_malloc(sizeof(*modifier_info)); + if (!modifier_info) + return AVERROR(ENOMEM); + append_to_structure_list((VkBaseOutStructure **)&hwctx->create_pnext, + (VkBaseOutStructure *)modifier_info); + modifier_info->pNext = NULL; + modifier_info->sType = VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT; + + modifiers = av_malloc(mod_props_list.drmFormatModifierCount * + sizeof(*modifiers)); + if (!modifiers) + return AVERROR(ENOMEM); + modifier_info->pDrmFormatModifiers = modifiers; + + mod_props = av_malloc(mod_props_list.drmFormatModifierCount * + sizeof(*mod_props)); + if (!mod_props) + return AVERROR(ENOMEM); + mod_props_list.pDrmFormatModifierProperties = mod_props; + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + for (int i = 0; i < mod_props_list.drmFormatModifierCount; i++) { + if (!(mod_props[i].drmFormatModifierTilingFeatures & FF_VK_DEFAULT_USAGE_FLAGS)) + continue; + modifiers[modifier_count++] = mod_props[i].drmFormatModifier; + } + if (!modifier_count) { + av_log(hwfc, AV_LOG_ERROR, "The supported modifiers doesn't support " + "default usage\n"); + av_freep(&mod_props); + return AVERROR(EINVAL); + } + modifier_info->drmFormatModifierCount = modifier_count; + av_freep(&mod_props); + } + + if (!hwctx->usage) hwctx->usage = FF_VK_DEFAULT_USAGE_FLAGS;