From patchwork Tue Nov 30 06:28:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 31827 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:d206:0:0:0:0:0 with SMTP id q6csp7254532iob; Mon, 29 Nov 2021 22:32:50 -0800 (PST) X-Google-Smtp-Source: ABdhPJxd59JKo9LZOC3GFLj9aBPxKOgj5tDIUMNIORAacuvzLvF4Pupo4gjTRfnmuyMziUe3EgME X-Received: by 2002:a17:907:9256:: with SMTP id kb22mr68352572ejb.423.1638253970229; Mon, 29 Nov 2021 22:32:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638253970; cv=none; d=google.com; s=arc-20160816; b=Sduv7ATGkZdpM7fxltmAxkJqaBTfcMZ4dj3ZgkDVNtv6sVqfgz92OVlvHfcRgCFmnn WGVPw8xMF/GxjCAYyfFM1FObVfpR/HnNM1q7SVHZwMLaBMfzXd35C8jQi7O1AJ3jEi+B /7zH4quuET/ZuFMCNraKM2v9E3pGwW3H5nAwnyx8MRJWyK1bg7sngLEliuHUm+pHd5uP DEwP8SBtFyYSvybp2TL7sUDYS88nDGNYjQbD51t6jI68bMMJdsdHzLlkhH8Uv1FLMMLP SIyhx3AKuvbxn2xsOeviigQDWdaqWsJZwNKz++Euwta3DTFeoJiRHvk+q2gQpNNNLubi mBZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=eK6DVZ84fkSkkl0MaTCcjPIt3l5P0p9/vsCtIu6/XTk=; b=yo+HB2QtqnkRRlgmTk+cjOwkvt0uq/T4YmYwzC0dIjh8TCu5HhNBRDwLi/5n+mGupJ pD6gFrcSGZv6cHU4rwOcRDbfwF6SpZiwkXMkmrY4mC7jYu7zT+tBL6oJ6OybAsX/8jKi ZSV8+4uW8fcpfbe+N5xiB5l3qgpe0H3JcMGKhf4q7NZc8P0vA4huwmXcxpHiPuqCvVrv TlHfev01o+JfelhrX5YQvQShEXSdJuv+ZtKMLK9CV7rBQhiDI94G/g9kpCm86bPtskTu itrpsyG9XAPHeHXHvdnLFKGDhTqvnxiBhaBko/sfylEXhgbjD0neUsWAJHncsgZ+H1E2 03DA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c14si35862909ede.582.2021.11.29.22.32.49; Mon, 29 Nov 2021 22:32:50 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B011C68ADE9; Tue, 30 Nov 2021 08:32:28 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 12B9168A7DE for ; Tue, 30 Nov 2021 08:32:20 +0200 (EET) X-IronPort-AV: E=McAfee;i="6200,9189,10183"; a="296958617" X-IronPort-AV: E=Sophos;i="5.87,275,1631602800"; d="scan'208";a="296958617" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Nov 2021 22:32:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,275,1631602800"; d="scan'208";a="653962402" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by fmsmga001.fm.intel.com with ESMTP; 29 Nov 2021 22:32:16 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Nov 2021 14:28:13 +0800 Message-Id: <20211130062815.708915-3-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211130062815.708915-1-wenbin.chen@intel.com> References: <20211130062815.708915-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V4 3/5] libavutil/hwcontext_vulkan: Allocate vkFrame in one memory X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: i45NJXtLPJob The vaapi can import external frame, but the planes of the external frames should be in the same drm object. A new option "contiguous_planes" is added to device. This flag tells device to allocate places in one memory. When device is derived from vaapi this flag will be enabled. A new flag frame_flag is also added to AVVulkanFramesContext. User can use this flag to force enable or disable this behaviour. A new variable "offset "is added to AVVKFrame. It describe describe the offset from the memory currently bound to the VkImage. Signed-off-by: Wenbin Chen --- libavutil/hwcontext_vulkan.c | 68 +++++++++++++++++++++++++++++++++++- libavutil/hwcontext_vulkan.h | 24 +++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index a0437c9661..eef9009ae1 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -103,8 +103,14 @@ typedef struct VulkanDevicePriv { /* Settings */ int use_linear_images; + /* allocate planes in a contiguous memory */ + int contiguous_planes; + /* Nvidia */ int dev_is_nvidia; + + /* Intel */ + int dev_is_intel; } VulkanDevicePriv; typedef struct VulkanFramesPriv { @@ -153,6 +159,8 @@ typedef struct AVVkFrameInternal { av_free((void *)props); \ } +#define VKF_FLAG(x, f) (((x) & (~AV_VK_FRAME_FLAG_NONE)) & (f)) + static const struct { enum AVPixelFormat pixfmt; const VkFormat vkfmts[4]; @@ -1374,6 +1382,13 @@ static int vulkan_device_create_internal(AVHWDeviceContext *ctx, if (opt_d) p->use_linear_images = strtol(opt_d->value, NULL, 10); + opt_d = av_dict_get(opts, "contiguous_planes", NULL, 0); + if (opt_d) + p->contiguous_planes = strtol(opt_d->value, NULL, 10); + else + p->contiguous_planes = -1; + + hwctx->enabled_dev_extensions = dev_info.ppEnabledExtensionNames; hwctx->nb_enabled_dev_extensions = dev_info.enabledExtensionCount; @@ -1425,6 +1440,8 @@ static int vulkan_device_init(AVHWDeviceContext *ctx) p->dev_is_nvidia = (p->props.properties.vendorID == 0x10de); + p->dev_is_intel = (p->props.properties.vendorID == 0x8086); + vk->GetPhysicalDeviceQueueFamilyProperties(hwctx->phys_dev, &queue_num, NULL); if (!queue_num) { av_log(ctx, AV_LOG_ERROR, "Failed to get queues!\n"); @@ -1742,8 +1759,12 @@ static int alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f, AVHWDeviceContext *ctx = hwfc->device_ctx; VulkanDevicePriv *p = ctx->internal->priv; FFVulkanFunctions *vk = &p->vkfn; + AVVulkanFramesContext *hwfctx = hwfc->hwctx; const int planes = av_pix_fmt_count_planes(hwfc->sw_format); VkBindImageMemoryInfo bind_info[AV_NUM_DATA_POINTERS] = { { 0 } }; + VkMemoryRequirements memory_requirements = { 0 }; + int mem_size = 0; + int mem_size_list[AV_NUM_DATA_POINTERS] = { 0 }; AVVulkanDeviceContext *hwctx = ctx->hwctx; @@ -1771,6 +1792,19 @@ static int alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f, req.memoryRequirements.size = FFALIGN(req.memoryRequirements.size, p->props.properties.limits.minMemoryMapAlignment); + if (VKF_FLAG(hwfctx->flags, AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) { + if (memory_requirements.size == 0) { + memory_requirements = req.memoryRequirements; + } else if (memory_requirements.memoryTypeBits != req.memoryRequirements.memoryTypeBits) { + av_log(hwfc, AV_LOG_ERROR, "the param for each planes are not the same\n"); + return AVERROR(EINVAL); + } + + mem_size_list[i] = req.memoryRequirements.size; + mem_size += mem_size_list[i]; + continue; + } + /* In case the implementation prefers/requires dedicated allocation */ use_ded_mem = ded_req.prefersDedicatedAllocation | ded_req.requiresDedicatedAllocation; @@ -1792,6 +1826,29 @@ static int alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f, bind_info[i].memory = f->mem[i]; } + if (VKF_FLAG(hwfctx->flags, AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) { + memory_requirements.size = mem_size; + + /* Allocate memory */ + if ((err = alloc_mem(ctx, &memory_requirements, + f->tiling == VK_IMAGE_TILING_LINEAR ? + VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT : + VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT, + (void *)(((uint8_t *)alloc_pnext)), + &f->flags, &f->mem[0]))) + return err; + + f->size[0] = memory_requirements.size; + + for (int i = 0; i < planes; i++) { + bind_info[i].sType = VK_STRUCTURE_TYPE_BIND_IMAGE_MEMORY_INFO; + bind_info[i].image = f->img[i]; + bind_info[i].memory = f->mem[0]; + bind_info[i].memoryOffset = i == 0 ? 0 : mem_size_list[i-1]; + f->offset[i] = bind_info[i].memoryOffset; + } + } + /* Bind the allocated memory to the images */ ret = vk->BindImageMemory2(hwctx->act_dev, planes, bind_info); if (ret != VK_SUCCESS) { @@ -2154,6 +2211,12 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) if (!hwctx->usage) hwctx->usage = FF_VK_DEFAULT_USAGE_FLAGS; + if (!(hwctx->flags & AV_VK_FRAME_FLAG_NONE)) { + if (p->contiguous_planes == 1 || + ((p->contiguous_planes == -1) && p->dev_is_intel)) + hwctx->flags |= AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY; + } + err = create_exec_ctx(hwfc, &fp->conv_ctx, dev_hwctx->queue_family_comp_index, dev_hwctx->nb_comp_queues); @@ -3074,6 +3137,7 @@ static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, FFVulkanFunctions *vk = &p->vkfn; VulkanFramesPriv *fp = hwfc->internal->priv; AVVulkanDeviceContext *hwctx = hwfc->device_ctx->hwctx; + AVVulkanFramesContext *hwfctx = hwfc->hwctx; const int planes = av_pix_fmt_count_planes(hwfc->sw_format); VkImageDrmFormatModifierPropertiesEXT drm_mod = { .sType = VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_PROPERTIES_EXT, @@ -3142,7 +3206,9 @@ static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, continue; vk->GetImageSubresourceLayout(hwctx->act_dev, f->img[i], &sub, &layout); - drm_desc->layers[i].planes[0].offset = layout.offset; + drm_desc->layers[i].planes[0].offset = layout.offset + + VKF_FLAG(hwfctx->flags, AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY) ? + f->offset[i] : 0; drm_desc->layers[i].planes[0].pitch = layout.rowPitch; } diff --git a/libavutil/hwcontext_vulkan.h b/libavutil/hwcontext_vulkan.h index fdf2a60156..c485ee7437 100644 --- a/libavutil/hwcontext_vulkan.h +++ b/libavutil/hwcontext_vulkan.h @@ -35,6 +35,17 @@ * with the data pointer set to an AVVkFrame. */ +/** + * Defines the behaviour of frame allocation + * AV_VK_FRAME_FLAG_NONE: planes will be allocated in separte memory + * AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY: planes will be allocated in a + * contiguous memory. + */ +typedef enum { + AV_VK_FRAME_FLAG_NONE = (1ULL << 0), + AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY = (1ULL << 1) | 1ULL +} AVVkFrameFlags; + /** * Main Vulkan context, allocated as AVHWDeviceContext.hwctx. * All of these can be set before init to change what the context uses @@ -157,6 +168,14 @@ typedef struct AVVulkanFramesContext { */ void *create_pnext; + /** + * Is a combination of AVVkFrameFlags. Defines the behaviour of frame + * allocation. + * If no flag is set, then the flags are automatically determined + * based on the device. + */ + int flags; + /** * Extension data for memory allocation. Must have as many entries as * the number of planes of the sw_format. @@ -198,6 +217,11 @@ typedef struct AVVkFrame { VkDeviceMemory mem[AV_NUM_DATA_POINTERS]; size_t size[AV_NUM_DATA_POINTERS]; + /** + * Describe the offset from the memory currently bound to the VkImage. + */ + size_t offset[AV_NUM_DATA_POINTERS]; + /** * OR'd flags for all memory allocated */