From patchwork Tue Dec 7 09:05:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 32096 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp5976215iog; Tue, 7 Dec 2021 01:10:12 -0800 (PST) X-Google-Smtp-Source: ABdhPJxGRfZULlYlq9hpTGOPe6dofD5mlKysEy+Mo90iTWRyH1DFInBdfAxABOvbNXH3QseXZejy X-Received: by 2002:a17:907:a426:: with SMTP id sg38mr52094328ejc.392.1638868212549; Tue, 07 Dec 2021 01:10:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638868212; cv=none; d=google.com; s=arc-20160816; b=b4txVDSuVhg9aO7g/u/OrGrw0d6x+zjwgzmZoCifrGVPepKGnvUufvItNqouzo8H5J oXyrEMkAFNK2vy7rOhIdIE9vS344wjmAzFUGneK/3orYY9eK2upv85PVxYab+krUSvoC tezjZopjUBL2t5Y4HD6Nsjf7KjDtLd7JpPmROjTMG6BqswOyyt4am1L59/8T9eTBw1DM jIR52WR5IJf1Xrn1r85OUbWTNPt1SjgLljeFUlQS05r9XgxiKyYiUVmKWPNBncLOyhmC rShTwwYXpeG+hHTaVuGhrvAH7Ym1P0NmmHJKZW0N7f7xfEvR96g4UhxZt36WMhqRMjzC hoqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=B9ZS4Q5STUEXuN7xPBnLS0OWXjWnqqhCX4dg195bg3Y=; b=JjCHvzzhXEbGtbvw6fXDC2FaTyflxutHFYCYsfAxACEsKDIcWzPl+UKwlgwH1Mf4qB 6gpzvT9+9r5US5EtbXViQqm8XNqaGsPYR3E8B4p+JoChLmoGaF5AsZwNgOG8hFP0s8LY UwOGfMCN9RwfeZThu9aLHhzswGzCQqTb4QOO7W35px7ZqBFhS7bht2TOV2edsxh7c+BN mc0e4pnFX077P7UvFz/gewylWUaHZwjKK3oG2x5zmvSKglqLN5Vk/LAtar5hQ5aFKqAK GvtIRlBlWDIKF4l1Ce99A5oMIGYnzEOHqrJLz3RFmYVw+HDsaKUUVJ1XraLIV5p5BEx4 b7iQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id w18si21758630edq.29.2021.12.07.01.10.11; Tue, 07 Dec 2021 01:10:12 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 58CBA68A980; Tue, 7 Dec 2021 11:10:09 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5824168A8FA for ; Tue, 7 Dec 2021 11:10:01 +0200 (EET) X-IronPort-AV: E=McAfee;i="6200,9189,10190"; a="224794090" X-IronPort-AV: E=Sophos;i="5.87,293,1631602800"; d="scan'208";a="224794090" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Dec 2021 01:09:59 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,293,1631602800"; d="scan'208";a="515198457" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by orsmga008.jf.intel.com with ESMTP; 07 Dec 2021 01:09:58 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Tue, 7 Dec 2021 17:05:49 +0800 Message-Id: <20211207090553.2138848-1-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V6 1/5] hwcontext_vaapi: Use PRIME_2 memory type for modifiers. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 6wV3DGEi8XVL From: Bas Nieuwenhuizen This way we can pass explicit modifiers in. Sometimes the modifier matters for the number of memory planes that libva accepts, in particular when dealing with driver-compressed textures. Furthermore the driver might not actually be able to determine the implicit modifier if all the buffer-passing has used explicit modifier. All these issues should be resolved by passing in the modifier, and for that we switch to using the PRIME_2 memory type. Tested with experimental radeonsi patches for modifiers and kmsgrab. Also tested with radeonsi without the patches to double-check it works without PRIME_2 support. v2: Cache PRIME_2 support to avoid doing two calls every time on libva drivers that do not support it. v3: Remove prime2_vas usage. Signed-off-by: Bas Nieuwenhuizen --- libavutil/hwcontext_vaapi.c | 158 ++++++++++++++++++++++++++---------- 1 file changed, 114 insertions(+), 44 deletions(-) diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c index 83e542876d..75acc851d6 100644 --- a/libavutil/hwcontext_vaapi.c +++ b/libavutil/hwcontext_vaapi.c @@ -79,6 +79,9 @@ typedef struct VAAPIFramesContext { unsigned int rt_format; // Whether vaDeriveImage works. int derive_works; + // Caches whether VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2 is unsupported for + // surface imports. + int prime_2_import_unsupported; } VAAPIFramesContext; typedef struct VAAPIMapping { @@ -1022,32 +1025,17 @@ static void vaapi_unmap_from_drm(AVHWFramesContext *dst_fc, static int vaapi_map_from_drm(AVHWFramesContext *src_fc, AVFrame *dst, const AVFrame *src, int flags) { + VAAPIFramesContext *src_vafc = src_fc->internal->priv; AVHWFramesContext *dst_fc = (AVHWFramesContext*)dst->hw_frames_ctx->data; AVVAAPIDeviceContext *dst_dev = dst_fc->device_ctx->hwctx; const AVDRMFrameDescriptor *desc; const VAAPIFormatDescriptor *format_desc; VASurfaceID surface_id; - VAStatus vas; + VAStatus vas = VA_STATUS_SUCCESS; + int use_prime2; uint32_t va_fourcc; - int err, i, j, k; - - unsigned long buffer_handle; - VASurfaceAttribExternalBuffers buffer_desc; - VASurfaceAttrib attrs[2] = { - { - .type = VASurfaceAttribMemoryType, - .flags = VA_SURFACE_ATTRIB_SETTABLE, - .value.type = VAGenericValueTypeInteger, - .value.value.i = VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME, - }, - { - .type = VASurfaceAttribExternalBufferDescriptor, - .flags = VA_SURFACE_ATTRIB_SETTABLE, - .value.type = VAGenericValueTypePointer, - .value.value.p = &buffer_desc, - } - }; + int err, i, j; desc = (AVDRMFrameDescriptor*)src->data[0]; @@ -1083,35 +1071,117 @@ static int vaapi_map_from_drm(AVHWFramesContext *src_fc, AVFrame *dst, format_desc = vaapi_format_from_fourcc(va_fourcc); av_assert0(format_desc); - buffer_handle = desc->objects[0].fd; - buffer_desc.pixel_format = va_fourcc; - buffer_desc.width = src_fc->width; - buffer_desc.height = src_fc->height; - buffer_desc.data_size = desc->objects[0].size; - buffer_desc.buffers = &buffer_handle; - buffer_desc.num_buffers = 1; - buffer_desc.flags = 0; - - k = 0; - for (i = 0; i < desc->nb_layers; i++) { - for (j = 0; j < desc->layers[i].nb_planes; j++) { - buffer_desc.pitches[k] = desc->layers[i].planes[j].pitch; - buffer_desc.offsets[k] = desc->layers[i].planes[j].offset; - ++k; + use_prime2 = !src_vafc->prime_2_import_unsupported && + desc->objects[0].format_modifier != DRM_FORMAT_MOD_INVALID; + if (use_prime2) { + VADRMPRIMESurfaceDescriptor prime_desc; + VASurfaceAttrib prime_attrs[2] = { + { + .type = VASurfaceAttribMemoryType, + .flags = VA_SURFACE_ATTRIB_SETTABLE, + .value.type = VAGenericValueTypeInteger, + .value.value.i = VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2, + }, + { + .type = VASurfaceAttribExternalBufferDescriptor, + .flags = VA_SURFACE_ATTRIB_SETTABLE, + .value.type = VAGenericValueTypePointer, + .value.value.p = &prime_desc, + } + }; + prime_desc.fourcc = va_fourcc; + prime_desc.width = src_fc->width; + prime_desc.height = src_fc->height; + prime_desc.num_objects = desc->nb_objects; + for (i = 0; i < desc->nb_objects; ++i) { + prime_desc.objects[i].fd = desc->objects[i].fd; + prime_desc.objects[i].size = desc->objects[i].size; + prime_desc.objects[i].drm_format_modifier = + desc->objects[i].format_modifier; } - } - buffer_desc.num_planes = k; - if (format_desc->chroma_planes_swapped && - buffer_desc.num_planes == 3) { - FFSWAP(uint32_t, buffer_desc.pitches[1], buffer_desc.pitches[2]); - FFSWAP(uint32_t, buffer_desc.offsets[1], buffer_desc.offsets[2]); + prime_desc.num_layers = desc->nb_layers; + for (i = 0; i < desc->nb_layers; ++i) { + prime_desc.layers[i].drm_format = desc->layers[i].format; + prime_desc.layers[i].num_planes = desc->layers[i].nb_planes; + for (j = 0; j < desc->layers[i].nb_planes; ++j) { + prime_desc.layers[i].object_index[j] = + desc->layers[i].planes[j].object_index; + prime_desc.layers[i].offset[j] = desc->layers[i].planes[j].offset; + prime_desc.layers[i].pitch[j] = desc->layers[i].planes[j].pitch; + } + + if (format_desc->chroma_planes_swapped && + desc->layers[i].nb_planes == 3) { + FFSWAP(uint32_t, prime_desc.layers[i].pitch[1], + prime_desc.layers[i].pitch[2]); + FFSWAP(uint32_t, prime_desc.layers[i].offset[1], + prime_desc.layers[i].offset[2]); + } + } + + /* + * We can query for PRIME_2 support with vaQuerySurfaceAttributes, but that + * that needs the config_id which we don't have here . Both Intel and + * Gallium seem to do the correct error checks, so lets just try the + * PRIME_2 import first. + */ + vas = vaCreateSurfaces(dst_dev->display, format_desc->rt_format, + src->width, src->height, &surface_id, 1, + prime_attrs, FF_ARRAY_ELEMS(prime_attrs)); + if (vas != VA_STATUS_SUCCESS) + src_vafc->prime_2_import_unsupported = 1; } - vas = vaCreateSurfaces(dst_dev->display, format_desc->rt_format, - src->width, src->height, - &surface_id, 1, - attrs, FF_ARRAY_ELEMS(attrs)); + if (!use_prime2 || vas != VA_STATUS_SUCCESS) { + int k; + unsigned long buffer_handle; + VASurfaceAttribExternalBuffers buffer_desc; + VASurfaceAttrib buffer_attrs[2] = { + { + .type = VASurfaceAttribMemoryType, + .flags = VA_SURFACE_ATTRIB_SETTABLE, + .value.type = VAGenericValueTypeInteger, + .value.value.i = VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME, + }, + { + .type = VASurfaceAttribExternalBufferDescriptor, + .flags = VA_SURFACE_ATTRIB_SETTABLE, + .value.type = VAGenericValueTypePointer, + .value.value.p = &buffer_desc, + } + }; + + buffer_handle = desc->objects[0].fd; + buffer_desc.pixel_format = va_fourcc; + buffer_desc.width = src_fc->width; + buffer_desc.height = src_fc->height; + buffer_desc.data_size = desc->objects[0].size; + buffer_desc.buffers = &buffer_handle; + buffer_desc.num_buffers = 1; + buffer_desc.flags = 0; + + k = 0; + for (i = 0; i < desc->nb_layers; i++) { + for (j = 0; j < desc->layers[i].nb_planes; j++) { + buffer_desc.pitches[k] = desc->layers[i].planes[j].pitch; + buffer_desc.offsets[k] = desc->layers[i].planes[j].offset; + ++k; + } + } + buffer_desc.num_planes = k; + + if (format_desc->chroma_planes_swapped && + buffer_desc.num_planes == 3) { + FFSWAP(uint32_t, buffer_desc.pitches[1], buffer_desc.pitches[2]); + FFSWAP(uint32_t, buffer_desc.offsets[1], buffer_desc.offsets[2]); + } + + vas = vaCreateSurfaces(dst_dev->display, format_desc->rt_format, + src->width, src->height, + &surface_id, 1, + buffer_attrs, FF_ARRAY_ELEMS(buffer_attrs)); + } if (vas != VA_STATUS_SUCCESS) { av_log(dst_fc, AV_LOG_ERROR, "Failed to create surface from DRM " "object: %d (%s).\n", vas, vaErrorStr(vas)); From patchwork Tue Dec 7 09:05:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 32097 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp5976446iog; Tue, 7 Dec 2021 01:10:22 -0800 (PST) X-Google-Smtp-Source: ABdhPJwDFwB0tt4TKJO0l9HGG+ojFq/7xCbUV+dJqN0wD1mMaoGRwTkXcczMeIVRfR4ImlOww+Pt X-Received: by 2002:a17:906:a215:: with SMTP id r21mr50700651ejy.21.1638868222166; Tue, 07 Dec 2021 01:10:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638868222; cv=none; d=google.com; s=arc-20160816; b=RAy8BV0HP8BzjNf3SWSi7TQnqlONzk4PrFgHdXD0G58VtMHY10kPdMBFMw2q1WzcAv V7ynbGKkmwUEc0Alyk5NaWb5iPtjs0JsDENM0CuEYdnJ0MAdoRz4dj+vmohszngaBKdc W9wsagDMkekieMkhdgT6RyztbJShU4kDVN2EUk1gLuEa1kdXqCAHY1h3dOrBvNhzSmi2 Qqp8mO4zlbllscqUu8AoFflwdmJgerBQMZZEMI0BAZbZXc0Oh3ymqkQRiL0MTsgF4Occ /s2bYQQvItA57KNDuZRfAMN4QS5AKM8MB87bIAlrXtPQBv7E1kZYDvrQqDcyw8tiNY7L 498A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=JrvPegIXt3WVsBm98MngVNyOCmhchkjbCrtfuvo3WjU=; b=YtlbxBmj/7QvAF2hz3sY6+4hC75i1chuj1ENDkGXeJTL+l9/cs01RCPHpfkSOt4TXq mWLja+vp3o8TsdxrjUqxUR0Wo730ooA5bbPaiWLJ0oMOW3nYBEUmE/PehrxBNqUCbrjB IewDIPPxTpRn/A7tS5lJmIMaBwYo5BQX+OJvqW0jsRiF9sbicGTU8JQc/ukFOS3tg49Z nkJxKVmus2qi6CWGDvUIoq0JVgV51bPKh2KpKZ5XKCYQ0g75vY5G1WQBG8DUJYCRBxHz KovymJArMPuC4iV18nDuohBeRB7+gngvMwA092LGcSnKXqqzLWBKpLsxrEOmvDFUj0Z3 Rf/g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cw13si20411670ejb.228.2021.12.07.01.10.21; Tue, 07 Dec 2021 01:10:22 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6B9E568ADDE; Tue, 7 Dec 2021 11:10:11 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6001A68A980 for ; Tue, 7 Dec 2021 11:10:03 +0200 (EET) X-IronPort-AV: E=McAfee;i="6200,9189,10190"; a="224794093" X-IronPort-AV: E=Sophos;i="5.87,293,1631602800"; d="scan'208";a="224794093" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Dec 2021 01:09:59 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,293,1631602800"; d="scan'208";a="515198460" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by orsmga008.jf.intel.com with ESMTP; 07 Dec 2021 01:09:59 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Tue, 7 Dec 2021 17:05:50 +0800 Message-Id: <20211207090553.2138848-2-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211207090553.2138848-1-wenbin.chen@intel.com> References: <20211207090553.2138848-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V6 2/5] libavutil/hwcontext_vaapi: Add a new nv12 format map to support vulkan frame X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Cp8VULctt6DI Vulkan will map nv12 to R8 and GR88, so add this map to vaapi to support vulkan frame. Signed-off-by: Wenbin Chen --- libavutil/hwcontext_vaapi.c | 1 + 1 file changed, 1 insertion(+) diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c index 75acc851d6..994b744e4d 100644 --- a/libavutil/hwcontext_vaapi.c +++ b/libavutil/hwcontext_vaapi.c @@ -992,6 +992,7 @@ static const struct { } vaapi_drm_format_map[] = { #ifdef DRM_FORMAT_R8 DRM_MAP(NV12, 2, DRM_FORMAT_R8, DRM_FORMAT_RG88), + DRM_MAP(NV12, 2, DRM_FORMAT_R8, DRM_FORMAT_GR88), #endif DRM_MAP(NV12, 1, DRM_FORMAT_NV12), #if defined(VA_FOURCC_P010) && defined(DRM_FORMAT_R16) From patchwork Tue Dec 7 09:05:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 32098 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp5976635iog; Tue, 7 Dec 2021 01:10:32 -0800 (PST) X-Google-Smtp-Source: ABdhPJxbMJy56EFk9iL0mnOj+Fm8ZxhmeRx7NIVYXjPTf+yavDwTAju2bS7rIoQPaXiSCcSEO47Y X-Received: by 2002:a05:6402:14f:: with SMTP id s15mr7350510edu.118.1638868232269; Tue, 07 Dec 2021 01:10:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638868232; cv=none; d=google.com; s=arc-20160816; b=AM3j4aYYCZQZcuJACPXXZY5fnVx/upgV+ZewX7Yeh5ahTrl2AyQiN9sekwmY6xV5l7 WDQl4m22/oY3IUtEV4QPzkJlFlRrN3eOU3pWDqhwbnlQlrkF1URp4w0NPzbouokSyPeP cBcSDidIgyE4Fg7flZaj77tVnowhR3ApLUjod7PxVBhVaB+akvcX5umvhs19FdpUzgw5 tE1C872pR3pTBEBk3h0uJnTSKgtlFpzvUQjgi/0admvaDhrrMQ2tlUsGNXNZlZkDxOw9 JUOFFje+xGAc0QvA3u2URvgz+reLWkplD8HZqQ49sKEsPCtAcL1zAwPzoVZajhNObmC4 du5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=0HRqub4o7oG/tVaF9WN1xDpdv1ZW90VATnG9bgiqgsc=; b=wrMg4Gz09/BVdqaMcFwm9A8WT47o6+8mIfuZxI6bz6uukPoJJAvs5Lwph9yNGK1ngs IiJXAsg3zycgPc/LqVRXki8+nGRkjQTPKtlu8SG7xpebCKqJTFaCzkmV3nbQSKn6OAW8 yKBO9mDjZBFMHfWotVR+TdCErGesvlIv1X+nMUr1DkJ/Ny+micYBwXCWul2HilatofNO vulAdsMocalwFaY7gI+ELwOq5nUWvmOgSDTunA9vWYW9JJ5/pv0/rUK+sRDDSc1W72W4 54v5FutUsbcRpCzW9cM3V0kzk8r+T1aV3H8K64VC9fR0krq9zZ0c1aHHhao7ujHbIlRx S7kg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b6si20542181eja.133.2021.12.07.01.10.31; Tue, 07 Dec 2021 01:10:32 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 871A168ADF2; Tue, 7 Dec 2021 11:10:14 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4314768AD93 for ; Tue, 7 Dec 2021 11:10:07 +0200 (EET) X-IronPort-AV: E=McAfee;i="6200,9189,10190"; a="224794095" X-IronPort-AV: E=Sophos;i="5.87,293,1631602800"; d="scan'208";a="224794095" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Dec 2021 01:10:00 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,293,1631602800"; d="scan'208";a="515198463" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by orsmga008.jf.intel.com with ESMTP; 07 Dec 2021 01:10:00 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Tue, 7 Dec 2021 17:05:51 +0800 Message-Id: <20211207090553.2138848-3-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211207090553.2138848-1-wenbin.chen@intel.com> References: <20211207090553.2138848-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V6 3/5] libavutil/hwcontext_vulkan: Allocate vkFrame in one memory X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: jqXeWzJWJNOi The vaapi can import external frame, but the planes of the external frames should be in the same drm object. A new option "contiguous_planes" is added to device. This flag tells device to allocate places in one memory. When device is derived from vaapi this flag will be enabled. A new flag frame_flag is also added to AVVulkanFramesContext. User can use this flag to force enable or disable this behaviour. A new variable "offset "is added to AVVKFrame. It describe describe the offset from the memory currently bound to the VkImage. Signed-off-by: Wenbin Chen --- libavutil/hwcontext_vulkan.c | 68 +++++++++++++++++++++++++++++++++++- libavutil/hwcontext_vulkan.h | 24 +++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index a0437c9661..eef9009ae1 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -103,8 +103,14 @@ typedef struct VulkanDevicePriv { /* Settings */ int use_linear_images; + /* allocate planes in a contiguous memory */ + int contiguous_planes; + /* Nvidia */ int dev_is_nvidia; + + /* Intel */ + int dev_is_intel; } VulkanDevicePriv; typedef struct VulkanFramesPriv { @@ -153,6 +159,8 @@ typedef struct AVVkFrameInternal { av_free((void *)props); \ } +#define VKF_FLAG(x, f) (((x) & (~AV_VK_FRAME_FLAG_NONE)) & (f)) + static const struct { enum AVPixelFormat pixfmt; const VkFormat vkfmts[4]; @@ -1374,6 +1382,13 @@ static int vulkan_device_create_internal(AVHWDeviceContext *ctx, if (opt_d) p->use_linear_images = strtol(opt_d->value, NULL, 10); + opt_d = av_dict_get(opts, "contiguous_planes", NULL, 0); + if (opt_d) + p->contiguous_planes = strtol(opt_d->value, NULL, 10); + else + p->contiguous_planes = -1; + + hwctx->enabled_dev_extensions = dev_info.ppEnabledExtensionNames; hwctx->nb_enabled_dev_extensions = dev_info.enabledExtensionCount; @@ -1425,6 +1440,8 @@ static int vulkan_device_init(AVHWDeviceContext *ctx) p->dev_is_nvidia = (p->props.properties.vendorID == 0x10de); + p->dev_is_intel = (p->props.properties.vendorID == 0x8086); + vk->GetPhysicalDeviceQueueFamilyProperties(hwctx->phys_dev, &queue_num, NULL); if (!queue_num) { av_log(ctx, AV_LOG_ERROR, "Failed to get queues!\n"); @@ -1742,8 +1759,12 @@ static int alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f, AVHWDeviceContext *ctx = hwfc->device_ctx; VulkanDevicePriv *p = ctx->internal->priv; FFVulkanFunctions *vk = &p->vkfn; + AVVulkanFramesContext *hwfctx = hwfc->hwctx; const int planes = av_pix_fmt_count_planes(hwfc->sw_format); VkBindImageMemoryInfo bind_info[AV_NUM_DATA_POINTERS] = { { 0 } }; + VkMemoryRequirements memory_requirements = { 0 }; + int mem_size = 0; + int mem_size_list[AV_NUM_DATA_POINTERS] = { 0 }; AVVulkanDeviceContext *hwctx = ctx->hwctx; @@ -1771,6 +1792,19 @@ static int alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f, req.memoryRequirements.size = FFALIGN(req.memoryRequirements.size, p->props.properties.limits.minMemoryMapAlignment); + if (VKF_FLAG(hwfctx->flags, AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) { + if (memory_requirements.size == 0) { + memory_requirements = req.memoryRequirements; + } else if (memory_requirements.memoryTypeBits != req.memoryRequirements.memoryTypeBits) { + av_log(hwfc, AV_LOG_ERROR, "the param for each planes are not the same\n"); + return AVERROR(EINVAL); + } + + mem_size_list[i] = req.memoryRequirements.size; + mem_size += mem_size_list[i]; + continue; + } + /* In case the implementation prefers/requires dedicated allocation */ use_ded_mem = ded_req.prefersDedicatedAllocation | ded_req.requiresDedicatedAllocation; @@ -1792,6 +1826,29 @@ static int alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f, bind_info[i].memory = f->mem[i]; } + if (VKF_FLAG(hwfctx->flags, AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) { + memory_requirements.size = mem_size; + + /* Allocate memory */ + if ((err = alloc_mem(ctx, &memory_requirements, + f->tiling == VK_IMAGE_TILING_LINEAR ? + VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT : + VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT, + (void *)(((uint8_t *)alloc_pnext)), + &f->flags, &f->mem[0]))) + return err; + + f->size[0] = memory_requirements.size; + + for (int i = 0; i < planes; i++) { + bind_info[i].sType = VK_STRUCTURE_TYPE_BIND_IMAGE_MEMORY_INFO; + bind_info[i].image = f->img[i]; + bind_info[i].memory = f->mem[0]; + bind_info[i].memoryOffset = i == 0 ? 0 : mem_size_list[i-1]; + f->offset[i] = bind_info[i].memoryOffset; + } + } + /* Bind the allocated memory to the images */ ret = vk->BindImageMemory2(hwctx->act_dev, planes, bind_info); if (ret != VK_SUCCESS) { @@ -2154,6 +2211,12 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) if (!hwctx->usage) hwctx->usage = FF_VK_DEFAULT_USAGE_FLAGS; + if (!(hwctx->flags & AV_VK_FRAME_FLAG_NONE)) { + if (p->contiguous_planes == 1 || + ((p->contiguous_planes == -1) && p->dev_is_intel)) + hwctx->flags |= AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY; + } + err = create_exec_ctx(hwfc, &fp->conv_ctx, dev_hwctx->queue_family_comp_index, dev_hwctx->nb_comp_queues); @@ -3074,6 +3137,7 @@ static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, FFVulkanFunctions *vk = &p->vkfn; VulkanFramesPriv *fp = hwfc->internal->priv; AVVulkanDeviceContext *hwctx = hwfc->device_ctx->hwctx; + AVVulkanFramesContext *hwfctx = hwfc->hwctx; const int planes = av_pix_fmt_count_planes(hwfc->sw_format); VkImageDrmFormatModifierPropertiesEXT drm_mod = { .sType = VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_PROPERTIES_EXT, @@ -3142,7 +3206,9 @@ static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, continue; vk->GetImageSubresourceLayout(hwctx->act_dev, f->img[i], &sub, &layout); - drm_desc->layers[i].planes[0].offset = layout.offset; + drm_desc->layers[i].planes[0].offset = layout.offset + + VKF_FLAG(hwfctx->flags, AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY) ? + f->offset[i] : 0; drm_desc->layers[i].planes[0].pitch = layout.rowPitch; } diff --git a/libavutil/hwcontext_vulkan.h b/libavutil/hwcontext_vulkan.h index fdf2a60156..acb4073e3d 100644 --- a/libavutil/hwcontext_vulkan.h +++ b/libavutil/hwcontext_vulkan.h @@ -35,6 +35,17 @@ * with the data pointer set to an AVVkFrame. */ +/** + * Defines the behaviour of frame allocation + * AV_VK_FRAME_FLAG_NONE: planes will be allocated in separte memory + * AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY: planes will be allocated in a + * contiguous memory. + */ +typedef enum { + AV_VK_FRAME_FLAG_NONE = (1ULL << 0), + AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY = (1ULL << 1) | 1ULL +} AVVkFrameFlags; + /** * Main Vulkan context, allocated as AVHWDeviceContext.hwctx. * All of these can be set before init to change what the context uses @@ -165,6 +176,14 @@ typedef struct AVVulkanFramesContext { * extensions are present in enabled_dev_extensions. */ void *alloc_pnext[AV_NUM_DATA_POINTERS]; + + /** + * Is a combination of AVVkFrameFlags. Defines the behaviour of frame + * allocation. + * If no flag is set, then the flags are automatically determined + * based on the device. + */ + int flags; } AVVulkanFramesContext; /* @@ -230,6 +249,11 @@ typedef struct AVVkFrame { * Internal data. */ struct AVVkFrameInternal *internal; + + /** + * Describe the offset from the memory currently bound to the VkImage. + */ + size_t offset[AV_NUM_DATA_POINTERS]; } AVVkFrame; /** From patchwork Tue Dec 7 09:05:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 32100 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp5976874iog; Tue, 7 Dec 2021 01:10:44 -0800 (PST) X-Google-Smtp-Source: ABdhPJx3lAs1WfU71NhGKbeVUlPfs0IRv3ckayxePYrlsG9lsxm3J9J1RNICOMFWk989XIOSDBM4 X-Received: by 2002:a17:906:fcc8:: with SMTP id qx8mr51275227ejb.370.1638868244051; Tue, 07 Dec 2021 01:10:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638868244; cv=none; d=google.com; s=arc-20160816; b=umTntCFRd1eydoDSJN82pU60F/i+1wzWBRdPL67FGXqU9dp0EOgcocKKIT6HkwXflb IJtCEtahYB8vV6b291IUF9IPEOOJc56O7MKScpIlfNLIciYCHOLni2uvAhPjXuO7tf/W nRiTiN2mSZWfXRiw68Q7KTUPGgYf+L/Sxc/zSKnDVmJrQNE1rolfL/vti0QTDaaavuvZ GNJSW0XXq6mBubjTi9/ZAbteUnu8bc5PFRu4eKjtrDILYFuXVO8Uof4e1VRi5sUuiA0T 4nXJ5AaaZLyCU0vnct7hRcIoru+gmry3ImteP4AmjfFOXu9ad8EagFA5TcmYMWpupSWb 457Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=stBrkhpINOmB/+B92/eZM2ix2cpztV/CtQ+mvHSHkKs=; b=OFuqoyqoBWfoOrhTHwKuY+Y4wfOh71p+0SfdzN3hoHnBnfljGWl1bu0nbZRRiGAeIx kZupWaXknLOvQdas1hastxZltqEMT8oMW5r97F1r9ogfEGBuuGpWdUlch/53a/Six8VG fwBRMU3HZNKIY8LBSn+ftMvOkZoJYJpCABnanPrPYD5QIgi23Z3pL+zZYbx3o7wsu6Vd azhwMCi6KmqtMk6y9HNPRvXxvVcd9oIl3xY5XwGa5KQ89pt2E/rov+GK/uwFjbgk8dRm 7Ddks5YZJt3Mi58bOcuityPv9MzbvZv7DcgduAuyrDp684hmcSfvLOEsXMuWNr7Ku2Gv ykfA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hp34si30922386ejc.640.2021.12.07.01.10.43; Tue, 07 Dec 2021 01:10:44 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 85B4668AE57; Tue, 7 Dec 2021 11:10:15 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C776568ADF2 for ; Tue, 7 Dec 2021 11:10:08 +0200 (EET) X-IronPort-AV: E=McAfee;i="6200,9189,10190"; a="224794096" X-IronPort-AV: E=Sophos;i="5.87,293,1631602800"; d="scan'208";a="224794096" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Dec 2021 01:10:01 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,293,1631602800"; d="scan'208";a="515198472" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by orsmga008.jf.intel.com with ESMTP; 07 Dec 2021 01:10:01 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Tue, 7 Dec 2021 17:05:52 +0800 Message-Id: <20211207090553.2138848-4-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211207090553.2138848-1-wenbin.chen@intel.com> References: <20211207090553.2138848-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V6 4/5] libavutil/hwcontext_vulkan: Add support to hwmap to software frame when using contiguous_planes flag. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: G3zcjKfQIeva Add support to map vulkan frames to software frames when using contiguous_planes flag. Signed-off-by: Wenbin Chen --- libavutil/hwcontext_vulkan.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index eef9009ae1..f980b72720 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -2327,9 +2327,10 @@ static int vulkan_map_frame_to_mem(AVHWFramesContext *hwfc, AVFrame *dst, const AVFrame *src, int flags) { VkResult ret; - int err, mapped_mem_count = 0; + int err, mapped_mem_count = 0, loop = 0; AVVkFrame *f = (AVVkFrame *)src->data[0]; AVVulkanDeviceContext *hwctx = hwfc->device_ctx->hwctx; + AVVulkanFramesContext *hwfctx = hwfc->hwctx; const int planes = av_pix_fmt_count_planes(hwfc->sw_format); VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; FFVulkanFunctions *vk = &p->vkfn; @@ -2356,7 +2357,9 @@ static int vulkan_map_frame_to_mem(AVHWFramesContext *hwfc, AVFrame *dst, dst->width = src->width; dst->height = src->height; - for (int i = 0; i < planes; i++) { + loop = VKF_FLAG(hwfctx->flags, AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY) ? + 1 : planes; + for (int i = 0; i < loop; i++) { ret = vk->MapMemory(hwctx->act_dev, f->mem[i], 0, VK_WHOLE_SIZE, 0, (void **)&dst->data[i]); if (ret != VK_SUCCESS) { @@ -2367,6 +2370,10 @@ static int vulkan_map_frame_to_mem(AVHWFramesContext *hwfc, AVFrame *dst, } mapped_mem_count++; } + if (VKF_FLAG(hwfctx->flags, AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) { + for (int i = 0; i < planes; i++) + dst->data[i] = dst->data[0] + f->offset[i]; + } /* Check if the memory contents matter */ if (((flags & AV_HWFRAME_MAP_READ) || !(flags & AV_HWFRAME_MAP_OVERWRITE)) && From patchwork Tue Dec 7 09:05:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 32099 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp5977118iog; Tue, 7 Dec 2021 01:10:57 -0800 (PST) X-Google-Smtp-Source: ABdhPJygwR1iCInT6XNzzzieyuA8V9VWt40c80CS6TnGwyIamzQU6DU2KEVK8qJDs0AvsxsRj8yQ X-Received: by 2002:a17:907:1b1f:: with SMTP id mp31mr51753985ejc.247.1638868256788; Tue, 07 Dec 2021 01:10:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638868256; cv=none; d=google.com; s=arc-20160816; b=YYR53YefWP/hW1yBgboPniFZc0QCfmjQLTy8aDlbjhLxiGS5Mps2OKxNjrXnnI4i7f bN66FArbUiMb542zpLRBlPHdcspnMtC8FXH5kIK7fHTrJTGW3Zb5QUScd/oIywELJN+0 c4VK7XVnwWe7L2vaUA8H4zyqOtcRIE/t9/IWTSEEVhGHYTHfh5k+CxAvOc8GXSrp0JoZ /6A2B6dXmHimrJk1ZnSymunnzWdyaMFd7GhVOA/mKIOU2O0DVhgCItcvUN8RsRI1totC ll1wxChbSe2uNAiV2RvzjBedSJDpyXUDoVkIfCgS4XpmZ0dUUmOD/W2v7HVevTZ1nURI D9Uw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=M7hr8oo2/BaImu1KDP9lVjekJhIwHLU33i3pL+JzJYw=; b=xidOyzDbMe4p+mDPKzsDDysofvUkhEYQW23XvlYrhVv/Jyy2MPucmS09b1+dq0i9B8 B4zvQw+iCGpD7K9ty+J3uN1edAEKnSTKu7YMDn0lv5IHjZCNBxIXF7ucuT4TfmjUQS/6 vT7/1HTQsMAiB8RV7OTyv5FqRquMuwAlf0bCNuISUiTAprXIjnr2Na5fzlZIgcXeiAS9 ID7KxpEa4RhaDEa3f84/jQpFDl0QsRhTL67VGFVK0i3UgREDicM4DYlrUy9THsbYMzSl nSAMbZJJm3bnL6OhnRHBP3KmmrBolg2NwfetpjE2EreB05xmqBSNxRTj7YlcJ+mvD4Z3 12+A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ji20si29608263ejc.138.2021.12.07.01.10.56; Tue, 07 Dec 2021 01:10:56 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7C24368AE5D; Tue, 7 Dec 2021 11:10:16 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 59F9968AE25 for ; Tue, 7 Dec 2021 11:10:09 +0200 (EET) X-IronPort-AV: E=McAfee;i="6200,9189,10190"; a="224794098" X-IronPort-AV: E=Sophos;i="5.87,293,1631602800"; d="scan'208";a="224794098" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Dec 2021 01:10:02 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,293,1631602800"; d="scan'208";a="515198479" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by orsmga008.jf.intel.com with ESMTP; 07 Dec 2021 01:10:01 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Tue, 7 Dec 2021 17:05:53 +0800 Message-Id: <20211207090553.2138848-5-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211207090553.2138848-1-wenbin.chen@intel.com> References: <20211207090553.2138848-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V6 5/5] libavutil/hwcontext_vulkan: specify the modifier to create VKImage X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: hfS857jXJkBi When vulkan image exports to drm, the tilling need to be VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. Now add code to create vulkan image using this format. Now the following command line works: ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format \ vaapi -i input_1080p.264 -vf "hwmap=derive_device=vulkan,format=vulkan, \ scale_vulkan=1920:1080,hwmap=derive_device=vaapi,format=vaapi" -c:v h264_vaapi output.264 Signed-off-by: Wenbin Chen --- libavutil/hwcontext_vulkan.c | 133 +++++++++++++++++++++++++++++++++-- 1 file changed, 127 insertions(+), 6 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index f980b72720..11c4260e86 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -120,6 +120,9 @@ typedef struct VulkanFramesPriv { /* Image transfers */ VulkanExecCtx upload_ctx; VulkanExecCtx download_ctx; + + /*modifier info*/ + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info; } VulkanFramesPriv; typedef struct AVVkFrameInternal { @@ -242,6 +245,31 @@ const VkFormat *av_vkfmt_from_pixfmt(enum AVPixelFormat p) return NULL; } +static const void *vk_find_struct(const void *chain, VkStructureType stype) +{ + const VkBaseInStructure *in = chain; + while (in) { + if (in->sType == stype) + return in; + + in = in->pNext; + } + + return NULL; +} + +static void vk_link_struct(void *chain, void *in) +{ + VkBaseOutStructure *out = chain; + if (!in) + return; + + while (out->pNext) + out = out->pNext; + + out->pNext = in; +} + static int pixfmt_is_supported(AVHWDeviceContext *dev_ctx, enum AVPixelFormat p, int linear) { @@ -2094,6 +2122,10 @@ static void try_export_flags(AVHWFramesContext *hwfc, AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; FFVulkanFunctions *vk = &p->vkfn; + const int has_modifiers = hwctx->tiling == VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT; + int loop_count; + const VkImageDrmFormatModifierListCreateInfoEXT *modifier_info = vk_find_struct(hwctx->create_pnext, + VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT); VkExternalImageFormatProperties eprops = { .sType = VK_STRUCTURE_TYPE_EXTERNAL_IMAGE_FORMAT_PROPERTIES_KHR, }; @@ -2101,9 +2133,18 @@ static void try_export_flags(AVHWFramesContext *hwfc, .sType = VK_STRUCTURE_TYPE_IMAGE_FORMAT_PROPERTIES_2, .pNext = &eprops, }; + VkPhysicalDeviceImageDrmFormatModifierInfoEXT phy_dev_mod_info = { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_DRM_FORMAT_MODIFIER_INFO_EXT, + .pNext = NULL, + .pQueueFamilyIndices = p->qfs, + .queueFamilyIndexCount = p->num_qfs, + .sharingMode = p->num_qfs > 1 ? VK_SHARING_MODE_CONCURRENT : + VK_SHARING_MODE_EXCLUSIVE, + }; VkPhysicalDeviceExternalImageFormatInfo enext = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_IMAGE_FORMAT_INFO, .handleType = exp, + .pNext = has_modifiers && modifier_info ? &phy_dev_mod_info : NULL, }; VkPhysicalDeviceImageFormatInfo2 pinfo = { .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_FORMAT_INFO_2, @@ -2115,11 +2156,16 @@ static void try_export_flags(AVHWFramesContext *hwfc, .flags = VK_IMAGE_CREATE_ALIAS_BIT, }; - ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, - &pinfo, &props); - if (ret == VK_SUCCESS) { - *iexp |= exp; - *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + loop_count = has_modifiers && modifier_info ? modifier_info->drmFormatModifierCount : 1; + for (int i = 0; i < loop_count; i++) { + if (has_modifiers && modifier_info) + phy_dev_mod_info.drmFormatModifier = modifier_info->pDrmFormatModifiers[i]; + ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, + &pinfo, &props); + if (ret == VK_SUCCESS) { + *iexp |= exp; + *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + } } } @@ -2190,6 +2236,12 @@ static void vulkan_frames_uninit(AVHWFramesContext *hwfc) { VulkanFramesPriv *fp = hwfc->internal->priv; + if (fp->modifier_info) { + if (fp->modifier_info->pDrmFormatModifiers) + av_freep(&fp->modifier_info->pDrmFormatModifiers); + av_freep(&fp->modifier_info); + } + free_exec_ctx(hwfc, &fp->conv_ctx); free_exec_ctx(hwfc, &fp->upload_ctx); free_exec_ctx(hwfc, &fp->download_ctx); @@ -2203,9 +2255,12 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) VulkanFramesPriv *fp = hwfc->internal->priv; AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx; VulkanDevicePriv *p = hwfc->device_ctx->internal->priv; + const VkImageDrmFormatModifierListCreateInfoEXT *modifier_info; + const int has_modifiers = !!(p->extensions & FF_VK_EXT_DRM_MODIFIER_FLAGS); /* Default pool flags */ - hwctx->tiling = hwctx->tiling ? hwctx->tiling : p->use_linear_images ? + hwctx->tiling = hwctx->tiling ? hwctx->tiling : has_modifiers ? + VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT : p->use_linear_images ? VK_IMAGE_TILING_LINEAR : VK_IMAGE_TILING_OPTIMAL; if (!hwctx->usage) @@ -2217,6 +2272,72 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) hwctx->flags |= AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY; } + modifier_info = vk_find_struct(hwctx->create_pnext, + VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT); + /* get the supported modifier */ + if (has_modifiers && !modifier_info) { + const VkFormat *fmt = av_vkfmt_from_pixfmt(hwfc->sw_format); + VkImageDrmFormatModifierListCreateInfoEXT *modifier_info; + FFVulkanFunctions *vk = &p->vkfn; + VkDrmFormatModifierPropertiesEXT *mod_props; + uint64_t *modifiers; + int modifier_count = 0; + + VkDrmFormatModifierPropertiesListEXT mod_props_list = { + .sType = VK_STRUCTURE_TYPE_DRM_FORMAT_MODIFIER_PROPERTIES_LIST_EXT, + .pNext = NULL, + .drmFormatModifierCount = 0, + .pDrmFormatModifierProperties = NULL, + }; + VkFormatProperties2 prop = { + .sType = VK_STRUCTURE_TYPE_FORMAT_PROPERTIES_2, + .pNext = &mod_props_list, + }; + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + if (!mod_props_list.drmFormatModifierCount) { + av_log(hwfc, AV_LOG_ERROR, "There are no supported modifiers for given sw_format\n"); + return AVERROR(EINVAL); + } + + modifier_info = av_malloc(sizeof(*modifier_info)); + if (!modifier_info) + return AVERROR(ENOMEM); + + if (!hwctx->create_pnext) + hwctx->create_pnext = modifier_info; + else + vk_link_struct(hwctx->create_pnext, (void *)modifier_info); + fp->modifier_info = modifier_info; + modifier_info->pNext = NULL; + modifier_info->sType = VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT; + + modifiers = av_malloc(mod_props_list.drmFormatModifierCount * + sizeof(*modifiers)); + if (!modifiers) + return AVERROR(ENOMEM); + modifier_info->pDrmFormatModifiers = modifiers; + + mod_props = av_malloc(mod_props_list.drmFormatModifierCount * + sizeof(*mod_props)); + if (!mod_props) + return AVERROR(ENOMEM); + mod_props_list.pDrmFormatModifierProperties = mod_props; + vk->GetPhysicalDeviceFormatProperties2(dev_hwctx->phys_dev, fmt[0], &prop); + for (int i = 0; i < mod_props_list.drmFormatModifierCount; i++) { + if (!(mod_props[i].drmFormatModifierTilingFeatures & hwctx->usage)) + continue; + modifiers[modifier_count++] = mod_props[i].drmFormatModifier; + } + if (!modifier_count) { + av_log(hwfc, AV_LOG_ERROR, "None of the given modifiers supports" + " the usage flags!\n"); + av_freep(&mod_props); + return AVERROR(EINVAL); + } + modifier_info->drmFormatModifierCount = modifier_count; + av_freep(&mod_props); + } + err = create_exec_ctx(hwfc, &fp->conv_ctx, dev_hwctx->queue_family_comp_index, dev_hwctx->nb_comp_queues);