From patchwork Thu Mar 11 22:09:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suji Velupillai X-Patchwork-Id: 26346 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id B080744B8A9 for ; Fri, 12 Mar 2021 00:10:11 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 785EF68AE34; Fri, 12 Mar 2021 00:10:11 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f98.google.com (mail-pj1-f98.google.com [209.85.216.98]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 31F4C68AD8F for ; Fri, 12 Mar 2021 00:10:05 +0200 (EET) Received: by mail-pj1-f98.google.com with SMTP id lr10-20020a17090b4b8ab02900dd61b95c5eso7509419pjb.4 for ; Thu, 11 Mar 2021 14:10:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id; bh=OgXPNA23fy8hKqXN0Y6N/lRPPbVilaH+7qp553X6Rmo=; b=Ab0EwkyJ+c8kn27GM2h8crRhdHZoH28agokUC/gqfdx0xsNYoa8FwPi4WqC8KFz0K7 qzLHJ2q1yimt/IiaFVijnP9RDco7mUIBzzxlWQ5Kl41D5eaG0J8d5XlcYLKqMi2K0Ahg AlhRiybG71gk0/Qr1doKJCkXS4rushfdFd4AQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=OgXPNA23fy8hKqXN0Y6N/lRPPbVilaH+7qp553X6Rmo=; b=AAT0/A0kiUuUhg8itJSqvviS/m6LT1Fn5CRauoP2O0oodmKU2EW2+djR4FxmX3YibS NiXnPpkGHvYBCt9FOfFmp1BxxWOcmCFh9meLX+Xw6WNfU5tp/L63WBErYn3Az7VrWzBW 3yKW35n+MYxOriSgoCjMdhBMBq865J1i363B9z66/Sg1xUSUdQmcrShl8A6Q6H1NxoNw mQsfcrg8+qzHAyDGJ/xyjJn7xmE8yUrj74pKZqpUVgQJYPI8NRvmw5L+XFKlAYGQRSrD 8D6Ci1BI8G22RG+4zdtAsJAsnpllli9xC3skqw90+W4sVNGJ30erV9gPVQUq7/lEyLD7 gJjg== X-Gm-Message-State: AOAM533Zg5qNoZRJ1OpiaUwpHV8+Ut+rLCIfqWgFVTPfujfMi0TdeJAV nh3tPv9pGf6+NZSKcSnkOkSrIlOwvV3lNxFz4sRPyNf1dcfyfBNdDrMfyTul4CWm8o1xTY4wCV3 gA0JPdgHxlaYvxYiYnQUJs019aXLnIrgS5SC/K2mm0u5jCaOXXA8dKTWhV568RxQ2DfDPTJFanq RmPwjZ304uBql3HNkuMi0= X-Google-Smtp-Source: ABdhPJz/Za6953n/9Q1spuFbWkXUlDHS7iSAH51QYuo1euccEYmIRVHwXGfEAdr8MLMUo6dWd4gYPi3EhtdS X-Received: by 2002:a17:90a:2f22:: with SMTP id s31mr10822444pjd.99.1615500602934; Thu, 11 Mar 2021 14:10:02 -0800 (PST) Received: from suji-Vostro-460.ric.broadcom.com ([192.19.228.250]) by smtp-relay.gmail.com with ESMTPS id x11sm58950pjk.5.2021.03.11.14.10.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 14:10:02 -0800 (PST) X-Relaying-Domain: broadcom.com From: suji.velupillai@broadcom.com To: ffmpeg-devel@ffmpeg.org Date: Thu, 11 Mar 2021 14:09:21 -0800 Message-Id: <20210311220921.28899-1-suji.velupillai@broadcom.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH] lavu: add VKAPI hwcontext implementation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: suji.velupillai@broadcom.com, shamez.kurji@broadcom.com, patrick.rault@broadcom.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Suji Velupillai Initial commit to add VKAPI hardware accelerator implementation. The depedency component vkil source code can be obtained from github https://github.com/Broadcom/vkil Signed-off-by: Suji Velupillai --- configure | 8 +- doc/APIchanges | 4 + libavutil/Makefile | 2 + libavutil/hwcontext.c | 4 + libavutil/hwcontext.h | 1 + libavutil/hwcontext_internal.h | 1 + libavutil/hwcontext_vkapi.c | 522 +++++++++++++++++++++++++++++++++ libavutil/hwcontext_vkapi.h | 104 +++++++ libavutil/pixdesc.c | 4 + libavutil/pixfmt.h | 6 + 10 files changed, 655 insertions(+), 1 deletion(-) create mode 100644 libavutil/hwcontext_vkapi.c create mode 100644 libavutil/hwcontext_vkapi.h diff --git a/configure b/configure index d11942fced..73eb4539ee 100755 --- a/configure +++ b/configure @@ -348,6 +348,7 @@ External library support: --disable-vaapi disable Video Acceleration API (mainly Unix/Intel) code [autodetect] --disable-vdpau disable Nvidia Video Decode and Presentation API for Unix code [autodetect] --disable-videotoolbox disable VideoToolbox code [autodetect] + --disable-vkapi disable VKAPI code [autodetect] Toolchain options: --arch=ARCH select architecture [$arch] @@ -1844,6 +1845,7 @@ HWACCEL_AUTODETECT_LIBRARY_LIST=" vaapi vdpau videotoolbox + vkapi v4l2_m2m xvmc " @@ -2919,6 +2921,7 @@ nvdec_deps="ffnvcodec" vaapi_x11_deps="xlib" videotoolbox_hwaccel_deps="videotoolbox pthreads" videotoolbox_hwaccel_extralibs="-framework QuartzCore" +vkapi_deps="libvkil" xvmc_deps="X11_extensions_XvMClib_h" av1_d3d11va_hwaccel_deps="d3d11va DXVA_PicParams_AV1" @@ -3708,7 +3711,7 @@ avformat_deps="avcodec avutil" avformat_suggest="libm network zlib" avresample_deps="avutil" avresample_suggest="libm" -avutil_suggest="clock_gettime ffnvcodec libm libdrm libmfx opencl user32 vaapi vulkan videotoolbox corefoundation corevideo coremedia bcrypt" +avutil_suggest="clock_gettime ffnvcodec libm libdrm libmfx opencl user32 vaapi vulkan vkapi videotoolbox corefoundation corevideo coremedia bcrypt" postproc_deps="avutil gpl" postproc_suggest="libm" swresample_deps="avutil" @@ -6734,6 +6737,9 @@ enabled vdpau && enabled vdpau && check_lib vdpau_x11 "vdpau/vdpau.h vdpau/vdpau_x11.h" vdp_device_create_x11 -lvdpau -lX11 +enabled vkapi && + check_lib libvkil vkil_api.h vkil_create_api -lvkil + enabled crystalhd && check_lib crystalhd "stdint.h libcrystalhd/libcrystalhd_if.h" DtsCrystalHDVersion -lcrystalhd enabled vulkan && diff --git a/doc/APIchanges b/doc/APIchanges index 13350c0db0..ccab2e6465 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -15,6 +15,10 @@ libavutil: 2017-10-21 API changes, most recent first: +2021-03-11 - xxxxxxxxxx - lavu yy.yy.yyy - hwcontext.h + Add AV_PIX_FMT_VKAPI + Add AV_HWDEVICE_TYPE_VKAPI and implementation. + 2021-03-10 - xxxxxxxxxx - lavf 58.72.100 - avformat.h Change AVBufferRef related AVStream function and struct size parameter and fields type to size_t at next major bump. diff --git a/libavutil/Makefile b/libavutil/Makefile index 27bafe9e12..4044b133a3 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -45,6 +45,7 @@ HEADERS = adler32.h \ hwcontext_vaapi.h \ hwcontext_videotoolbox.h \ hwcontext_vdpau.h \ + hwcontext_vkapi.h \ hwcontext_vulkan.h \ imgutils.h \ intfloat.h \ @@ -185,6 +186,7 @@ OBJS-$(CONFIG_QSV) += hwcontext_qsv.o OBJS-$(CONFIG_VAAPI) += hwcontext_vaapi.o OBJS-$(CONFIG_VIDEOTOOLBOX) += hwcontext_videotoolbox.o OBJS-$(CONFIG_VDPAU) += hwcontext_vdpau.o +OBJS-$(CONFIG_VKAPI) += hwcontext_vkapi.o OBJS-$(CONFIG_VULKAN) += hwcontext_vulkan.o OBJS += $(COMPAT_OBJS:%=../compat/%) diff --git a/libavutil/hwcontext.c b/libavutil/hwcontext.c index d13d0f7c9b..4c13db6578 100644 --- a/libavutil/hwcontext.c +++ b/libavutil/hwcontext.c @@ -59,6 +59,9 @@ static const HWContextType * const hw_table[] = { #if CONFIG_MEDIACODEC &ff_hwcontext_type_mediacodec, #endif +#if CONFIG_VKAPI + &ff_hwcontext_type_vkapi, +#endif #if CONFIG_VULKAN &ff_hwcontext_type_vulkan, #endif @@ -76,6 +79,7 @@ static const char *const hw_type_names[] = { [AV_HWDEVICE_TYPE_VDPAU] = "vdpau", [AV_HWDEVICE_TYPE_VIDEOTOOLBOX] = "videotoolbox", [AV_HWDEVICE_TYPE_MEDIACODEC] = "mediacodec", + [AV_HWDEVICE_TYPE_VKAPI] = "vkapi", [AV_HWDEVICE_TYPE_VULKAN] = "vulkan", }; diff --git a/libavutil/hwcontext.h b/libavutil/hwcontext.h index 04d19d89c2..b862f45aa3 100644 --- a/libavutil/hwcontext.h +++ b/libavutil/hwcontext.h @@ -36,6 +36,7 @@ enum AVHWDeviceType { AV_HWDEVICE_TYPE_DRM, AV_HWDEVICE_TYPE_OPENCL, AV_HWDEVICE_TYPE_MEDIACODEC, + AV_HWDEVICE_TYPE_VKAPI, AV_HWDEVICE_TYPE_VULKAN, }; diff --git a/libavutil/hwcontext_internal.h b/libavutil/hwcontext_internal.h index e6266494ac..c5270759a4 100644 --- a/libavutil/hwcontext_internal.h +++ b/libavutil/hwcontext_internal.h @@ -173,6 +173,7 @@ extern const HWContextType ff_hwcontext_type_vaapi; extern const HWContextType ff_hwcontext_type_vdpau; extern const HWContextType ff_hwcontext_type_videotoolbox; extern const HWContextType ff_hwcontext_type_mediacodec; +extern const HWContextType ff_hwcontext_type_vkapi; extern const HWContextType ff_hwcontext_type_vulkan; #endif /* AVUTIL_HWCONTEXT_INTERNAL_H */ diff --git a/libavutil/hwcontext_vkapi.c b/libavutil/hwcontext_vkapi.c new file mode 100644 index 0000000000..089b0a47d6 --- /dev/null +++ b/libavutil/hwcontext_vkapi.c @@ -0,0 +1,522 @@ +/* + * Copyright (c) 2018 Broadcom + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "avassert.h" +#include "buffer.h" +#include "buffer_internal.h" +#include "common.h" +#include "hwcontext.h" +#include "hwcontext_internal.h" +#include "hwcontext_vkapi.h" +#include "imgutils.h" +#include "mem.h" +#include "pixdesc.h" +#include "pixfmt.h" + +#define VKAPI_FRAME_ALIGNMENT 256 + +static const enum AVPixelFormat supported_formats[] = { + AV_PIX_FMT_NV12, + AV_PIX_FMT_NV21, + AV_PIX_FMT_P010, + AV_PIX_FMT_P016, + AV_PIX_FMT_VKAPI, + AV_PIX_FMT_NONE, +}; + +/** + * Check if the format fmt is in an array + * + * @param fmt the format to check if exists + * @param fmts the array to check the format + * @return 1 if format is presnet in an array, 0 otherwise + */ +static int vkapi_fmt_is_in(int fmt, const int *fmts) +{ + const int *p; + + for (p = fmts; *p != -1; p++) { + if (fmt == *p) + return 1; + } + + return 0; +} + +/** + * Convert the pixel format from the AVPixelFormat to a vk_format_type + * equivalent. + * + * @param pixel_format an AVPixelFormat + * @return a vkil_format_type if match found, otherwise return an error code + */ +static int vkapi_av2vk_fmt(enum AVPixelFormat pixel_format) +{ + switch (pixel_format) { + case AV_PIX_FMT_NV12: + return VK_FORMAT_NV12; + case AV_PIX_FMT_NV21: + return VK_FORMAT_NV21; + case AV_PIX_FMT_P010: + return VK_FORMAT_P010; + case AV_PIX_FMT_VKAPI: + return VK_FORMAT_YOL2; + default: + return AVERROR(EINVAL); + } +} + +/** + * Provide the number of buffer currently referenced in the hardware pool + * + * @param ctx the hardware frames context + * @return number of buffer in use in the internal pool + */ +static int vkapi_get_pool_occupancy(AVHWFramesContext *ctx) +{ + int val = atomic_load(&ctx->internal->pool_internal->refcount); + + return (val - 1); +} + +/** + * Get the reference on the frame hwprops + * + * @param frame the ffmpeg AVFrame structure + * @param phw_surface_desc vkil buffer surface associated with the AVFrame + * @return 0 if succesfull, otherwise return an error code + */ +static int vkapi_frame_ref_hwprops(const AVFrame *frame, void *phw_surface_desc) +{ + void **hw_surface_desc = phw_surface_desc; + + if (frame->format != AV_PIX_FMT_VKAPI) + return AVERROR_BUG; + + *hw_surface_desc = frame->data[3]; + + if (!(*hw_surface_desc)) + return AVERROR_BUG; + + return 0; +} + +/** + * Set a ffmpeg AVFrame with the vkil buffer surface properties + * + * @param frame the ffmpeg AVFrame structure + * @param hw_surface_desc the vkil buffer surface to be associated with the AVFrame + * @return 0 if succesfull, otherwise return an error code + */ +static int vkapi_frame_set_hwprops(AVFrame *frame, + const vkil_buffer_surface *hw_surface_desc) +{ + if (frame->format != AV_PIX_FMT_VKAPI) + return AVERROR_BUG; + + // vkil buffer surface description is stored in ffmpeg AVframe:data[3] + av_assert0(frame->data[3]); + + // the "hard copy" only the buffer descriptor + memcpy((vkil_buffer_surface *)frame->data[3], hw_surface_desc, + sizeof(vkil_buffer_surface)); + + return 0; +} + +/** + * Retrieve vkil buffer surface properties associated with an ffmpeg AVFrame + * + * @param frame the ffmpeg AVFrame structure + * @param hw_surface_desc a vkil buffer surface, where where to extract the property + * @return 0 if succesfull, otherwise return an error code + */ +static int vkapi_frame_get_hwprops(const AVFrame *frame, + vkil_buffer_surface *hw_surface_desc) +{ + if (frame->format != AV_PIX_FMT_VKAPI) + return AVERROR_BUG; + + // the frame is hw tunneled (hw format) + // vkil buffer surface description is stored in ffmpeg AVframe:data[3] + av_assert0(frame->data[3]); + + // the "hard copy" only the buffer descriptor + memcpy(hw_surface_desc, (vkil_buffer_surface *)frame->data[3], + sizeof(vkil_buffer_surface)); + + return 0; +} + +static void vkapi_pool_release(void *opaque, uint8_t *data) +{ + av_free(data); +} + +static AVBufferRef *vkapi_pool_alloc(void *opaque, int size) +{ + AVHWFramesContext *ctx = opaque; + AVBufferRef *ret_ptr = NULL; + vkil_buffer_surface *hw_surface_desc; + + hw_surface_desc = av_mallocz(sizeof(vkil_buffer_surface)); + if (!hw_surface_desc) { + av_log(hw_surface_desc, AV_LOG_ERROR, "av_mallocz failed\n"); + goto out; + } + + ret_ptr = av_buffer_create((uint8_t *)hw_surface_desc, + sizeof(*hw_surface_desc), vkapi_pool_release, + NULL, AV_BUFFER_FLAG_READONLY); + if (!ret_ptr) { + av_log(ctx, AV_LOG_ERROR, "av_buffer_create failed\n"); + av_free(hw_surface_desc); + } + +out: + return ret_ptr; +} + +static int vkapi_frames_init(AVHWFramesContext *ctx) +{ + int aligned_width = FFALIGN(ctx->width, VKAPI_FRAME_ALIGNMENT); + int ret = AVERROR(EINVAL); + int valid_format = vkapi_fmt_is_in(ctx->sw_format, supported_formats); + VKAPIFramesContext *vk_framectx = ctx->hwctx; + + if (!valid_format) + goto fail; + + // set default VKAPIFramesContext + vk_framectx->color.range = AVCOL_RANGE_UNSPECIFIED; + vk_framectx->color.primaries = AVCOL_PRI_UNSPECIFIED; + vk_framectx->color.trc = AVCOL_TRC_UNSPECIFIED; + vk_framectx->color.space = AVCOL_SPC_UNSPECIFIED; + + if (!ctx->pool) { + int size; + + switch (ctx->sw_format) { + case AV_PIX_FMT_VKAPI: + size = sizeof(void *); + break; + case AV_PIX_FMT_NV12: + case AV_PIX_FMT_NV21: + size = (aligned_width * ctx->height * 3) >> 1; + break; + case AV_PIX_FMT_P010: + case AV_PIX_FMT_P016: + size = aligned_width * ctx->height * 3; + break; + default: + goto fail; + } + + ctx->internal->pool_internal = av_buffer_pool_init2(size, ctx, vkapi_pool_alloc, NULL); + if (!ctx->internal->pool_internal) { + ret = AVERROR(ENOMEM); + goto fail; + } + } + + return 0; + +fail: + av_log(ctx, AV_LOG_ERROR, "vkapi_frames_init failed on error %d\n", ret); + return ret; + +} + +static int vkapi_get_buffer(AVHWFramesContext *ctx, AVFrame *frame) +{ + int ret = 0; + + frame->buf[0] = av_buffer_pool_get(ctx->pool); + if (!frame->buf[0]) { + av_log(ctx, AV_LOG_ERROR, "av_buffer_pool_get failed\n"); + ret = AVERROR(ENOMEM); + } else { + // vkil data are stored in frame->data[3] + frame->data[3] = frame->buf[0]->data; + frame->format = AV_PIX_FMT_VKAPI; + frame->width = ctx->width; + frame->height = ctx->height; + } + + return ret; +} + +static int vkapi_transfer_get_formats(AVHWFramesContext *ctx, + enum AVHWFrameTransferDirection dir, + enum AVPixelFormat **formats) +{ + enum AVPixelFormat *fmts; + int ret = 0; + + // this allocation freed in hwcontext::transfer_data_alloc + fmts = av_malloc_array(2, sizeof(*fmts)); + if (!fmts) { + av_log(ctx, AV_LOG_ERROR, "vkapi_transfer_get_formats failed\n"); + ret = AVERROR(ENOMEM); + } else { + fmts[0] = ctx->sw_format; + fmts[1] = AV_PIX_FMT_NONE; + *formats = fmts; + } + + return ret; +} + +static int vkapi_convert_AV2VK_Frame(vkil_buffer_surface *dst, + const AVFrame *src) +{ + int i; + + av_assert0(src->interlaced_frame == 0); + + dst->max_size.width = src->width; + dst->max_size.height = src->height; + dst->format = vkapi_av2vk_fmt(src->format); + dst->quality = src->quality; + dst->prefix.type = VKIL_BUF_SURFACE; + dst->prefix.port_id = 0; // set to default (first port) + dst->prefix.flags = 0; // other fields set to zero (default value) + + for (i = 0; i < VKIL_BUF_NPLANES; i++) { + dst->plane_top[i] = src->data[i]; + dst->plane_bot[i] = src->data[i + VKIL_BUF_NPLANES]; + dst->stride[i] = src->linesize[i]; + // sanity check: vk structure requires 32 bits alignment + if (((uintptr_t)dst->plane_top[i] & (VKIL_BUF_ALIGN -1)) || + ((uintptr_t)dst->plane_bot[i] & (VKIL_BUF_ALIGN -1)) || + (dst->stride[i] &(VKIL_BUF_ALIGN -1))) + return AVERROR(EINVAL); + } + + return 0; +} + +static int vkapi_transfer_data_from(AVHWFramesContext *ctx, AVFrame *dst, + const AVFrame *src) +{ + int ret; + int32_t size; + VKAPIFramesContext *hw_framectx; + AVHWFramesContext *avhw_framectx; + VKAPIDeviceContext *hw_devicectx; + vkil_buffer_surface *surface; + vkil_buffer metadata; + + hw_devicectx = ctx->device_ctx->hwctx; + + av_assert0(src->hw_frames_ctx->data); + avhw_framectx = (AVHWFramesContext *)src->hw_frames_ctx->data; + + av_assert0(avhw_framectx->hwctx); + hw_framectx = avhw_framectx->hwctx; + vkapi_frame_ref_hwprops(src, &surface); + + // populate the vkil_surface structure with the destination pointer on the host + vkapi_convert_AV2VK_Frame(surface, dst); + + // blocking call as ffmpeg assume the transfer is complete on return + ret = hw_devicectx->ilapi->transfer_buffer2(hw_framectx->ilctx, surface, + VK_CMD_DOWNLOAD | VK_CMD_OPT_BLOCKING, + &size); + if (ret < 0) + goto fail; + + + if (src->data[VKAPI_METADATA_PLANE]) { + // Dereference if any transferred frame that has an associated metadata buffer + // no need to transfer it back to host + metadata.handle = (uint32_t)src->data[VKAPI_METADATA_PLANE]; + metadata.type = VKIL_BUF_META_DATA; + metadata.ref = 1; + ret = hw_devicectx->ilapi->xref_buffer(hw_framectx->ilctx, &metadata, -1, + VK_CMD_RUN | VK_CMD_OPT_BLOCKING); + if (ret) + goto fail; + } + + return 0; + +fail: + av_log(ctx, AV_LOG_ERROR, "failure %d on vkapi_transfer_data_to\n", ret); + return AVERROR(EINVAL); +} + +static int vkapi_transfer_data_to(AVHWFramesContext *ctx, AVFrame *dst, + const AVFrame *src) +{ + int ret; + int32_t i, size = 0; + VKAPIFramesContext *hw_framectx; + AVHWFramesContext *avhw_framectx; + VKAPIDeviceContext *hw_devicectx; + vkil_buffer_surface *surface; + vkil_context *ilctx; + uint8_t *tmp_data[4] = {NULL, NULL, NULL, NULL}; + int linesize[4]; + + av_assert0(VKIL_BUF_NPLANES * 2 <= 4); + + hw_devicectx = ctx->device_ctx->hwctx; + + av_assert0(dst->hw_frames_ctx->data); + avhw_framectx = (AVHWFramesContext *)dst->hw_frames_ctx->data; + + av_assert0(avhw_framectx->hwctx); + hw_framectx = avhw_framectx->hwctx; + + ret = vkapi_frame_ref_hwprops(dst, &surface); + if (ret < 0) { + ret = AVERROR(EINVAL); + goto fail; + } + + // populate the vkil_surface structure with the origin pointer on the host + ret = vkapi_convert_AV2VK_Frame(surface, src); + if (ret) { + ret = av_image_alloc(tmp_data, linesize, src->width, src->height, + src->format, VKIL_BUF_ALIGN); + if (ret < 0) + goto fail; + + av_image_copy(tmp_data, linesize, (const uint8_t **)src->data, + src->linesize, src->format, src->width, src->height); + + for (i = 0; i < VKIL_BUF_NPLANES; i++) { + surface->plane_top[i]= tmp_data[i]; + surface->plane_bot[i]= tmp_data[VKIL_BUF_NPLANES + i]; + surface->stride[i] = linesize[i]; + } + } + + surface->quality = dst->quality; + + ilctx = hw_framectx->ilctx; + if (!ilctx) { + ret = AVERROR(EINVAL); + goto fail; + } + + // blocking call as ffmpeg assumes the transfer is complete on return + ret = hw_devicectx->ilapi->transfer_buffer2(ilctx, surface, + VK_CMD_UPLOAD | VK_CMD_OPT_BLOCKING, + &size); + if (ret < 0) { + ret = AVERROR(EINVAL); + goto fail; + } + + hw_framectx->handle = surface->prefix.handle; + + if (tmp_data[0]) + av_free(tmp_data[0]); + + return 0; + +fail: + if (tmp_data[0]) + av_free(tmp_data[0]); + + av_log(ctx, AV_LOG_ERROR, "failure %d on vkapi_transfer_data_to\n", ret); + return ret; +} + +static void vkapi_device_uninit(AVHWDeviceContext *ctx) +{ + VKAPIDeviceContext *hwctx; + + av_assert0(ctx); + av_assert0(ctx->hwctx); + + hwctx = ctx->hwctx; + + if (hwctx->ilapi) + vkil_destroy_api((void **)&hwctx->ilapi); +} + +static int vkapi_device_init(AVHWDeviceContext *ctx) +{ + VKAPIDeviceContext *hwctx; + int ret = AVERROR_EXTERNAL; + + av_assert0(ctx); + av_assert0(ctx->hwctx); + + hwctx = ctx->hwctx; + hwctx->frame_set_hwprops = vkapi_frame_set_hwprops; + hwctx->frame_get_hwprops = vkapi_frame_get_hwprops; + hwctx->frame_ref_hwprops = vkapi_frame_ref_hwprops; + hwctx->fmt_is_in = vkapi_fmt_is_in; + hwctx->av2vk_fmt = vkapi_av2vk_fmt; + hwctx->get_pool_occupancy = vkapi_get_pool_occupancy; + + if (!(hwctx->ilapi = vkil_create_api())) { + av_log(ctx, AV_LOG_ERROR, "ctx->ilapi failed to be created\n"); + goto out; + } + + if (!hwctx->ilapi->init) { + av_log(ctx, AV_LOG_ERROR, "ctx->ilapi not properly initialized\n"); + goto out; + } + + ret = 0; + +out: + if (ret) + vkapi_device_uninit(ctx); + + return ret; +} + +static int vkapi_device_create(AVHWDeviceContext *ctx, const char *device, + AVDictionary *opts, int flags) +{ + /* nothing to do at this time */ + return 0; +} + +const HWContextType ff_hwcontext_type_vkapi = { + .type = AV_HWDEVICE_TYPE_VKAPI, + .name = "VKAPI", + + .device_hwctx_size = sizeof(VKAPIDeviceContext), + .frames_hwctx_size = sizeof(VKAPIFramesContext), + + .device_create = vkapi_device_create, + .device_init = vkapi_device_init, + .device_uninit = vkapi_device_uninit, + + .frames_init = vkapi_frames_init, + .frames_get_buffer = vkapi_get_buffer, + + .transfer_get_formats = vkapi_transfer_get_formats, + .transfer_data_to = vkapi_transfer_data_to, + .transfer_data_from = vkapi_transfer_data_from, + + .pix_fmts = (const enum AVPixelFormat[]) { + AV_PIX_FMT_VKAPI, + AV_PIX_FMT_NONE + }, +}; diff --git a/libavutil/hwcontext_vkapi.h b/libavutil/hwcontext_vkapi.h new file mode 100644 index 0000000000..096602b42e --- /dev/null +++ b/libavutil/hwcontext_vkapi.h @@ -0,0 +1,104 @@ +/* + * Copyright (c) 2018 Broadcom + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_HWCONTEXT_VKAPI_H +#define AVUTIL_HWCONTEXT_VKAPI_H + +#include + +#define VKAPI_METADATA_PLANE 4 + +/** + * @file + * API-specific header for AV_HWDEVICE_TYPE_VKAPI. + */ + +/** + * Allocated as AVHWDeviceContext.hwctx + */ +typedef struct VKAPIDeviceContext { + /** + * Holds pointers to hardware specific functions + */ + vkil_api *ilapi; + /** + * Internal functions definitions + */ + /** + * Get the hwprops reference from the AVFrame:data[3] + */ + int (*frame_ref_hwprops)(const AVFrame *frame, void *phw_surface_desc); + /** + * Set the hwprops into AVFrame:data[3] + */ + int (*frame_set_hwprops)(AVFrame *frame, const vkil_buffer_surface *hw_surface_desc); + /** + * Get the hwprops from AVFrame:data[3] + */ + int (*frame_get_hwprops)(const AVFrame *frame, vkil_buffer_surface *hw_surface_desc); + /** + * Check if format is in an array + */ + int (*fmt_is_in)(int fmt, const int *fmts); + /** + * Convert AVPixelFormat to VKAPI equivalent pixel format + */ + int (*av2vk_fmt)(enum AVPixelFormat pixel_format); + /** + * Get no of buffer count reference in the hardware pool + */ + int (*get_pool_occupancy)(AVHWFramesContext *ctx); +} VKAPIDeviceContext; + +/** + * Contains color information for hardware + */ +typedef struct VKAPIColorContext { + enum AVColorRange range; + enum AVColorPrimaries primaries; + enum AVColorTransferCharacteristic trc; + enum AVColorSpace space; + enum AVChromaLocation chroma_location; +} VKAPIColorContext; + +/** + * Allocated as AVHWFramesContext.hwctx + */ +typedef struct VKAPIFramesContext { + /** + * Handle to a hardware frame context + */ + uint32_t handle; + /** + * Hardware component port associated to the frame context + */ + uint32_t port_id; + uint32_t extra_port_id; + /** + * Color information + */ + VKAPIColorContext color; + /** + * ilcontext associated to the frame context + */ + vkil_context *ilctx; +} VKAPIFramesContext; + +#endif /* AVUTIL_HWCONTEXT_VKAPI_H */ diff --git a/libavutil/pixdesc.c b/libavutil/pixdesc.c index 2a919461a5..1d2f242e57 100644 --- a/libavutil/pixdesc.c +++ b/libavutil/pixdesc.c @@ -2391,6 +2391,10 @@ static const AVPixFmtDescriptor av_pix_fmt_descriptors[AV_PIX_FMT_NB] = { }, .flags = AV_PIX_FMT_FLAG_PLANAR, }, + [AV_PIX_FMT_VKAPI] = { + .name = "vkapi", + .flags = AV_PIX_FMT_FLAG_HWACCEL, + }, [AV_PIX_FMT_VULKAN] = { .name = "vulkan", .flags = AV_PIX_FMT_FLAG_HWACCEL, diff --git a/libavutil/pixfmt.h b/libavutil/pixfmt.h index 46ef211add..3ae607a3d6 100644 --- a/libavutil/pixfmt.h +++ b/libavutil/pixfmt.h @@ -348,6 +348,12 @@ enum AVPixelFormat { AV_PIX_FMT_NV24, ///< planar YUV 4:4:4, 24bpp, 1 plane for Y and 1 plane for the UV components, which are interleaved (first byte U and the following byte V) AV_PIX_FMT_NV42, ///< as above, but U and V bytes are swapped + /** + * VKAPI hardware acceleration. + * data[3] contains a pointer to the vkil_buffer_surface structure + */ + AV_PIX_FMT_VKAPI, + /** * Vulkan hardware images. *