From patchwork Wed Oct 18 16:55:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 44292 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:2a09:b0:15d:8365:d4b8 with SMTP id e9csp12760pzh; Wed, 18 Oct 2023 09:56:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHJjNBT0YABgbB/SqDMDsXHCW27MjpRIHdILWH+l+umU1hmV0FlodiLyBHD1+Hfxuz4B/v8 X-Received: by 2002:ac2:4282:0:b0:500:863e:fc57 with SMTP id m2-20020ac24282000000b00500863efc57mr4149934lfh.25.1697648166794; Wed, 18 Oct 2023 09:56:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697648166; cv=none; d=google.com; s=arc-20160816; b=nOB6Ww/quHLJcdIx8jI+b8nVDJ6kWOXcVD32yR5BC01M7K93ZH3624q1fNWD6rFh9E er+TmuA2Vp9kqLl11weK4u8ujcZlyogypAPlKOHaIqkoWN2Hn7l90yo1ccIRgG04BKGG 98lXlVEs4qRurKiBhG1map1XTXVE3/5gGw2TYs5jcyisIppQbD7DE4A/6KrQAwspuY28 6GDhOxA0TjnKrYsu8ZaTiPpcGxS5YnQ1s2Pf7FEPAwZ2saqhoSVAGxjWcNd9Pmcb0o7u HS0EULfkV5vW67RHznuOt044QwZw2hGaESL/+kCOdkjwPZvl0kpYbYYfFuIZ+DV+tfJJ DBpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:date:to:from:message-id :dkim-signature:delivered-to; bh=NNgueInZFmBGHSmoCvW0iKm7tB9MiIpAT8knY4EA1PA=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=CTNAVyjQUF1knAuHaNBw4JNmCJOkT3Eqw3pFDeIgsjokb+4OPxDwrCF/rvTfaSRFr4 V0UbiN7t4juixGUCX92tSuK28C196GCUlSbmS4GT174jkq8sT4OM1P+2zHFvhRHi1Gho XP+hz5ztcLgTz59mr8iyB9CbeI5pza8fT0R5mKgRUaBnjreLWVSVzSz/Ae3UAsKWxQmN TV2NzVhOzslpnIDDxtMoXVLvaKgctJ96owADsG3GoKoeO0mESoggh1znk/sl+8ecbKvu rO6GQtbwJ/D/J46BI0/XyTFDwcGd2HxyiQtqaPtCEAgN9nnh/ZyzzwrrLYwYUtjmmYKp OLJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=S0f1CTTU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id i7-20020a05640242c700b0053641221ff8si2075458edc.315.2023.10.18.09.55.42; Wed, 18 Oct 2023 09:56:06 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=S0f1CTTU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5288868CA93; Wed, 18 Oct 2023 19:55:38 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-58-211.mail.qq.com (out162-62-58-211.mail.qq.com [162.62.58.211]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A9A0B68C902 for ; Wed, 18 Oct 2023 19:55:30 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1697648119; bh=nSReDLNYAHOi/zU7dfJMnajLD0xprnPSlTCt1c8MH34=; h=From:To:Cc:Subject:Date; b=S0f1CTTU3YNExdRDC+L60xNJaUokSEK7q78YWdCwAiAhDK3jfVFYiV8IN84EnYHrb Pw/cgpUuclVGvWitbU4wNOJ42H/2s6ZNAEr+9FiHBIatwBCH3KLVnRdhfeXZdE3xWK jLhScua/4J9v6sV8C6Oovn4Coo8O687T5tM9mljU= Received: from localhost.localdomain ([113.118.106.95]) by newxmesmtplogicsvrszb9-0.qq.com (NewEsmtp) with SMTP id DD2B24AA; Thu, 19 Oct 2023 00:55:18 +0800 X-QQ-mid: xmsmtpt1697648118tk7eujdv6 Message-ID: X-QQ-XMAILINFO: NUe3UHuMggO/lm0iLH+2XgamhMDKGMzo3VGdfQoSlfdxZisd6ZApkbytVh9C4t SwxPy4E2bHbI94Nilv6tXmxzLCAR+NgV0/0ozSsfKqDOOPPggzsGYUeSvYm7oBrQxfkU4whd11KA 7zAbVXLhzrfi76oZ6ZLRfdPBUy2aJziNVT2TeojTwatB2iKC+du9rYFhkDo48THn5ExHy9D3MfAW ZU3QrcP7EmSkNlSs4DfPpo+CD2yBuzW5kfs7GFNVzb17vCYdR4a8QvbWSIjb6XRuF/3V7N1gqmv0 DhEHjwTeDn7dbszdcmd8mtPmozc6f3Fi9WrRm4S7pCNbylhhqO+vwi5Em0XrD8SOlYuvnVz+tB7F Vg9KcTwjZo0v09y2RfBSSCJ1W4zTio3748S2AgeyYIkdFbGvTw8ySD24X7LaowzM8GXG22f/AGSy T60M4bWGqbKxkj7bgXO5WgqGOIojQiT5934IDelGJzQm24ItJL9o9U8/KFnqmZok0SjNVHuexL+Z vS5xltApqvRfBshojBjkfhFOUpJXW0FlIZTK4M71x5IFgO7rTOKn7Ud6rq/b3CB/H+IEnbVySvBR Iokd1nWyzb6LYFyVRI4pZ9ntjbsKBeRnDYjvPGxWgipfULVT/7w0LtpJ5fGBn0H4pYgPyCKJOToM VHQaHptBW7Vj8yRt9EEBOb975UBCLxavnSRuIzhOB9Kn6F07jhCYgM/k1TRqYQ5LVm/uq93B4a2M AM7Im7TbBfj9QSca03VSqUxs/SWQa1fNzdKQtcbkurSILVdVfWpNGDW0J9fEym7r14Up0OrXA4+w 6eQTAO5MmmzKCfsbHFoh/bkwmO+wXnWB8K3ly/e5q+GOG8LxN9mm/N0+r34gEvS7tITvxavgUVZh mIXblIdZ8hWr7gXEAnEmEJ2wDmeeMPe73AxhewCnPzDuDYQWz4P9Z2x4TmStaRuH7vqo1NviNp7d 9cgA9cZFXFTxfi4IuuEcNZbPzg7bBHkjeY92M/kZc= X-QQ-XMRINFO: NI4Ajvh11aEj8Xl/2s1/T8w= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Thu, 19 Oct 2023 00:55:15 +0800 X-OQ-MSGID: <20231018165516.275945-1-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 1/2] fftools/ffplay: add vulkan renderer via libplacebo X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 3RXkhoD9TP4y From: Zhao Zhili --- configure | 2 +- fftools/Makefile | 2 + fftools/ffplay.c | 46 ++++++- fftools/ffplay_renderer.c | 265 ++++++++++++++++++++++++++++++++++++++ fftools/ffplay_renderer.h | 38 ++++++ 5 files changed, 347 insertions(+), 6 deletions(-) create mode 100644 fftools/ffplay_renderer.c create mode 100644 fftools/ffplay_renderer.h diff --git a/configure b/configure index 953104150f..a3b3726e2c 100755 --- a/configure +++ b/configure @@ -3904,7 +3904,7 @@ ffmpeg_select="aformat_filter anull_filter atrim_filter format_filter ffmpeg_suggest="ole32 psapi shell32" ffplay_deps="avcodec avformat avfilter swscale swresample sdl2" ffplay_select="crop_filter transpose_filter hflip_filter vflip_filter rotate_filter" -ffplay_suggest="shell32" +ffplay_suggest="shell32 libplacebo vulkan" ffprobe_deps="avcodec avformat" ffprobe_suggest="shell32" diff --git a/fftools/Makefile b/fftools/Makefile index 56820e6bc8..3c763e3db9 100644 --- a/fftools/Makefile +++ b/fftools/Makefile @@ -22,6 +22,8 @@ OBJS-ffmpeg += \ fftools/sync_queue.o \ fftools/thread_queue.o \ +OBJS-ffplay += fftools/ffplay_renderer.o + define DOFFTOOL OBJS-$(1) += fftools/cmdutils.o fftools/opt_common.o fftools/$(1).o $(OBJS-$(1)-yes) ifdef HAVE_GNU_WINDRES diff --git a/fftools/ffplay.c b/fftools/ffplay.c index d8c69e10bc..305d72d8b8 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -58,6 +58,7 @@ #include #include "cmdutils.h" +#include "ffplay_renderer.h" #include "opt_common.h" const char program_name[] = "ffplay"; @@ -350,6 +351,7 @@ static char *afilters = NULL; static int autorotate = 1; static int find_stream_info = 1; static int filter_nbthreads = 0; +static int enable_vulkan = 0; /* current context */ static int is_full_screen; @@ -362,6 +364,8 @@ static SDL_Renderer *renderer; static SDL_RendererInfo renderer_info = {0}; static SDL_AudioDeviceID audio_dev; +static VkRenderer *vk_renderer; + static const struct TextureFormatEntry { enum AVPixelFormat format; int texture_fmt; @@ -954,6 +958,11 @@ static void video_image_display(VideoState *is) SDL_Rect rect; vp = frame_queue_peek_last(&is->pictq); + if (vk_renderer) { + vk_renderer_display(vk_renderer, vp->frame); + return; + } + if (is->subtitle_st) { if (frame_queue_nb_remaining(&is->subpq) > 0) { sp = frame_queue_peek(&is->subpq); @@ -1289,6 +1298,8 @@ static void do_exit(VideoState *is) } if (renderer) SDL_DestroyRenderer(renderer); + if (vk_renderer) + vk_renderer_destroy(vk_renderer); if (window) SDL_DestroyWindow(window); uninit_opts(); @@ -3447,6 +3458,8 @@ static void event_loop(VideoState *cur_stream) SDL_DestroyTexture(cur_stream->vis_texture); cur_stream->vis_texture = NULL; } + if (vk_renderer) + vk_renderer_resize(vk_renderer, screen_width, screen_height); case SDL_WINDOWEVENT_EXPOSED: cur_stream->force_refresh = 1; } @@ -3611,6 +3624,7 @@ static const OptionDef options[] = { { "find_stream_info", OPT_BOOL | OPT_INPUT | OPT_EXPERT, { &find_stream_info }, "read and decode the streams to fill missing information with heuristics" }, { "filter_threads", HAS_ARG | OPT_INT | OPT_EXPERT, { &filter_nbthreads }, "number of filter threads per graph" }, + { "enable_vulkan", OPT_BOOL, { &enable_vulkan }, "enable vulkan render" }, { NULL, }, }; @@ -3725,9 +3739,31 @@ int main(int argc, char **argv) #ifdef SDL_HINT_VIDEO_X11_NET_WM_BYPASS_COMPOSITOR SDL_SetHint(SDL_HINT_VIDEO_X11_NET_WM_BYPASS_COMPOSITOR, "0"); #endif + if (enable_vulkan) { + vk_renderer = vk_get_renderer(); + if (vk_renderer) { +#if SDL_VERSION_ATLEAST(2, 0, 6) + flags |= SDL_WINDOW_VULKAN; +#endif + } else { + av_log(NULL, AV_LOG_WARNING, "Doesn't support vulkan renderer, fallback to SDL renderer\n"); + enable_vulkan = 0; + } + } window = SDL_CreateWindow(program_name, SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED, default_width, default_height, flags); SDL_SetHint(SDL_HINT_RENDER_SCALE_QUALITY, "linear"); - if (window) { + if (!window) { + av_log(NULL, AV_LOG_FATAL, "Failed to create window: %s", SDL_GetError()); + do_exit(NULL); + } + + if (vk_renderer) { + ret = vk_renderer_create(vk_renderer, window); + if (ret < 0) { + av_log(NULL, AV_LOG_FATAL, "Failed to create vulkan renderer, %s\n", av_err2str(ret)); + do_exit(NULL); + } + } else { renderer = SDL_CreateRenderer(window, -1, SDL_RENDERER_ACCELERATED | SDL_RENDERER_PRESENTVSYNC); if (!renderer) { av_log(NULL, AV_LOG_WARNING, "Failed to initialize a hardware accelerated renderer: %s\n", SDL_GetError()); @@ -3737,10 +3773,10 @@ int main(int argc, char **argv) if (!SDL_GetRendererInfo(renderer, &renderer_info)) av_log(NULL, AV_LOG_VERBOSE, "Initialized %s renderer.\n", renderer_info.name); } - } - if (!window || !renderer || !renderer_info.num_texture_formats) { - av_log(NULL, AV_LOG_FATAL, "Failed to create window or renderer: %s", SDL_GetError()); - do_exit(NULL); + if (!renderer || !renderer_info.num_texture_formats) { + av_log(NULL, AV_LOG_FATAL, "Failed to create window or renderer: %s", SDL_GetError()); + do_exit(NULL); + } } } diff --git a/fftools/ffplay_renderer.c b/fftools/ffplay_renderer.c new file mode 100644 index 0000000000..796a964a7b --- /dev/null +++ b/fftools/ffplay_renderer.c @@ -0,0 +1,265 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "config.h" +#include "ffplay_renderer.h" + +#define HAVE_VULKAN_RENDERER (SDL_VERSION_ATLEAST(2, 0, 6) && CONFIG_LIBPLACEBO) + +struct VkRenderer { + const AVClass *class; + + int (*create)(VkRenderer *renderer, SDL_Window *window); + + int (*display)(VkRenderer *renderer, AVFrame *frame); + + int (*resize)(VkRenderer *renderer, int width, int height); + + void (*destroy)(VkRenderer *renderer); +}; + +#if HAVE_VULKAN_RENDERER + +#include +#include +#include +#include + +typedef struct RendererContext { + VkRenderer api; + pl_vk_inst vk_inst; + pl_vulkan pl_vk; + pl_swapchain swapchain; + VkSurfaceKHR vk_surface; + pl_renderer renderer; + pl_tex tex[4]; + + pl_log vk_log; +} RendererContext; + +static void vk_log_cb(void *log_priv, enum pl_log_level level, const char *msg) { + static const int level_map[] = { + AV_LOG_QUIET, + AV_LOG_FATAL, + AV_LOG_ERROR, + AV_LOG_WARNING, + AV_LOG_INFO, + AV_LOG_DEBUG, + AV_LOG_TRACE, + }; + + if (level > 0 && level < FF_ARRAY_ELEMS(level_map)) + av_log(log_priv, level_map[level], "%s\n", msg); +} + +static int create(VkRenderer *renderer, SDL_Window *window) +{ + int ret = 0; + unsigned ext_num = 0; + const char **ext = NULL; + int w, h; + struct pl_log_params vk_log_params = { + .log_cb = vk_log_cb, + .log_level = PL_LOG_WARN, + .log_priv = renderer, + }; + RendererContext *ctx = (RendererContext *)renderer; + + if (!SDL_Vulkan_GetInstanceExtensions(window, &ext_num, NULL)) { + av_log(NULL, AV_LOG_FATAL, "Failed to get vulkan extensions: %s\n", SDL_GetError()); + return AVERROR_EXTERNAL; + } + + ext = av_calloc(ext_num, sizeof(*ext)); + if (!ext) { + ret = AVERROR(ENOMEM); + goto out; + } + + SDL_Vulkan_GetInstanceExtensions(window, &ext_num, ext); + + ctx->vk_log = pl_log_create(PL_API_VER, &vk_log_params); + ctx->vk_inst = pl_vk_inst_create(ctx->vk_log, pl_vk_inst_params( + .get_proc_addr = SDL_Vulkan_GetVkGetInstanceProcAddr(), + .debug = false, + .extensions = ext, + .num_extensions = ext_num, + )); + if (!ctx->vk_inst) { + ret = AVERROR_EXTERNAL; + goto out; + } + + if (!SDL_Vulkan_CreateSurface(window, ctx->vk_inst->instance, &ctx->vk_surface)) { + ret = AVERROR_EXTERNAL; + goto out; + } + + ctx->pl_vk = pl_vulkan_create(ctx->vk_log, pl_vulkan_params( + .instance = ctx->vk_inst->instance, + .get_proc_addr = ctx->vk_inst->get_proc_addr, + .surface = ctx->vk_surface, + .allow_software = false, + )); + if (!ctx->pl_vk) { + ret = AVERROR_EXTERNAL; + goto out; + } + + ctx->swapchain = pl_vulkan_create_swapchain(ctx->pl_vk, + pl_vulkan_swapchain_params(.surface = ctx->vk_surface, + .present_mode = VK_PRESENT_MODE_FIFO_KHR)); + + SDL_Vulkan_GetDrawableSize(window, &w, &h); + pl_swapchain_resize(ctx->swapchain, &w, &h); + + ctx->renderer = pl_renderer_create(ctx->vk_log, ctx->pl_vk->gpu); + if (!ctx->renderer) { + ret = AVERROR_EXTERNAL; + goto out; + } + ret = 0; + +out: + av_free(ext); + return ret; +} + +static int display(VkRenderer *renderer, AVFrame *frame) +{ + struct pl_swapchain_frame swap_frame = {0}; + struct pl_frame pl_frame = {0}; + struct pl_frame target = {0}; + RendererContext *ctx = (RendererContext *)renderer; + int ret = 0; + + if (!pl_map_avframe_ex(ctx->pl_vk->gpu, &pl_frame, pl_avframe_params( + .frame = frame, + .tex = ctx->tex))) { + av_log(NULL, AV_LOG_ERROR, "pl_map_avframe_ex failed\n"); + return AVERROR_EXTERNAL; + } + + if (!pl_swapchain_start_frame(ctx->swapchain, &swap_frame)) { + av_log(NULL, AV_LOG_ERROR, "start frame failed\n"); + ret = AVERROR_EXTERNAL; + goto out; + } + + pl_frame_from_swapchain(&target, &swap_frame); + if (!pl_render_image(ctx->renderer, &pl_frame, &target, &pl_render_default_params)) { + av_log(NULL, AV_LOG_ERROR, "pl_render_image failed\n"); + ret = AVERROR_EXTERNAL; + goto out; + } + + if (!pl_swapchain_submit_frame(ctx->swapchain)) { + av_log(NULL, AV_LOG_ERROR, "pl_swapchain_submit_frame failed\n"); + ret = AVERROR_EXTERNAL; + goto out; + } + pl_swapchain_swap_buffers(ctx->swapchain); + +out: + pl_unmap_avframe(ctx->pl_vk->gpu, &pl_frame); + return ret; +} + +static int resize(VkRenderer *renderer, int width, int height) +{ + RendererContext *ctx = (RendererContext *)renderer; + + if (!pl_swapchain_resize(ctx->swapchain, &width, &height)) + return AVERROR_EXTERNAL; + return 0; +} + +static void destroy(VkRenderer *renderer) +{ + PFN_vkDestroySurfaceKHR vkDestroySurfaceKHR; + RendererContext *ctx = (RendererContext *)renderer; + + for (int i = 0; i < FF_ARRAY_ELEMS(ctx->tex); i++) + pl_tex_destroy(ctx->pl_vk->gpu, &ctx->tex[i]); + pl_renderer_destroy(&ctx->renderer); + pl_swapchain_destroy(&ctx->swapchain); + pl_vulkan_destroy(&ctx->pl_vk); + + if (ctx->vk_surface) { + vkDestroySurfaceKHR = (PFN_vkDestroySurfaceKHR) + ctx->vk_inst->get_proc_addr(ctx->vk_inst->instance, "vkDestroySurfaceKHR"); + vkDestroySurfaceKHR(ctx->vk_inst->instance, ctx->vk_surface, NULL); + ctx->vk_surface = NULL; + } + + pl_vk_inst_destroy(&ctx->vk_inst); + pl_log_destroy(&ctx->vk_log); +} + +static const AVClass vulkan_renderer_class = { + .class_name = "Vulkan Renderer", + .item_name = av_default_item_name, + .version = LIBAVUTIL_VERSION_INT, +}; + +VkRenderer *vk_get_renderer(void) +{ + RendererContext *ctx = av_mallocz(sizeof(*ctx)); + VkRenderer *renderer; + + if (!ctx) + return NULL; + + renderer = &ctx->api; + renderer->class = &vulkan_renderer_class; + renderer->create = create; + renderer->display = display; + renderer->resize = resize; + renderer->destroy = destroy; + + return renderer; +} + +#else + +VkRenderer *vk_get_renderer(void) +{ + return NULL; +} + +#endif + +int vk_renderer_create(VkRenderer *renderer, SDL_Window *window) +{ + return renderer->create(renderer, window); +} + +int vk_renderer_display(VkRenderer *renderer, AVFrame *frame) +{ + return renderer->display(renderer, frame); +} + +int vk_renderer_resize(VkRenderer *renderer, int width, int height) +{ + return renderer->resize(renderer, width, height); +} + +void vk_renderer_destroy(VkRenderer *renderer) +{ + renderer->destroy(renderer); +} diff --git a/fftools/ffplay_renderer.h b/fftools/ffplay_renderer.h new file mode 100644 index 0000000000..4029f154a9 --- /dev/null +++ b/fftools/ffplay_renderer.h @@ -0,0 +1,38 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef FFTOOLS_FFPLAY_RENDERER_H +#define FFTOOLS_FFPLAY_RENDERER_H + +#include + +#include "libavutil/frame.h" + +typedef struct VkRenderer VkRenderer; + +VkRenderer *vk_get_renderer(void); + +int vk_renderer_create(VkRenderer *renderer, SDL_Window *window); + +int vk_renderer_display(VkRenderer *renderer, AVFrame *frame); + +int vk_renderer_resize(VkRenderer *renderer, int width, int height); + +void vk_renderer_destroy(VkRenderer *renderer); + +#endif /* FFTOOLS_FFPLAY_RENDERER_H */ From patchwork Wed Oct 18 16:55:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 44293 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:2a09:b0:15d:8365:d4b8 with SMTP id e9csp12923pzh; Wed, 18 Oct 2023 09:56:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF3NBLVdiOXGInzvUJQ5+SruAQwe41XjbEXfelux17QvBolW5TWZg1ti1bLhZRobRUAQs+I X-Received: by 2002:a17:907:78d1:b0:9c6:3c94:69de with SMTP id kv17-20020a17090778d100b009c63c9469demr3276068ejc.53.1697648185681; Wed, 18 Oct 2023 09:56:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697648185; cv=none; d=google.com; s=arc-20160816; b=NVpBdyS7qaicRgjUZR4uXYphRtpx6zuSIUDub/ryq3J93htHbo13iigwM4H+U5SsMT y8SwbaYWp8LPL2pK8Ef6XvNF/KXXrvQr6Q0nCYpJCxsioF7HaZ0eTH4iJaS2L8nEbFAA 4J1JsLLKqgFhwlZ7+Lt+6Jtpx5xVcsnFLq7QmKCxWXOrlSLzX23ZIkL9lVTDW/Zz3YmW vv2kH4Hn7KFE9f9LZxYrESnuwRX6Mp0TVz742e6WSMt0En8YIEJGphumHaKXZ45za60d 4ucql4jRAP8v3VTPESkYo6XtEddN6rpjFFHCzMmKiNlTLkIPXk/Lfu+gTovdeZOpAVvA +5ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=s+Ka7MFMKuBKL6jhOpoMpqyQ8Yc0kBJ589xCkWEZ9IY=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=C+JWvU65uHtEGtNcQoUtLt2NKSrprCZxlEWd46gYdK5dSMOOW89KmW6v8E2+dfyvvL nnIW7Fs55gajN2TaYxePFFSfrdVsoCUP7WBZDkUQlB4UCVrPZ4vrQDOY/n6zfleNivLS +7oiKw7SeiP8TL9xq3jMib5r8UJoeg6QdfJqSO3RmKj56xZExPqxpupApRB6Xz6EUsmX HiBfUbrImcIFWH7OH2DDk7s0tVYHHYQjBcHnbo90JGALfurJCS0uK9h5Qkee+sEJAEC9 8KOYIwhrXD6hbZKVR51RbpXdm9hV1sx+ql8asHuyOEK6B/zgp++Q3ZPo8/8Cju84ogXM A7Yg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=aBi2w1a6; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id qk36-20020a1709077fa400b0099cb1e742e7si1246476ejc.224.2023.10.18.09.56.02; Wed, 18 Oct 2023 09:56:25 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=aBi2w1a6; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8E2DF68CA9B; Wed, 18 Oct 2023 19:55:39 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-251-53.mail.qq.com (out203-205-251-53.mail.qq.com [203.205.251.53]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3561468CA86 for ; Wed, 18 Oct 2023 19:55:30 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1697648120; bh=ATLwNzBhUMaZra4W0RKjV8G3BrY0bv8RgKbXljYnqK0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=aBi2w1a6IPkRYcs90KX4z7YRD0aLYssR8V8f1AkP+xu/ZeaYCLCqDTlaTGRT1Blpm 6Pl06rhaKV4u3nTKsLRxkIiM9XPui3+Bw1+ShhI/dfUIIP+ZSdJZsDO/K2jekPyNqk hCwAfUY7TBhkvJcgvnsCaWq4CDGeAPDeBR8lsGiU= Received: from localhost.localdomain ([113.118.106.95]) by newxmesmtplogicsvrszb9-0.qq.com (NewEsmtp) with SMTP id DD2B24AA; Thu, 19 Oct 2023 00:55:18 +0800 X-QQ-mid: xmsmtpt1697648119t221nwh1n Message-ID: X-QQ-XMAILINFO: M0msBj/q2J/mK9ZaKgqG3FOBo3rZjVXOaezzpGFu0YPd624JIHtuxq7pZQaXy4 OSdlyid4fa9IWhn4ZhZizbO26T0ad/kA/oHxbrtGH6DAU2amwgU25ip22rxAMTPzjbLmaTrM49DC NUQIrJjUJazQrdSOquMp070Tuz7yZULVTHKI/PZZKfI5y+zMVNtDj1OQpI5C7eyRtoCHlHBQ/FCj cRQKyJtz5GrCNbjp+P9Aa6XGPHeE8Wgeuz8+U1pkA/Ltf+JQVgjYAsVqaYHk5wbk4ceEC9q6fK11 K5aJIOkGtNlwwJ7pvgvXE0V2ltIC58bs6bCjd42s0iMuq9+JzsAZ/xbp9ZIlmDnpg6q9IFVv6LxI TB6I0PD4rqk7RVzfSvKJnnETtfWmohsFDBoBcVGQfDQFbZ6pq4lPWSYLOpHKkJt4icOuCJDx26ZW TV9B013oH8nYL3esmjjgdBxnknxLVONkwrTDMXvHp9/e3rE7Ml2azfE0eusCa5M/GM46XdclklsS 7Mn5egAYJyiDprJZMy7s8wYhwicbb4sV4gxXhyOOXUzKJ54P8ekeWdhUDg3NmZPGJbGKCUNV28mK yoXe9cCZ1jeoRFv0/e4UR25KvP4CZ29YU3Wqqawv0fm5KvYmhU4urqkxc571nH9BATpwGM7qwgkS R2glJXwsYcfGfYqHEMypl84hOzTjs9rTFLH19a8a92MlsDB+S078dBDOOxmpetLB1DCnKbnoghPI FVL8BDXEZYenZaz7dVF043HN0RBWrBANLYRIzA4hYFkaWw0K6QuKuSLWdbr2DFCN020+iee3o1qD IBq1YtFYUmSaGvJMrmZY6Dpp7IC3LKkrPZHPSz72/QlSCRSdVOjhji0duOI8BteEGmLEvVYfTKXi SzDT1s0HvtV/aaTycDpJ6F/+p75MmaDYSO9gg4U+rXA1XVg2HimSKfskSojhr30GZwv3koHW9rgl SQ+TGOYf7JK7lGdyleqaUnJlzJAo+Fu1hDkbqmM06E82W0DOvuNb7xsi72sVKpZLwjq2ehLXk= X-QQ-XMRINFO: NS+P29fieYNw95Bth2bWPxk= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Thu, 19 Oct 2023 00:55:16 +0800 X-OQ-MSGID: <20231018165516.275945-2-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231018165516.275945-1-quinkblack@foxmail.com> References: <20231018165516.275945-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 2/2] fftools/ffplay: add hwaccel decoding support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: +zWjRihhLsFJ From: Zhao Zhili --- fftools/ffplay.c | 30 ++++++++++ fftools/ffplay_renderer.c | 117 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+) diff --git a/fftools/ffplay.c b/fftools/ffplay.c index 305d72d8b8..0f9584b57e 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -352,6 +352,7 @@ static int autorotate = 1; static int find_stream_info = 1; static int filter_nbthreads = 0; static int enable_vulkan = 0; +static const char *hwaccel = NULL; /* current context */ static int is_full_screen; @@ -2557,6 +2558,24 @@ static int audio_open(void *opaque, AVChannelLayout *wanted_channel_layout, int return spec.size; } +static int create_hwaccel(AVBufferRef **device_ctx) +{ + enum AVHWDeviceType type; + int ret; + + *device_ctx = NULL; + + if (!hwaccel) + return 0; + + type = av_hwdevice_find_type_by_name(hwaccel); + if (type == AV_HWDEVICE_TYPE_NONE) + return AVERROR(ENOTSUP); + + ret = av_hwdevice_ctx_create(device_ctx, type, NULL, NULL, 0); + return ret; +} + /* open a given stream. Return 0 if OK */ static int stream_component_open(VideoState *is, int stream_index) { @@ -2624,6 +2643,12 @@ static int stream_component_open(VideoState *is, int stream_index) av_dict_set(&opts, "flags", "+copy_opaque", AV_DICT_MULTIKEY); + if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) { + ret = create_hwaccel(&avctx->hw_device_ctx); + if (ret < 0) + goto fail; + } + if ((ret = avcodec_open2(avctx, codec, &opts)) < 0) { goto fail; } @@ -3625,6 +3650,7 @@ static const OptionDef options[] = { "read and decode the streams to fill missing information with heuristics" }, { "filter_threads", HAS_ARG | OPT_INT | OPT_EXPERT, { &filter_nbthreads }, "number of filter threads per graph" }, { "enable_vulkan", OPT_BOOL, { &enable_vulkan }, "enable vulkan render" }, + { "hwaccel", HAS_ARG | OPT_STRING | OPT_EXPERT, { &hwaccel }, "use HW accelerated decoding" }, { NULL, }, }; @@ -3739,6 +3765,10 @@ int main(int argc, char **argv) #ifdef SDL_HINT_VIDEO_X11_NET_WM_BYPASS_COMPOSITOR SDL_SetHint(SDL_HINT_VIDEO_X11_NET_WM_BYPASS_COMPOSITOR, "0"); #endif + if (hwaccel && !enable_vulkan) { + av_log(NULL, AV_LOG_INFO, "Enable vulkan renderer to support hwaccel %s\n", hwaccel); + enable_vulkan = 1; + } if (enable_vulkan) { vk_renderer = vk_get_renderer(); if (vk_renderer) { diff --git a/fftools/ffplay_renderer.c b/fftools/ffplay_renderer.c index 796a964a7b..bf6ad7fbaf 100644 --- a/fftools/ffplay_renderer.c +++ b/fftools/ffplay_renderer.c @@ -50,6 +50,12 @@ typedef struct RendererContext { pl_tex tex[4]; pl_log vk_log; + + AVBufferRef *hw_device; + AVBufferRef *hw_frame; + int hw_failed; + + AVFrame *vk_frame; } RendererContext; static void vk_log_cb(void *log_priv, enum pl_log_level level, const char *msg) { @@ -133,6 +139,13 @@ static int create(VkRenderer *renderer, SDL_Window *window) ret = AVERROR_EXTERNAL; goto out; } + + ctx->vk_frame = av_frame_alloc(); + if (!ctx->vk_frame) { + ret = AVERROR(ENOMEM); + goto out; + } + ret = 0; out: @@ -140,6 +153,102 @@ out: return ret; } +static int create_hw(VkRenderer *renderer, AVFrame *frame) +{ + RendererContext *ctx = (RendererContext *)renderer; + AVHWFramesContext *src_hw_frame = (AVHWFramesContext *)frame->hw_frames_ctx->data; + AVBufferRef *src_dev = src_hw_frame->device_ref; + int ret; + + if (ctx->hw_failed) + return ctx->hw_failed; + + if (!ctx->hw_device) { + ret = av_hwdevice_ctx_create_derived(&ctx->hw_device, AV_HWDEVICE_TYPE_VULKAN, src_dev, 0); + if (ret < 0) { + av_log(renderer, AV_LOG_ERROR, "Derive hwaccel failed, %s\n", av_err2str(ret)); + ctx->hw_failed = ret; + return ret; + } + } + + if (!ctx->hw_frame) { + AVHWFramesContext *hw_frame; + + ctx->hw_frame = av_hwframe_ctx_alloc(ctx->hw_device); + if (!ctx->hw_frame) { + ctx->hw_failed = AVERROR(ENOMEM); + return AVERROR(ENOMEM); + } + + hw_frame = (AVHWFramesContext *)ctx->hw_frame->data; + hw_frame->format = AV_PIX_FMT_VULKAN; + hw_frame->sw_format = src_hw_frame->sw_format; + hw_frame->width = frame->width; + hw_frame->height = frame->height; + + ret = av_hwframe_ctx_init(ctx->hw_frame); + if (ret < 0) { + av_log(renderer, AV_LOG_ERROR, "Create hwframe context failed, %s\n", av_err2str(ret)); + ctx->hw_failed = ret; + return ret; + } + } + + return 0; +} + +static int transfer_frame(VkRenderer *renderer, AVFrame *frame) +{ + RendererContext *ctx = (RendererContext *)renderer; + int ret; + + if (!frame->hw_frames_ctx) + return 0; + + if (frame->format == AV_PIX_FMT_VULKAN) + return 0; + + ret = create_hw(renderer, frame); + if (ret < 0 && ret != AVERROR(ENOSYS)) + return ret; + + // Try map data first + av_frame_unref(ctx->vk_frame); + if (ctx->hw_frame) { + ctx->vk_frame->hw_frames_ctx = av_buffer_ref(ctx->hw_frame); + ctx->vk_frame->format = AV_PIX_FMT_VULKAN; + } + ret = av_hwframe_map(ctx->vk_frame, frame, AV_HWFRAME_MAP_READ); + if (!ret) { + goto out; + } + + if (ret != AVERROR(ENOSYS)) { + av_log(NULL, AV_LOG_FATAL, "Map data to vulkan failed: %s\n", av_err2str(ret)); + return ret; + } + + // Try transfer data + av_frame_unref(ctx->vk_frame); + if (ctx->hw_frame) + av_hwframe_get_buffer(ctx->hw_frame, ctx->vk_frame, 0); + ret = av_hwframe_transfer_data(ctx->vk_frame, frame, 0); + if (ret < 0) { + av_log(NULL, AV_LOG_FATAL, "Transfer data to vulkan failed: %s\n", av_err2str(ret)); + return ret; + } + +out: + ret = av_frame_copy_props(ctx->vk_frame, frame); + if (ret < 0) + return ret; + av_frame_unref(frame); + av_frame_move_ref(frame, ctx->vk_frame); + + return 0; +} + static int display(VkRenderer *renderer, AVFrame *frame) { struct pl_swapchain_frame swap_frame = {0}; @@ -148,6 +257,10 @@ static int display(VkRenderer *renderer, AVFrame *frame) RendererContext *ctx = (RendererContext *)renderer; int ret = 0; + ret = transfer_frame(renderer, frame); + if (ret < 0) + return ret; + if (!pl_map_avframe_ex(ctx->pl_vk->gpu, &pl_frame, pl_avframe_params( .frame = frame, .tex = ctx->tex))) { @@ -194,6 +307,10 @@ static void destroy(VkRenderer *renderer) PFN_vkDestroySurfaceKHR vkDestroySurfaceKHR; RendererContext *ctx = (RendererContext *)renderer; + av_buffer_unref(&ctx->hw_frame); + av_buffer_unref(&ctx->hw_device); + av_frame_free(&ctx->vk_frame); + for (int i = 0; i < FF_ARRAY_ELEMS(ctx->tex); i++) pl_tex_destroy(ctx->pl_vk->gpu, &ctx->tex[i]); pl_renderer_destroy(&ctx->renderer);