From patchwork Fri May 10 21:12:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Fran=C3=A7ois-Simon_Fauteux-Chapleau?= X-Patchwork-Id: 48714 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1706:b0:1af:cdee:28c5 with SMTP id nv6csp842125pzb; Fri, 10 May 2024 14:12:42 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUakB46loAMhdUvO+IbWQQ0jp6p4nK05dyopQMXGECqLftuaAgtw5aNl5Wnv4PquAj3Oho5g3ChQw4H21RRbwRIbCdMp7AzHCvxxA== X-Google-Smtp-Source: AGHT+IEoBa66V2KDbkZ/qi+3PRmHvzUEJreUpri9+TP5q53JYC/1pQEOw1KgR1vXeM3Hg/K16US9 X-Received: by 2002:a50:9353:0:b0:568:1248:9f49 with SMTP id 4fb4d7f45d1cf-5734d5bfbf9mr2328275a12.18.1715375562344; Fri, 10 May 2024 14:12:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1715375562; cv=none; d=google.com; s=arc-20160816; b=kjJujS6NOfa3aJ5KOXe3SPvBKv5N8QIGXqWmlX1u7+FsdhALs8v9SgyVv6Ty/f55fJ o8czTWYjv84FxqdA2oG2TWyNYz2g0yCFqcsNjDUq125jAy033fJa/qr4B8KMBUuWX8Us 7vL1mDlNSzrpb7x7cL8hH5MAkAkmtGJUE3E0ZCLhHHhfgLJve/YjJDDbHvGK8jm5QgoI XL3p84dDSoObY0RY0MnYFvEBCVnAUyaDF1C14r96Z1wbxdv6NGzQag1x2lFtcnhmjLrw pQcPYFC42cYxpjmM/ox2lvMhunfPri9de0jOOwOSpkJlz8qsI8d2UJ6JsQAvwf9wfdg1 mKsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:dkim-filter:delivered-to; bh=z+HhuxXs3HZmd5UPy46q+dOyPj8SC1rGg+rSg1ioB5g=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=KW3Kj5pKK2oQjbZFHXoZ2OpVZQ6NTWmp6Cf9nO/X7/3Vy2Uw2sAe7Rut2DL7MX7JaH xK/Csjb7K+li19HVl8Qckqk2U85Hax8x7i5w2ITP59HFdCAryv4UM2YmeyeAgH2c40jl RhgK4I80MdDH4pI6llKtEj7kDzcVNvEQCi5Npn061e4QnwA/QeazzVDZE0ol+psIENCl RNhq1cE3bomlUX2XLjeXMc8QMwNr409P7V3hA0pjMGBek9+Oiz2jljgiNWdi4g+87pdM xk2vhKP9E6U3ZXj90AanEOGwdeWhpE53n6NM4QvyP29OyfWs9jk5Tl+NtV2opKNKXKa1 gY8Q==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=dJZfLoPb; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5733beac6ecsi2293892a12.62.2024.05.10.14.12.41; Fri, 10 May 2024 14:12:42 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=dJZfLoPb; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 512DC68D546; Sat, 11 May 2024 00:12:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 45E8E68BDB6 for ; Sat, 11 May 2024 00:12:29 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 9A6699C41A0 for ; Fri, 10 May 2024 17:12:27 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id mE3rYMQnPcsb for ; Fri, 10 May 2024 17:12:19 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id D0BA89C5649 for ; Fri, 10 May 2024 17:12:19 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com D0BA89C5649 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1715375539; bh=7c/oVAVSLQ16VvJDjPVIu5V35H6meOPfrHMKqGNnqHU=; h=From:To:Date:Message-Id:MIME-Version; b=dJZfLoPb8Q82d+NYFHkVOXGaCaoffdlbJanh1BbYi1g+Aze/0g3pwp52vmqQmUMa8 ewBWOZOqk7FT5kWGs4p7U71A+bf/38wx9iaaC9rzT43ZqYSPxqwToPX0zfPPN/r7L+ hNdaJltFDj/O8QrKBG9GIIe9Gxugujfbwdglck6MLZR5yKdfBCvm5pYjCrv5eon5bP F+vNmXcPamAM5WuMlPd36VHHnzt7dmPUANxc7dqItXk4TSPz2nAd5wDHilXjAj8ZE5 weYwiu5zXTntjW8ZfxlA4ve+KbdNg1BFP6GjRM8Ld3qgdDzNszmZBFCKXw36KBlxKe ANkP4KOn0Bxog== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id SEGa4w0jHYXW for ; Fri, 10 May 2024 17:12:19 -0400 (EDT) Received: from strix670.mtl.sfl (unknown [192.168.51.254]) by mail.savoirfairelinux.com (Postfix) with ESMTP id AE6A69C41A0 for ; Fri, 10 May 2024 17:12:19 -0400 (EDT) From: =?utf-8?q?Fran=C3=A7ois-Simon_Fauteux-Chapleau?= To: ffmpeg-devel@ffmpeg.org Date: Fri, 10 May 2024 17:12:19 -0400 Message-Id: <20240510211219.213409-1-francois-simon.fauteux-chapleau@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2] libavfilter: add PipeWire-based grab X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: rj5OcJd6wfyQ This is a revised version of the "pipewiregrab" patch submitted by Abhishek Ojha a few months ago: https://patchwork.ffmpeg.org/project/ffmpeg/patch/20231227162504.690730-1-abhishek.ojha@savoirfairelinux.com/ https://patchwork.ffmpeg.org/project/ffmpeg/patch/20231227162504.690730-2-abhishek.ojha@savoirfairelinux.com/ The main change is that the patch is now implemented as a libavfilter source filter instead of a libavdevice input device, as was requested in a comment on the previous version. This version also adds support for DMA buffer sharing and uses sd-bus instead of GDBus. There are also several small changes meant to fix bugs or simplify the code, but the overall structure remains the same as before: we use the ScreenCast interface provided by XDG Desktop Portal to obtain a file descriptor, which is then used to create a PipeWire stream. The data from that stream can then be used to generate frames for FFmpeg. Example usage: ffmpeg -f lavfi -i pipewiregrab \ -vf 'hwmap=derive_device=vaapi,scale_vaapi=format=nv12' \ -c:v h264_vaapi -t 10 output.mp4 Signed-off-by: François-Simon Fauteux-Chapleau --- configure | 16 + libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vsrc_pipewiregrab.c | 1433 +++++++++++++++++++++++++++++++ 4 files changed, 1451 insertions(+) create mode 100644 libavfilter/vsrc_pipewiregrab.c diff --git a/configure b/configure index beb1fa6d3c..028020455e 100755 --- a/configure +++ b/configure @@ -304,6 +304,7 @@ External library support: --enable-libxcb-shm enable X11 grabbing shm communication [autodetect] --enable-libxcb-xfixes enable X11 grabbing mouse rendering [autodetect] --enable-libxcb-shape enable X11 grabbing shape rendering [autodetect] + --enable-libpipewire enable screen grabbing using PipeWire [autodetect] --enable-libxvid enable Xvid encoding via xvidcore, native MPEG-4/Xvid encoder exists [no] --enable-libxml2 enable XML parsing using the C library libxml2, needed @@ -1845,6 +1846,8 @@ EXTERNAL_AUTODETECT_LIBRARY_LIST=" libxcb_shm libxcb_shape libxcb_xfixes + libpipewire + libsystemd lzma mediafoundation metal @@ -3895,6 +3898,7 @@ pad_opencl_filter_deps="opencl" pan_filter_deps="swresample" perspective_filter_deps="gpl" phase_filter_deps="gpl" +pipewiregrab_filter_deps="libpipewire libsystemd pthreads" pp7_filter_deps="gpl" pp_filter_deps="gpl postproc" prewitt_opencl_filter_deps="opencl" @@ -7230,6 +7234,18 @@ if enabled libxcb; then enabled libxcb_xfixes && check_pkg_config libxcb_xfixes xcb-xfixes xcb/xfixes.h xcb_xfixes_get_cursor_image fi +# Starting with version 0.3.52, PipeWire's spa library uses the __LOCALE_C_ONLY macro to determine +# whether the locale_t type (introduced in POSIX.1-2008) and some related functions are available (see +# https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/2390 for more information). +# Unfortunately, this macro is specific to uclibc, which can cause build issues on systems that use a +# different implementation of libc if POSIX 2008 support isn't enabled (which is the case for FFmpeg currently). +# As a workaround for this problem, we add a compilation flag to ensure that __LOCALE_C_ONLY is always defined. +add_cppflags -D__LOCALE_C_ONLY +enabled libpipewire && check_pkg_config libpipewire "libpipewire-0.3 >= 0.3.40" pipewire/pipewire.h pw_init +if enabled libpipewire; then + enabled libsystemd && check_pkg_config libsystemd "libsystemd >= 246" systemd/sd-bus.h sd_bus_call_method +fi + check_func_headers "windows.h" CreateDIBSection "$gdigrab_indev_extralibs" # check if building for desktop or uwp diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 5992fd161f..6352e91586 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -603,6 +603,7 @@ OBJS-$(CONFIG_NULLSRC_FILTER) += vsrc_testsrc.o OBJS-$(CONFIG_OPENCLSRC_FILTER) += vf_program_opencl.o opencl.o OBJS-$(CONFIG_PAL75BARS_FILTER) += vsrc_testsrc.o OBJS-$(CONFIG_PAL100BARS_FILTER) += vsrc_testsrc.o +OBJS-$(CONFIG_PIPEWIREGRAB_FILTER) += vsrc_pipewiregrab.o OBJS-$(CONFIG_QRENCODE_FILTER) += qrencode.o textutils.o OBJS-$(CONFIG_QRENCODESRC_FILTER) += qrencode.o textutils.o OBJS-$(CONFIG_RGBTESTSRC_FILTER) += vsrc_testsrc.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index c532682fc2..3670a6d7e7 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -569,6 +569,7 @@ extern const AVFilter ff_vsrc_openclsrc; extern const AVFilter ff_vsrc_qrencodesrc; extern const AVFilter ff_vsrc_pal75bars; extern const AVFilter ff_vsrc_pal100bars; +extern const AVFilter ff_vsrc_pipewiregrab; extern const AVFilter ff_vsrc_rgbtestsrc; extern const AVFilter ff_vsrc_sierpinski; extern const AVFilter ff_vsrc_smptebars; diff --git a/libavfilter/vsrc_pipewiregrab.c b/libavfilter/vsrc_pipewiregrab.c new file mode 100644 index 0000000000..51073c22b1 --- /dev/null +++ b/libavfilter/vsrc_pipewiregrab.c @@ -0,0 +1,1433 @@ +/* + * PipeWire input grabber (ScreenCast) + * Copyright (C) 2024 Savoir-faire Linux, Inc. + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * PipeWireGrab video source + * @author Firas Ashkar + * @author Abhishek Ojha + * @author François-Simon Fauteux-Chapleau + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "avfilter.h" +#include "formats.h" +#include "video.h" + +#include "libavformat/avformat.h" +#include "libavutil/avassert.h" +#include "libavutil/avstring.h" +#include "libavutil/hwcontext.h" +#include "libavutil/hwcontext_drm.h" +#include "libavutil/mem.h" +#include "libavutil/opt.h" +#include "libavutil/time.h" + +#ifndef __USE_XOPEN2K8 +#define F_DUPFD_CLOEXEC \ + 1030 /* Duplicate file descriptor with close-on-exit set. */ +#endif + +#define DESTINATION "org.freedesktop.portal.Desktop" +#define SENDER DESTINATION +#define OBJECT_PATH "/org/freedesktop/portal/desktop" +#define INTERFACE "org.freedesktop.portal.ScreenCast" +#define REQUEST_PATH "/org/freedesktop/portal/desktop/request/%s/%s" + +#define BYTES_PER_PIXEL 4 /* currently all formats assume 4 bytes per pixel */ +#define MAX_SPA_PARAM 4 /* max number of params for spa pod */ + +/** + * PipeWire capture types + */ +typedef enum { + DESKTOP_CAPTURE = 1, + WINDOW_CAPTURE = 2, +} pw_capture_type; + +/** + * XDG Desktop Portal supported cursor modes + */ +enum PortalCursorMode { + PORTAL_CURSOR_MODE_HIDDEN = 1 << 0, + PORTAL_CURSOR_MODE_EMBEDDED = 1 << 1, +}; + +typedef struct PipewireGrabContext { + const AVClass *class; + + sd_bus *connection; + atomic_int dbus_event_loop_running; + char *sender_name; + char *session_handle; + + uint64_t pipewire_node; + int pipewire_fd; + + pthread_cond_t pipewire_initialization_cond_var; + pthread_mutex_t pipewire_initialization_mutex; + atomic_int pipewire_initialization_over; + int pw_init_called; + struct pw_thread_loop *thread_loop; + struct pw_context *context; + struct pw_core *core; + struct spa_hook core_listener; + struct pw_stream *stream; + struct spa_hook stream_listener; + struct spa_video_info format; + + uint32_t available_cursor_modes; + pw_capture_type capture_type; + int draw_mouse; + + uint32_t width, height; + size_t frame_size; + uint8_t Bpp; + enum AVPixelFormat av_pxl_format; + + int64_t time_frame; + int64_t frame_duration; + AVRational framerate; + pthread_mutex_t current_frame_mutex; + AVFrame *current_frame; + AVBufferRef *hw_device_ref; + AVBufferRef *hw_frames_ref; + int enable_dmabuf; + const char *device_path; + + int portal_error; + int pipewire_error; +} PipewireGrabContext; + +/** + * Data for DBus signals callbacks + */ +struct DbusSignalData { + AVFilterContext *ctx; + sd_bus_slot *slot; +}; + +#define OFFSET(x) offsetof(PipewireGrabContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM +static const AVOption pipewiregrab_options[] = { + { "framerate", "set video frame rate", OFFSET(framerate), AV_OPT_TYPE_VIDEO_RATE, { .str = "ntsc" }, 0, INT_MAX, FLAGS }, + { "draw_mouse", "draw the mouse pointer", OFFSET(draw_mouse), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, FLAGS }, + { "capture_type", "set the capture type (1 for screen, 2 for window)", OFFSET(capture_type), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, 2, FLAGS }, + { "fd", "set file descriptor to be used by PipeWire", OFFSET(pipewire_fd), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, + { "node", "set PipeWire node (required when using the 'fd' option)", OFFSET(pipewire_node), AV_OPT_TYPE_UINT64, { .i64 = 0 }, 0, 0xffffffff, FLAGS }, + { "enable_dmabuf", "enable DMA-BUF sharing", OFFSET(enable_dmabuf), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, FLAGS }, + { "device", "DRM device path", OFFSET(device_path), AV_OPT_TYPE_STRING, { .str = "/dev/dri/card0" }, 0, 0, FLAGS }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(pipewiregrab); + +/** + * Helper function to allow portal_init_screencast to stop and return an error + * code if a DBus operation/callback fails. + * + * @param ctx + * @param error AVERROR code (negative) + * @param message error message + */ +static void portal_abort(AVFilterContext *ctx, int error, const char *message) +{ + PipewireGrabContext *pw_ctx = ctx->priv; + + pw_ctx->portal_error = error; + av_log(ctx, AV_LOG_ERROR, "Aborting: %s\n", message); + + atomic_store(&pw_ctx->dbus_event_loop_running, 0); +} + +/** + * Callback to handle PipeWire core info events + * + * @param user_data pointer to AVFilterContext + * @param info pw_core_info + */ +static void on_core_info_callback(void *user_data, const struct pw_core_info *info) +{ + AVFilterContext *ctx = user_data; + av_log(ctx, AV_LOG_DEBUG, "Server version: %s\n", info->version); + av_log(ctx, AV_LOG_INFO, "Library version: %s\n", pw_get_library_version()); + av_log(ctx, AV_LOG_DEBUG, "Header version: %s\n", pw_get_headers_version()); +} + +/** + * Callback to handle PipeWire core done events + * + * @param user_data pointer to AVFilterContext + * @param id PipeWire object id of calling + * @param seq PipeWire object sequence + */ +static void on_core_done_callback(void *user_data, uint32_t id, int seq) +{ + AVFilterContext *ctx = user_data; + PipewireGrabContext *pw_ctx; + + if (!ctx || !ctx->priv) + return; + pw_ctx = ctx->priv; + + if (id == PW_ID_CORE) + pw_thread_loop_signal(pw_ctx->thread_loop, false); +} + +/** + * Callback to handle Pipewire core error events + * + * @param user_data pointer to AVFilterContext + * @param id id of PipeWire proxy object where the error occured + * @param seq PipeWire sequence number which produced the error + * @param res error number + * @param message error message + */ +static void on_core_error_callback(void *user_data, uint32_t id, int seq, + int res, const char *message) +{ + AVFilterContext *ctx = user_data; + PipewireGrabContext *pw_ctx; + + if (!ctx) + return; + + av_log(ctx, AV_LOG_ERROR, + "PipeWire core error: %s (id=%u, seq=%d, res=%d: %s)\n", + message, id, seq, res, strerror(-res)); + + pw_ctx = ctx->priv; + if (!pw_ctx) + return; + + pw_thread_loop_signal(pw_ctx->thread_loop, false); + pw_ctx->pipewire_error = res; + atomic_store(&pw_ctx->pipewire_initialization_over, 1); + pthread_cond_signal(&pw_ctx->pipewire_initialization_cond_var); +} + +/** + * PipeWire core events callbacks + */ +static const struct pw_core_events core_events = { + PW_VERSION_CORE_EVENTS, + .info = on_core_info_callback, + .done = on_core_done_callback, + .error = on_core_error_callback, +}; + +/** + * Helper function: convert spa video format to AVPixelFormat + * + * @param video_format spa video format to convert + * @return the corresponding AVPixelFormat + */ +static enum AVPixelFormat +spa_video_format_to_av_pixel_format(enum spa_video_format video_format) +{ + switch (video_format) { + case SPA_VIDEO_FORMAT_RGBA: + case SPA_VIDEO_FORMAT_RGBx: + return AV_PIX_FMT_RGBA; + + case SPA_VIDEO_FORMAT_BGRA: + case SPA_VIDEO_FORMAT_BGRx: + return AV_PIX_FMT_BGRA; + + default: + return AV_PIX_FMT_NONE; + } +} + +static uint32_t spa_video_format_to_drm_format(enum spa_video_format video_format) +{ + switch (video_format) { + case SPA_VIDEO_FORMAT_RGBA: + return DRM_FORMAT_ABGR8888; + case SPA_VIDEO_FORMAT_RGBx: + return DRM_FORMAT_XBGR8888; + case SPA_VIDEO_FORMAT_BGRA: + return DRM_FORMAT_ARGB8888; + case SPA_VIDEO_FORMAT_BGRx: + return DRM_FORMAT_XRGB8888; + default: + return DRM_FORMAT_INVALID; + } +} + +static const uint32_t pipewiregrab_formats[] = { + SPA_VIDEO_FORMAT_RGBA, + SPA_VIDEO_FORMAT_RGBx, + SPA_VIDEO_FORMAT_BGRx, + SPA_VIDEO_FORMAT_BGRA, +}; + +static const uint64_t pipewiregrab_default_modifiers[] = { + DRM_FORMAT_MOD_LINEAR, + DRM_FORMAT_MOD_INVALID, +}; + +/** + * PipeWire callback of parameters changed events + * + * @param user_data pointer to AVFilterContext + * @param id type of changed param + * @param param pointer to changed param structure + */ +static void on_stream_param_changed_callback(void *user_data, uint32_t id, + const struct spa_pod *param) +{ + struct spa_pod_builder pod_builder; + const struct spa_pod *params[MAX_SPA_PARAM]; + uint32_t n_params = 0; + uint32_t buffer_types; + uint8_t params_buffer[4096]; + int result; + int err; + PipewireGrabContext *pw_ctx; + AVFilterContext *ctx = user_data; + AVHWFramesContext *frames_ctx = NULL; + + if (!ctx || !ctx->priv || !param) + return; + + if (id != SPA_PARAM_Format) { + av_log(ctx, AV_LOG_WARNING, + "Ignoring non-Format param change\n"); + return; + } + + pw_ctx = ctx->priv; + + result = spa_format_parse(param, &pw_ctx->format.media_type, + &pw_ctx->format.media_subtype); + if (result < 0) { + av_log(ctx, AV_LOG_ERROR, "Unable to parse media type\n"); + pw_ctx->pipewire_error = AVERROR(EINVAL); + goto end; + } + + if (pw_ctx->format.media_type != SPA_MEDIA_TYPE_video || + pw_ctx->format.media_subtype != SPA_MEDIA_SUBTYPE_raw) { + av_log(ctx, AV_LOG_ERROR, "Unexpected media type\n"); + pw_ctx->pipewire_error = AVERROR(EINVAL); + goto end; + } + + spa_format_video_raw_parse(param, &pw_ctx->format.info.raw); + + av_log(ctx, AV_LOG_INFO, "Negotiated format:\n"); + + av_log(ctx, AV_LOG_INFO, "Format: %d (%s)\n", + pw_ctx->format.info.raw.format, + spa_debug_type_find_name(spa_type_video_format, + pw_ctx->format.info.raw.format)); + av_log(ctx, AV_LOG_INFO, "Size: %dx%d\n", + pw_ctx->format.info.raw.size.width, + pw_ctx->format.info.raw.size.height); + av_log(ctx, AV_LOG_INFO, "Framerate: %d/%d\n", + pw_ctx->format.info.raw.framerate.num, + pw_ctx->format.info.raw.framerate.denom); + + pw_ctx->width = pw_ctx->format.info.raw.size.width; + pw_ctx->height = pw_ctx->format.info.raw.size.height; + pw_ctx->Bpp = BYTES_PER_PIXEL; + pw_ctx->frame_size = pw_ctx->width * pw_ctx->height * pw_ctx->Bpp; + if (pw_ctx->frame_size + AV_INPUT_BUFFER_PADDING_SIZE > INT_MAX) { + av_log(ctx, AV_LOG_ERROR, "Captured area is too large\n"); + pw_ctx->pipewire_error = AVERROR(EINVAL); + goto end; + } + + pw_ctx->av_pxl_format = + spa_video_format_to_av_pixel_format(pw_ctx->format.info.raw.format); + if (pw_ctx->av_pxl_format == AV_PIX_FMT_NONE) { + av_log(ctx, AV_LOG_ERROR, + "Unsupported buffer format: %d\n", pw_ctx->format.info.raw.format); + pw_ctx->pipewire_error = AVERROR(EINVAL); + goto end; + } + + /* Video crop */ + pod_builder = SPA_POD_BUILDER_INIT(params_buffer, sizeof(params_buffer)); + params[n_params++] = spa_pod_builder_add_object( + &pod_builder, SPA_TYPE_OBJECT_ParamMeta, SPA_PARAM_Meta, + SPA_PARAM_META_type, SPA_POD_Id(SPA_META_VideoCrop), + SPA_PARAM_META_size, SPA_POD_Int(sizeof(struct spa_meta_region))); + + /* Buffer options */ + buffer_types = (1 << SPA_DATA_MemPtr) | (1 << SPA_DATA_MemFd); + if (spa_pod_find_prop(param, NULL, SPA_FORMAT_VIDEO_modifier)) { + err = av_hwdevice_ctx_create(&pw_ctx->hw_device_ref, AV_HWDEVICE_TYPE_DRM, + pw_ctx->device_path, NULL, 0); + if (err < 0) + goto hw_fail; + + pw_ctx->hw_frames_ref = av_hwframe_ctx_alloc(pw_ctx->hw_device_ref); + if (!pw_ctx->hw_frames_ref) { + err = AVERROR(ENOMEM); + goto hw_fail; + } + frames_ctx = (AVHWFramesContext*)pw_ctx->hw_frames_ref->data; + frames_ctx->format = AV_PIX_FMT_DRM_PRIME; + frames_ctx->sw_format = pw_ctx->av_pxl_format; + frames_ctx->width = pw_ctx->width; + frames_ctx->height = pw_ctx->height; + err = av_hwframe_ctx_init(pw_ctx->hw_frames_ref); +hw_fail: + if (!err) { + buffer_types |= 1 << SPA_DATA_DmaBuf; + } else { + av_log(ctx, AV_LOG_WARNING, + "Failed to initialize hardware frames context: %s. " + "Falling back to shared memory\n", av_err2str(err)); + } + } + + params[n_params++] = spa_pod_builder_add_object( + &pod_builder, SPA_TYPE_OBJECT_ParamBuffers, SPA_PARAM_Buffers, + SPA_PARAM_BUFFERS_dataType, + SPA_POD_Int(buffer_types)); + + /* Meta header */ + params[n_params++] = spa_pod_builder_add_object( + &pod_builder, SPA_TYPE_OBJECT_ParamMeta, SPA_PARAM_Meta, + SPA_PARAM_META_type, SPA_POD_Id(SPA_META_Header), + SPA_PARAM_META_size, + SPA_POD_Int(sizeof(struct spa_meta_header))); + + pw_stream_update_params(pw_ctx->stream, params, n_params); + +end: + // Signal pipewiregrab_init that PipeWire initialization is over (either + // because it was completed successfully or because there was an error, in + // which case pw_ctx->pipewire_error will have been set to a nonzero value). + atomic_store(&pw_ctx->pipewire_initialization_over, 1); + pthread_cond_signal(&pw_ctx->pipewire_initialization_cond_var); +} + +/** + * PipeWire callback of state changed events + * + * @param user_data pointer to AVFilterContext + * @param old old PipeWire stream state + * @param state current PipeWire stream state + * @param error received error information + */ +static void on_stream_state_changed_callback(void *user_data, + enum pw_stream_state old, + enum pw_stream_state state, + const char *error) +{ + AVFilterContext *ctx = user_data; + if (!ctx) + return; + + av_log(ctx, AV_LOG_INFO, "stream state: \"%s\"\n", + pw_stream_state_as_string(state)); +} + +/** + * Find most recent buffer received in a PipeWire stream + * + * @param stream stream to get buffer from + * @return most recent buffer in the stream + */ +static struct pw_buffer *find_most_recent_buffer_and_recycle_olders(struct pw_stream *stream) +{ + struct pw_buffer *pw_buf = NULL; + while (1) { + struct pw_buffer *aux = pw_stream_dequeue_buffer(stream); + if (!aux) + break; + if (pw_buf) + pw_stream_queue_buffer(stream, pw_buf); + pw_buf = aux; + } + return pw_buf; +} + +static void free_frame_desc(void *opaque, uint8_t *data) +{ + AVDRMFrameDescriptor *frame_desc = (AVDRMFrameDescriptor *)data; + + for (int i = 0; i < frame_desc->nb_objects; i++) + close(frame_desc->objects[i].fd); + av_free(frame_desc); +} + +static void process_dma_buffer(AVFilterContext *ctx, struct spa_buffer *spa_buf) +{ + AVFrame *frame = NULL; + AVDRMFrameDescriptor *frame_desc = NULL; + int ret; + int n_planes; + size_t size; + uint32_t offset, pitch; + PipewireGrabContext *pw_ctx = ctx->priv; + + n_planes = spa_buf->n_datas; + av_assert0(n_planes <= AV_DRM_MAX_PLANES); + + // Create frame descriptor + frame_desc = av_mallocz(sizeof(*frame_desc)); + if (!frame_desc) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate frame descriptor\n"); + goto fail; + } + *frame_desc = (AVDRMFrameDescriptor) { + .nb_objects = n_planes, + .nb_layers = 1, + .layers[0] = { + .format = spa_video_format_to_drm_format(pw_ctx->format.info.raw.format), + .nb_planes = n_planes, + }, + }; + for (int i = 0; i < n_planes; i++) { + offset = spa_buf->datas[i].chunk->offset; + pitch = spa_buf->datas[i].chunk->stride; + size = offset + pitch * pw_ctx->height; + + frame_desc->objects[i] = (AVDRMObjectDescriptor) { + .fd = spa_buf->datas[i].fd, + .size = size, + .format_modifier = pw_ctx->format.info.raw.modifier, + }; + frame_desc->layers[0].planes[i] = (AVDRMPlaneDescriptor) { + .object_index = i, + .offset = offset, + .pitch = pitch, + }; + } + + // Create frame + frame = av_frame_alloc(); + if (!frame) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate frame\n"); + goto fail; + } + frame->hw_frames_ctx = av_buffer_ref(pw_ctx->hw_frames_ref); + if (!frame->hw_frames_ctx) { + av_log(ctx, AV_LOG_ERROR, "Failed to create buffer reference\n"); + goto fail; + } + frame->buf[0] = av_buffer_create((uint8_t *)frame_desc, sizeof(*frame_desc), + free_frame_desc, NULL, 0); + if (!frame->buf[0]) { + av_log(ctx, AV_LOG_ERROR, "Failed to create buffer\n"); + goto fail; + } + frame->data[0] = (uint8_t *)frame_desc; + frame->format = AV_PIX_FMT_DRM_PRIME; + frame->width = pw_ctx->width; + frame->height = pw_ctx->height; + + // Update current_frame + pthread_mutex_lock(&pw_ctx->current_frame_mutex); + av_frame_unref(pw_ctx->current_frame); + ret = av_frame_ref(pw_ctx->current_frame, frame); + pthread_mutex_unlock(&pw_ctx->current_frame_mutex); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Failed to create frame reference\n"); + av_frame_free(&frame); + } + return; + +fail: + av_freep(&frame_desc); + av_frame_free(&frame); +} + +static void process_shm_buffer(AVFilterContext *ctx, struct spa_buffer *spa_buf) +{ + uint8_t *map = NULL; + void *sdata = NULL; + struct spa_meta_region *region; + int crop_left, crop_right, crop_top, crop_bottom; + PipewireGrabContext *pw_ctx = ctx->priv; + + // Get data + if (spa_buf->datas[0].type == SPA_DATA_MemFd ) { + map = mmap(NULL, spa_buf->datas[0].maxsize + spa_buf->datas[0].mapoffset, + PROT_READ, MAP_PRIVATE, spa_buf->datas[0].fd, 0); + if (map == MAP_FAILED) { + av_log(ctx, AV_LOG_ERROR, "mmap failed: %s\n", strerror(errno)); + return; + } + sdata = SPA_PTROFF(map, spa_buf->datas[0].mapoffset, uint8_t); + } else if (spa_buf->datas[0].type == SPA_DATA_MemPtr) { + if (spa_buf->datas[0].data == NULL) { + av_log(ctx, AV_LOG_ERROR, "No data in buffer\n"); + return; + } + sdata = spa_buf->datas[0].data; + } else { + av_log(ctx, AV_LOG_ERROR, "Buffer is not valid\n"); + return; + } + + region = spa_buffer_find_meta_data(spa_buf, SPA_META_VideoCrop, sizeof(*region)); + if (region && spa_meta_region_is_valid(region)) { + crop_left = region->region.position.x; + crop_top = region->region.position.y; + crop_right = pw_ctx->width - crop_left - region->region.size.width; + crop_bottom = pw_ctx->height - crop_top - region->region.size.height; + } + + // Update current_frame with the new data + pthread_mutex_lock(&pw_ctx->current_frame_mutex); + memcpy(pw_ctx->current_frame->data[0], sdata, spa_buf->datas[0].chunk->size); + pw_ctx->current_frame->crop_top = crop_top; + pw_ctx->current_frame->crop_bottom = crop_bottom; + pw_ctx->current_frame->crop_left = crop_left; + pw_ctx->current_frame->crop_right = crop_right; + pthread_mutex_unlock(&pw_ctx->current_frame_mutex); + + // Cleanup + if (spa_buf->datas[0].type == SPA_DATA_MemFd) + munmap(map, spa_buf->datas[0].maxsize + spa_buf->datas[0].mapoffset); +} + +/** + * This function is called by PipeWire when a buffer + * is ready to be dequeued and processed. + * + * @param user_data pointer to AVFilterContext + */ +static void on_stream_process_callback(void *user_data) +{ + struct spa_buffer *spa_buf; + struct pw_buffer *pw_buf = NULL; + struct spa_meta_header *header = NULL; + + AVFilterContext *ctx = user_data; + PipewireGrabContext *pw_ctx; + if (!ctx || !ctx->priv) + return; + pw_ctx = ctx->priv; + + // We need to wait for pw_ctx->current_frame to have been allocated before + // we can use it to get frames from the PipeWire thread to FFmpeg + pthread_mutex_lock(&pw_ctx->current_frame_mutex); + if (!pw_ctx->current_frame) { + pthread_mutex_unlock(&pw_ctx->current_frame_mutex); + return; + } + pthread_mutex_unlock(&pw_ctx->current_frame_mutex); + + pw_buf = find_most_recent_buffer_and_recycle_olders(pw_ctx->stream); + if (!pw_buf) { + av_log(ctx, AV_LOG_ERROR, "Out of buffers\n"); + return; + } + + spa_buf = pw_buf->buffer; + header = spa_buffer_find_meta_data(spa_buf, SPA_META_Header, sizeof(*header)); + if (header && (header->flags & SPA_META_HEADER_FLAG_CORRUPTED)) { + av_log(ctx, AV_LOG_ERROR, "Corrupted PipeWire buffer\n"); + goto end; + } + + if (spa_buf->datas[0].type == SPA_DATA_DmaBuf) + process_dma_buffer(ctx, spa_buf); + else + process_shm_buffer(ctx, spa_buf); + +end: + pw_stream_queue_buffer(pw_ctx->stream, pw_buf); +} + +static const struct pw_stream_events stream_events = { + PW_VERSION_STREAM_EVENTS, + .state_changed = on_stream_state_changed_callback, + .param_changed = on_stream_param_changed_callback, + .process = on_stream_process_callback, +}; + +static int subscribe_to_signal(AVFilterContext *ctx, + const char *sender_name, + const char *request_token, + sd_bus_message_handler_t callback) +{ + int ret; + char *request_path; + struct DbusSignalData *dbus_signal_data; + PipewireGrabContext *pw_ctx = ctx->priv; + + dbus_signal_data = (struct DbusSignalData *)av_mallocz(sizeof(struct DbusSignalData)); + if (!dbus_signal_data) + return AVERROR(ENOMEM); + + dbus_signal_data->ctx = ctx; + request_path = av_asprintf(REQUEST_PATH, sender_name, request_token); + + ret = sd_bus_match_signal(pw_ctx->connection, + &dbus_signal_data->slot, + SENDER, + request_path, + "org.freedesktop.portal.Request", + "Response", + callback, + dbus_signal_data); + av_free(request_path); + return (ret < 0) ? ret : 0; +} + +static struct spa_pod *build_format(PipewireGrabContext *pw_ctx, + struct spa_pod_builder *builder, + uint32_t format, + const uint64_t *modifiers, + int n_modifiers) +{ + struct spa_pod_frame format_frame; + struct spa_pod_frame modifier_frame; + + spa_pod_builder_push_object(builder, &format_frame, + SPA_TYPE_OBJECT_Format, SPA_PARAM_EnumFormat); + spa_pod_builder_add(builder, SPA_FORMAT_mediaType, + SPA_POD_Id(SPA_MEDIA_TYPE_video), 0); + spa_pod_builder_add(builder, SPA_FORMAT_mediaSubtype, + SPA_POD_Id(SPA_MEDIA_SUBTYPE_raw), 0); + spa_pod_builder_add(builder, SPA_FORMAT_VIDEO_format, + SPA_POD_Id(format), 0); + spa_pod_builder_add(builder, SPA_FORMAT_VIDEO_size, + SPA_POD_CHOICE_RANGE_Rectangle( + &SPA_RECTANGLE(320, 240), + &SPA_RECTANGLE(1, 1), + &SPA_RECTANGLE(4096, 4096) + ), 0); + spa_pod_builder_add(builder, SPA_FORMAT_VIDEO_framerate, + SPA_POD_CHOICE_RANGE_Fraction( + &SPA_FRACTION(pw_ctx->framerate.num, pw_ctx->framerate.den), + &SPA_FRACTION(0, 1), + &SPA_FRACTION(144, 1) + ), 0); + if (n_modifiers > 0) { + spa_pod_builder_prop(builder, SPA_FORMAT_VIDEO_modifier, + SPA_POD_PROP_FLAG_MANDATORY | SPA_POD_PROP_FLAG_DONT_FIXATE); + spa_pod_builder_push_choice(builder, &modifier_frame, SPA_CHOICE_Enum, 0); + + // A choice POD consists of a "default" value followed by the list of + // all possible values (https://docs.pipewire.org/page_spa_pod.html) + // This is why we need to add one of the modifiers twice. + spa_pod_builder_long(builder, modifiers[0]); + for (int i = 0; i < n_modifiers; i++) + spa_pod_builder_long(builder, modifiers[i]); + + spa_pod_builder_pop(builder, &modifier_frame); + } + return spa_pod_builder_pop(builder, &format_frame); +} + +static int play_pipewire_stream(AVFilterContext *ctx) +{ + int ret; + uint8_t buffer[4096]; + struct spa_pod_builder pod_builder; + const struct spa_pod **params; + uint32_t n_params; + + PipewireGrabContext *pw_ctx = ctx->priv; + + pw_init(NULL, NULL); + pw_ctx->pw_init_called = 1; + + pw_ctx->thread_loop = + pw_thread_loop_new("thread loop", NULL); + if (!pw_ctx->thread_loop) { + av_log(ctx, AV_LOG_ERROR, "pw_thread_loop_new failed\n"); + return AVERROR(ENOMEM); + } + + pw_ctx->context = + pw_context_new(pw_thread_loop_get_loop(pw_ctx->thread_loop), NULL, 0); + if (!pw_ctx->context) { + av_log(ctx, AV_LOG_ERROR, "pw_context_new failed\n"); + ret = AVERROR(ENOMEM); + goto fail; + } + + if (pw_thread_loop_start(pw_ctx->thread_loop) < 0) { + av_log(ctx, AV_LOG_ERROR, "pw_thread_loop_start failed\n"); + ret = AVERROR(EFAULT); + goto fail; + } + + pw_thread_loop_lock(pw_ctx->thread_loop); + + // Core + pw_ctx->core = + pw_context_connect_fd(pw_ctx->context, + fcntl(pw_ctx->pipewire_fd, F_DUPFD_CLOEXEC, 3), + NULL, 0); + if (!pw_ctx->core) { + ret = AVERROR(errno); + av_log(ctx, AV_LOG_ERROR, "pw_context_connect_fd failed\n"); + pw_thread_loop_unlock(pw_ctx->thread_loop); + goto fail; + } + + pw_core_add_listener(pw_ctx->core, &pw_ctx->core_listener, &core_events, + ctx /* user_data */); + + // Stream + pw_ctx->stream = pw_stream_new( + pw_ctx->core, "wayland grab", + pw_properties_new(PW_KEY_MEDIA_TYPE, "Video", PW_KEY_MEDIA_CATEGORY, + "Capture", PW_KEY_MEDIA_ROLE, "Screen", NULL)); + + if (!pw_ctx->stream) { + av_log(ctx, AV_LOG_ERROR, "pw_stream_new failed\n"); + ret = AVERROR(ENOMEM); + pw_thread_loop_unlock(pw_ctx->thread_loop); + goto fail; + } + + pw_stream_add_listener(pw_ctx->stream, &pw_ctx->stream_listener, + &stream_events, ctx /* user_data */); + + // Stream parameters + pod_builder = SPA_POD_BUILDER_INIT(buffer, sizeof(buffer)); + params = av_mallocz(2 * FF_ARRAY_ELEMS(pipewiregrab_formats) * sizeof(*params)); + n_params = 0; + + for (int i = 0; i < FF_ARRAY_ELEMS(pipewiregrab_formats); i++) { + if (pw_ctx->enable_dmabuf) + params[n_params++] = build_format(pw_ctx, &pod_builder, pipewiregrab_formats[i], + pipewiregrab_default_modifiers, + FF_ARRAY_ELEMS(pipewiregrab_default_modifiers)); + params[n_params++] = build_format(pw_ctx, &pod_builder, pipewiregrab_formats[i], + NULL, 0); + } + + ret = pw_stream_connect( + pw_ctx->stream, PW_DIRECTION_INPUT, (uint32_t)pw_ctx->pipewire_node, + PW_STREAM_FLAG_AUTOCONNECT | PW_STREAM_FLAG_MAP_BUFFERS, params, n_params); + if (ret != 0) { + av_log(ctx, AV_LOG_ERROR, "pw_stream_connect failed\n"); + pw_thread_loop_unlock(pw_ctx->thread_loop); + goto fail; + } + + av_log(ctx, AV_LOG_INFO, "Starting screen capture ...\n"); + pw_thread_loop_unlock(pw_ctx->thread_loop); + return 0; + +fail: + if (pw_ctx->core) { + pw_core_disconnect(pw_ctx->core); + pw_ctx->core = NULL; + } + if (pw_ctx->context) { + pw_context_destroy(pw_ctx->context); + pw_ctx->context = NULL; + } + if (pw_ctx->thread_loop) { + pw_thread_loop_destroy(pw_ctx->thread_loop); + pw_ctx->thread_loop = NULL; + } + + return ret; +} + +static void portal_open_pipewire_remote(AVFilterContext *ctx) +{ + int ret; + int fd; + sd_bus_message *reply = NULL; + sd_bus_error err = SD_BUS_ERROR_NULL; + PipewireGrabContext *pw_ctx = ctx->priv; + + const char *method_name = "OpenPipeWireRemote"; + ret = sd_bus_call_method(pw_ctx->connection, + DESTINATION, + OBJECT_PATH, + INTERFACE, + method_name, + &err, + &reply, + "oa{sv}", + pw_ctx->session_handle, + 0); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Call to DBus method '%s' failed: %s\n", + method_name, err.message); + sd_bus_error_free(&err); + portal_abort(ctx, ret, "Failed to open PipeWire remote"); + return; + } + + ret = sd_bus_message_read(reply, "h", &fd); + if (ret < 0) { + portal_abort(ctx, ret, "Failed to read file descriptor"); + return; + } else + av_log(ctx, AV_LOG_DEBUG, "PipeWire fd: %d\n", fd); + + pw_ctx->pipewire_fd = fd; + atomic_store(&pw_ctx->dbus_event_loop_running, 0); +} + +static void dbus_signal_data_free(struct DbusSignalData *dbus_signal_data) +{ + sd_bus_slot_unref(dbus_signal_data->slot); + av_free(dbus_signal_data); +} + +static int on_start_response_received_callback( + sd_bus_message *message, void *user_data, sd_bus_error *err) +{ + int ret; + uint32_t response; + uint32_t node; + struct DbusSignalData *dbus_signal_data = user_data; + AVFilterContext *ctx = dbus_signal_data->ctx; + PipewireGrabContext *pw_ctx = ctx->priv; + + dbus_signal_data_free(dbus_signal_data); + + ret = sd_bus_message_read(message, "u", &response); + if (ret < 0) { + portal_abort(ctx, ret, "Failed to read DBus response"); + return -1; + } + if (response != 0) { + portal_abort(ctx, AVERROR(EACCES), + "Failed to start screen cast, denied or cancelled by user"); + return -1; + } + + sd_bus_message_enter_container(message, SD_BUS_TYPE_ARRAY, "{sv}"); + sd_bus_message_enter_container(message, SD_BUS_TYPE_DICT_ENTRY, "sv"); + sd_bus_message_skip(message, "s"); + sd_bus_message_enter_container(message, SD_BUS_TYPE_VARIANT, "a(ua{sv})"); + sd_bus_message_enter_container(message, SD_BUS_TYPE_ARRAY, "(ua{sv})"); + sd_bus_message_enter_container(message, SD_BUS_TYPE_STRUCT, "ua{sv}"); + + ret = sd_bus_message_read(message, "u", &node); + if (ret < 0) { + portal_abort(ctx, ret, "Failed to read PipeWire node: %s"); + return -1; + } + pw_ctx->pipewire_node = node; + + av_log(ctx, AV_LOG_DEBUG, "PipeWire node: %"PRIu64"\n", pw_ctx->pipewire_node); + av_log(ctx, AV_LOG_INFO, "Monitor selected, setting up screen cast\n\n"); + + portal_open_pipewire_remote(ctx); + return 0; +} + +static void portal_start(AVFilterContext *ctx) +{ + int ret; + sd_bus_error err = SD_BUS_ERROR_NULL; + PipewireGrabContext *pw_ctx = ctx->priv; + + const char *method_name = "Start"; + const char *request_token = "pipewiregrabStart"; + + ret = subscribe_to_signal(ctx, pw_ctx->sender_name, request_token, + on_start_response_received_callback); + if (ret < 0) { + portal_abort(ctx, ret, "Failed to subscribe to DBus signal"); + return; + } + + av_log(ctx, AV_LOG_INFO, "Asking for monitor…\n"); + ret = sd_bus_call_method(pw_ctx->connection, + DESTINATION, + OBJECT_PATH, + INTERFACE, + method_name, + &err, + NULL, + "osa{sv}", + pw_ctx->session_handle, + "", + 1, + "handle_token", "s", request_token); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Call to DBus method '%s' failed: %s\n", + method_name, err.message); + sd_bus_error_free(&err); + portal_abort(ctx, ret, "Failed to start screen cast session"); + } +} + +static int on_select_sources_response_received_callback( + sd_bus_message *message, void *user_data, sd_bus_error *err) +{ + int ret; + uint32_t response; + struct DbusSignalData *dbus_signal_data = user_data; + AVFilterContext *ctx = dbus_signal_data->ctx; + + dbus_signal_data_free(dbus_signal_data); + + ret = sd_bus_message_read(message, "u", &response); + if (ret < 0) { + portal_abort(ctx, ret, "Failed to read DBus response"); + return -1; + } + if (response != 0) { + portal_abort(ctx, AVERROR(EACCES), + "Failed to select screen cast sources"); + return -1; + } + + portal_start(ctx); + return 0; +} + +static void portal_select_sources(AVFilterContext *ctx) +{ + int ret; + uint32_t cursor_mode; + sd_bus_error err = SD_BUS_ERROR_NULL; + PipewireGrabContext *pw_ctx = ctx->priv; + + const char *method_name = "SelectSources"; + const char *request_token = "pipewiregrabSelectSources"; + + ret = subscribe_to_signal(ctx, pw_ctx->sender_name, request_token, + on_select_sources_response_received_callback); + if (ret < 0) { + portal_abort(ctx, ret, "Failed to subscribe to DBus signal"); + return; + } + + if ((pw_ctx->available_cursor_modes & PORTAL_CURSOR_MODE_EMBEDDED) + && pw_ctx->draw_mouse) + cursor_mode = PORTAL_CURSOR_MODE_EMBEDDED; + else + cursor_mode = PORTAL_CURSOR_MODE_HIDDEN; + + ret = sd_bus_call_method(pw_ctx->connection, + DESTINATION, + OBJECT_PATH, + INTERFACE, + method_name, + &err, + NULL, + "oa{sv}", + pw_ctx->session_handle, + 4, + "types", "u", pw_ctx->capture_type, + "multiple", "b", 0, + "handle_token", "s", request_token, + "cursor_mode", "u", cursor_mode); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Call to DBus method '%s' failed: %s\n", + method_name, err.message); + sd_bus_error_free(&err); + portal_abort(ctx, ret, "Failed to select sources for screen cast session"); + } +} + +static int on_create_session_response_received_callback( + sd_bus_message *message, void *user_data, sd_bus_error *err) +{ + int ret; + uint32_t response; + const char *session_handle; + const char *type; + struct DbusSignalData *dbus_signal_data = user_data; + AVFilterContext *ctx = dbus_signal_data->ctx; + PipewireGrabContext *pw_ctx = ctx->priv; + + dbus_signal_data_free(dbus_signal_data); + + ret = sd_bus_message_read(message, "u", &response); + if (ret < 0) { + portal_abort(ctx, ret, "Failed to read DBus response"); + return -1; + } + if (response != 0) { + portal_abort(ctx, AVERROR(EACCES), + "Failed to create screen cast session"); + return -1; + } + + sd_bus_message_enter_container(message, SD_BUS_TYPE_ARRAY, "{sv}"); + sd_bus_message_enter_container(message, SD_BUS_TYPE_DICT_ENTRY, "sv"); + sd_bus_message_skip(message, "s"); + // The XDG Desktop Portal documentation says that the type of `session_handle` + // is "o" (object path), but at least on some systems it's actually "s" (string), + // so we need to check to make sure we're using the right one. + sd_bus_message_peek_type(message, NULL, &type); + ret = sd_bus_message_read(message, "v", type, &session_handle); + if (ret < 0) { + portal_abort(ctx, ret, "Failed to read session handle"); + return -1; + } + pw_ctx->session_handle = av_strdup(session_handle); + + portal_select_sources(ctx); + return 0; +} + +/** + * Function to create a screen cast session + * + * @param ctx + */ +static void portal_create_session(AVFilterContext *ctx) +{ + int ret; + sd_bus_error err = SD_BUS_ERROR_NULL; + PipewireGrabContext *pw_ctx = ctx->priv; + + const char *method_name = "CreateSession"; + const char *request_token = "pipewiregrabCreateSession"; + + ret = subscribe_to_signal(ctx, pw_ctx->sender_name, request_token, + on_create_session_response_received_callback); + if (ret < 0) { + portal_abort(ctx, ret, "Failed to subscribe to DBus signal"); + return; + } + + ret = sd_bus_call_method(pw_ctx->connection, + DESTINATION, + OBJECT_PATH, + INTERFACE, + method_name, + &err, + NULL, + "a{sv}", + 2, + "handle_token", "s", request_token, + "session_handle_token", "s", "pipewiregrab"); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Call to DBus method '%s' failed: %s\n", + method_name, err.message); + sd_bus_error_free(&err); + portal_abort(ctx, ret, "Failed to create screen cast session"); + } +} + +/** + * Helper function: get available cursor modes and update the + * PipewireGrabContext accordingly + * + * @param ctx + */ +static int portal_update_available_cursor_modes(AVFilterContext *ctx) +{ + int ret; + sd_bus_error err = SD_BUS_ERROR_NULL; + PipewireGrabContext *pw_ctx = ctx->priv; + + ret = sd_bus_get_property_trivial(pw_ctx->connection, + DESTINATION, + OBJECT_PATH, + INTERFACE, + "AvailableCursorModes", + &err, + 'u', + &pw_ctx->available_cursor_modes); + if (ret < 0) + av_log(ctx, AV_LOG_ERROR, + "Couldn't retrieve available cursor modes: %s\n", err.message); + + sd_bus_error_free(&err); + return ret; +} + +static int create_dbus_connection(AVFilterContext *ctx) +{ + const char *aux; + int ret; + PipewireGrabContext *pw_ctx = ctx->priv; + + ret = sd_bus_open_user(&pw_ctx->connection); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Failed to create DBus connection: %s\n", strerror(-ret)); + return ret; + } + + ret = sd_bus_get_unique_name(pw_ctx->connection, &aux); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Failed to get bus name: %s\n", strerror(-ret)); + return ret; + } + // From https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.Request.html: + // "SENDER is the caller's unique name, with the initial ':' removed and all '.' replaced by '_'" + pw_ctx->sender_name = av_strireplace(aux + 1, ".", "_"); + av_log(ctx, AV_LOG_DEBUG, + "DBus connection created (sender name: %s)\n", pw_ctx->sender_name); + return 0; +} + + +/** + * Use XDG Desktop Portal's ScreenCast interface to open a file descriptor that + * can be used by PipeWire to access the screen cast streams. + * (https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.ScreenCast.html) + * + * @param ctx + */ +static int portal_init_screencast(AVFilterContext *ctx) +{ + int ret; + PipewireGrabContext *pw_ctx = ctx->priv; + + ret = create_dbus_connection(ctx); + if (ret < 0) + return ret; + + ret = portal_update_available_cursor_modes(ctx); + if (ret < 0) + return ret; + + portal_create_session(ctx); + if (pw_ctx->portal_error) + return pw_ctx->portal_error; + + // The event loop will run until it's stopped by portal_open_pipewire_remote (if + // all DBus method calls completed successfully) or portal_abort (in case of error). + // In the latter case, pw_ctx->portal_error gets set to a negative value. + atomic_store(&pw_ctx->dbus_event_loop_running, 1); + while(atomic_load(&pw_ctx->dbus_event_loop_running)) { + ret = sd_bus_process(pw_ctx->connection, NULL); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Failed to process DBus event: %s\n", strerror(-ret)); + return ret; + } + + ret = sd_bus_wait(pw_ctx->connection, 2000); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Error while waiting on bus: %s\n", strerror(-ret)); + return ret; + } + } + return pw_ctx->portal_error; +} + +static av_cold int pipewiregrab_init(AVFilterContext *ctx) +{ + int ret; + PipewireGrabContext *pw_ctx = ctx->priv; + if (!pw_ctx) { + av_log(ctx, AV_LOG_ERROR, + "Invalid private context data\n"); + return AVERROR(EINVAL); + } + + atomic_init(&pw_ctx->dbus_event_loop_running, 0); + atomic_init(&pw_ctx->pipewire_initialization_over, 0); + pthread_cond_init(&pw_ctx->pipewire_initialization_cond_var, NULL); + pthread_mutex_init(&pw_ctx->pipewire_initialization_mutex, NULL); + pthread_mutex_init(&pw_ctx->current_frame_mutex, NULL); + + if (pw_ctx->pipewire_fd == 0) { + ret = portal_init_screencast(ctx); + if (ret != 0) { + av_log(ctx, AV_LOG_ERROR, "Couldn't init screen cast\n"); + return ret; + } + } + + ret = play_pipewire_stream(ctx); + if (ret != 0) + return ret; + + // Wait until PipeWire initialization is over + pthread_mutex_lock(&pw_ctx->pipewire_initialization_mutex); + while (!atomic_load(&pw_ctx->pipewire_initialization_over)) { + pthread_cond_wait(&pw_ctx->pipewire_initialization_cond_var, + &pw_ctx->pipewire_initialization_mutex); + } + pthread_mutex_unlock(&pw_ctx->pipewire_initialization_mutex); + + return pw_ctx->pipewire_error; +} + +static void pipewiregrab_uninit(AVFilterContext *ctx) +{ + int ret; + PipewireGrabContext *pw_ctx = ctx->priv; + if (!pw_ctx) + return; + + // PipeWire cleanup + if (pw_ctx->thread_loop) { + pw_thread_loop_signal(pw_ctx->thread_loop, false); + pw_thread_loop_unlock(pw_ctx->thread_loop); + pw_thread_loop_stop(pw_ctx->thread_loop); + } + if (pw_ctx->stream) { + pw_stream_disconnect(pw_ctx->stream); + pw_stream_destroy(pw_ctx->stream); + pw_ctx->stream = NULL; + } + if (pw_ctx->core){ + pw_core_disconnect(pw_ctx->core); + pw_ctx->core = NULL; + } + if (pw_ctx->context) { + pw_context_destroy(pw_ctx->context); + pw_ctx->context = NULL; + } + if (pw_ctx->thread_loop) { + pw_thread_loop_destroy(pw_ctx->thread_loop); + pw_ctx->thread_loop = NULL; + } + if (pw_ctx->pw_init_called) { + pw_deinit(); + pw_ctx->pw_init_called = 0; + } + if (pw_ctx->pipewire_fd > 0) { + close(pw_ctx->pipewire_fd); + pw_ctx->pipewire_fd = 0; + } + av_frame_free(&pw_ctx->current_frame); + av_buffer_unref(&pw_ctx->hw_frames_ref); + av_buffer_unref(&pw_ctx->hw_device_ref); + + // DBus cleanup + if (pw_ctx->session_handle) { + ret = sd_bus_call_method(pw_ctx->connection, + DESTINATION, + pw_ctx->session_handle, + "org.freedesktop.portal.Session", + "Close", + NULL, NULL, NULL); + if (ret < 0) + av_log(ctx, AV_LOG_DEBUG, + "Failed to close portal session: %s\n", strerror(-ret)); + + av_freep(&pw_ctx->session_handle); + } + sd_bus_flush_close_unref(pw_ctx->connection); + av_freep(&pw_ctx->sender_name); +} + +static int pipewiregrab_config_props(AVFilterLink *outlink) +{ + AVFrame *frame; + PipewireGrabContext *pw_ctx = outlink->src->priv; + + AVRational time_base = av_inv_q(pw_ctx->framerate); + pw_ctx->frame_duration = av_rescale_q(1, time_base, AV_TIME_BASE_Q); + pw_ctx->time_frame = av_gettime_relative(); + + outlink->w = pw_ctx->width; + outlink->h = pw_ctx->height; + outlink->time_base = AV_TIME_BASE_Q; + outlink->frame_rate = pw_ctx->framerate; + + frame = ff_get_video_buffer(outlink, pw_ctx->width, pw_ctx->height); + if (!frame) + return AVERROR(ENOMEM); + pthread_mutex_lock(&pw_ctx->current_frame_mutex); + pw_ctx->current_frame = frame; + pthread_mutex_unlock(&pw_ctx->current_frame_mutex); + + return 0; +} + +static int pipewiregrab_request_frame(AVFilterLink *outlink) +{ + int ret; + int64_t curtime, delay; + PipewireGrabContext *pw_ctx = outlink->src->priv; + AVFrame *frame = av_frame_alloc(); + if (!frame) + return AVERROR(ENOMEM); + + pw_ctx->time_frame += pw_ctx->frame_duration; + while (1) { + curtime = av_gettime_relative(); + delay = pw_ctx->time_frame - curtime; + if (delay <= 0) + break; + av_usleep(delay); + } + + pthread_mutex_lock(&pw_ctx->current_frame_mutex); + ret = av_frame_ref(frame, pw_ctx->current_frame); + pthread_mutex_unlock(&pw_ctx->current_frame_mutex); + if (ret < 0) { + av_frame_free(&frame); + return ret; + } + + frame->pts = av_gettime(); + frame->duration = pw_ctx->frame_duration; + frame->sample_aspect_ratio = (AVRational) {1, 1}; + + return ff_filter_frame(outlink, frame); +} + +static int pipewiregrab_query_formats(AVFilterContext *ctx) +{ + PipewireGrabContext *pw_ctx = ctx->priv; + enum AVPixelFormat pix_fmts[] = {pw_ctx->av_pxl_format, AV_PIX_FMT_NONE}; + + return ff_set_common_formats_from_list(ctx, pix_fmts); +} + +static const AVFilterPad pipewiregrab_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .request_frame = pipewiregrab_request_frame, + .config_props = pipewiregrab_config_props, + }, +}; + +const AVFilter ff_vsrc_pipewiregrab= { + .name = "pipewiregrab", + .description = NULL_IF_CONFIG_SMALL("Capture screen or window using PipeWire."), + .priv_size = sizeof(struct PipewireGrabContext), + .priv_class = &pipewiregrab_class, + .init = pipewiregrab_init, + .uninit = pipewiregrab_uninit, + .inputs = NULL, + FILTER_OUTPUTS(pipewiregrab_outputs), + FILTER_QUERY_FUNC(pipewiregrab_query_formats), +};