From patchwork Tue Jul 19 16:56:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov X-Patchwork-Id: 36846 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1649:b0:8b:613a:194d with SMTP id no9csp1878989pzb; Tue, 19 Jul 2022 09:57:11 -0700 (PDT) X-Google-Smtp-Source: AGRyM1vYPwF56FdZSbAEzruMZCoZXEv4Wl0gQRq3Tb7T1AnZaCU/8M2WXGtXHRIAd2N5nKBE3yJZ X-Received: by 2002:a05:6402:11c7:b0:43a:c61c:21cd with SMTP id j7-20020a05640211c700b0043ac61c21cdmr44833297edw.108.1658249831139; Tue, 19 Jul 2022 09:57:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1658249831; cv=none; d=google.com; s=arc-20160816; b=yPYjIaAkZEranY8Y/lDiDK7YMpuoFmlyMoNXjPd8+kYY5wzsRhCqrsWK9LM3ap4Pqb 0CjeSkcjrv/B1xUZ/yQUBWJAO3hHS+DFm1lO+Rd10X5LPxM5885+QW0hcIvyz1Phr96X zm5mE4YBpdwIwZ3RLP6+WGkbW/WTa8dNnrC+FmrDikXGH4pJEAlW7R48IBQv6Mv+Btys ORkVY+CP8zaQzRBkkI7hTrB7c/7PCS8EQ9opPbH+hAzNql73+DJA7al2pCskWeCGmTFn AktDqZ8yux8qxCBmBVlqbmhIhnbiq/z4fv824gWx2zK7DGB3bdEnY8eU80FgwMMmCiEO u1gA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=gZmEDcMCHy2NxZPiz/Mgi3+0eSBH69TTg4TZPbejbh4=; b=c8u/IttOoZ9Z11fmClj8XBRjkmxVo23sUDrCh1tRNp+jCdDsREDIWKhT3l066arXUA NX1nfQ5jZN8uDeGCxGGfp+h2/g1WxHkq8awn2RwPCBn04pLj4aKU5JCEqS/1bV6sdQ6Q 3Tt8TuMNidVJLDPM9bwnZCuYsecUP9cbNMv7Wuj3LQG+7vFL/hRym7KlWU3q1vCePD8o hgbDPUHvJgznvaX0laJ9owRA6rebrSgNZnSog1UOKy4MFDE/cKS3jk9sbp9qVK0qMcw5 PCg4cpT3lZjgyA2zB26omVdbno18npEms3iaRlv5Xzfbgjwl1ZFxCSv7+FyCFBSSwNt8 1FMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=cEK4Bhjn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e19-20020a50d4d3000000b0043a0e3729f1si16858614edj.493.2022.07.19.09.57.10; Tue, 19 Jul 2022 09:57:11 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=cEK4Bhjn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EE94068B6F3; Tue, 19 Jul 2022 19:56:59 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6C6FC68B64E for ; Tue, 19 Jul 2022 19:56:58 +0300 (EEST) Received: by mail-wr1-f49.google.com with SMTP id j1so18058254wrs.4 for ; Tue, 19 Jul 2022 09:56:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YITKel03Snormo2Fa/J70IbQzMTC/5cw4tARiiJyb8s=; b=cEK4BhjnRncQYLOx4sa7Ci1jT/oxt1rCFwNj0eFn4I1IDcOphZZ6FsQeg0ozMyV93l FnbVc/i7QtFHw+kj6HMG6m1cfkKUmnGi67m7XTyVp5nczsGi69dC7za9ORx2yGszJWAA 63sDukuZ4RC5qyUl6kXT/XnSnUWvggl7YAagtK03DgwkB4jpqe5Sz0Od9rnHmdAJCOeS eLFSd3ijGw3tEcDtjpKU+Il2WD706KklEPjzb0liyMfhP+qS0Wv+TKqkIf0SEUli1OIO OONslAwHHnSL4kKPzZH8LBVzb9fuQQhe2YfGv1U7Hkm3xO3VedzkFEaWibJXxm/SKmv8 LPrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YITKel03Snormo2Fa/J70IbQzMTC/5cw4tARiiJyb8s=; b=WxY41Btkf8YlGKvqOok0PlNQBNyr2w8bC9/pdE1GAtxzylqgA+q3Ls4TYl/yflljKB 4XN7unfgFBXM34xCIwWKzWQ8CtqUQtHUUCOdcGlhVTrBzktzu09b+HnFTj71689i1KOZ rJwkr5iEr76kK5uhc5OkPseJ2Rkf8rkTp2krp3IXhum259bV8d5WtWvjEFGRT1QqTawE Jag93k7jK75kswu3WRYoRA3TJoYaPYz9zozre2le6DyCc5+NZ+yx8Jiw8v8PWMCzSuk+ IJ9NddEM078wLbd0uSxYIx1CQQriuFcs8is2uWe9/vfTVZxYoHf6uNkdcoqFHDfz08lj NTIw== X-Gm-Message-State: AJIora9zi1J2pQYMfnODpfN+C9ygs7mq17d1BRu/Ok2CuhYNtQlM+Svw D5oOtepJbSGMURatX1BE4Uc/OdZVtd7taQ== X-Received: by 2002:adf:b613:0:b0:21d:6da6:d541 with SMTP id f19-20020adfb613000000b0021d6da6d541mr26938158wre.485.1658249817675; Tue, 19 Jul 2022 09:56:57 -0700 (PDT) Received: from localhost ([2a00:5f00:102:0:ed02:7072:7dbf:2431]) by smtp.gmail.com with UTF8SMTPSA id h4-20020a1c2104000000b00397402ae674sm23169629wmh.11.2022.07.19.09.56.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 19 Jul 2022 09:56:57 -0700 (PDT) From: Emil Velikov To: ffmpeg-devel@ffmpeg.org Date: Tue, 19 Jul 2022 17:56:54 +0100 Message-Id: <20220719165656.640869-2-emil.l.velikov@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: <20220719165656.640869-1-emil.l.velikov@gmail.com> References: <20220719165656.640869-1-emil.l.velikov@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/3] hwcontext_vaapi: do not link against libva-x11.so X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Mark Thompson , emil.l.velikov@gmail.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: WiaQu/1ENH// From: Emil Velikov There is an internal ABI between libva.so the libva-XXX.so libraries. With a recent change, the internal va_fool API was removed breaking the ABI. So if libva.so and libva-x11.so are from different version, the whole stack will crash. Instead we can dlopen() the libva-x11 library and gracefully error out. Cc: Mark Thompson Signed-off-by: Emil Velikov --- Greetings team, please consider backporting this for the stable releases. I've noticed that we've got plenty of pre-existing memory and state leaks in vaapi_device_create(). Would it make sense to fix those up? If so I can follow-up with a patch. --- configure | 2 +- libavutil/hwcontext_vaapi.c | 34 +++++++++++++++++++++++++++++++++- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/configure b/configure index 18d9b61a99..ab2a29544c 100755 --- a/configure +++ b/configure @@ -3816,7 +3816,7 @@ swscale_suggest="libm stdatomic" avcodec_extralibs="pthreads_extralibs iconv_extralibs dxva2_extralibs" avfilter_extralibs="pthreads_extralibs" -avutil_extralibs="d3d11va_extralibs nanosleep_extralibs pthreads_extralibs vaapi_drm_extralibs vaapi_x11_extralibs vdpau_x11_extralibs" +avutil_extralibs="d3d11va_extralibs nanosleep_extralibs pthreads_extralibs vaapi_drm_extralibs vdpau_x11_extralibs" # programs ffmpeg_deps="avcodec avfilter avformat" diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c index c3a98bc4b1..e44d324928 100644 --- a/libavutil/hwcontext_vaapi.c +++ b/libavutil/hwcontext_vaapi.c @@ -18,8 +18,16 @@ #include "config.h" +#if CONFIG_VAAPI_1 +# define VA_ABI ".2" +#else +# define VA_ABI ".1" +#endif + #if HAVE_VAAPI_X11 # include +# include +# define VA_X11_LIB "libva-x11.so" VA_ABI #endif #if HAVE_VAAPI_DRM # include @@ -54,6 +62,7 @@ typedef struct VAAPIDevicePriv { #if HAVE_VAAPI_X11 + void *libva_x11; Display *x11_display; #endif @@ -1565,6 +1574,8 @@ static void vaapi_device_free(AVHWDeviceContext *ctx) vaTerminate(hwctx->display); #if HAVE_VAAPI_X11 + if (priv->libva_x11) + dlclose(priv->libva_x11); if (priv->x11_display) XCloseDisplay(priv->x11_display); #endif @@ -1723,14 +1734,35 @@ static int vaapi_device_create(AVHWDeviceContext *ctx, const char *device, #if HAVE_VAAPI_X11 if (!display && try_x11) { + VADisplay (*GetDisplay)(Display *dpy); + + priv->libva_x11 = dlopen(VA_X11_LIB, RTLD_NOW | RTLD_LOCAL); + if (!priv->libva_x11) { + av_log(ctx, AV_LOG_ERROR, "Cannot open %s library %s.\n", + VA_X11_LIB, dlerror()); + return AVERROR_UNKNOWN; + } + + GetDisplay = dlsym(priv->libva_x11, "vaGetDisplay"); + if (!GetDisplay) { + av_log(ctx, AV_LOG_ERROR, "Cannot retrieve %s entrypoint %s.\n", + "vaGetDisplay", dlerror()); + // Always dlclose after the dlerror(). The former can alter the + // error string returned by the latter. + dlclose(priv->libva_x11); + return AVERROR_UNKNOWN; + } + // Try to open the device as an X11 display. priv->x11_display = XOpenDisplay(device); if (!priv->x11_display) { + dlclose(priv->libva_x11); av_log(ctx, AV_LOG_VERBOSE, "Cannot open X11 display " "%s.\n", XDisplayName(device)); } else { - display = vaGetDisplay(priv->x11_display); + display = GetDisplay(priv->x11_display); if (!display) { + dlclose(priv->libva_x11); av_log(ctx, AV_LOG_ERROR, "Cannot open a VA display " "from X11 display %s.\n", XDisplayName(device)); return AVERROR_UNKNOWN;