From patchwork Tue Jul 19 16:56:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov X-Patchwork-Id: 36843 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1649:b0:8b:613a:194d with SMTP id no9csp1878528pzb; Tue, 19 Jul 2022 09:56:17 -0700 (PDT) X-Google-Smtp-Source: AGRyM1uRIYuYSxVmAryZWUO7ojqCP7Q09a5veREVTqlbuipaLkq7rcY0PWJoL11dMhXYSS2HZ5+p X-Received: by 2002:a17:907:28ca:b0:72b:110a:b34e with SMTP id en10-20020a17090728ca00b0072b110ab34emr31591528ejc.113.1658249776815; Tue, 19 Jul 2022 09:56:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1658249776; cv=none; d=google.com; s=arc-20160816; b=dfQPvakrnVM0PgtMW4srof0m6E4MTunkS+UMRHErHLm4haRNh3PJqkbi208L4POmUe /EU3RxlEj2WiNr0/AbMBd0/mVtC5evnEhuJrxqtPL7n6spW/iCnOp8pQcp41Nnr25ezA m/gDpEt306K39NHJWauU1FdBzi8MGBNkznaLUpAemytLC8MrGCYzn9sBE9s6//k/+upb di10fYkRMrjwxxqwD0AbGwGdnjb4mzXJRvahtToc9HjtW8cL54Mf+HzJESioU1WaqZEe dKZQw85rYwU12UhkL6TKhFCHalpcL4/TiOX1PnJlutO894XFupRqVyHbIcAK155UvEPW xvAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=gZmEDcMCHy2NxZPiz/Mgi3+0eSBH69TTg4TZPbejbh4=; b=J4VJRgeyxQmpluTnXtbaFAQgg91JH1YuRGSLB0KZ3mRThjTsibq8iNtjmJJU5ov0iT kENhzbS/QG3Q2BNQ82s8en32hF5nOIti4Fk1ybRpFjgcz0uXEvO9Gc4hD2ZLuhsyM+a8 ozhrpcuQVgU+v2lbZty+XLyl/uhhRj7+d941R62XtTNhPdOqoZh6GJ6wHzw8iT6jmqXZ CxxNPJzxapYPxKOU36/Ji/yFcr79ibRjZxjYs+VOj9TvoBLjAp/fnGWazcn3bOLa5jOM HjGthX+OFNAFf22qXeq7VgTlmOycF7+vZHNZKXSrFravTnN5FZTh9/YGfZPCVzGpKJt8 gNeA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=MxFBMVo9; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l24-20020a170907915800b00711d89b830asi15942727ejs.819.2022.07.19.09.56.16; Tue, 19 Jul 2022 09:56:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=MxFBMVo9; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D561F68B44A; Tue, 19 Jul 2022 19:56:12 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7976E68A467 for ; Tue, 19 Jul 2022 19:56:06 +0300 (EEST) Received: by mail-wm1-f50.google.com with SMTP id p26-20020a1c545a000000b003a2fb7c1274so7788851wmi.1 for ; Tue, 19 Jul 2022 09:56:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=YITKel03Snormo2Fa/J70IbQzMTC/5cw4tARiiJyb8s=; b=MxFBMVo9O6np8214hCs66N0ah5CO9LpxDlHv8nv7zesWUa2bcLLmuuRLW+m+sNnsKL c2FqZK3bArD0y5mnoQ3iu3dlnpHGy1b1Nm0WO1MQF0wfmu0yR3T87piIZ49LfhC+I2It +SYBY0jxxTeZbEuXg7r6O6BE5AFku4HXMWiw8lbn2pc2uGWgWyt+JB9ERsAt7FCvy64w iUaZdyQH6fWqRyKjwFRfnHjxKXGUBAd0ZvtIWqoraGSYDAiSeCJs3C8ZtnDS3RXVR0SY MJIvV4GgceeTB8XASy/QNmg16CRVuyhPOXAwXMWNyWHYALx19vdOjeRFnDlK9/yI4VJ7 +MoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=YITKel03Snormo2Fa/J70IbQzMTC/5cw4tARiiJyb8s=; b=LL7aLmvX51/p0aPgk1e4G4b3bDGsgIqPF3Uegm/Ci7HtUoHymZbblAvL5YiCW/Mgcs Zb2MUJ0Cm5SC+OJfY7Oo31Gq5yFumUcFkg2vQ7CTSX9VbXxKb0s2oIJ6NNwYmttCEnAb KUj9C+QCJ3ocCO/8bqZEqwhQlYIxKECOh5uDG7Rj+SkD+cCmJgivpPgqj57+QgGlNkxR fiigs8FbAqoTAB7iGa3p9qZhKBysSUz45PjO+nKkkNlk9y+uLs4TNcUAf/98H6NvbVYu AggrM7BwXzvjYYnXOy5p7o35VHkJPl+3TpXPbdK8htfiiy2GOJXVc3M30r+AY301RUHk jfvQ== X-Gm-Message-State: AJIora8v7xhZWqSGH/VtbLlc+dOmIFWDnY5nb2mIB2A17BZrk80MOwmp WFeMVjV6FiKEN457g2cLExwioB5R7NyL6Q== X-Received: by 2002:a05:600c:3845:b0:3a3:227d:c1c6 with SMTP id s5-20020a05600c384500b003a3227dc1c6mr226619wmr.9.1658249765597; Tue, 19 Jul 2022 09:56:05 -0700 (PDT) Received: from localhost ([2a00:5f00:102:0:ed02:7072:7dbf:2431]) by smtp.gmail.com with UTF8SMTPSA id n11-20020a05600c294b00b003a325bd8517sm2389553wmd.5.2022.07.19.09.56.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 19 Jul 2022 09:56:05 -0700 (PDT) From: Emil Velikov To: ffmpeg-devel@ffmpeg.org Date: Tue, 19 Jul 2022 17:56:02 +0100 Message-Id: <20220719165604.640770-1-emil.l.velikov@gmail.com> X-Mailer: git-send-email 2.37.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/3] hwcontext_vaapi: do not link against libva-x11.so X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Mark Thompson , emil.l.velikov@gmail.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 6bgMyJNzQSpz From: Emil Velikov There is an internal ABI between libva.so the libva-XXX.so libraries. With a recent change, the internal va_fool API was removed breaking the ABI. So if libva.so and libva-x11.so are from different version, the whole stack will crash. Instead we can dlopen() the libva-x11 library and gracefully error out. Cc: Mark Thompson Signed-off-by: Emil Velikov --- Greetings team, please consider backporting this for the stable releases. I've noticed that we've got plenty of pre-existing memory and state leaks in vaapi_device_create(). Would it make sense to fix those up? If so I can follow-up with a patch. --- configure | 2 +- libavutil/hwcontext_vaapi.c | 34 +++++++++++++++++++++++++++++++++- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/configure b/configure index 18d9b61a99..ab2a29544c 100755 --- a/configure +++ b/configure @@ -3816,7 +3816,7 @@ swscale_suggest="libm stdatomic" avcodec_extralibs="pthreads_extralibs iconv_extralibs dxva2_extralibs" avfilter_extralibs="pthreads_extralibs" -avutil_extralibs="d3d11va_extralibs nanosleep_extralibs pthreads_extralibs vaapi_drm_extralibs vaapi_x11_extralibs vdpau_x11_extralibs" +avutil_extralibs="d3d11va_extralibs nanosleep_extralibs pthreads_extralibs vaapi_drm_extralibs vdpau_x11_extralibs" # programs ffmpeg_deps="avcodec avfilter avformat" diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c index c3a98bc4b1..e44d324928 100644 --- a/libavutil/hwcontext_vaapi.c +++ b/libavutil/hwcontext_vaapi.c @@ -18,8 +18,16 @@ #include "config.h" +#if CONFIG_VAAPI_1 +# define VA_ABI ".2" +#else +# define VA_ABI ".1" +#endif + #if HAVE_VAAPI_X11 # include +# include +# define VA_X11_LIB "libva-x11.so" VA_ABI #endif #if HAVE_VAAPI_DRM # include @@ -54,6 +62,7 @@ typedef struct VAAPIDevicePriv { #if HAVE_VAAPI_X11 + void *libva_x11; Display *x11_display; #endif @@ -1565,6 +1574,8 @@ static void vaapi_device_free(AVHWDeviceContext *ctx) vaTerminate(hwctx->display); #if HAVE_VAAPI_X11 + if (priv->libva_x11) + dlclose(priv->libva_x11); if (priv->x11_display) XCloseDisplay(priv->x11_display); #endif @@ -1723,14 +1734,35 @@ static int vaapi_device_create(AVHWDeviceContext *ctx, const char *device, #if HAVE_VAAPI_X11 if (!display && try_x11) { + VADisplay (*GetDisplay)(Display *dpy); + + priv->libva_x11 = dlopen(VA_X11_LIB, RTLD_NOW | RTLD_LOCAL); + if (!priv->libva_x11) { + av_log(ctx, AV_LOG_ERROR, "Cannot open %s library %s.\n", + VA_X11_LIB, dlerror()); + return AVERROR_UNKNOWN; + } + + GetDisplay = dlsym(priv->libva_x11, "vaGetDisplay"); + if (!GetDisplay) { + av_log(ctx, AV_LOG_ERROR, "Cannot retrieve %s entrypoint %s.\n", + "vaGetDisplay", dlerror()); + // Always dlclose after the dlerror(). The former can alter the + // error string returned by the latter. + dlclose(priv->libva_x11); + return AVERROR_UNKNOWN; + } + // Try to open the device as an X11 display. priv->x11_display = XOpenDisplay(device); if (!priv->x11_display) { + dlclose(priv->libva_x11); av_log(ctx, AV_LOG_VERBOSE, "Cannot open X11 display " "%s.\n", XDisplayName(device)); } else { - display = vaGetDisplay(priv->x11_display); + display = GetDisplay(priv->x11_display); if (!display) { + dlclose(priv->libva_x11); av_log(ctx, AV_LOG_ERROR, "Cannot open a VA display " "from X11 display %s.\n", XDisplayName(device)); return AVERROR_UNKNOWN;