From patchwork Wed Jan 3 23:22:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Humberto Ribeiro X-Patchwork-Id: 7111 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.79.195 with SMTP id r64csp16475484jad; Wed, 3 Jan 2018 15:22:56 -0800 (PST) X-Google-Smtp-Source: ACJfBou0Pm4k7MoST5O26rc9WLJKcSZ/zQ69Tm/8k3REmyM3cVSGP/RQKFVOXg6Cq4iLx+iWC6ET X-Received: by 10.223.142.176 with SMTP id q45mr3198008wrb.6.1515021776724; Wed, 03 Jan 2018 15:22:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515021776; cv=none; d=google.com; s=arc-20160816; b=hNC7h9+rnfjlcuwXmI+sPBVVLYepKW6vv5r6giKXDs84SXEaNRX+IT9Q/F5OZZyDUC c+8gavaPhi8zEd3WYyvE7wx7ZCJLhhrE5PzFwC9zVY+wGbeAzvDcqyqJoYssH+6LQiUI GtDStqiI4gn2Gxzql+A8Cczhdn2GT+Mq9KgPpxgWQJII/bgW27/uhRbC60CO3PgQX8tL LbxubNfb+wd0xDIGpsb/4OezwPziEMwhEyI0SAOCMFjPeTmWM9HMAPulwk5kZUEubulX fCkRaVn72AA5TjQAVbTNfS01qj7vX7rfP5FKHkYNGAJSpLWiynbw9UgamU7DGXgwi63K slsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=kMRUFeeeHznwGbY+BgMxgLIsNvOzRO8p4oZBzoWJjlg=; b=fDcOqAwoh8iTJmugI4D7YGnmqJf11Ab6GFxfofvGignuu5C/l718sLqVkg7D2Ri0zN 4zaBRVAhAci7UyuxxDihK216viWnpDH62++PKYsYK3XWXOIt9eXy1k58l04Qr9MhB2wI R6ByStt3+7FBwby8GyNEBesUlPd8MlTT4umynWSUYS+Uo/qLaR1uYX3PG9WJs34r7OPJ aAA3n5xyQhX/xTPCTFv+oHvun3r/0Kvt8KbviFP76DUwLZi9pje/fHj2PZuzFcaOidPh VBBDruFC1c7gDE2ZEyDc97AnR4bTmYnAWkLMWYkIDGt3K79YqMLH0nw9ndvELhmT59My fACw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=i4+gSnba; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j142si1494352wmf.33.2018.01.03.15.22.55; Wed, 03 Jan 2018 15:22:56 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=i4+gSnba; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9225568839D; Thu, 4 Jan 2018 01:22:37 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl0-f52.google.com (mail-pl0-f52.google.com [209.85.160.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1019668064D for ; Thu, 4 Jan 2018 01:22:31 +0200 (EET) Received: by mail-pl0-f52.google.com with SMTP id o2so2131650plk.12 for ; Wed, 03 Jan 2018 15:22:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=sY01UEi6OL7QkyjKww2//rPKBhaoFkVs7+aXjOR5dYs=; b=i4+gSnba1Ygptx5y7DRJRPcEeLpOkUqF8R24ilE8mQxOzVYtxE6QdYE8rRlJsQuMbB LAa2X/ABWaG/xaGqPck30r5nHEkBgyfJ2tVvHwBna9WD1Jztzyo2d/Q/s+fLM4ApWHrV CnNTYNToDQorjuzoPzx0nfGKsZMcW4OJ0XGXT0ZdgjeT2PZbVqqB2s1dECpD+NSwxHQ0 YDs8dyyUcP4OylY6U6lxncDmfIiSTWkfXQudM7oC6eaCk8AJA1I54+U9qJMjnE0iDYQH /CCjJF8WwcM7ySkEud6zKyoAtb9OODFTwZNv254E2vc7j+RSxUyBvXlyYa64o9xVcuqf axSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=sY01UEi6OL7QkyjKww2//rPKBhaoFkVs7+aXjOR5dYs=; b=Xmku7+DL3rJ9WBb2FIj8Z/oQKe6U1hxSuwlUs63xVmV8jQTCXagdN8hPAh+5J/Yqs6 Ss/SM2KRRuv4L9R+T2ddDcJiUNPddhu9RIldLYDA1p5XE8LiOWBNCDnKZHg3Y6sEWQrC 4s8fmiJh4Q/C4qKxIqv1aayWoHRvkveilHwUtAx0iBvfl7P0D7XqDiQDPIkrIz22N4e8 EY9d83rnGuVprAkZ2clXCAxSfnZjs7dwVnx0n3NMe+HwaSOwku2idLU1pv3eZxcxzl9b 2NH6uk8AfGPYlwPbKmtXrOIDSTiJF3IovCT/3Ij1HtwMus2RTSeKZ/bwa8en/02lsGuu SqVA== X-Gm-Message-State: AKGB3mLgWDzZxXGcd537q8eCxKbXOeJJA3Rutd9cjcg96TNV2mdelOSX /UAVYlw730EWlXuDMv+uBoX8lg== X-Received: by 10.159.214.148 with SMTP id n20mr2760198plp.101.1515021765872; Wed, 03 Jan 2018 15:22:45 -0800 (PST) Received: from hribeiro-LinuxHostVM.york.lan ([64.114.31.58]) by smtp.gmail.com with ESMTPSA id w10sm4005943pgc.28.2018.01.03.15.22.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 15:22:45 -0800 (PST) From: Humberto Ribeiro To: ffmpeg-devel@ffmpeg.org Date: Wed, 3 Jan 2018 15:22:15 -0800 Message-Id: <1515021735-11532-1-git-send-email-mont3z.claro5@gmail.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH] libavutil/hwcontext_dxva2: Add check for possible errors from GetAdapterDisplayModeEx X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Humberto Ribeiro MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This prevents a possible crash in CreateDeviceEx when using faulty response from GetAdapterDisplayModeEx and allows ffmpeg to fallback to classic d3d9. --- libavutil/hwcontext_dxva2.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c index 2ddd4be..44ebdbc 100644 --- a/libavutil/hwcontext_dxva2.c +++ b/libavutil/hwcontext_dxva2.c @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter) if (FAILED(hr)) return AVERROR_UNKNOWN; - IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); + hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); + if (FAILED(hr)) { + IDirect3D9Ex_Release(d3d9ex); + return AVERROR_UNKNOWN; + } d3dpp.BackBufferFormat = modeex.Format;