diff mbox

[FFmpeg-devel] libavutil/hwcontext_dxva2: Add check for possible errors from GetAdapterDisplayModeEx

Message ID 20180104155053.2916320f@debian
State Accepted
Commit 18fbfd7bf86effb49c0341b37b9c1d73940d688e
Headers show

Commit Message

wm4 Jan. 4, 2018, 2:50 p.m. UTC 
On Wed,  3 Jan 2018 15:22:15 -0800
Humberto Ribeiro <mont3z.claro5@gmail.com> wrote:

> This prevents a possible crash in CreateDeviceEx when using faulty response from GetAdapterDisplayModeEx and allows ffmpeg to fallback to classic d3d9.
> ---
>  libavutil/hwcontext_dxva2.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c
> index 2ddd4be..44ebdbc 100644
> --- a/libavutil/hwcontext_dxva2.c
> +++ b/libavutil/hwcontext_dxva2.c
> @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter)
>      if (FAILED(hr))
>          return AVERROR_UNKNOWN;
>  
> -    IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
> +    hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
> +    if (FAILED(hr)) {
> +        IDirect3D9Ex_Release(d3d9ex);
> +        return AVERROR_UNKNOWN;
> +    }
>  
>      d3dpp.BackBufferFormat = modeex.Format;
>  

I'll apply this and this trivial fix in an hour or so (after testing
it):
diff mbox

Patch

--- a/libavutil/hwcontext_dxva2.c
+++ b/libavutil/hwcontext_dxva2.c
@@ -485,6 +485,7 @@  static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter)
     if (FAILED(hr))
         return AVERROR_UNKNOWN;
 
+    modeex.Size = sizeof(D3DDISPLAYMODEEX);
     hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
     if (FAILED(hr)) {
         IDirect3D9Ex_Release(d3d9ex);