| Message ID | 1515016774-11084-1-git-send-email-mont3z.claro5@gmail.com |
|---|---|
| State | Accepted |
| Commit | 59b126f92225316e0cd77bb952d630553801dc85 |
| Headers | show |
On Wed, 3 Jan 2018 13:59:34 -0800 Humberto Ribeiro <mont3z.claro5@gmail.com> wrote: > --- > libavutil/hwcontext_dxva2.c | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c > index 2ddd4be..44ebdbc 100644 > --- a/libavutil/hwcontext_dxva2.c > +++ b/libavutil/hwcontext_dxva2.c > @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter) > if (FAILED(hr)) > return AVERROR_UNKNOWN; > > - IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); > + hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); > + if (FAILED(hr)) { > + IDirect3D9Ex_Release(d3d9ex); > + return AVERROR_UNKNOWN; > + } > > d3dpp.BackBufferFormat = modeex.Format; > LGTM, but when does it happen? Why would creation of IDirect3D9Ex succeed, but this method fail? The strangest thing being that CreateDeviceEx apparently fails, even though modeex is zero-initialized and doesn't contain any pointers. Anyway, please format the commit message according to project standards, and add anything you might know about this to the commit message.
> LGTM, but when does it happen? Why would creation of IDirect3D9Ex > succeed, but this method fail? The strangest thing being that > CreateDeviceEx apparently fails, even though modeex is > zero-initialized and doesn't contain any pointers. > > Anyway, please format the commit message according to project > standards, and add anything you might know about this to the commit > message. > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel Thx for your reply. Our software uses an old Directx SDK (June 2010) and it works fine with ffmpeg 3.1.4. However it was crashing with the new ffmpeg 3.4. I've noticed that there was a crash with a memory violation when executing IDirect3D9Ex_CreateDeviceEx. Unhandled exception at 0x000007FEF10974A3 (d3d9.dll) in crash_dump.dmp: 0xC0000005: Access violation reading location 0x0000000000000001. This crash prevent ffmpeg from falling back to the classic format. Actually an error was being issued by IDirect3D9Ex_GetAdapterDisplayModeEx but this was never caught. After this patch, ffmpeg is falling back to the classic format and no crashes take place. I'll update the message and resubmit the patch.
On Wed, 3 Jan 2018 14:34:59 -0800 Mont3z Claros <mont3z.claro5@gmail.com> wrote: > > LGTM, but when does it happen? Why would creation of IDirect3D9Ex > > succeed, but this method fail? The strangest thing being that > > CreateDeviceEx apparently fails, even though modeex is > > zero-initialized and doesn't contain any pointers. > > > > Anyway, please format the commit message according to project > > standards, and add anything you might know about this to the commit > > message. > > _______________________________________________ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > Thx for your reply. > Our software uses an old Directx SDK (June 2010) and it works fine > with ffmpeg 3.1.4. > However it was crashing with the new ffmpeg 3.4. I've noticed that > there was a crash with a memory violation when executing > IDirect3D9Ex_CreateDeviceEx. > > Unhandled exception at 0x000007FEF10974A3 (d3d9.dll) in > crash_dump.dmp: 0xC0000005: Access violation reading location > 0x0000000000000001. > > This crash prevent ffmpeg from falling back to the classic format. > Actually an error was being issued by > IDirect3D9Ex_GetAdapterDisplayModeEx but this was never caught. > After this patch, ffmpeg is falling back to the classic format and no > crashes take place. > > I'll update the message and resubmit the patch. Not really much useful information (it's true that D3D9Ex was added some time, probably in ffmpeg 3.4). Bit sure, if it's known to crash in CreateDeviceEx, that's seems good enough. (Probably a buggy driver.)
On Wed, Jan 3, 2018 at 11:18 PM, wm4 <nfxjfg@googlemail.com> wrote: > On Wed, 3 Jan 2018 13:59:34 -0800 > Humberto Ribeiro <mont3z.claro5@gmail.com> wrote: > >> --- >> libavutil/hwcontext_dxva2.c | 6 +++++- >> 1 file changed, 5 insertions(+), 1 deletion(-) >> >> diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c >> index 2ddd4be..44ebdbc 100644 >> --- a/libavutil/hwcontext_dxva2.c >> +++ b/libavutil/hwcontext_dxva2.c >> @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter) >> if (FAILED(hr)) >> return AVERROR_UNKNOWN; >> >> - IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); >> + hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); >> + if (FAILED(hr)) { >> + IDirect3D9Ex_Release(d3d9ex); >> + return AVERROR_UNKNOWN; >> + } >> >> d3dpp.BackBufferFormat = modeex.Format; >> > > LGTM, but when does it happen? Why would creation of IDirect3D9Ex > succeed, but this method fail? The strangest thing being that > CreateDeviceEx apparently fails, even though modeex is > zero-initialized and doesn't contain any pointers. > I noticed something about this some time ago - D3DDISPLAYMODEEX should not be entirely zero-initialized before this call, instead its "Size" member should be set to sizeof(D3DDISPLAYMODEEX) - see https://msdn.microsoft.com/en-us/library/windows/desktop/bb172549(v=vs.85).aspx Currently it always causes IDirect3D9Ex_GetAdapterDisplayModeEx to fail due to that (at least on my hardware), but it didn't seem to have any impact on the later create device call in my case. - Hendrik
On Thu, 4 Jan 2018 00:26:01 +0100 Hendrik Leppkes <h.leppkes@gmail.com> wrote: > On Wed, Jan 3, 2018 at 11:18 PM, wm4 <nfxjfg@googlemail.com> wrote: > > On Wed, 3 Jan 2018 13:59:34 -0800 > > Humberto Ribeiro <mont3z.claro5@gmail.com> wrote: > > > >> --- > >> libavutil/hwcontext_dxva2.c | 6 +++++- > >> 1 file changed, 5 insertions(+), 1 deletion(-) > >> > >> diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c > >> index 2ddd4be..44ebdbc 100644 > >> --- a/libavutil/hwcontext_dxva2.c > >> +++ b/libavutil/hwcontext_dxva2.c > >> @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter) > >> if (FAILED(hr)) > >> return AVERROR_UNKNOWN; > >> > >> - IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); > >> + hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); > >> + if (FAILED(hr)) { > >> + IDirect3D9Ex_Release(d3d9ex); > >> + return AVERROR_UNKNOWN; > >> + } > >> > >> d3dpp.BackBufferFormat = modeex.Format; > >> > > > > LGTM, but when does it happen? Why would creation of IDirect3D9Ex > > succeed, but this method fail? The strangest thing being that > > CreateDeviceEx apparently fails, even though modeex is > > zero-initialized and doesn't contain any pointers. > > > > I noticed something about this some time ago - D3DDISPLAYMODEEX should > not be entirely zero-initialized before this call, instead its "Size" > member should be set to sizeof(D3DDISPLAYMODEEX) - see > https://msdn.microsoft.com/en-us/library/windows/desktop/bb172549(v=vs.85).aspx > Currently it always causes IDirect3D9Ex_GetAdapterDisplayModeEx to > fail due to that (at least on my hardware), but it didn't seem to have > any impact on the later create device call in my case. Oh, so possibly entirely our/my fault.
Thx for this inside. Unfortunately my application still crashes even after setting Size in modeex. In my case, IDirect3D9Ex_GetAdapterDisplayModeEx always returns D3DERR_INVALIDCALL. Humberto On Wed, Jan 3, 2018 at 4:17 PM, wm4 <nfxjfg@googlemail.com> wrote: > On Thu, 4 Jan 2018 00:26:01 +0100 > Hendrik Leppkes <h.leppkes@gmail.com> wrote: > >> On Wed, Jan 3, 2018 at 11:18 PM, wm4 <nfxjfg@googlemail.com> wrote: >> > On Wed, 3 Jan 2018 13:59:34 -0800 >> > Humberto Ribeiro <mont3z.claro5@gmail.com> wrote: >> > >> >> --- >> >> libavutil/hwcontext_dxva2.c | 6 +++++- >> >> 1 file changed, 5 insertions(+), 1 deletion(-) >> >> >> >> diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c >> >> index 2ddd4be..44ebdbc 100644 >> >> --- a/libavutil/hwcontext_dxva2.c >> >> +++ b/libavutil/hwcontext_dxva2.c >> >> @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter) >> >> if (FAILED(hr)) >> >> return AVERROR_UNKNOWN; >> >> >> >> - IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); >> >> + hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); >> >> + if (FAILED(hr)) { >> >> + IDirect3D9Ex_Release(d3d9ex); >> >> + return AVERROR_UNKNOWN; >> >> + } >> >> >> >> d3dpp.BackBufferFormat = modeex.Format; >> >> >> > >> > LGTM, but when does it happen? Why would creation of IDirect3D9Ex >> > succeed, but this method fail? The strangest thing being that >> > CreateDeviceEx apparently fails, even though modeex is >> > zero-initialized and doesn't contain any pointers. >> > >> >> I noticed something about this some time ago - D3DDISPLAYMODEEX should >> not be entirely zero-initialized before this call, instead its "Size" >> member should be set to sizeof(D3DDISPLAYMODEEX) - see >> https://msdn.microsoft.com/en-us/library/windows/desktop/bb172549(v=vs.85).aspx >> Currently it always causes IDirect3D9Ex_GetAdapterDisplayModeEx to >> fail due to that (at least on my hardware), but it didn't seem to have >> any impact on the later create device call in my case. > > Oh, so possibly entirely our/my fault. > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On 1/3/2018 10:21 PM, Mont3z Claros wrote: > Thx for this inside. Unfortunately my application still crashes even > after setting Size in modeex. > In my case, IDirect3D9Ex_GetAdapterDisplayModeEx always returns > D3DERR_INVALIDCALL. > > Humberto It does solve IDirect3D9Ex_CreateDeviceEx() failing for me (Win10 x86_64, AMD GPU), so it should be applied alongside your patch in any case.
diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c index 2ddd4be..44ebdbc 100644 --- a/libavutil/hwcontext_dxva2.c +++ b/libavutil/hwcontext_dxva2.c @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter) if (FAILED(hr)) return AVERROR_UNKNOWN; - IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); + hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL); + if (FAILED(hr)) { + IDirect3D9Ex_Release(d3d9ex); + return AVERROR_UNKNOWN; + } d3dpp.BackBufferFormat = modeex.Format;