diff mbox

[FFmpeg-devel] vdpau: Fix buffer overflow with old hwaccel_context API

Message ID 20170908190813.26742-1-spaz16@wp.pl
State New
Headers show

Commit Message

Błażej Szczygieł Sept. 8, 2017, 7:08 p.m. UTC 
VDPAUHWContext struct is used internally, so allocate required amount
of memory. Also move hwctx->reset as it was prior to
7e4ba776a2240d40124d5540ea6b2118fa2fe26a to make sure that buffer
overflow doesn't happen if application allocates only
sizeof(AVVDPAUContext) bytes of memory.
---
 libavcodec/vdpau.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

Comments

wm4 Sept. 22, 2017, 7:39 a.m. UTC | #1 
On Fri,  8 Sep 2017 21:08:13 +0200
Błażej Szczygieł <spaz16@wp.pl> wrote:

> VDPAUHWContext struct is used internally, so allocate required amount
> of memory. Also move hwctx->reset as it was prior to
> 7e4ba776a2240d40124d5540ea6b2118fa2fe26a to make sure that buffer
> overflow doesn't happen if application allocates only
> sizeof(AVVDPAUContext) bytes of memory.
> ---
>  libavcodec/vdpau.c | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/libavcodec/vdpau.c b/libavcodec/vdpau.c
> index 9c7804a287..8d1748347f 100644
> --- a/libavcodec/vdpau.c
> +++ b/libavcodec/vdpau.c
> @@ -142,14 +142,13 @@ int ff_vdpau_common_init(AVCodecContext *avctx, VdpDecoderProfile profile,
>          return AVERROR(ENOSYS);
>  
>      if (hwctx) {
> -        hwctx->reset            = 0;
> -
>          if (hwctx->context.decoder != VDP_INVALID_HANDLE) {
>              vdctx->decoder = hwctx->context.decoder;
>              vdctx->render  = hwctx->context.render;
>              vdctx->device  = VDP_INVALID_HANDLE;
>              return 0; /* Decoder created by user */
>          }
> +        hwctx->reset            = 0;
>  
>          vdctx->device           = hwctx->device;
>          vdctx->get_proc_address = hwctx->get_proc_address;
> @@ -797,7 +796,7 @@ do {                                       \
>  
>  AVVDPAUContext *av_vdpau_alloc_context(void)
>  {
> -    return av_mallocz(sizeof(AVVDPAUContext));
> +    return av_mallocz(sizeof(VDPAUHWContext));
>  }
>  
>  int av_vdpau_bind_context(AVCodecContext *avctx, VdpDevice device,

I realize now this is with the old-old API. av_vdpau_alloc_context() is
really not supposed to be used. You're supposed to call
av_vdpau_bind_context(), which allocates the context for you. So I
don't think applying this patch makes much sense or helps anyone. We
should just deprecate all the old crappy API instead.
diff mbox

Patch

diff --git a/libavcodec/vdpau.c b/libavcodec/vdpau.c
index 9c7804a287..8d1748347f 100644
--- a/libavcodec/vdpau.c
+++ b/libavcodec/vdpau.c
@@ -142,14 +142,13 @@  int ff_vdpau_common_init(AVCodecContext *avctx, VdpDecoderProfile profile,
         return AVERROR(ENOSYS);
 
     if (hwctx) {
-        hwctx->reset            = 0;
-
         if (hwctx->context.decoder != VDP_INVALID_HANDLE) {
             vdctx->decoder = hwctx->context.decoder;
             vdctx->render  = hwctx->context.render;
             vdctx->device  = VDP_INVALID_HANDLE;
             return 0; /* Decoder created by user */
         }
+        hwctx->reset            = 0;
 
         vdctx->device           = hwctx->device;
         vdctx->get_proc_address = hwctx->get_proc_address;
@@ -797,7 +796,7 @@  do {                                       \
 
 AVVDPAUContext *av_vdpau_alloc_context(void)
 {
-    return av_mallocz(sizeof(AVVDPAUContext));
+    return av_mallocz(sizeof(VDPAUHWContext));
 }
 
 int av_vdpau_bind_context(AVCodecContext *avctx, VdpDevice device,