From patchwork Fri Sep 8 19:08:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?QsWCYcW8ZWogU3pjenlnaWXFgg==?= X-Patchwork-Id: 5061 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.36.26 with SMTP id f26csp878150jaa; Fri, 8 Sep 2017 20:50:12 -0700 (PDT) X-Google-Smtp-Source: ADKCNb6Ubucspy8LrmS7iRGRUi2/cOVsrraNwanYDTUtgyfGRtTETBxDGmuAgew6nXrlwHkyGj67 X-Received: by 10.223.131.65 with SMTP id 59mr3480319wrd.137.1504929012004; Fri, 08 Sep 2017 20:50:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504929011; cv=none; d=google.com; s=arc-20160816; b=vb9qewtT81zXB3gQoFukEqA5rHjIDTeLY0AYa0rwKNQSRBc/BVnDpWCLAfP0qnHY3G o5C4vkdWnKJdTcZnm83mx2B7V1fkdWuaFLjxvmWiIX6Gwak/tAIq21M9yrc6DwOh1p5e iCGeMnULRixuQLrg+Lq4UvT6OuXMfcCEg3FOlfZnCyjy+VpZGEZTqjtijvnfMF/dZdeK buAfTV09GpKm1SRpAspc71ZWwed5MQNg4WRXLeTBSomE5C/ILj8ZOFrXJ9gLxtYOt9HF 6cdzikrudUpGCz0nqBXPqtri/YFZ7VSN9kHdn8t2xnr3CLDrWABqcKHZAnKb5dW7XFKl 478g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=u4T24me+fJs55xWvGTFYkiLtJY8GJ+3cO5zflY4+oj0=; b=AhvHvd/+Vrj3QpGv6uJjXz/sBOcUdCP1PfSjcDDznTY1ia8GPxlemQ/j16CnD0hsth cr5pT28Zq4h1IXPJm8HqLRRf5Fw9SSXc4QA54AYP4Uj5+xfOYmBOuc1nOrlOivI6SVMB D6EmD7xDO7RFe24G9NU15X+/GIxNEuMlyXdDlpiMiwjNKriau4yiMwPKxPLpv12dXSO7 v1ZxRJDyTiqErtePNdxjertCD95tbqGAJVDakEnYvUnhHBgNcupLVFJfxBp9QRtWzEly EQPGGNqAM+uZgC6G/nbj2tlcKiMXsJA31bwFpnxOl7L9pP5Wx5Pc99RQJmF2tVn+yGOb ck0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@wp.pl header.s=1024a header.b=Jz6ds7b9; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 69si2733006wrk.554.2017.09.08.20.50.09; Fri, 08 Sep 2017 20:50:11 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@wp.pl header.s=1024a header.b=Jz6ds7b9; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 78DC7689D7D; Sat, 9 Sep 2017 06:50:01 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mx3.wp.pl (mx3.wp.pl [212.77.101.10]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AA7116806AF for ; Fri, 8 Sep 2017 22:08:17 +0300 (EEST) Received: (wp-smtpd smtp.wp.pl 25605 invoked from network); 8 Sep 2017 21:08:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wp.pl; s=1024a; t=1504897700; bh=TgmQnq5p0V/jUSenVjwMVs07vnYx+OStkz4BahvTKHM=; h=From:To:Cc:Subject; b=Jz6ds7b98n6Hq/8h9/xyKBCQmm9Pt+SPfi7Vr0AYreofXVnEA/dKQdxKyc1ueQe3j qsmmsg2FkaHSiHc34AFhbzsRk4FWkJq14WAiHI+kkkwkdvcn6TV81r3fABuDYGqITM wilSbowZUcUzIKJfQm7pLuU3e+tmDQ72zOec31Vw= Received: from unknown (HELO localhost.localdomain) (spaz16@wp.pl@[89.25.249.201]) (envelope-sender ) by smtp.wp.pl (WP-SMTPD) with ECDHE-RSA-AES256-GCM-SHA384 encrypted SMTP for ; 8 Sep 2017 21:08:20 +0200 From: =?UTF-8?q?B=C5=82a=C5=BCej=20Szczygie=C5=82?= To: ffmpeg-devel@ffmpeg.org Date: Fri, 8 Sep 2017 21:08:13 +0200 Message-Id: <20170908190813.26742-1-spaz16@wp.pl> X-Mailer: git-send-email 2.14.1 X-WP-DKIM-Status: good (id: wp.pl) X-WP-MailID: 83fa31a4573d50d61198de4482ae04df X-WP-AV: skaner antywirusowy Poczty Wirtualnej Polski X-WP-SPAM: NO 000000B [wWNE] X-Mailman-Approved-At: Sat, 09 Sep 2017 06:50:00 +0300 Subject: [FFmpeg-devel] [PATCH] vdpau: Fix buffer overflow with old hwaccel_context API X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: =?UTF-8?q?B=C5=82a=C5=BCej=20Szczygie=C5=82?= MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" VDPAUHWContext struct is used internally, so allocate required amount of memory. Also move hwctx->reset as it was prior to 7e4ba776a2240d40124d5540ea6b2118fa2fe26a to make sure that buffer overflow doesn't happen if application allocates only sizeof(AVVDPAUContext) bytes of memory. --- libavcodec/vdpau.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/libavcodec/vdpau.c b/libavcodec/vdpau.c index 9c7804a287..8d1748347f 100644 --- a/libavcodec/vdpau.c +++ b/libavcodec/vdpau.c @@ -142,14 +142,13 @@ int ff_vdpau_common_init(AVCodecContext *avctx, VdpDecoderProfile profile, return AVERROR(ENOSYS); if (hwctx) { - hwctx->reset = 0; - if (hwctx->context.decoder != VDP_INVALID_HANDLE) { vdctx->decoder = hwctx->context.decoder; vdctx->render = hwctx->context.render; vdctx->device = VDP_INVALID_HANDLE; return 0; /* Decoder created by user */ } + hwctx->reset = 0; vdctx->device = hwctx->device; vdctx->get_proc_address = hwctx->get_proc_address; @@ -797,7 +796,7 @@ do { \ AVVDPAUContext *av_vdpau_alloc_context(void) { - return av_mallocz(sizeof(AVVDPAUContext)); + return av_mallocz(sizeof(VDPAUHWContext)); } int av_vdpau_bind_context(AVCodecContext *avctx, VdpDevice device,