From patchwork Tue May 16 12:04:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Levinson X-Patchwork-Id: 3678 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.10.2 with SMTP id 2csp424453vsk; Tue, 16 May 2017 05:04:49 -0700 (PDT) X-Received: by 10.223.133.35 with SMTP id 32mr7305022wrh.200.1494936289168; Tue, 16 May 2017 05:04:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1494936289; cv=none; d=google.com; s=arc-20160816; b=q96jNoPd+tMYUeVfnWy05znaWbWfVjwKTbofjM2WYUjkgMFCou3X06EngrCgK3IP0J XNGEQDMgpPDgX+KNiDhNaFLWbXxoQWVPlIc82BcGkGDIiIs2YWIqFS00/O4n3/XnMC89 7DNT22KU9D8iZB6b4myK2KJYOZpugAybh1E+V2QHvC3YNFe8OlnVZY/G9hVPlvZ1zTsO OjiH3yDywWVxunVOsGqQUdO5rY5t9yOuqBFiRjXk1rms+yma+eg/NnNSkodMpjmPoEuT +YrLjcScQGmC7P3SKANz6bOlP5PNs4IIic/mZBGBXyc91YtVk9RcB/TOjSlj+2EO3+sF dbdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:user-agent:date:message-id:from:to :delivered-to:arc-authentication-results; bh=U5OYdrVf6gv1Ol+slV4D7Mh9WYWWJyKKEa6Ztp6YJD8=; b=MnUAAYNWAZczdBKKIJ/IFnfQAFLlfTAaFKiR26pbBinvv/SHcPmSEoBgRvG3Xl15Cx azJLTRo1qLPMx66g7WbB+pMBCquux+8QXDCCyX7byhqUqcaofV5Q6XlJdzmIanbIR9Wq S+gowuubAbxJksc2fOI7JoVYB4q6ISl0dYh9HNxfVTV13JBLNoeASZH2oi9KC2GPt52C sZkenqC1jv9DfTmQMCIAjD1cAI9CNNngYKebLugicyQH6j6sMKl/R/y2myMS0jVaXCAI 1f3413iag/+ulxI3fgGIdkB9UYJ8NXIZRcd6RzIGMndNIseOZojOs8kuwyh9UEINfk1C 6VLw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id v10si2108016wma.121.2017.05.16.05.04.47; Tue, 16 May 2017 05:04:49 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A3889689834; Tue, 16 May 2017 15:04:45 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from white.spiritone.com (white.spiritone.com [216.99.193.38]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 897716883C0 for ; Tue, 16 May 2017 15:04:39 +0300 (EEST) Received: from [192.168.3.101] (184-100-204-251.ptld.qwest.net [184.100.204.251]) by white.spiritone.com (Postfix) with ESMTPSA id 0D588734070E for ; Tue, 16 May 2017 05:04:38 -0700 (PDT) To: FFmpeg development discussions and patches From: Aaron Levinson Message-ID: Date: Tue, 16 May 2017 05:04:36 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avutil/hwcontext_dxva2: Don't improperly free IDirect3DSurface9 objects X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Add dxva2_pool_release_dummy() and use it in call to av_buffer_create() in dxva2_pool_alloc(). Prior to this change, av_buffer_create() was called with NULL for the third argument, which indicates that av_buffer_default_free() should be used to free the buffer's data. Eventually, it gets to buffer_pool_free() and calls buf->free() on a surface object (which is av_buffer_default_free()). This can result in a crash when the debug version of the C-runtime is used on Windows. While it doesn't appear to result in a crash when the release version of the C-runtime is used on Windows, it likely results in memory corruption, since av_free() is being called on memory that was allocated using IDirectXVideoAccelerationService::CreateSurface(). Signed-off-by: Aaron Levinson --- libavutil/hwcontext_dxva2.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c index 4ed0d56..6c41788 100644 --- a/libavutil/hwcontext_dxva2.c +++ b/libavutil/hwcontext_dxva2.c @@ -121,6 +121,13 @@ static void dxva2_frames_uninit(AVHWFramesContext *ctx) } } +static void dxva2_pool_release_dummy(void *opaque, uint8_t *data) +{ + // important not to free anything here--data is a surface object + // associated with the call to CreateSurface(), and these surfaces are + // released in dxva2_frames_uninit() +} + static AVBufferRef *dxva2_pool_alloc(void *opaque, int size) { AVHWFramesContext *ctx = (AVHWFramesContext*)opaque; @@ -130,7 +137,7 @@ static AVBufferRef *dxva2_pool_alloc(void *opaque, int size) if (s->nb_surfaces_used < hwctx->nb_surfaces) { s->nb_surfaces_used++; return av_buffer_create((uint8_t*)s->surfaces_internal[s->nb_surfaces_used - 1], - sizeof(*hwctx->surfaces), NULL, 0, 0); + sizeof(*hwctx->surfaces), dxva2_pool_release_dummy, 0, 0); } return NULL;