From patchwork Mon Nov 13 20:47:15 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 6033 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.161.94 with SMTP id m30csp2536304jah; Mon, 13 Nov 2017 12:47:33 -0800 (PST) X-Google-Smtp-Source: AGs4zMYae37z1IgtkOaw/cUSlpdq01Y12FTk1ACG5xuuJNsU/tpJcyNnIm48W5xHw7y4nBO/G+di X-Received: by 10.223.199.133 with SMTP id l5mr2113051wrg.20.1510606053332; Mon, 13 Nov 2017 12:47:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510606053; cv=none; d=google.com; s=arc-20160816; b=fQfu0VdYF5Xxnt0bUr16enWW8SA4d2PmJ+mu4LfC6r5OcZpT6lfz63Me4vE4cFc9AH 8UceU4jHxG+Bk+TIuylpA/zVzTM9Gb/8DUsKcbrsYH4jdV+Z2eBAu66TcvYG5mltEKRQ B113PFhsTaI38fzI7T1U4VZkdPGtwZzN4I9JRp3UmE4obLnPcSVHdaAf396yf/V7Y7nC ay3aZpRPUIhf1yq5hwB0H/q6eZqzV/Xm4lRiVw/5PZxFvwJ/hP1cz/yW2Zgoe0HJ7yKj dUhloBHR6TWmy3buyZjEniLzwywjDoHto8hIJXQ6Jo4zofP/8RCkCeKc/nESyzHBBDHx M67g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:delivered-to :arc-authentication-results; bh=5Mita00YULmQlPVGutk1hphB1K+jR5aaZ7brslc1mpo=; b=qrRk4c7URwmEc47tC1RDuPXfjBO+P8KNvoFdswD232J3MmXcHMPgB5wt3GDsEv9YAS LVSenBCIvEXIOLLdajKkDM8/TU+VBDIJMALXsbBM8T14baTUnvtkzUFPK/W0E3m2So5b 1WNLXaA4Q+moKpPNSa6Arte8bpe+o+pMWww9I3ABj04O38iGhhmAMO4Qnr4TexyQr9bI 9eNXit0Tferxu++P5Qhi4EZaL1giqPxLvhywpQn2UylHUw2YV7YdOtI8LiznVYnKe1WL 3/8cywjOeL26nC5qmsYXTd6gAlVIZF8uBq4S50ahK9Vj3543NUvezHnPnnrmZJ2PtMv0 oxWQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z56si13950684wrz.517.2017.11.13.12.47.32; Mon, 13 Nov 2017 12:47:33 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A879568A605; Mon, 13 Nov 2017 22:47:16 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe05-1.mx.upcmail.net (vie01a-dmta-pe05-1.mx.upcmail.net [84.116.36.11]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id DECD268A5FA for ; Mon, 13 Nov 2017 22:47:10 +0200 (EET) Received: from [172.31.216.43] (helo=vie01a-pemc-psmtp-pe01) by vie01a-dmta-pe05.mx.upcmail.net with esmtp (Exim 4.88) (envelope-from ) id 1eELdd-0006yv-Bz for ffmpeg-devel@ffmpeg.org; Mon, 13 Nov 2017 21:47:25 +0100 Received: from localhost ([213.47.41.20]) by vie01a-pemc-psmtp-pe01 with SMTP @ mailcloud.upcmail.net id ZYnG1w00K0S5wYM01YnHSU; Mon, 13 Nov 2017 21:47:17 +0100 X-SourceIP: 213.47.41.20 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Mon, 13 Nov 2017 21:47:15 +0100 Message-Id: <20171113204715.25185-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.15.0 Subject: [FFmpeg-devel] [PATCH] avcodec/x86/mpegvideodsp: Fix signedness bug in need_emu X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes: out of array read Fixes: 3516/attachment-311488.dat Found-by: Insu Yun, Georgia Tech. Signed-off-by: Michael Niedermayer Tested-by: wuninsu@gmail.com --- libavcodec/x86/mpegvideodsp.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/libavcodec/x86/mpegvideodsp.c b/libavcodec/x86/mpegvideodsp.c index e0498f3849..6009b64e07 100644 --- a/libavcodec/x86/mpegvideodsp.c +++ b/libavcodec/x86/mpegvideodsp.c @@ -52,8 +52,9 @@ static void gmc_mmx(uint8_t *dst, uint8_t *src, const int dyh = (dyy - (1 << (16 + shift))) * (h - 1); const int dxh = dxy * (h - 1); const int dyw = dyx * (w - 1); - int need_emu = (unsigned) ix >= width - w || - (unsigned) iy >= height - h; + int need_emu = (unsigned) ix >= width - w || width < w || + (unsigned) iy >= height - h || height< h + ; if ( // non-constant fullpel offset (3% of blocks) ((ox ^ (ox + dxw)) | (ox ^ (ox + dxh)) | (ox ^ (ox + dxw + dxh)) |