From patchwork Thu Apr 26 14:49:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jerome Borsboom X-Patchwork-Id: 8658 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:155:0:0:0:0:0 with SMTP id c82-v6csp2143701jad; Thu, 26 Apr 2018 07:49:15 -0700 (PDT) X-Google-Smtp-Source: AIpwx48zULZGrW/WdS3787mMaWosAFyS1/YCwP4LJ39dI3xzt6cK6tScrGAOIzcWWOzg6WODD9tG X-Received: by 10.28.156.195 with SMTP id f186mr17717433wme.87.1524754155360; Thu, 26 Apr 2018 07:49:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524754155; cv=none; d=google.com; s=arc-20160816; b=FQzPL648UwokN9R2eOGSEheDW0wO/ojbd+5Kj7wropeRj3KGDDC+tdgWCuDMEPaD7j eDXSi3t4d+S4CHi2ey2R4mMtTenRFcsSRX45bBAZzuvBFI/PVanFwGplmMlkNwMEVVLf 1+iVvmAGxJXrxX/iCnbCK0nFF15f9pwXIgUbTB35coIqGsaLgVjnYpY9WOJxauAgc6SI JAinVw6xml9b7GC4mpoCmj8TNd+4nxsoBbceHrhV/G/cPanryOkATeeHbkPPNJj+WfmC tZa5AQfdS59jv+UdHVkFfBDkjJ8HOKkhLCqF/O4nJOVuueZ8Lk8EwJMFTBwShVpVSL3X Np6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:content-language:mime-version:user-agent:date :message-id:from:to:dkim-signature:delivered-to :arc-authentication-results; bh=Pb+8oAE98d6HXqdoI9Zzj8zrbr1OX/V98ihYyNUDwe8=; b=NvfeiPpLOGZLuPSpdIpMjlO01rFW5Z6dn0qCOb7Ic9buxN972LZmuECnTtWoV6ypal 7WItAGC6s0Rmx2DMXr37knJbNqoLkBwkL3HLAX6XrEHb/q+msfwFsb3gS/gtDPYIaO94 FKFj50cTAd/8NIsngbaGMv1C9fOwmwgaKYUCxH7Dz/N7FOsHLITADhQN8t2O0ixRrveP Ye2uwCJhwQdUFJCStsSBm9fvLx/0J04D2UfldO0mnXY8Wznb2QGc8zTgSsMWcXw6/XWS UWH/ofSCC3ezwluvIbtrS6nM/eCO121fuWXnqorPA9N2hmMH42jlvY/xa9a2kvYhZo3C YXWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@carpalis.nl header.s=default header.b=pNQ+pDgy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s101-v6si10738146wrc.164.2018.04.26.07.49.14; Thu, 26 Apr 2018 07:49:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@carpalis.nl header.s=default header.b=pNQ+pDgy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5BBE968A4F1; Thu, 26 Apr 2018 17:48:42 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from kyoto.xs4all.nl (kyoto.xs4all.nl [83.161.153.34]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C96E768A4D5 for ; Thu, 26 Apr 2018 17:48:35 +0300 (EEST) Received: from [10.1.48.247] ([87.215.30.74]) (authenticated bits=0) by kyoto.xs4all.nl (8.14.7/8.14.7) with ESMTP id w3QEn4Ip012102 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 26 Apr 2018 16:49:05 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=carpalis.nl; s=default; t=1524754145; bh=6bY8VTiufGuIsrmfGMoRNPpcd+rdGu8hg4ho2++Z/CY=; h=To:From:Subject:Date; b=pNQ+pDgys971ynH3A59TIiuLunH96avaBpcTSnUZYbkIXzevwyHM+gYh11MxfPv04 woVbXPXn9m8aryftKboieMVTNc8QmMcHpLOGg9goqvvaGQRietydFc2xszzgysRsf6 uCr4zDoHN98rY859/uvlz/R7WKN4FX4EMUAPHXOHFZpIr+3t/rRTNBbEtCWN14pA22 Xx3td0XTf+xWTGmddIatkAVX8MJsHn3V+woM7bbioRvIphB26SOKH112VtCRAXEXP3 rhLyVcO53SSiCldA/wizmhOEtb9B7j3VJvfb/3cPXgsw/E5d2LQSENlsmpMDCTQPxY jTfXsajTw5nvA== To: ffmpeg-devel@ffmpeg.org From: Jerome Borsboom Message-ID: <5d293b4f-0ded-096e-b1df-d20e2932e2dd@carpalis.nl> Date: Thu, 26 Apr 2018 16:49:04 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 Content-Language: nl Subject: [FFmpeg-devel] [PATCH] avcodec/vc1: fix out of bounds access of overlap filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Overlap filtering of the first row and first column must be guarded for out of bounds access of v->over_flags_plane. Signed-off-by: Jerome Borsboom --- libavcodec/vc1_loopfilter.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libavcodec/vc1_loopfilter.c b/libavcodec/vc1_loopfilter.c index bab28a649f..4c0de7c025 100644 --- a/libavcodec/vc1_loopfilter.c +++ b/libavcodec/vc1_loopfilter.c @@ -110,19 +110,19 @@ void ff_vc1_i_overlap_filter(VC1Context *v) * we run the put_pixels loop, i.e. delayed by one row and one column. */ for (i = 0; i < block_count; i++) if (v->pq >= 9 || v->condover == CONDOVER_ALL || - (v->over_flags_plane[mb_pos] && ((i & 5) == 1 || v->over_flags_plane[mb_pos - 1]))) + (v->over_flags_plane[mb_pos] && ((i & 5) == 1 || (s->mb_x && v->over_flags_plane[mb_pos - 1])))) vc1_h_overlap_filter(v, s->mb_x ? left_blk : cur_blk, cur_blk, i); if (v->fcm != ILACE_FRAME) for (i = 0; i < block_count; i++) { if (s->mb_x && (v->pq >= 9 || v->condover == CONDOVER_ALL || (v->over_flags_plane[mb_pos - 1] && - ((i & 2) || v->over_flags_plane[mb_pos - 1 - s->mb_stride])))) + ((i & 2) || (!s->first_slice_line && v->over_flags_plane[mb_pos - 1 - s->mb_stride]))))) vc1_v_overlap_filter(v, s->first_slice_line ? left_blk : topleft_blk, left_blk, i); if (s->mb_x == s->mb_width - 1) if (v->pq >= 9 || v->condover == CONDOVER_ALL || (v->over_flags_plane[mb_pos] && - ((i & 2) || v->over_flags_plane[mb_pos - s->mb_stride]))) + ((i & 2) || (!s->first_slice_line && v->over_flags_plane[mb_pos - s->mb_stride])))) vc1_v_overlap_filter(v, s->first_slice_line ? cur_blk : top_blk, cur_blk, i); } }