From patchwork Thu Oct 13 20:25:56 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Cadhalpun X-Patchwork-Id: 994 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.140.66 with SMTP id o63csp336219vsd; Thu, 13 Oct 2016 13:26:07 -0700 (PDT) X-Received: by 10.194.16.161 with SMTP id h1mr8127336wjd.164.1476390367853; Thu, 13 Oct 2016 13:26:07 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f3si19877493wjt.126.2016.10.13.13.26.07; Thu, 13 Oct 2016 13:26:07 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@googlemail.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE dis=NONE) header.from=googlemail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 963B568988E; Thu, 13 Oct 2016 23:26:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf0-f66.google.com (mail-lf0-f66.google.com [209.85.215.66]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 55D1B687ED3 for ; Thu, 13 Oct 2016 23:25:57 +0300 (EEST) Received: by mail-lf0-f66.google.com with SMTP id l131so11609148lfl.0 for ; Thu, 13 Oct 2016 13:25:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=from:to:subject:message-id:date:user-agent:mime-version :content-transfer-encoding; bh=XY3DL6x05SAjwk/EqtVX3pl5EwNUykIrxqlCTNyMWNE=; b=VwUBCUKw77trXcSBjckmHnJzMNeWOq9HnbemjM+ThydOLiEwU5RVhtXYX7lIZ16OW1 KlKg88Jpx1mffcG2lUtAyBrXfiSYeqVpTsW+eSyhAL4zr7thmVyjxaU/Uh8sg5rXY6kc 76U7TPjNUwIJZlRfXHGR5XnJhNxzCV9oOhfPM7aDP5hBO4mSjm+Zf6ZpqAHbxYr0DIh4 3vu9IXLMP6+iHJCXWaBq/GZimwg5UT5G64xs/TvwpnUcDmv357m8KVaF1PGZM5LQcMj8 WP4kbDc2mV+sPDGoo1V1lYduUvt/hcOGC6MSrlkUaOqdKfA8QDAYyNU2Rr+J/3ekygzu 0UkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:message-id:date:user-agent :mime-version:content-transfer-encoding; bh=XY3DL6x05SAjwk/EqtVX3pl5EwNUykIrxqlCTNyMWNE=; b=MjfvaOjKnXlqH2rxp2j4+QEUlOylOs6oBqJua0OO90dT5gBYDs6uhsagb56DbqlnQr TjY58YfMnFC8h/Fs9qHu/vXA3TBVbnajyC5Mjy4ntLT9IrR1YNTCgfbSgpm2f0AZnIRo tlFXut/srVf72aSgmPUU3dobALZHZarrJoZll7Js5sMFZQ1W5Bk1VH5VJpa5aUO9NI4D Cr81PWzt5elVw2yu1YmXbxgDJyyUt+uyDZi0mQ4l1ZgRWNfjwXDWeo6PnqRVeg+e191C OL0jJN5sTSdUlRSFLNvrp7+rk0seCDVC/hOeY3367zaVLbM/OatDEncK3VfgeIQeZxlU woqg== X-Gm-Message-State: AA6/9Rkjfat3CCe7R6b0aeMqEap95mChtLvErpxfN23IM/cLDx5TyC7ouWV0GaEVTwdUOw== X-Received: by 10.28.66.68 with SMTP id p65mr3139623wma.126.1476390357298; Thu, 13 Oct 2016 13:25:57 -0700 (PDT) Received: from [192.168.2.21] (p5B09552C.dip0.t-ipconnect.de. [91.9.85.44]) by smtp.googlemail.com with ESMTPSA id y2sm25343129wji.42.2016.10.13.13.25.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Oct 2016 13:25:56 -0700 (PDT) From: Andreas Cadhalpun X-Google-Original-From: Andreas Cadhalpun To: FFmpeg development discussions and patches Message-ID: <99563f2a-b72a-398d-b9ee-b2f9004bc52d@googlemail.com> Date: Thu, 13 Oct 2016 22:25:56 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.4.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] libopenjpegenc: fix out-of-bounds reads when filling the edges X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" If x is 0, 'x - 1' is in the previous line, or worse outside the buffer for the first line. If y is 0, 'x - image->comps[compno].w' is outside the buffer. Finally, image->comps[compno].w is unsigned (at least in openjpeg2), so the calculation could silently wrap around without the explicit cast to int. Signed-off-by: Andreas Cadhalpun --- libavcodec/libopenjpegenc.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/libavcodec/libopenjpegenc.c b/libavcodec/libopenjpegenc.c index 857ee1a..83c965d 100644 --- a/libavcodec/libopenjpegenc.c +++ b/libavcodec/libopenjpegenc.c @@ -415,13 +415,13 @@ static int libopenjpeg_copy_packed8(AVCodecContext *avctx, const AVFrame *frame, frame_index += numcomps; } for (; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - 1]; + image_line[x] = x > 0 ? image_line[x - 1] : 0; } } for (; y < image->comps[compno].h; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; for (x = 0; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - image->comps[compno].w]; + image_line[x] = y > 0 ? image_line[x - (int)image->comps[compno].w] : 0; } } } @@ -455,13 +455,13 @@ static int libopenjpeg_copy_packed12(AVCodecContext *avctx, const AVFrame *frame frame_index += numcomps; } for (; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - 1]; + image_line[x] = x > 0 ? image_line[x - 1] : 0; } } for (; y < image->comps[compno].h; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; for (x = 0; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - image->comps[compno].w]; + image_line[x] = y > 0 ? image_line[x - (int)image->comps[compno].w] : 0; } } } @@ -495,13 +495,13 @@ static int libopenjpeg_copy_packed16(AVCodecContext *avctx, const AVFrame *frame frame_index += numcomps; } for (; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - 1]; + image_line[x] = x > 0 ? image_line[x - 1] : 0; } } for (; y < image->comps[compno].h; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; for (x = 0; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - image->comps[compno].w]; + image_line[x] = y > 0 ? image_line[x - (int)image->comps[compno].w] : 0; } } } @@ -536,13 +536,13 @@ static int libopenjpeg_copy_unpacked8(AVCodecContext *avctx, const AVFrame *fram for (x = 0; x < width; ++x) image_line[x] = frame->data[compno][frame_index++]; for (; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - 1]; + image_line[x] = x > 0 ? image_line[x - 1] : 0; } } for (; y < image->comps[compno].h; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; for (x = 0; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - image->comps[compno].w]; + image_line[x] = y > 0 ? image_line[x - (int)image->comps[compno].w] : 0; } } } @@ -579,13 +579,13 @@ static int libopenjpeg_copy_unpacked16(AVCodecContext *avctx, const AVFrame *fra for (x = 0; x < width; ++x) image_line[x] = frame_ptr[frame_index++]; for (; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - 1]; + image_line[x] = x > 0 ? image_line[x - 1] : 0; } } for (; y < image->comps[compno].h; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; for (x = 0; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - image->comps[compno].w]; + image_line[x] = y > 0 ? image_line[x - (int)image->comps[compno].w] : 0; } } }