From patchwork Fri Oct 14 00:00:49 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Cadhalpun X-Patchwork-Id: 998 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.140.66 with SMTP id o63csp20682vsd; Thu, 13 Oct 2016 17:01:04 -0700 (PDT) X-Received: by 10.28.224.139 with SMTP id x133mr3679191wmg.6.1476403264102; Thu, 13 Oct 2016 17:01:04 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cw8si15125801wjb.50.2016.10.13.17.01.00; Thu, 13 Oct 2016 17:01:04 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@googlemail.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE dis=NONE) header.from=googlemail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 68C4F68993A; Fri, 14 Oct 2016 03:00:57 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf0-f42.google.com (mail-lf0-f42.google.com [209.85.215.42]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 80C5B689210 for ; Fri, 14 Oct 2016 03:00:51 +0300 (EEST) Received: by mail-lf0-f42.google.com with SMTP id x79so167500756lff.0 for ; Thu, 13 Oct 2016 17:00:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=from:subject:to:references:message-id:date:user-agent:mime-version :in-reply-to; bh=lqpuO62tIOVA7uCYmdMvu5FtUr5WJ7gVW0Ou4xKMt3g=; b=U4QpYLzkqATSrcTQ/BqZUDMRpBAqLRAjuINYU3z0NlL0pp1IvgtNFX1ArMGl5PRQGs KUd6D9yBGd/EfjAh2sE3rvP+13VU5CgCtJxpG7/Mfp5vIXhdOK9nj+TqF9TmhLO2rlv8 Eo7sndnnhf5EPkVz1hXS+vPeUWwEMgacAM7UFgye7k1EndP8232abq3rYWcui0EfLy4S ioG/eLBurH/7sTFyGwWgAuaf+bRK6Y72VMnSRpF1tzkLiIPfoJ25fMMJ/iRCR6I7zhhm T7m24fvD8KMUN2oGDKB8FPsZg0KxJW51VRmscvj2NAmEqZgjmomKdF7GDp4ol2d6KzjU ZjEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:subject:to:references:message-id:date :user-agent:mime-version:in-reply-to; bh=lqpuO62tIOVA7uCYmdMvu5FtUr5WJ7gVW0Ou4xKMt3g=; b=elJT6dTp3UBJWKvq6KJjZfeINXhyyggUGo5fy4M4A11/S6V/y73QJgS5zpj4/u0hdG 7V47YkxkjBj/lYpkxs8Y8xcl2+2kEMKFQKTycvBKfbTi8HoDJ7kSt+c8G7vGtmGKMqwX 03U1JU4JeAI7lmevU8kYAaDmBpU20yi1jLiz3KbkCTLcWRS9fBJtvXKuOpd9GJ2U+NBj tf4JjGOtP8A+hlLYiPQOSps6xxq6oL/31SdVPFy0hiykADTWEgaqpg6KiIhpMirVVStA qdAoPrEpdKFsoQZptckasI0ZziKtZrE4gHe224MaBQipCMrzrJDHu2cBNercSuU882+u j8iw== X-Gm-Message-State: AA6/9Rm7hYFCt0KTcAkBOUhZueUCvNuTaXEQYZNrCkBlxFD1KgfCPVvAVJF25hB2kYnHUg== X-Received: by 10.28.111.8 with SMTP id k8mr3722640wmc.4.1476403251450; Thu, 13 Oct 2016 17:00:51 -0700 (PDT) Received: from [192.168.2.21] (p5B09552C.dip0.t-ipconnect.de. [91.9.85.44]) by smtp.googlemail.com with ESMTPSA id y2sm26710712wji.42.2016.10.13.17.00.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Oct 2016 17:00:50 -0700 (PDT) From: Andreas Cadhalpun X-Google-Original-From: Andreas Cadhalpun To: ffmpeg-devel@ffmpeg.org References: <99563f2a-b72a-398d-b9ee-b2f9004bc52d@googlemail.com> <20161013224938.GY4602@nb4> Message-ID: <91054da1-baa0-ecea-4ee1-cf1e58c0e328@googlemail.com> Date: Fri, 14 Oct 2016 02:00:49 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161013224938.GY4602@nb4> Subject: Re: [FFmpeg-devel] [PATCH] libopenjpegenc: fix out-of-bounds reads when filling the edges X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" On 14.10.2016 00:49, Michael Niedermayer wrote: > On Fri, Oct 14, 2016 at 12:23:02AM +0200, Andreas Cadhalpun wrote: >> The avctx->width/avctx->height is not zero, but libopenjpeg_copy_unpacked8 >> does: > >> width = avctx->width / image->comps[compno].dx; >> height = avctx->height / image->comps[compno].dy; > > this looks wrong to me > the code in mj2_create_image() looks better: > cmptparm[i].dx = sub_dx[i]; > cmptparm[i].dy = sub_dy[i]; > cmptparm[i].w = (avctx->width + sub_dx[i] - 1) / sub_dx[i]; > cmptparm[i].h = (avctx->height + sub_dy[i] - 1) / sub_dy[i]; Indeed this looks better, so I updated the patch (attached) to change the calculation of width/height. Best regards, Andreas From 1461064c1eaabb71661f9ff68b94f35a1b98e3b5 Mon Sep 17 00:00:00 2001 From: Andreas Cadhalpun Date: Thu, 13 Oct 2016 22:14:46 +0200 Subject: [PATCH] libopenjpegenc: fix out-of-bounds reads when filling the edges The calculation of width/height should round up, not round down to prevent setting width or height to 0. Also image->comps[compno].w is unsigned (at least in openjpeg2), so the calculation could silently wrap around without the explicit cast to int. Signed-off-by: Andreas Cadhalpun --- libavcodec/libopenjpegenc.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/libavcodec/libopenjpegenc.c b/libavcodec/libopenjpegenc.c index 857ee1a..1b7e168 100644 --- a/libavcodec/libopenjpegenc.c +++ b/libavcodec/libopenjpegenc.c @@ -421,7 +421,7 @@ static int libopenjpeg_copy_packed8(AVCodecContext *avctx, const AVFrame *frame, for (; y < image->comps[compno].h; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; for (x = 0; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - image->comps[compno].w]; + image_line[x] = image_line[x - (int)image->comps[compno].w]; } } } @@ -461,7 +461,7 @@ static int libopenjpeg_copy_packed12(AVCodecContext *avctx, const AVFrame *frame for (; y < image->comps[compno].h; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; for (x = 0; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - image->comps[compno].w]; + image_line[x] = image_line[x - (int)image->comps[compno].w]; } } } @@ -501,7 +501,7 @@ static int libopenjpeg_copy_packed16(AVCodecContext *avctx, const AVFrame *frame for (; y < image->comps[compno].h; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; for (x = 0; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - image->comps[compno].w]; + image_line[x] = image_line[x - (int)image->comps[compno].w]; } } } @@ -528,8 +528,8 @@ static int libopenjpeg_copy_unpacked8(AVCodecContext *avctx, const AVFrame *fram } for (compno = 0; compno < numcomps; ++compno) { - width = avctx->width / image->comps[compno].dx; - height = avctx->height / image->comps[compno].dy; + width = (avctx->width + image->comps[compno].dx - 1) / image->comps[compno].dx; + height = (avctx->height + image->comps[compno].dy - 1) / image->comps[compno].dy; for (y = 0; y < height; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; frame_index = y * frame->linesize[compno]; @@ -542,7 +542,7 @@ static int libopenjpeg_copy_unpacked8(AVCodecContext *avctx, const AVFrame *fram for (; y < image->comps[compno].h; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; for (x = 0; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - image->comps[compno].w]; + image_line[x] = image_line[x - (int)image->comps[compno].w]; } } } @@ -570,8 +570,8 @@ static int libopenjpeg_copy_unpacked16(AVCodecContext *avctx, const AVFrame *fra } for (compno = 0; compno < numcomps; ++compno) { - width = avctx->width / image->comps[compno].dx; - height = avctx->height / image->comps[compno].dy; + width = (avctx->width + image->comps[compno].dx - 1) / image->comps[compno].dx; + height = (avctx->height + image->comps[compno].dy - 1) / image->comps[compno].dy; frame_ptr = (uint16_t *)frame->data[compno]; for (y = 0; y < height; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; @@ -585,7 +585,7 @@ static int libopenjpeg_copy_unpacked16(AVCodecContext *avctx, const AVFrame *fra for (; y < image->comps[compno].h; ++y) { image_line = image->comps[compno].data + y * image->comps[compno].w; for (x = 0; x < image->comps[compno].w; ++x) { - image_line[x] = image_line[x - image->comps[compno].w]; + image_line[x] = image_line[x - (int)image->comps[compno].w]; } } } -- 2.9.3