From patchwork Tue Jul 14 16:43:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gautam Ramakrishnan X-Patchwork-Id: 21011 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 4832E44AD8E for ; Tue, 14 Jul 2020 19:43:44 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 30C0668B255; Tue, 14 Jul 2020 19:43:44 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BEAF568A383 for ; Tue, 14 Jul 2020 19:43:37 +0300 (EEST) Received: by mail-pl1-f194.google.com with SMTP id x9so7263278plr.2 for ; Tue, 14 Jul 2020 09:43:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=W0vkLVpbGC8iplakE5HggLPIkZErAF5DCoPm30qoMME=; b=ZkOupz1pcN0JLf+4PGSUSC+Q5K30/QIDnAfSFIvXJOBhhdsHoF+diZeaQkNYJom5b+ pm3gKKMx7gOWNxuOh5aodw4VuBE0bkImmTg2GL5HqSy+XPtl0h7JFhGlCR3AgejLsPA4 flOPioidz/X4u98jhFbWzx4X3S1DqvbVGQhjKyaw8Q+Y/xKkluCmkrPxjX/ILeB0yzQw xhfB+35BijFMvQuRFleRm/jutAw759MPlSvCiudjrq5I66/LMwd+G34GA/lLCsULYHqh 8SdgFGiq9PbEWezYQxqd9CS3BZdp9+tJ3JnAESdNOujTvfhSEnSY+ixwvXr6xPR/80Ey Ax5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=W0vkLVpbGC8iplakE5HggLPIkZErAF5DCoPm30qoMME=; b=b+TMReM09XkdFh64JQtMDEkHsAPN2yoqQOAG+sR10gklzgEJ3xm4M69DE1gfC8yz2b 0uWyWhmjnFh50akZgod/wuw42p0kDyqpRWdzM46yIRuhUTvfOL+dYaBjJXgI1rBiWTdf 3vPahTPfFmdILd5tXLeMnSyI6aOPWmv7qzboh3kAZ402K5VHxC5DzKVjQPcbpuOh4bpB QIKmzuCMNGrMXXGwRC3yvwn15QPsphtufkdO+njhZIpa9shzmAaverSIgQq+Qy570jXC /3CWZf1KRCRkz8Q71grvqamBOt9SBXO9Wm3na3LHj5mLXFoYwJ4Jd0WWcP6/+39aeFz8 G0fA== X-Gm-Message-State: AOAM532czbOeM/ODygG2kLk5ZZzIR+R+whkzxS/1kAVZg1r/IbD5zqgh EV8yUn5EehkFGBikbYdQu1e9ts4kTBUppw== X-Google-Smtp-Source: ABdhPJzOnBV/jlTgYGoBsMEaY1vk5S0BgkJT5ZMZGeiD1yEYX6OF/HH8b5mhjjqYLa4Zu7xr06kAQA== X-Received: by 2002:a17:902:8491:: with SMTP id c17mr4528811plo.262.1594745015861; Tue, 14 Jul 2020 09:43:35 -0700 (PDT) Received: from localhost.localdomain ([122.172.60.196]) by smtp.gmail.com with ESMTPSA id c1sm3295983pje.9.2020.07.14.09.43.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jul 2020 09:43:35 -0700 (PDT) From: gautamramk@gmail.com To: ffmpeg-devel@ffmpeg.org Date: Tue, 14 Jul 2020 22:13:14 +0530 Message-Id: <20200714164316.32120-2-gautamramk@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200714164316.32120-1-gautamramk@gmail.com> References: <20200714164316.32120-1-gautamramk@gmail.com> Subject: [FFmpeg-devel] [PATCH v2 2/4] libavcodec/jpeg2000 Fix PCRL Progression Order check X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Gautam Ramakrishnan MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Gautam Ramakrishnan The PCRL progression checks were incomplete. This patch modifes completes the check. Tested on p1_05.j2k. --- libavcodec/jpeg2000dec.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/libavcodec/jpeg2000dec.c b/libavcodec/jpeg2000dec.c index 48ca1c37a5..adaac192e9 100644 --- a/libavcodec/jpeg2000dec.c +++ b/libavcodec/jpeg2000dec.c @@ -1465,23 +1465,30 @@ static int jpeg2000_decode_packets_po_iteration(Jpeg2000DecoderContext *s, Jpeg2 Jpeg2000Component *comp = tile->comp + compno; Jpeg2000CodingStyle *codsty = tile->codsty + compno; Jpeg2000QuantStyle *qntsty = tile->qntsty + compno; - int xc = x / s->cdx[compno]; - int yc = y / s->cdy[compno]; + + if (!s->cdx[compno] || !s->cdy[compno]) + return AVERROR_INVALIDDATA; for (reslevelno = RSpoc; reslevelno < FFMIN(codsty->nreslevels, REpoc); reslevelno++) { unsigned prcx, prcy; uint8_t reducedresno = codsty->nreslevels - 1 -reslevelno; // ==> N_L - r Jpeg2000ResLevel *rlevel = comp->reslevel + reslevelno; + int trx0, try0; - if (yc % (1LL << (rlevel->log2_prec_height + reducedresno)) && y != tile->coord[1][0]) //FIXME this is a subset of the check - continue; + trx0 = ff_jpeg2000_ceildiv(tile->coord[0][0], s->cdx[compno] << reducedresno); + try0 = ff_jpeg2000_ceildiv(tile->coord[1][0], s->cdy[compno] << reducedresno); - if (xc % (1LL << (rlevel->log2_prec_width + reducedresno)) && x != tile->coord[0][0]) //FIXME this is a subset of the check - continue; + if (!(y % ((uint64_t)s->cdy[compno] << (rlevel->log2_prec_height + reducedresno)) == 0 || + (y == tile->coord[1][0] && (try0 << reducedresno) % (1U << (reducedresno + rlevel->log2_prec_height))))) + continue; + + if (!(x % ((uint64_t)s->cdx[compno] << (rlevel->log2_prec_width + reducedresno)) == 0 || + (x == tile->coord[0][0] && (trx0 << reducedresno) % (1U << (reducedresno + rlevel->log2_prec_width))))) + continue; // check if a precinct exists - prcx = ff_jpeg2000_ceildivpow2(xc, reducedresno) >> rlevel->log2_prec_width; - prcy = ff_jpeg2000_ceildivpow2(yc, reducedresno) >> rlevel->log2_prec_height; + prcx = ff_jpeg2000_ceildiv(x, s->cdx[compno] << reducedresno) >> rlevel->log2_prec_width; + prcy = ff_jpeg2000_ceildiv(y, s->cdy[compno] << reducedresno) >> rlevel->log2_prec_height; prcx -= ff_jpeg2000_ceildivpow2(comp->coord_o[0][0], reducedresno) >> rlevel->log2_prec_width; prcy -= ff_jpeg2000_ceildivpow2(comp->coord_o[1][0], reducedresno) >> rlevel->log2_prec_height;