From patchwork Wed Dec 19 22:00:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: matthew.w.fearnley@gmail.com X-Patchwork-Id: 11483 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 3BB6844D4F3 for ; Thu, 20 Dec 2018 00:06:28 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5917068ABF8; Thu, 20 Dec 2018 00:06:28 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0441C68AAE5 for ; Thu, 20 Dec 2018 00:06:22 +0200 (EET) Received: by mail-wm1-f68.google.com with SMTP id f81so19289wmd.4 for ; Wed, 19 Dec 2018 14:06:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=rvrhONm91cRrDX0D0yXlpRgglRQwGpWKy5OrM/yUPD4=; b=VMtFYxH9z0PVhCgUY3kd+qGsQBUuZQTtam14j3/DIy2zKSB27+LPo2pncZKEnI0DCE SelD4Jsid4BdT7QOr+KRsFA/j7RI/8KJD30cA60KTMp/up8iKhiUjxRAgy52ih37u+YN ce4jIvDTFqIl4rgIo53M1W5pR4ljv/kkEWOZVaahLPRnjcm6HdcgNlCSysKm4NDIZTcU IKcraScT+ljUoOctZUF/iN49Ioov4fln0aOqtnaH40hMOouGg4S5MpU31tzXv9HgnTPk R8/rG91nHXXqIP3RF8jduJzv1cxRsIL0mI4k6KwCNtB6azyd3XWmFOMADrXHJ1+FX6uy fTLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=rvrhONm91cRrDX0D0yXlpRgglRQwGpWKy5OrM/yUPD4=; b=kxOQrXfaqzS2Sa1xvCqYLtDT4RKMJxzCc5eycL9IClgz16m7a6ExFA7oAPWcg5YeTZ tcSSOgLYqhdtyIbWeC0Elti7ZJNH/dO860ay3AGSbOYY0Aeb/JZ/ILqaq3S1iiXcY8RU K1rasupKwFM9imqgPguXo0RlAqdc4xUfBDLls2NRhdtVP7+wAN9cEBTcx06w1FGvYRwb gpn1/mAlZIIdtxMbVPwwAKOCrWbnMpDa22zAywmgWLarSKtUdoh7RMjHacw9RyiVm3t8 7Bo8oJu2v3A5HXOrU8Y1hop2pv1MwSTm1sOMWOWiiZ3muEr2Qc4HNA5acrwbPZJunwA7 evTg== X-Gm-Message-State: AA+aEWYk8qzVdJhQ4VFhSi0GXKqPWFHSZDOdjArPG9ZuhIfvvH5u/T1V dkpxSsGtNYveUXQj2LzgphyYMv6s X-Google-Smtp-Source: AFSGD/XJymOXgSFkQlMmUr8u1892Wr8TZHGmfeI6ZAdYaPB6OYlqzDL0F1FHOvS9quRG/X2oKSgOeQ== X-Received: by 2002:a1c:4046:: with SMTP id n67mr8382523wma.123.1545256839630; Wed, 19 Dec 2018 14:00:39 -0800 (PST) Received: from localhost.localdomain (cpc131498-bagu18-2-0-cust88.know.cable.virginm.net. [86.9.33.89]) by smtp.gmail.com with ESMTPSA id k135sm8967078wmd.42.2018.12.19.14.00.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Dec 2018 14:00:38 -0800 (PST) From: matthew.w.fearnley@gmail.com To: ffmpeg-devel@ffmpeg.org Date: Wed, 19 Dec 2018 22:00:02 +0000 Message-Id: <20181219220003.27225-3-matthew.w.fearnley@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181219220003.27225-1-matthew.w.fearnley@gmail.com> References: <20181219220003.27225-1-matthew.w.fearnley@gmail.com> Subject: [FFmpeg-devel] [PATCH 3/4] zmbvenc: Prevent memory/math overflows in block_cmp() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Matthew Fearnley MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Matthew Fearnley score_tab[] was only declared/initialised for elements 0..255, but with block sizes set to 16*16, it was possible to reach 256. This limit could also be overflowed in the histogram, because it was declared with a uint8_t type. This can be fixed, and also allow different ZMBV_BLOCK sizes, by making score_tab[] with (ZMBV_BLOCK*ZMBV_BLOCK+1) elements, and declaring histogram[] to use a uint16_t type. Note: the maximum block size possible for PAL8 is 255*255 bytes, which is close to the uint16_t limit. To support full-colour pixel formats, a uint32_t could potentially be required. --- libavcodec/zmbvenc.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/libavcodec/zmbvenc.c b/libavcodec/zmbvenc.c index 0e8ee5ce31..0ebae1b254 100644 --- a/libavcodec/zmbvenc.c +++ b/libavcodec/zmbvenc.c @@ -55,7 +55,7 @@ typedef struct ZmbvEncContext { int keyint, curfrm; z_stream zstream; - int score_tab[256]; + int score_tab[ZMBV_BLOCK * ZMBV_BLOCK + 1]; } ZmbvEncContext; @@ -69,7 +69,7 @@ static inline int block_cmp(ZmbvEncContext *c, uint8_t *src, int stride, { int sum = 0; int i, j; - uint8_t histogram[256] = {0}; + uint16_t histogram[256] = {0}; /* build frequency histogram of byte values for src[] ^ src2[] */ *xored = 0; @@ -285,7 +285,9 @@ static av_cold int encode_init(AVCodecContext *avctx) int i; int lvl = 9; - for(i=1; i<256; i++) + /* entropy score table for block_cmp() */ + c->score_tab[0] = 0; + for(i = 1; i <= ZMBV_BLOCK * ZMBV_BLOCK; i++) c->score_tab[i] = -i * log2(i / (double)(ZMBV_BLOCK * ZMBV_BLOCK)) * 256; c->avctx = avctx;