From patchwork Sat Sep 28 02:26:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Rheinhardt X-Patchwork-Id: 15349 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 9E3F4449F5A for ; Sat, 28 Sep 2019 05:34:08 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8BC9468A5CC; Sat, 28 Sep 2019 05:34:08 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1E959688324 for ; Sat, 28 Sep 2019 05:33:58 +0300 (EEST) Received: by mail-wm1-f65.google.com with SMTP id v17so7281387wml.4 for ; Fri, 27 Sep 2019 19:33:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iMr0sMXN8m937Vyc2NtQKmDsOBDAUk9Qbslos60ZwKY=; b=XVTdpPdH/nyDBwJVy/t/rSFE40RNvsksZ0V++vn45lN2D0SOJmiAi16x0fow73I72w KU5tpLkCQlTd9+3hIjXpZ8WFs52K5fA90MQqUXIUjFfT9pK7CBd7B3zD7JqWekD9bp/F DwnMZGafWluQfdjZ963EV+vGHAAxUtZ+x4lK7XrLZfJ4AvzyR3xR4WOrvAYetM5RSAWo 8sacr0tq4vAe0Lz6Xky2nGmejV9yj5bNtCPySvX6fLWkekDg+JHMriktLubmCNiiLQUm YZQb8vuHLoLglZhrV/7WPalIjLkDxWyZSthPeYUqcCd++cXVjW8C14c2uNOyD+DKvwyZ +fEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iMr0sMXN8m937Vyc2NtQKmDsOBDAUk9Qbslos60ZwKY=; b=NAFZ0qX71OGBGGpDMSbwnJK/EaINYriCksp+HX6bHq0d/DiVoZDZIMIhBIgw04H42j /mcFWAc8pC238ENCuickdO0djcfqcfm6UqQVR74pMQDCyxaU+BDAYMxyIzsCuLS7OXhm /Lg0oO5neYVjmSLffJUwU8FcEfnH+k/+y6EkpCajCG7aFBoilxfN62jfbjfItqll/52s QzuBxCAewnXd/pLBbvLNszgRYLsQUEOhieEFpjA5vMpRak7mZzYLUWRXn3ao5cy2/Wjj nfvC5MUsxCHlHH8u3JBvrJfxz097hCKC4Cp7ZZCOADV1qLxvrA/GVRTbUOwxHUsdz+WO 1pRw== X-Gm-Message-State: APjAAAWhQ5/po6KpjD3gTaXDA/re/0AjWi1WMM1XAqBl0MEadUGgRqtQ M94+9RiQ+S5UdT+iI2Bq0oPZBJiz X-Google-Smtp-Source: APXvYqzaIfLCsn/SYTad+TbI5CPuR8g+3OQ3MdlwIzngaHoVXoxlXm5UeiJhdumVboIMRSFO3A49bg== X-Received: by 2002:a05:600c:248a:: with SMTP id 10mr9201727wms.97.1569638037510; Fri, 27 Sep 2019 19:33:57 -0700 (PDT) Received: from sblaptop.fritz.box (ipbcc0f857.dynamic.kabel-deutschland.de. [188.192.248.87]) by smtp.gmail.com with ESMTPSA id r7sm4484231wrx.87.2019.09.27.19.33.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Sep 2019 19:33:57 -0700 (PDT) From: Andreas Rheinhardt To: ffmpeg-devel@ffmpeg.org Date: Sat, 28 Sep 2019 04:26:01 +0200 Message-Id: <20190928022610.5903-6-andreas.rheinhardt@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190928022610.5903-1-andreas.rheinhardt@gmail.com> References: <20190928022610.5903-1-andreas.rheinhardt@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 06/15] avcodec/ffv1enc: Fix out-of-bounds-array access X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" libavcodec/ffv1enc.c accessed an array of uint8_t [32] via array[0][j] in order to loop over all the uint8_t in this array of arrays. Of course this implied an out-of-bounds access for array[0] and UBSan complained about this. So replace this with nested loops; furthermore, factor this out into a function of its own to easily break out of the nested loops. This affected the FATE-tests vsynth1-ffv1, vsynth1-ffv1-v3-yuv420p, vsynth1-ffv1-v3-yuv422p10, vsynth1-ffv1-v3-yuv444p16, vsynth1-ffv1-v3-bgr0, vsynth1-ffv1-ffv1-v3-rgb48 as well as the corresponding vsynth2-*, vsynth3-* and the vsynth_lena-* tests. Signed-off-by: Andreas Rheinhardt --- libavcodec/ffv1enc.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/libavcodec/ffv1enc.c b/libavcodec/ffv1enc.c index 796d81f7c6..c521b7d445 100644 --- a/libavcodec/ffv1enc.c +++ b/libavcodec/ffv1enc.c @@ -334,6 +334,18 @@ static void write_quant_tables(RangeCoder *c, write_quant_table(c, quant_table[i]); } +static int contains_non_128(uint8_t (*initial_state)[CONTEXT_SIZE], + int nb_contexts) +{ + if (!initial_state) + return 0; + for (int i = 0; i < nb_contexts; i++) + for (int j = 0; j < CONTEXT_SIZE; j++) + if (initial_state[i][j] != 128) + return 1; + return 0; +} + static void write_header(FFV1Context *f) { uint8_t state[CONTEXT_SIZE]; @@ -428,10 +440,7 @@ static int write_extradata(FFV1Context *f) write_quant_tables(c, f->quant_tables[i]); for (i = 0; i < f->quant_table_count; i++) { - for (j = 0; j < f->context_count[i] * CONTEXT_SIZE; j++) - if (f->initial_states[i] && f->initial_states[i][0][j] != 128) - break; - if (j < f->context_count[i] * CONTEXT_SIZE) { + if (contains_non_128(f->initial_states[i], f->context_count[i])) { put_rac(c, state, 1); for (j = 0; j < f->context_count[i]; j++) for (k = 0; k < CONTEXT_SIZE; k++) {