From patchwork Fri Dec 6 13:55:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andriy Gelman X-Patchwork-Id: 16634 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 6694744ADCF for ; Fri, 6 Dec 2019 16:02:37 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 55F0168B65C; Fri, 6 Dec 2019 16:02:37 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oi1-f193.google.com (mail-oi1-f193.google.com [209.85.167.193]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E20896880EF for ; Fri, 6 Dec 2019 16:02:30 +0200 (EET) Received: by mail-oi1-f193.google.com with SMTP id v10so6183529oiv.12 for ; Fri, 06 Dec 2019 06:02:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=7WgUIaNpZfjQ6RzpKCWmh9rF9/eUtf4txVMAu3GGjCg=; b=nx48/5FaK5ROZlQkVvYXvfOYq5dCFVs9Xr3nvitPeeDXFoyuM8maTTiEmBnut5laA9 O2VriMQjyRPsny9aYGv/JpsBx8zp7GiwZiszOhCdlEqfNRVUiS5txjSHez9/SYlTFn42 sVFpLv7UuvWMu0vo8k1IEfJaGdinJlXXxINpsB8l/xnhHPFobyFJrnEDaD409IU1oXWw z3ju8TX1zSIqyovvQ+0NZv28t5QQiX7gjxSSOgBc0pq32fMYqUau5DatENqi8oAZkKbK bQe+ryTzSEJc3j6VZkhjWdLkB8BdeTLwwDUkgatcA0yQ/70gFv28C6u1o0R0vdBKXQVK UJcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=7WgUIaNpZfjQ6RzpKCWmh9rF9/eUtf4txVMAu3GGjCg=; b=oH9OkI8BUmfCyQwFMECXVRk17UTAHQItIzjV5o4G3LdenIiK9wt9Teq1ekYaeJyhNl z2sGf4uPcMhdW50M1DT90aHSOQYzckDXgv16koJRRlz0v5E4Y4aVRZTECjARCYhsrNcY /+8eB0A64Jy37ERNE3JAVf0/0TWOeaMUYQl8+yeDvD+m+2ZUREz8qr2Aj5V9C1QYVM7Y vlVyRFdMbJ5SESvNHSE7s81/rLP8VxF8+/YOWtjrzM4RbjRChyAmjygO9+tTMsEcVmYQ 95mAo5t25XxPON4ECq0iibg/lZtW8pc80Y3wayBPGV20QUVow8fYUHFH10cwK0KzR3yV j0Nw== X-Gm-Message-State: APjAAAW9c1lFX+42xScmAN45OcbmMoi/oaIJmce4QNhz8oH1iP37refB V8ulizPCUrJFyQwU1nHNR6tWZ0sT X-Google-Smtp-Source: APXvYqw+TsdVR+6SxItKvm83Vr+CAB5EIF+x1yDXgQEI1PyIl8/j46tkTP3XeavQQt3K1G4zTWQ4Wg== X-Received: by 2002:aca:568d:: with SMTP id k135mr12451526oib.45.1575640554968; Fri, 06 Dec 2019 05:55:54 -0800 (PST) Received: from manj.guest.slb.com ([192.23.22.48]) by smtp.gmail.com with ESMTPSA id t9sm1883753otm.76.2019.12.06.05.55.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2019 05:55:54 -0800 (PST) From: Andriy Gelman X-Google-Original-From: Andriy Gelman To: ffmpeg-devel@ffmpeg.org Date: Fri, 6 Dec 2019 08:55:40 -0500 Message-Id: <20191206135541.8202-1-andriy.gelman@gmail.com> X-Mailer: git-send-email 2.24.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 1/2] lavc/hevc_mp4toannexb: Fix integer overflow X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andriy Gelman Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Andriy Gelman Fixes a check where the grow_by packet size cannot exceed INT_MAX. Although the complete heap allocation check should ensure that (uint64_t)nalu_size + extra_size + 4 + out->size + AV_INPUT_BUFFER_PADDING_SIZE <= INT_MAX, it suffices to only check that the grow_by size does exceed INT_MAX. The final memory check is done internally in av_packet_grow(). This approach means that no casts to uint64_t are needed. Also, note that extra_size < INT_MAX - 4, because from hevc_extradata_to_annexb(): extra_size <= INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE. Found with libFuzzer: 4294967044 cannot be represented as int. Signed-off-by: Andriy Gelman --- libavcodec/hevc_mp4toannexb_bsf.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/libavcodec/hevc_mp4toannexb_bsf.c b/libavcodec/hevc_mp4toannexb_bsf.c index 09bce5b34c2..f1e1a45abd1 100644 --- a/libavcodec/hevc_mp4toannexb_bsf.c +++ b/libavcodec/hevc_mp4toannexb_bsf.c @@ -66,7 +66,7 @@ static int hevc_extradata_to_annexb(AVBSFContext *ctx) for (j = 0; j < cnt; j++) { int nalu_len = bytestream2_get_be16(&gb); - if (4 + AV_INPUT_BUFFER_PADDING_SIZE + nalu_len > SIZE_MAX - new_extradata_size) { + if (4 + AV_INPUT_BUFFER_PADDING_SIZE + nalu_len > INT_MAX - new_extradata_size) { ret = AVERROR_INVALIDDATA; goto fail; } @@ -152,8 +152,7 @@ static int hevc_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out) extra_size = add_extradata * ctx->par_out->extradata_size; got_irap |= is_irap; - if (SIZE_MAX - nalu_size < 4 || - SIZE_MAX - 4 - nalu_size < extra_size) { + if (INT_MAX - 4 - extra_size < nalu_size) { ret = AVERROR_INVALIDDATA; goto fail; }