From patchwork Thu Oct 19 18:46:47 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niki Bowe X-Patchwork-Id: 5631 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.161.90 with SMTP id m26csp2884036jah; Thu, 19 Oct 2017 11:53:58 -0700 (PDT) X-Received: by 10.223.131.166 with SMTP id 35mr2469386wre.84.1508439238516; Thu, 19 Oct 2017 11:53:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1508439238; cv=none; d=google.com; s=arc-20160816; b=P+BkkXDP+2riF44Vld69XQZqbGC7Nl2rsBjfVfIB7neiZq2Haq5d//QLzcmrvYUQ32 UofmIqPfA1Bg0xMTpzUX+Ash3JYkxZw5loIGaF1Jusy/mvmvVZC1BcG3FD8p/m44lNJ6 oe/omcoKYtu4gVWTM2qd0iFLKKv0ZyN4OEqdnZFUdqTRjhUzhI+P0mrMifkZ/SQJUHLT gYuJiti4QztUhh7ECSfQ2rOporPX911tPEZu9DIzLvzDGyjrfq19eM1BySgm3zDHcP2D o7yLDVhplXZXKCR3jLehFB5f+1P+mv7Qf4w6lniwz/1IMpMhpoj0wcUjNXzV9hc6va19 8jVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=lX5T+VZMlrdBdj+SMGQ/B0sihhmT9FsiZoqAML4dlUg=; b=MImJsLkpMBrZIbJam53H+ARWPgaK2Rtm3HIkYM38L1b490uODT4yK51EhphOsKwN2M l4UbP4X13qoQL0yaI49mVF4rqw28jvrK7HWSQ37eV+50kPESDUsrG6IaA9dVXQ1S3eAK Brr0+GGdBeax6vhJjDkZlBp2Hrnz47+AwO0YGuNKcy0CWCK3Xy7aHu6VBL8QHCeZchmK CGBxMLqk6f/CKUkLVJ8ptBlcQrl/PSkVHtjSKTocOzLfn3ebA/Ylc+xuFsHhN3oAA2Uh Bx7EfZiCBOauNeHNS7GxcblEfRaeppWnuN787ULMBdFmtKXI93RpGXNpdImTMFi8VjIT FZKw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20161025 header.b=CQFFsRQH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id k4si3774597wrh.27.2017.10.19.11.53.57; Thu, 19 Oct 2017 11:53:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20161025 header.b=CQFFsRQH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7480A689F41; Thu, 19 Oct 2017 21:53:50 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-it0-f54.google.com (mail-it0-f54.google.com [209.85.214.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 02E0A689EAE for ; Thu, 19 Oct 2017 21:53:44 +0300 (EEST) Received: by mail-it0-f54.google.com with SMTP id n195so10912879itg.2 for ; Thu, 19 Oct 2017 11:53:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=5v0uzi3jgWNqdURpQkmqGUhYR42e/4kIeVELwFCf1P4=; b=CQFFsRQHkFUjBhhFjcxllStjTEDwSgT5Nx5x91MxkJMNp4sF5BpjDSwmGEqnwUkkBw RLURBa59k3moS4c1I1OoUiNcw+362MvuczQx1mOpQheYi6W2cOqaP9/kYl1a5SG+IvJF Kua3AKrT5j/cJVuyecS1v+beuUYU54yHLh1IBYg7/zQfjVTbsomrqnm/Z5nkqKpFVcNH 2leKiKmx2KtKJXBxwQRTcXlmgecf8jxKEuZX/tgnPFPHsGWEM5vKhBEonrlLLTPpPrfW XUphEJH3jMFCLyX6gm8D+VYPuMHU1b2dCLUjPcCuoR8YCnU2RcQW3D5fk7cuJy61WRTx GKUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=5v0uzi3jgWNqdURpQkmqGUhYR42e/4kIeVELwFCf1P4=; b=ggkVo+24zZrdX4hj8pZhgrVhNHk+C8BI8xNLuz63IyrWXG1Cws4LKpPDxiMrQsQi2B qBHzo4wewbdNBDn7wkIBnGEXf3vd3OkFmBPQKRovVanesc24lxQ75z0Ynj1d2s+E3jCO oZks2+pGrRKicmeIApIWXEYhaWvnWoWWXUFRaWEjllolzsreFxdDK/jCjwlDVTeuLHAr YZkgox/0clm1LruBAIjCYFjLjwuK6qm8SNsWZdtx8gzyAR4RxQ2OVqTTzArwG+6QJ0jn lcQYmsC54e+Bif4fpK7K/mUm4GDyVulVcrnEGm1eO0QWITomf8t4k0cBLHAsYhmS0cHj PNLw== X-Gm-Message-State: AMCzsaXGrzPRTKV62jCzVdr5CcIZezaD89uZhzeaV78E4jAHgX56PPrB b4w7g6yt5RLLtUylNmJOjLN5mhtv1Uc= X-Google-Smtp-Source: ABhQp+TulXpe08uFlkaK4yd9MqNdyHLHolTNHMzujLFaBtRD4OpgLfo2SYq1x5qXXy/gw+pOFV7hSQ== X-Received: by 10.36.254.140 with SMTP id w134mr3621608ith.73.1508438830147; Thu, 19 Oct 2017 11:47:10 -0700 (PDT) Received: from nbowe.mtv.corp.google.com ([100.98.2.75]) by smtp.gmail.com with ESMTPSA id q187sm7154178iof.26.2017.10.19.11.47.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 19 Oct 2017 11:47:09 -0700 (PDT) From: Nikolas Bowe To: ffmpeg-devel@ffmpeg.org Date: Thu, 19 Oct 2017 11:46:47 -0700 Message-Id: <20171019184647.80641-1-nbowe@google.com> X-Mailer: git-send-email 2.15.0.rc1.287.g2b38de12cc-goog Subject: [FFmpeg-devel] [PATCH] Fix quadratic memory use in ff_h2645_extract_rbsp() when multiple NALUs exist in packet. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Nikolas Bowe MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Found via fuzzing. /tmp/poc is a 1 MB mpegts file generated via fuzzing, where 1 packet has many NALUs Before this change: $ /usr/bin/time -f "\t%M Max Resident Set Size (Kb)" ./ffprobe /tmp/poc 2>&1 | tail -n 1 2158192 Max Resident Set Size (Kb) After this change: $ /usr/bin/time -f "\t%M Max Resident Set Size (Kb)" ./ffprobe /tmp/poc 2>&1 | tail -n 1 1046812 Max Resident Set Size (Kb) --- libavcodec/h2645_parse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/libavcodec/h2645_parse.c b/libavcodec/h2645_parse.c index b0d9ff66f0..e77689f347 100644 --- a/libavcodec/h2645_parse.c +++ b/libavcodec/h2645_parse.c @@ -32,7 +32,7 @@ int ff_h2645_extract_rbsp(const uint8_t *src, int length, H2645NAL *nal, int small_padding) { - int i, si, di; + int i, si, di, nsc; uint8_t *dst; int64_t padding = small_padding ? 0 : MAX_MBPAIR_SIZE; @@ -91,8 +91,17 @@ int ff_h2645_extract_rbsp(const uint8_t *src, int length, } else if (i > length) i = length; + // Find next NAL start code, if present, to reduce rbsp_buffer size when + // multiple NALUs. + for (nsc = i; nsc + 2 < length; nsc++) { + if (src[nsc] == 0 && src[nsc + 1] == 0 && src[nsc + 2] == 1) + break; + } + if (nsc + 2 == length) + nsc = length; + av_fast_padded_malloc(&nal->rbsp_buffer, &nal->rbsp_buffer_size, - length + padding); + nsc + padding); if (!nal->rbsp_buffer) return AVERROR(ENOMEM);