From patchwork Sat Nov 19 23:44:21 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Cadhalpun X-Patchwork-Id: 1492 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.90.1 with SMTP id o1csp850904vsb; Sat, 19 Nov 2016 15:44:33 -0800 (PST) X-Received: by 10.194.90.231 with SMTP id bz7mr4065412wjb.35.1479599073089; Sat, 19 Nov 2016 15:44:33 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f125si7677293wmf.44.2016.11.19.15.44.32; Sat, 19 Nov 2016 15:44:33 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@googlemail.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE dis=NONE) header.from=googlemail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 329E2689CBD; Sun, 20 Nov 2016 01:44:28 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C4825689A66 for ; Sun, 20 Nov 2016 01:44:21 +0200 (EET) Received: by mail-wm0-f67.google.com with SMTP id m203so16308321wma.3 for ; Sat, 19 Nov 2016 15:44:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=from:subject:to:references:message-id:date:user-agent:mime-version :in-reply-to; bh=FCKXUGB7CltwiAcf51YfF7DENw+TVRZFBb0l6Dx3H5E=; b=nMujnBRdIxzab8d9ZDVbgYQJKj5SMk8fK7ntKha9yLuWZDN2xNAqdvsllOYKFtuRn6 NoGHeLFnyDraWicIQGwpxETZmGGYPqOVOQdQO/BQ9+GZgzKHWW8FkCNbi8IrEUaZb+7t iN4BjTeoUsr8v2OgRYXVXo+wwwRVc8R72YS9mF02NzFM7dVMWWVMi9ocoStQJtPQD1ku gl4iVR1MnTcp7TIv+MOpN9qegqP5h6HKLl3qsTuOtpc6ubx0mzi60JKtYkWpQX73kpoI UKBxkj5/YmfqRBFIKhTa7nmtVE35r/uKaFQxWeOs6cf0foQ2QmRfYvKNFzUWn+2CLlB8 bw1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:subject:to:references:message-id:date :user-agent:mime-version:in-reply-to; bh=FCKXUGB7CltwiAcf51YfF7DENw+TVRZFBb0l6Dx3H5E=; b=m9hy7jwqt9vPlB3UKumu1+8zGHejSNMncEAt4cojZoWL3R/+Q5vCYSCBG6+Rpz//hv 5NO0ElFMjTT+NmVpnpZ4mxRhvTJff5W6WaYMsfNZZqag9yaYSsn2SeFf/7t4oMV/5qJ1 azDF4J2QVyJlYCmUYd9IBv008VAJTMNGQK7hyyNc6RD689TnOZiyOqDEJzBd6Ekbd5Nk zu3VoBSa+Dcf5QQ2LvzGGbZF6FjAkZVhajxQEh5ys/AJ9F6ZMlw0mLfDOrCkYgQQGBNU 0dUJxVh3awhuj1fbx/KnS2gGQ8/xhO9h+BBsZORkDbgCRTFsGz80C73l/Eq8rP96dfRl 0YMg== X-Gm-Message-State: AKaTC01zSQYZhlU3kylHhdmFi+xD5fjQo8g+SeK/uDeNEfPaK/Z6K95SIR5ptxcvYo/csQ== X-Received: by 10.28.111.139 with SMTP id c11mr6340730wmi.139.1479599063539; Sat, 19 Nov 2016 15:44:23 -0800 (PST) Received: from [192.168.2.21] (p5B095CF1.dip0.t-ipconnect.de. [91.9.92.241]) by smtp.googlemail.com with ESMTPSA id t82sm11088588wmd.17.2016.11.19.15.44.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 Nov 2016 15:44:22 -0800 (PST) From: Andreas Cadhalpun X-Google-Original-From: Andreas Cadhalpun To: ffmpeg-devel@ffmpeg.org, libav development References: <451af605-69ea-c944-9e1b-0fb67ae1abcf@googlemail.com> <20161119151321.GP4824@nb4> <20161119223453.GV4824@nb4> Message-ID: Date: Sun, 20 Nov 2016 00:44:21 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161119223453.GV4824@nb4> Subject: Re: [FFmpeg-devel] [PATCH] smacker: limit recursion depth of smacker_decode_bigtree X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" On 19.11.2016 23:34, Michael Niedermayer wrote: > On Sat, Nov 19, 2016 at 05:27:19PM +0100, Andreas Cadhalpun wrote: >> diff --git a/libavcodec/smacker.c b/libavcodec/smacker.c >> index b8a0c55..be3914b 100644 >> --- a/libavcodec/smacker.c >> +++ b/libavcodec/smacker.c >> @@ -129,8 +129,12 @@ static int smacker_decode_tree(GetBitContext *gb, HuffContext *hc, uint32_t pref >> /** >> * Decode header tree >> */ >> -static int smacker_decode_bigtree(GetBitContext *gb, HuffContext *hc, DBCtx *ctx) >> +static int smacker_decode_bigtree(GetBitContext *gb, HuffContext *hc, DBCtx *ctx, int length) >> { >> + if(length > 5000) { // Larger length can cause segmentation faults due to too deep recursion. >> + av_log(NULL, AV_LOG_ERROR, "length too long\n"); >> + return AVERROR_INVALIDDATA; >> + } > > are you sure this is not too large for some platforms ? I don't think it's even possible to make this small enough for all cases, as the stack size can be arbitrarily changed with 'ulimit -s'. This value was chosen so that it works with the default stack size of 8 MB, but if you think that's too much, it can be made smaller. Attached is a variant reducing the 5000 to 500 and thus still working with a stack size of only 0.8 MB. Best regards, Andreas From 031676592dc15ea11fd677d37d094772478a16e0 Mon Sep 17 00:00:00 2001 From: Andreas Cadhalpun Date: Sat, 19 Nov 2016 14:21:11 +0100 Subject: [PATCH] smacker: limit recursion depth of smacker_decode_bigtree This fixes segmentation faults due to stack-overflow caused by too deep recursion. Signed-off-by: Andreas Cadhalpun --- libavcodec/smacker.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/libavcodec/smacker.c b/libavcodec/smacker.c index b8a0c55..2d20be9 100644 --- a/libavcodec/smacker.c +++ b/libavcodec/smacker.c @@ -129,8 +129,12 @@ static int smacker_decode_tree(GetBitContext *gb, HuffContext *hc, uint32_t pref /** * Decode header tree */ -static int smacker_decode_bigtree(GetBitContext *gb, HuffContext *hc, DBCtx *ctx) +static int smacker_decode_bigtree(GetBitContext *gb, HuffContext *hc, DBCtx *ctx, int length) { + if(length > 500) { // Larger length can cause segmentation faults due to too deep recursion. + av_log(NULL, AV_LOG_ERROR, "length too long\n"); + return AVERROR_INVALIDDATA; + } if (hc->current + 1 >= hc->length) { av_log(NULL, AV_LOG_ERROR, "Tree size exceeded!\n"); return AVERROR_INVALIDDATA; @@ -159,12 +163,12 @@ static int smacker_decode_bigtree(GetBitContext *gb, HuffContext *hc, DBCtx *ctx int r = 0, r_new, t; t = hc->current++; - r = smacker_decode_bigtree(gb, hc, ctx); + r = smacker_decode_bigtree(gb, hc, ctx, length + 1); if(r < 0) return r; hc->values[t] = SMK_NODE | r; r++; - r_new = smacker_decode_bigtree(gb, hc, ctx); + r_new = smacker_decode_bigtree(gb, hc, ctx, length + 1); if (r_new < 0) return r_new; return r + r_new; @@ -275,7 +279,7 @@ static int smacker_decode_header_tree(SmackVContext *smk, GetBitContext *gb, int goto error; } - if (smacker_decode_bigtree(gb, &huff, &ctx) < 0) + if (smacker_decode_bigtree(gb, &huff, &ctx, 0) < 0) err = -1; skip_bits1(gb); if(ctx.last[0] == -1) ctx.last[0] = huff.current++; -- 2.10.2