From patchwork Fri Mar 8 08:14:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jun Zhao X-Patchwork-Id: 12242 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id D0797448499 for ; Fri, 8 Mar 2019 10:14:26 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A69A868A741; Fri, 8 Mar 2019 10:14:26 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 82075680B80 for ; Fri, 8 Mar 2019 10:14:19 +0200 (EET) Received: by mail-pg1-f194.google.com with SMTP id m2so13532846pgl.5 for ; Fri, 08 Mar 2019 00:14:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=R9nL/cK8ONP86Lt1uksgUZTFv9P3QT+HOrZrniZWgXY=; b=m7Ny6nAjP+bbTPWQ9oM3GHuEWfSzqfFlBQUvL6PgiumaH1WbjVz2zZn6jq1T5GvJ9e GW0jXpD5OXpeEvzzWXS37rC6S73XNkp9QRftmHREFf+r+sf/iVkJ05B76X6gkriaHfmw IaSTEZRDaGnv/QEWV2c4nY27a9p2q27y7uk0Dl+V7bymawLZQFlkpWseEN5b8E+rHyeF zlHuhZ/N3NJ5Y+kwnpf+BdT7BWMBlMcO5OVs9KNukb9k/9Vym7tH3pjd5Avv9csLn+ec UilBdH4upQ1tDbQ5xcnv1EMfej6sR6hypWaQuI70pMUH7O2wYhOIdrQi1FLlASmoomVq CwMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=R9nL/cK8ONP86Lt1uksgUZTFv9P3QT+HOrZrniZWgXY=; b=kB86barsJ69BJBU1y/jQwZ2cPnr+XtU7/zui7Un/3tZMdjrlS3yNALz11WmAH5NIf9 5yijoVFW8KytJdQHyUNyzTqw4AHZ84d68a+vYTYBeKBjmXXPziJFemn7H13UYxgF1OHn LP0+JIBwZ9RTakitMetYBE2SiaGjLyX1aZnFbzThG5YcgnJS3nFdInSyfBho8ZK6RhtE Avx0tzrTKG54UO5w8kSsYOVKMVz6WKjmDjcfGSjBNBA2akLu/5US9WffV21fXr99VeoG 9BmrZAlBrVWYuTpCU08Z47vjU9OwiuRwxlUPIotHjdhnWakyf8EvSHZQi7yw5xMhweq4 6rug== X-Gm-Message-State: APjAAAWPQ1S26Uql+c1z4m1eR1FOOH2X55gJ7UivaNfYzFiV/XyyHK03 nnKVNZUg3er7ZcnOzoZJ7XnyX1Tw X-Google-Smtp-Source: APXvYqzuHi7j8azH1GjTWdcQIMvrZBwsVN9ebtdyOWuuJKFbg04oNesOIT5g/2Q2E3SCQtQ4yGrx/Q== X-Received: by 2002:a17:902:d88c:: with SMTP id b12mr17678807plz.339.1552032857471; Fri, 08 Mar 2019 00:14:17 -0800 (PST) Received: from localhost.localdomain ([47.90.47.25]) by smtp.gmail.com with ESMTPSA id j14sm11044027pfn.77.2019.03.08.00.14.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 08 Mar 2019 00:14:16 -0800 (PST) From: Jun Zhao To: ffmpeg-devel@ffmpeg.org Date: Fri, 8 Mar 2019 16:14:11 +0800 Message-Id: <1552032852-18731-1-git-send-email-mypopydev@gmail.com> X-Mailer: git-send-email 1.7.1 Subject: [FFmpeg-devel] [PATCH V4 1/2] lavfi/nlmeans: Checking number precision when computing integral images X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Jun Zhao MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Jun Zhao accumulation of 8-bits uint_8 (uint8_t *src) into 32-bits (uint32_t *ii) data type, it will have a risk of an integral value becoming larger than the 32-bits integer capacity and resulting in an integer overflow. For this risk, add a checking with warning message. Signed-off-by: Jun Zhao --- libavfilter/vf_nlmeans.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/libavfilter/vf_nlmeans.c b/libavfilter/vf_nlmeans.c index dcb5a03..8d47f9d 100644 --- a/libavfilter/vf_nlmeans.c +++ b/libavfilter/vf_nlmeans.c @@ -477,6 +477,17 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) NLMeansContext *s = ctx->priv; AVFilterLink *outlink = ctx->outputs[0]; + // accumulation of 8-bits uint_8 into 32-bits data type, it will have + // a risk of an integral value becoming larger than the 32-bits integer + // capacity and resulting in an integer overflow, so limit the image size + if ((UINT32_MAX / (uint64_t)inlink->w) < (255 * (uint64_t)inlink->h)) { + av_log(ctx, AV_LOG_ERROR, + "image size (%d x %d) integral value may overflow.\n", + inlink->w, inlink->h); + av_frame_free(&in); + return AVERROR(EINVAL); + } + AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); if (!out) { av_frame_free(&in);