From patchwork Mon Jul 13 17:09:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Kim X-Patchwork-Id: 20991 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id B414344BAD2 for ; Mon, 13 Jul 2020 20:10:03 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9E3C868A0BE; Mon, 13 Jul 2020 20:10:03 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id F3A2E689F05 for ; Mon, 13 Jul 2020 20:09:56 +0300 (EEST) Received: by mail-yb1-f202.google.com with SMTP id u64so17645340ybf.13 for ; Mon, 13 Jul 2020 10:09:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=cScOrf5yw1fhKdnCsZegpa21iGFMqbKRqKrgygcJcNU=; b=P+FprLniACBKpRfR00zbn1XoOFYA2ka+L3ro8/CCRtLekCtxI/R5Eer+kJQuVVBDH1 0dXq+x7w9AkTAi1T+RGsFT9zvJwsKxdp+5+/XLwLTmbpoLtB0mD9qjvV9s0qImtcUYPW OPHDtTdQ8ttZy+G9TmqdlhH+q2ej9ZFmbXWH+GHINV4MZJJIkGDvq+K+7raJ8Vm+0aMO Efua4v/0jW654crBEePx7d5QZsNAqDGkTmh7SC7i58Kq1KueLDyUe0wzE2iQLCJ4iyAO Q3KOCn2/3Bq5dlX8UOyuf77XZd9fEtBAenDO5j994C3lRvT13opFjLNwTv1Vevyut1/Z zGxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=cScOrf5yw1fhKdnCsZegpa21iGFMqbKRqKrgygcJcNU=; b=XcBX5ZZo/FPhOTOwLTSfpNLpcARUy0CjiJopmvg2d4irWbI+H2JXvu3JgdzQfkfZjo TIIK8iS+DyRX1VOmGmyZ+fUZojxRveerzCT8ozVBUpmzThDb2wvT41X5JdhS6UYtzClJ bd+8cay9DPoGrJZNNtyEcgbPDtNDOF+iRQIDi+lt3+a5zlESm4ZTgsa71tnaws4o45Hs 5NY7g2N/XgigBx/K1XmOBfzJGsolFMszLf4H9Q187kW1hUBxPKFc/Ehaqaitye1ZYMAw Cj1UsnkRT6YUUxbkQ628q1vtPz6Pm+ns6QubPnyYkYZhbz6doOItPl0/6t7dTMPVe88R Xxow== X-Gm-Message-State: AOAM531gQQbuuyuvU/c2wLt2PDRXoeZVWlzQcx4/pVQInKy9es6YfBX/ Kd4NZBX2YKyYcbOdudhVTgrN1Ehh59VM/QCRjc/j3423LNLTMkdXeBFsmbg4K7yPQnCbIb7sUR9 Frs3KZSVHUomebCeiqD3a0YuxrvJrWFhyyDo8fQ0BpKRaNZhAs5Lm4RGduRYV X-Google-Smtp-Source: ABdhPJxqQ9KiKQ+Eak1TzsYxYvYC+9cbihZxQe1uqibF1oIntkQEbEqR67Q0FLvF/Ug5zEcIfsnijZhk8A== X-Received: by 2002:a25:d081:: with SMTP id h123mr1479666ybg.88.1594660195464; Mon, 13 Jul 2020 10:09:55 -0700 (PDT) Date: Mon, 13 Jul 2020 10:09:38 -0700 In-Reply-To: <11b6ffd9e674f497508f1b7cbad1dee4284f78c7.1594660141.git.bkkim@google.com> Message-Id: <91953b81c47f87fc10001298e608e57c3453237b.1594660141.git.bkkim@google.com> Mime-Version: 1.0 References: <11b6ffd9e674f497508f1b7cbad1dee4284f78c7.1594660141.git.bkkim@google.com> X-Mailer: git-send-email 2.27.0.389.gc38d7665816-goog From: Brian Kim To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v3 2/4] libavutil/frame: avoid UB when getting plane sizes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Brian Kim Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This uses av_image_fill_plane_sizes instead of av_image_fill_pointers when we are getting plane sizes to avoid UB from adding offsets to NULL. Signed-off-by: Brian Kim --- libavutil/frame.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/libavutil/frame.c b/libavutil/frame.c index 9884eae054..3ab1aa3242 100644 --- a/libavutil/frame.c +++ b/libavutil/frame.c @@ -212,8 +212,10 @@ void av_frame_free(AVFrame **frame) static int get_video_buffer(AVFrame *frame, int align) { const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(frame->format); - int ret, i, padded_height; + int ret, i, padded_height, total_size; int plane_padding = FFMAX(16 + 16/*STRIDE_ALIGN*/, align); + ptrdiff_t linesizes[4]; + size_t sizes[4]; if (!desc) return AVERROR(EINVAL); @@ -238,12 +240,22 @@ static int get_video_buffer(AVFrame *frame, int align) frame->linesize[i] = FFALIGN(frame->linesize[i], align); } + for (i = 0; i < 4; i++) + linesizes[i] = frame->linesize[i]; + padded_height = FFALIGN(frame->height, 32); - if ((ret = av_image_fill_pointers(frame->data, frame->format, padded_height, - NULL, frame->linesize)) < 0) + if ((ret = av_image_fill_plane_sizes(sizes, frame->format, + padded_height, linesizes)) < 0) return ret; - frame->buf[0] = av_buffer_alloc(ret + 4*plane_padding); + total_size = 4*plane_padding; + for (i = 0; i < 4; i++) { + if (sizes[i] > INT_MAX - total_size) + return AVERROR(EINVAL); + total_size += sizes[i]; + } + + frame->buf[0] = av_buffer_alloc(total_size); if (!frame->buf[0]) { ret = AVERROR(ENOMEM); goto fail;