From patchwork Mon Jul 13 17:09:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Kim X-Patchwork-Id: 20990 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 9ABEA44BAD2 for ; Mon, 13 Jul 2020 20:10:02 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 702AE68A295; Mon, 13 Jul 2020 20:10:02 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0486C687F65 for ; Mon, 13 Jul 2020 20:09:54 +0300 (EEST) Received: by mail-yb1-f202.google.com with SMTP id d202so17891479ybh.12 for ; Mon, 13 Jul 2020 10:09:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=J+t6ijdSq79BUlFBhKYJal4TwsxKXLpCYKPd2hZFCTg=; b=fRNS3yQJ6Q26xsi9B3GDM1cYs4fu8I1+yFojHn0jr11BA1I2MLUXs39LPMunsDvCHJ /12SrWko80jasAOF30uIqpByz1+RPYuav6s4Jaleo6GwUc9YtfRXmwxXPADwezjrGP/Q 5oL5XyIHQHtSAQ00uxvDG8la2PE9pV9UmSHTmEMRgvgHqXgMEU94rIG8hCVFTpPmUZLU 1UPDrTiqUedOGOppHZz9JTtKhKJ1uf2RVFmo3+ZWRhlq8HVHHTVER6Sewd+NcCVxL4nm iYY9rCZroPlPwevdrukTMNq5unr/+CcMW7lKWqsg/EZs1dUJhTHkTQysxFXKM8dQPT8i sZ7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=J+t6ijdSq79BUlFBhKYJal4TwsxKXLpCYKPd2hZFCTg=; b=OA/+cDzbglGBhHIkqzy+i39YqzDedA09rb9Qjd21zIHU4eouctKGQZ+lSl5EzmPfoB 2WR6u+RZRx7PERUMkSCgTLdjiE+gtxxAPZSew/vCu8b+VOzhdtZPc2px7iuhizvQwqBg wz7ApnbmigyVyx6/qVYOHS553rCQdyjzpK2j3CkiganLlhqy5TwjG5DiRm2hUHrK45Du y7WlxYZu/GFncZBPZwZ8VwwWdwUCZyKfqM5KGDooYb/VuwlDYOFshET6zuZd/IeOC29D 6jreSvhxeoORLSozsgr24YlAp6pL8Ff1rUsQ/u/QLkwdXmrz3/0u36wGw7H4BgjblGPc CorQ== X-Gm-Message-State: AOAM533upPT8BF9nqHyecuScVd9+WiIu1atGzAU+wDcv5mzIklQ3pqJT EreETRDrUGDsUJ7BEp5ANO129W5JQ4gF2EfsPlV3LldgWNnhCFoBz7BgrNRpobR9juf87sZDlng jcGcEFHLkCEGnKW5j6ww71iPPvoWkkm3n3fPSmEdQ47kxoqx45KWIqlKUhzB8 X-Google-Smtp-Source: ABdhPJz5dl2LhfI8hP1eaJ/RD3AJdT8RMYMxXTZDtT83gSbIxuHq0GMORFoX0CU4DHbC1i+kwP/g+tXpTw== X-Received: by 2002:a25:9302:: with SMTP id f2mr1527118ybo.238.1594660193473; Mon, 13 Jul 2020 10:09:53 -0700 (PDT) Date: Mon, 13 Jul 2020 10:09:37 -0700 Message-Id: <11b6ffd9e674f497508f1b7cbad1dee4284f78c7.1594660141.git.bkkim@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.27.0.389.gc38d7665816-goog From: Brian Kim To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v3 1/4] libavutil/imgutils: add utility to get plane sizes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Brian Kim Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This utility helps avoid undefined behavior when doing things like checking how much memory we need to allocate for an image before we have allocated a buffer. Signed-off-by: Brian Kim --- doc/APIchanges | 3 ++ libavutil/imgutils.c | 98 +++++++++++++++++++++++++++++++++----------- libavutil/imgutils.h | 11 +++++ libavutil/version.h | 2 +- 4 files changed, 90 insertions(+), 24 deletions(-) diff --git a/doc/APIchanges b/doc/APIchanges index 1d6cc36b8c..44defd9ca8 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -15,6 +15,9 @@ libavutil: 2017-10-21 API changes, most recent first: +2020-07-xx - xxxxxxxxxx - lavu 56.56.100 - imgutils.h + Add av_image_fill_plane_sizes(). + 2020-06-12 - b09fb030c1 - lavu 56.55.100 - pixdesc.h Add AV_PIX_FMT_X2RGB10. diff --git a/libavutil/imgutils.c b/libavutil/imgutils.c index 7f9c1b632c..345b7fa94c 100644 --- a/libavutil/imgutils.c +++ b/libavutil/imgutils.c @@ -108,45 +108,69 @@ int av_image_fill_linesizes(int linesizes[4], enum AVPixelFormat pix_fmt, int wi return 0; } -int av_image_fill_pointers(uint8_t *data[4], enum AVPixelFormat pix_fmt, int height, - uint8_t *ptr, const int linesizes[4]) +int av_image_fill_plane_sizes(size_t sizes[4], enum AVPixelFormat pix_fmt, + int height, const ptrdiff_t linesizes[4]) { - int i, total_size, size[4] = { 0 }, has_plane[4] = { 0 }; + int i, has_plane[4] = { 0 }; const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(pix_fmt); - memset(data , 0, sizeof(data[0])*4); + memset(sizes , 0, sizeof(sizes[0])*4); if (!desc || desc->flags & AV_PIX_FMT_FLAG_HWACCEL) return AVERROR(EINVAL); - data[0] = ptr; - if (linesizes[0] > (INT_MAX - 1024) / height) + if (linesizes[0] > SIZE_MAX / height) return AVERROR(EINVAL); - size[0] = linesizes[0] * height; + sizes[0] = linesizes[0] * height; if (desc->flags & AV_PIX_FMT_FLAG_PAL || desc->flags & FF_PSEUDOPAL) { - data[1] = ptr + size[0]; /* palette is stored here as 256 32 bits words */ - return size[0] + 256 * 4; + sizes[1] = 256 * 4; /* palette is stored here as 256 32 bits words */ + return 0; } for (i = 0; i < 4; i++) has_plane[desc->comp[i].plane] = 1; - total_size = size[0]; for (i = 1; i < 4 && has_plane[i]; i++) { int h, s = (i == 1 || i == 2) ? desc->log2_chroma_h : 0; - data[i] = data[i-1] + size[i-1]; h = (height + (1 << s) - 1) >> s; - if (linesizes[i] > INT_MAX / h) + if (linesizes[i] > SIZE_MAX / h) return AVERROR(EINVAL); - size[i] = h * linesizes[i]; - if (total_size > INT_MAX - size[i]) + sizes[i] = h * linesizes[i]; + } + + return 0; +} + +int av_image_fill_pointers(uint8_t *data[4], enum AVPixelFormat pix_fmt, int height, + uint8_t *ptr, const int linesizes[4]) +{ + int i, ret; + ptrdiff_t linesizes1[4]; + size_t sizes[4]; + + for (i = 0; i < 4; i++) + linesizes1[i] = linesizes[i]; + + ret = av_image_fill_plane_sizes(sizes, pix_fmt, height, linesizes1); + if (ret < 0) + return ret; + + ret = 0; + for (i = 0; i < 4; i++) { + if (sizes[i] > INT_MAX - ret) return AVERROR(EINVAL); - total_size += size[i]; + ret += sizes[i]; } - return total_size; + memset(data , 0, sizeof(data[0])*4); + + data[0] = ptr; + for (i = 1; i < 4 && sizes[i - 1] > 0; i++) + data[i] = data[i - 1] + sizes[i - 1]; + + return ret; } int avpriv_set_systematic_pal2(uint32_t pal[256], enum AVPixelFormat pix_fmt) @@ -194,6 +218,8 @@ int av_image_alloc(uint8_t *pointers[4], int linesizes[4], { const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(pix_fmt); int i, ret; + ptrdiff_t linesizes1[4]; + size_t total_size, sizes[4]; uint8_t *buf; if (!desc) @@ -204,12 +230,20 @@ int av_image_alloc(uint8_t *pointers[4], int linesizes[4], if ((ret = av_image_fill_linesizes(linesizes, pix_fmt, align>7 ? FFALIGN(w, 8) : w)) < 0) return ret; - for (i = 0; i < 4; i++) + for (i = 0; i < 4; i++) { linesizes[i] = FFALIGN(linesizes[i], align); + linesizes1[i] = linesizes[i]; + } - if ((ret = av_image_fill_pointers(pointers, pix_fmt, h, NULL, linesizes)) < 0) + if ((ret = av_image_fill_plane_sizes(sizes, pix_fmt, h, linesizes1)) < 0) return ret; - buf = av_malloc(ret + align); + total_size = align; + for (i = 0; i < 4; i++) { + if (total_size > SIZE_MAX - sizes[i]) + return AVERROR(EINVAL); + total_size += sizes[i]; + } + buf = av_malloc(total_size); if (!buf) return AVERROR(ENOMEM); if ((ret = av_image_fill_pointers(pointers, pix_fmt, h, buf, linesizes)) < 0) { @@ -220,6 +254,7 @@ int av_image_alloc(uint8_t *pointers[4], int linesizes[4], avpriv_set_systematic_pal2((uint32_t*)pointers[1], pix_fmt); if (align < 4) { av_log(NULL, AV_LOG_ERROR, "Formats with a palette require a minimum alignment of 4\n"); + av_free(buf); return AVERROR(EINVAL); } } @@ -431,9 +466,10 @@ int av_image_fill_arrays(uint8_t *dst_data[4], int dst_linesize[4], int av_image_get_buffer_size(enum AVPixelFormat pix_fmt, int width, int height, int align) { - uint8_t *data[4]; + int ret, i; int linesize[4]; - int ret; + ptrdiff_t aligned_linesize[4]; + size_t sizes[4]; const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(pix_fmt); if (!desc) return AVERROR(EINVAL); @@ -446,8 +482,24 @@ int av_image_get_buffer_size(enum AVPixelFormat pix_fmt, if (desc->flags & FF_PSEUDOPAL) return FFALIGN(width, align) * height; - return av_image_fill_arrays(data, linesize, NULL, pix_fmt, - width, height, align); + ret = av_image_fill_linesizes(linesize, pix_fmt, width); + if (ret < 0) + return ret; + + for (i = 0; i < 4; i++) + aligned_linesize[i] = FFALIGN(linesize[i], align); + + ret = av_image_fill_plane_sizes(sizes, pix_fmt, height, aligned_linesize); + if (ret < 0) + return ret; + + ret = 0; + for (i = 0; i < 4; i++) { + if (sizes[i] > INT_MAX - ret) + return AVERROR(EINVAL); + ret += sizes[i]; + } + return ret; } int av_image_copy_to_buffer(uint8_t *dst, int dst_size, diff --git a/libavutil/imgutils.h b/libavutil/imgutils.h index 5b790ecf0a..b09ca11e90 100644 --- a/libavutil/imgutils.h +++ b/libavutil/imgutils.h @@ -67,6 +67,17 @@ int av_image_get_linesize(enum AVPixelFormat pix_fmt, int width, int plane); */ int av_image_fill_linesizes(int linesizes[4], enum AVPixelFormat pix_fmt, int width); +/** + * Fill plane sizes for an image with pixel format pix_fmt and height height. + * + * @param size the array to be filled with the size of each image plane + * @param linesizes the array containing the linesize for each + * plane, should be filled by av_image_fill_linesizes() + * @return >= 0 in case of success, a negative error code otherwise + */ +int av_image_fill_plane_sizes(size_t size[4], enum AVPixelFormat pix_fmt, + int height, const ptrdiff_t linesizes[4]); + /** * Fill plane data pointers for an image with pixel format pix_fmt and * height height. diff --git a/libavutil/version.h b/libavutil/version.h index 3ce9b1831e..a63f79feb1 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -79,7 +79,7 @@ */ #define LIBAVUTIL_VERSION_MAJOR 56 -#define LIBAVUTIL_VERSION_MINOR 55 +#define LIBAVUTIL_VERSION_MINOR 56 #define LIBAVUTIL_VERSION_MICRO 100 #define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \ From patchwork Mon Jul 13 17:09:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Kim X-Patchwork-Id: 20991 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id B414344BAD2 for ; Mon, 13 Jul 2020 20:10:03 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9E3C868A0BE; Mon, 13 Jul 2020 20:10:03 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id F3A2E689F05 for ; Mon, 13 Jul 2020 20:09:56 +0300 (EEST) Received: by mail-yb1-f202.google.com with SMTP id u64so17645340ybf.13 for ; Mon, 13 Jul 2020 10:09:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=cScOrf5yw1fhKdnCsZegpa21iGFMqbKRqKrgygcJcNU=; b=P+FprLniACBKpRfR00zbn1XoOFYA2ka+L3ro8/CCRtLekCtxI/R5Eer+kJQuVVBDH1 0dXq+x7w9AkTAi1T+RGsFT9zvJwsKxdp+5+/XLwLTmbpoLtB0mD9qjvV9s0qImtcUYPW OPHDtTdQ8ttZy+G9TmqdlhH+q2ej9ZFmbXWH+GHINV4MZJJIkGDvq+K+7raJ8Vm+0aMO Efua4v/0jW654crBEePx7d5QZsNAqDGkTmh7SC7i58Kq1KueLDyUe0wzE2iQLCJ4iyAO Q3KOCn2/3Bq5dlX8UOyuf77XZd9fEtBAenDO5j994C3lRvT13opFjLNwTv1Vevyut1/Z zGxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=cScOrf5yw1fhKdnCsZegpa21iGFMqbKRqKrgygcJcNU=; b=XcBX5ZZo/FPhOTOwLTSfpNLpcARUy0CjiJopmvg2d4irWbI+H2JXvu3JgdzQfkfZjo TIIK8iS+DyRX1VOmGmyZ+fUZojxRveerzCT8ozVBUpmzThDb2wvT41X5JdhS6UYtzClJ bd+8cay9DPoGrJZNNtyEcgbPDtNDOF+iRQIDi+lt3+a5zlESm4ZTgsa71tnaws4o45Hs 5NY7g2N/XgigBx/K1XmOBfzJGsolFMszLf4H9Q187kW1hUBxPKFc/Ehaqaitye1ZYMAw Cj1UsnkRT6YUUxbkQ628q1vtPz6Pm+ns6QubPnyYkYZhbz6doOItPl0/6t7dTMPVe88R Xxow== X-Gm-Message-State: AOAM531gQQbuuyuvU/c2wLt2PDRXoeZVWlzQcx4/pVQInKy9es6YfBX/ Kd4NZBX2YKyYcbOdudhVTgrN1Ehh59VM/QCRjc/j3423LNLTMkdXeBFsmbg4K7yPQnCbIb7sUR9 Frs3KZSVHUomebCeiqD3a0YuxrvJrWFhyyDo8fQ0BpKRaNZhAs5Lm4RGduRYV X-Google-Smtp-Source: ABdhPJxqQ9KiKQ+Eak1TzsYxYvYC+9cbihZxQe1uqibF1oIntkQEbEqR67Q0FLvF/Ug5zEcIfsnijZhk8A== X-Received: by 2002:a25:d081:: with SMTP id h123mr1479666ybg.88.1594660195464; Mon, 13 Jul 2020 10:09:55 -0700 (PDT) Date: Mon, 13 Jul 2020 10:09:38 -0700 In-Reply-To: <11b6ffd9e674f497508f1b7cbad1dee4284f78c7.1594660141.git.bkkim@google.com> Message-Id: <91953b81c47f87fc10001298e608e57c3453237b.1594660141.git.bkkim@google.com> Mime-Version: 1.0 References: <11b6ffd9e674f497508f1b7cbad1dee4284f78c7.1594660141.git.bkkim@google.com> X-Mailer: git-send-email 2.27.0.389.gc38d7665816-goog From: Brian Kim To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v3 2/4] libavutil/frame: avoid UB when getting plane sizes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Brian Kim Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This uses av_image_fill_plane_sizes instead of av_image_fill_pointers when we are getting plane sizes to avoid UB from adding offsets to NULL. Signed-off-by: Brian Kim --- libavutil/frame.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/libavutil/frame.c b/libavutil/frame.c index 9884eae054..3ab1aa3242 100644 --- a/libavutil/frame.c +++ b/libavutil/frame.c @@ -212,8 +212,10 @@ void av_frame_free(AVFrame **frame) static int get_video_buffer(AVFrame *frame, int align) { const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(frame->format); - int ret, i, padded_height; + int ret, i, padded_height, total_size; int plane_padding = FFMAX(16 + 16/*STRIDE_ALIGN*/, align); + ptrdiff_t linesizes[4]; + size_t sizes[4]; if (!desc) return AVERROR(EINVAL); @@ -238,12 +240,22 @@ static int get_video_buffer(AVFrame *frame, int align) frame->linesize[i] = FFALIGN(frame->linesize[i], align); } + for (i = 0; i < 4; i++) + linesizes[i] = frame->linesize[i]; + padded_height = FFALIGN(frame->height, 32); - if ((ret = av_image_fill_pointers(frame->data, frame->format, padded_height, - NULL, frame->linesize)) < 0) + if ((ret = av_image_fill_plane_sizes(sizes, frame->format, + padded_height, linesizes)) < 0) return ret; - frame->buf[0] = av_buffer_alloc(ret + 4*plane_padding); + total_size = 4*plane_padding; + for (i = 0; i < 4; i++) { + if (sizes[i] > INT_MAX - total_size) + return AVERROR(EINVAL); + total_size += sizes[i]; + } + + frame->buf[0] = av_buffer_alloc(total_size); if (!frame->buf[0]) { ret = AVERROR(ENOMEM); goto fail; From patchwork Mon Jul 13 17:09:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Kim X-Patchwork-Id: 20992 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 851E844BAD2 for ; Mon, 13 Jul 2020 20:10:04 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7437168A3AF; Mon, 13 Jul 2020 20:10:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B466868A494 for ; Mon, 13 Jul 2020 20:09:58 +0300 (EEST) Received: by mail-yb1-f201.google.com with SMTP id e81so7516832ybb.3 for ; Mon, 13 Jul 2020 10:09:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qf1cClQM3nz5RMEA+kWr2X2NwGnsdbqix4LUlmbJU78=; b=hRob738JTDHjH6uzSX8p+lKLrY6erpNiQ/txNPcJt8kWNDaRlUZB05rUttxgrnfBS0 7S21N218Mdss4+4H4FrGjxIJBz8CRHJ9zK+NVS7EPxIjGHnzbsqUj8AGMUUV3u4O9qST dxjcn3pCqkN+U/f8tpc+nL2VmpPxwozkw/tZBngOWZ9XQ72j7HV6ndh0ER4S1kca5Z2X r1L10Pi/IhJ06K7fcTt4QT6/hIHRvZGti7odBlMMB0JNU2SDZ4iAJ8WE0Qc8LiQh6EnN VK/H4s+I8hG7luMYXX+5HxSF96zWvtZqywVIbdaJxyl5Acd7Av66IQj88jOcorEN6Vmk 6Fhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qf1cClQM3nz5RMEA+kWr2X2NwGnsdbqix4LUlmbJU78=; b=o1YMB6lFiZNURxshnZL6Q9fBi3iX+biizih8MELj/z2p7QaWwqQhHZYOI5eqVr4HDr w+atQpH4Za5F+Z89xYseiau/V/0zJtAmYEUElWStH9F6FVhHfblnbxDAJXAOtIPTn5lc R0JYAuWKRwHrxwMl0paNYczg3XD7jwQk3cRvegojfCpnBUmXnZzNEezb//gcytfsymeD m3Pfm2I8ayjq+617N1QoZn242rhM74xNgoq4Ad/RnkBPAGdXBBGBgtlINVF0eFzAlwqN t3PY6dpKUqePQTv6w2DDuAQGvk/1L72UZXTADTBEF/08EIXtr21aH9Af3ymwp+lDOszA ZAmw== X-Gm-Message-State: AOAM5305bDi5wqVow9sRRD04nHjXaZ7uTeEl0bA1gnPGvY5He81M4nkO PVcaBDjpKJnnR98lcEFxmrhffd2X80Oz9AxKU4DVsMx7NBajK+dv2SkoODBylUCoKwyVIFS9B83 2kEhpJLDM9xWzcVBHN3SwiCtocgCKUd3Bf/RlVUfdwbY//tKYReyAP7Zc1//C X-Google-Smtp-Source: ABdhPJwYTpJmWIffiu3f2YW5DSaBYsRfgWXmdb27uwTF1TvpDKeTDUMhU5e1VAzvXK8E3IvFL//osV2V7w== X-Received: by 2002:a25:8808:: with SMTP id c8mr1489628ybl.422.1594660197262; Mon, 13 Jul 2020 10:09:57 -0700 (PDT) Date: Mon, 13 Jul 2020 10:09:39 -0700 In-Reply-To: <11b6ffd9e674f497508f1b7cbad1dee4284f78c7.1594660141.git.bkkim@google.com> Message-Id: <7075f517618e9ebe0de46857d5d73e71f3b76470.1594660141.git.bkkim@google.com> Mime-Version: 1.0 References: <11b6ffd9e674f497508f1b7cbad1dee4284f78c7.1594660141.git.bkkim@google.com> X-Mailer: git-send-email 2.27.0.389.gc38d7665816-goog From: Brian Kim To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v3 3/4] libavcodec/decode: avoid UB when getting plane sizes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Brian Kim Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This uses av_image_fill_plane_sizes instead of av_image_fill_pointers when we are getting plane sizes to avoid UB from adding offsets to NULL. Signed-off-by: Brian Kim --- libavcodec/decode.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/libavcodec/decode.c b/libavcodec/decode.c index de9c079f9d..8ea9293ebf 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -1471,12 +1471,12 @@ static int update_frame_pool(AVCodecContext *avctx, AVFrame *frame) switch (avctx->codec_type) { case AVMEDIA_TYPE_VIDEO: { - uint8_t *data[4]; int linesize[4]; - int size[4] = { 0 }; int w = frame->width; int h = frame->height; - int tmpsize, unaligned; + int unaligned; + ptrdiff_t tmpsize, linesize1[4]; + size_t sizes[4]; avcodec_align_dimensions2(avctx, &w, &h, pool->stride_align); @@ -1494,21 +1494,20 @@ static int update_frame_pool(AVCodecContext *avctx, AVFrame *frame) unaligned |= linesize[i] % pool->stride_align[i]; } while (unaligned); - tmpsize = av_image_fill_pointers(data, avctx->pix_fmt, h, - NULL, linesize); - if (tmpsize < 0) { - ret = tmpsize; + for (i = 0; i < 4; i++) + linesize1[i] = linesize[i]; + ret = av_image_fill_plane_sizes(sizes, avctx->pix_fmt, h, linesize1); + if (ret < 0) goto fail; - } - - for (i = 0; i < 3 && data[i + 1]; i++) - size[i] = data[i + 1] - data[i]; - size[i] = tmpsize - (data[i] - data[0]); for (i = 0; i < 4; i++) { pool->linesize[i] = linesize[i]; - if (size[i]) { - pool->pools[i] = av_buffer_pool_init(size[i] + 16 + STRIDE_ALIGN - 1, + if (sizes[i]) { + if (sizes[i] > INT_MAX - (16 + STRIDE_ALIGN - 1)) { + ret = AVERROR(EINVAL); + goto fail; + } + pool->pools[i] = av_buffer_pool_init(sizes[i] + 16 + STRIDE_ALIGN - 1, CONFIG_MEMORY_POISONING ? NULL : av_buffer_allocz); From patchwork Mon Jul 13 17:09:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Kim X-Patchwork-Id: 20993 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 63BA244BAD2 for ; Mon, 13 Jul 2020 20:10:08 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4632668A494; Mon, 13 Jul 2020 20:10:08 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 25E8F68A345 for ; Mon, 13 Jul 2020 20:10:01 +0300 (EEST) Received: by mail-pl1-f202.google.com with SMTP id f2so7619722plt.2 for ; Mon, 13 Jul 2020 10:10:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=FXjRTZaUo86ggwwhs+uYxJI4OUZP0c8TGK+wd81c7nk=; b=pPJsmEE45wKk2wPJ2lqiHLANElapJ4c8rQR9zoCxzi/GjSNH1T7wKZXM1Yh4Qvgeca CCLt/792COyMSFvioqMUseGP6WixuQ7Bj5+SEVzbiqix0E1BITECCUnjCUnzP3yoYr/g OvsCHZqgSoC+93Ed4IcP7aeibHTj35ns6fsONn6+oP7vrTp+/Zgf5q2U7RM44gTrRmJB wF2SdIDjq9V0CYcq4pfaHUtsJNgLZQlRjE/3V4QEe6tP74AUYt0NrZdi+Qp0pKy4cOCE 530ozi+BXzcHTW9gQh8WPqr41n4fPhTAVHcAEBwr3qo3wIj7sQXogMyOYbvsaJ2pM25v LFVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=FXjRTZaUo86ggwwhs+uYxJI4OUZP0c8TGK+wd81c7nk=; b=GYldLbqbfur3QwnDQPjM+9ipTXbeRrlIuKIU1CBE2fyiCCVIDe+3YfLeOpcXxIkUin KJY6aSEG9eeMWQl3NEy85QgusAvmjPqZ8Yz9yjBsPq2Kx8NeO1LLblW+u/ccdoatg+Tw vss5y++ai9xRTx18h5B9ry/EtyxOUTo826iUISQmF4lAj53primrnZvmaYdV/ouzh1zx zSHUlVIf4ETtJoanw0Fh0BEBvpc9/z9Hgsb/Ty9GgWUJnS/1gbBdr/6nQkQFgBjKuAki VxYVGAOygesuao4ybq+1IBAScCl1OSFKE5W8BmYERTjjFK200g1bC7WsmwO5vRotalt/ CS6A== X-Gm-Message-State: AOAM532xbB/rSQxz7eS5JpLBBhnJyNB5+5F/UHET3MTAsOSNHGPoi0ri uyKHwgnAER2ZeN0RnCSsD+VZUFkp6khriu9Vs0DkikLoY7A56n2WP5Qp68W1kL4aUdR9ESoE8wu akmhA7+krYGtyoLo7bOjTVDSjUIhdrzOxnuNeM8cCaf3/14eROFk6JHUQCljt X-Google-Smtp-Source: ABdhPJxuVAqoEw3/iqo3LYt36wJay/LMnoJ5gt3FIekGcuk5V43hN1I4D5tXOALwjgcuxT4Iv19QEGENDg== X-Received: by 2002:a62:8482:: with SMTP id k124mr732143pfd.285.1594660199134; Mon, 13 Jul 2020 10:09:59 -0700 (PDT) Date: Mon, 13 Jul 2020 10:09:40 -0700 In-Reply-To: <11b6ffd9e674f497508f1b7cbad1dee4284f78c7.1594660141.git.bkkim@google.com> Message-Id: Mime-Version: 1.0 References: <11b6ffd9e674f497508f1b7cbad1dee4284f78c7.1594660141.git.bkkim@google.com> X-Mailer: git-send-email 2.27.0.389.gc38d7665816-goog From: Brian Kim To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v3 4/4] libavutil/imgutils: check for non-null buffer in av_image_fill_pointers X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Brian Kim Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" We were previously always filling data by adding offsets to ptr, which was undefined behavior when ptr was NULL. Signed-off-by: Brian Kim --- libavutil/imgutils.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/libavutil/imgutils.c b/libavutil/imgutils.c index 345b7fa94c..721dc2784a 100644 --- a/libavutil/imgutils.c +++ b/libavutil/imgutils.c @@ -166,6 +166,9 @@ int av_image_fill_pointers(uint8_t *data[4], enum AVPixelFormat pix_fmt, int hei memset(data , 0, sizeof(data[0])*4); + if (!ptr) + return ret; + data[0] = ptr; for (i = 1; i < 4 && sizes[i - 1] > 0; i++) data[i] = data[i - 1] + sizes[i - 1];