From patchwork Mon Jul 13 17:09:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Kim X-Patchwork-Id: 20990 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 9ABEA44BAD2 for ; Mon, 13 Jul 2020 20:10:02 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 702AE68A295; Mon, 13 Jul 2020 20:10:02 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0486C687F65 for ; Mon, 13 Jul 2020 20:09:54 +0300 (EEST) Received: by mail-yb1-f202.google.com with SMTP id d202so17891479ybh.12 for ; Mon, 13 Jul 2020 10:09:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=J+t6ijdSq79BUlFBhKYJal4TwsxKXLpCYKPd2hZFCTg=; b=fRNS3yQJ6Q26xsi9B3GDM1cYs4fu8I1+yFojHn0jr11BA1I2MLUXs39LPMunsDvCHJ /12SrWko80jasAOF30uIqpByz1+RPYuav6s4Jaleo6GwUc9YtfRXmwxXPADwezjrGP/Q 5oL5XyIHQHtSAQ00uxvDG8la2PE9pV9UmSHTmEMRgvgHqXgMEU94rIG8hCVFTpPmUZLU 1UPDrTiqUedOGOppHZz9JTtKhKJ1uf2RVFmo3+ZWRhlq8HVHHTVER6Sewd+NcCVxL4nm iYY9rCZroPlPwevdrukTMNq5unr/+CcMW7lKWqsg/EZs1dUJhTHkTQysxFXKM8dQPT8i sZ7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=J+t6ijdSq79BUlFBhKYJal4TwsxKXLpCYKPd2hZFCTg=; b=OA/+cDzbglGBhHIkqzy+i39YqzDedA09rb9Qjd21zIHU4eouctKGQZ+lSl5EzmPfoB 2WR6u+RZRx7PERUMkSCgTLdjiE+gtxxAPZSew/vCu8b+VOzhdtZPc2px7iuhizvQwqBg wz7ApnbmigyVyx6/qVYOHS553rCQdyjzpK2j3CkiganLlhqy5TwjG5DiRm2hUHrK45Du y7WlxYZu/GFncZBPZwZ8VwwWdwUCZyKfqM5KGDooYb/VuwlDYOFshET6zuZd/IeOC29D 6jreSvhxeoORLSozsgr24YlAp6pL8Ff1rUsQ/u/QLkwdXmrz3/0u36wGw7H4BgjblGPc CorQ== X-Gm-Message-State: AOAM533upPT8BF9nqHyecuScVd9+WiIu1atGzAU+wDcv5mzIklQ3pqJT EreETRDrUGDsUJ7BEp5ANO129W5JQ4gF2EfsPlV3LldgWNnhCFoBz7BgrNRpobR9juf87sZDlng jcGcEFHLkCEGnKW5j6ww71iPPvoWkkm3n3fPSmEdQ47kxoqx45KWIqlKUhzB8 X-Google-Smtp-Source: ABdhPJz5dl2LhfI8hP1eaJ/RD3AJdT8RMYMxXTZDtT83gSbIxuHq0GMORFoX0CU4DHbC1i+kwP/g+tXpTw== X-Received: by 2002:a25:9302:: with SMTP id f2mr1527118ybo.238.1594660193473; Mon, 13 Jul 2020 10:09:53 -0700 (PDT) Date: Mon, 13 Jul 2020 10:09:37 -0700 Message-Id: <11b6ffd9e674f497508f1b7cbad1dee4284f78c7.1594660141.git.bkkim@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.27.0.389.gc38d7665816-goog From: Brian Kim To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v3 1/4] libavutil/imgutils: add utility to get plane sizes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Brian Kim Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This utility helps avoid undefined behavior when doing things like checking how much memory we need to allocate for an image before we have allocated a buffer. Signed-off-by: Brian Kim --- doc/APIchanges | 3 ++ libavutil/imgutils.c | 98 +++++++++++++++++++++++++++++++++----------- libavutil/imgutils.h | 11 +++++ libavutil/version.h | 2 +- 4 files changed, 90 insertions(+), 24 deletions(-) diff --git a/doc/APIchanges b/doc/APIchanges index 1d6cc36b8c..44defd9ca8 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -15,6 +15,9 @@ libavutil: 2017-10-21 API changes, most recent first: +2020-07-xx - xxxxxxxxxx - lavu 56.56.100 - imgutils.h + Add av_image_fill_plane_sizes(). + 2020-06-12 - b09fb030c1 - lavu 56.55.100 - pixdesc.h Add AV_PIX_FMT_X2RGB10. diff --git a/libavutil/imgutils.c b/libavutil/imgutils.c index 7f9c1b632c..345b7fa94c 100644 --- a/libavutil/imgutils.c +++ b/libavutil/imgutils.c @@ -108,45 +108,69 @@ int av_image_fill_linesizes(int linesizes[4], enum AVPixelFormat pix_fmt, int wi return 0; } -int av_image_fill_pointers(uint8_t *data[4], enum AVPixelFormat pix_fmt, int height, - uint8_t *ptr, const int linesizes[4]) +int av_image_fill_plane_sizes(size_t sizes[4], enum AVPixelFormat pix_fmt, + int height, const ptrdiff_t linesizes[4]) { - int i, total_size, size[4] = { 0 }, has_plane[4] = { 0 }; + int i, has_plane[4] = { 0 }; const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(pix_fmt); - memset(data , 0, sizeof(data[0])*4); + memset(sizes , 0, sizeof(sizes[0])*4); if (!desc || desc->flags & AV_PIX_FMT_FLAG_HWACCEL) return AVERROR(EINVAL); - data[0] = ptr; - if (linesizes[0] > (INT_MAX - 1024) / height) + if (linesizes[0] > SIZE_MAX / height) return AVERROR(EINVAL); - size[0] = linesizes[0] * height; + sizes[0] = linesizes[0] * height; if (desc->flags & AV_PIX_FMT_FLAG_PAL || desc->flags & FF_PSEUDOPAL) { - data[1] = ptr + size[0]; /* palette is stored here as 256 32 bits words */ - return size[0] + 256 * 4; + sizes[1] = 256 * 4; /* palette is stored here as 256 32 bits words */ + return 0; } for (i = 0; i < 4; i++) has_plane[desc->comp[i].plane] = 1; - total_size = size[0]; for (i = 1; i < 4 && has_plane[i]; i++) { int h, s = (i == 1 || i == 2) ? desc->log2_chroma_h : 0; - data[i] = data[i-1] + size[i-1]; h = (height + (1 << s) - 1) >> s; - if (linesizes[i] > INT_MAX / h) + if (linesizes[i] > SIZE_MAX / h) return AVERROR(EINVAL); - size[i] = h * linesizes[i]; - if (total_size > INT_MAX - size[i]) + sizes[i] = h * linesizes[i]; + } + + return 0; +} + +int av_image_fill_pointers(uint8_t *data[4], enum AVPixelFormat pix_fmt, int height, + uint8_t *ptr, const int linesizes[4]) +{ + int i, ret; + ptrdiff_t linesizes1[4]; + size_t sizes[4]; + + for (i = 0; i < 4; i++) + linesizes1[i] = linesizes[i]; + + ret = av_image_fill_plane_sizes(sizes, pix_fmt, height, linesizes1); + if (ret < 0) + return ret; + + ret = 0; + for (i = 0; i < 4; i++) { + if (sizes[i] > INT_MAX - ret) return AVERROR(EINVAL); - total_size += size[i]; + ret += sizes[i]; } - return total_size; + memset(data , 0, sizeof(data[0])*4); + + data[0] = ptr; + for (i = 1; i < 4 && sizes[i - 1] > 0; i++) + data[i] = data[i - 1] + sizes[i - 1]; + + return ret; } int avpriv_set_systematic_pal2(uint32_t pal[256], enum AVPixelFormat pix_fmt) @@ -194,6 +218,8 @@ int av_image_alloc(uint8_t *pointers[4], int linesizes[4], { const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(pix_fmt); int i, ret; + ptrdiff_t linesizes1[4]; + size_t total_size, sizes[4]; uint8_t *buf; if (!desc) @@ -204,12 +230,20 @@ int av_image_alloc(uint8_t *pointers[4], int linesizes[4], if ((ret = av_image_fill_linesizes(linesizes, pix_fmt, align>7 ? FFALIGN(w, 8) : w)) < 0) return ret; - for (i = 0; i < 4; i++) + for (i = 0; i < 4; i++) { linesizes[i] = FFALIGN(linesizes[i], align); + linesizes1[i] = linesizes[i]; + } - if ((ret = av_image_fill_pointers(pointers, pix_fmt, h, NULL, linesizes)) < 0) + if ((ret = av_image_fill_plane_sizes(sizes, pix_fmt, h, linesizes1)) < 0) return ret; - buf = av_malloc(ret + align); + total_size = align; + for (i = 0; i < 4; i++) { + if (total_size > SIZE_MAX - sizes[i]) + return AVERROR(EINVAL); + total_size += sizes[i]; + } + buf = av_malloc(total_size); if (!buf) return AVERROR(ENOMEM); if ((ret = av_image_fill_pointers(pointers, pix_fmt, h, buf, linesizes)) < 0) { @@ -220,6 +254,7 @@ int av_image_alloc(uint8_t *pointers[4], int linesizes[4], avpriv_set_systematic_pal2((uint32_t*)pointers[1], pix_fmt); if (align < 4) { av_log(NULL, AV_LOG_ERROR, "Formats with a palette require a minimum alignment of 4\n"); + av_free(buf); return AVERROR(EINVAL); } } @@ -431,9 +466,10 @@ int av_image_fill_arrays(uint8_t *dst_data[4], int dst_linesize[4], int av_image_get_buffer_size(enum AVPixelFormat pix_fmt, int width, int height, int align) { - uint8_t *data[4]; + int ret, i; int linesize[4]; - int ret; + ptrdiff_t aligned_linesize[4]; + size_t sizes[4]; const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(pix_fmt); if (!desc) return AVERROR(EINVAL); @@ -446,8 +482,24 @@ int av_image_get_buffer_size(enum AVPixelFormat pix_fmt, if (desc->flags & FF_PSEUDOPAL) return FFALIGN(width, align) * height; - return av_image_fill_arrays(data, linesize, NULL, pix_fmt, - width, height, align); + ret = av_image_fill_linesizes(linesize, pix_fmt, width); + if (ret < 0) + return ret; + + for (i = 0; i < 4; i++) + aligned_linesize[i] = FFALIGN(linesize[i], align); + + ret = av_image_fill_plane_sizes(sizes, pix_fmt, height, aligned_linesize); + if (ret < 0) + return ret; + + ret = 0; + for (i = 0; i < 4; i++) { + if (sizes[i] > INT_MAX - ret) + return AVERROR(EINVAL); + ret += sizes[i]; + } + return ret; } int av_image_copy_to_buffer(uint8_t *dst, int dst_size, diff --git a/libavutil/imgutils.h b/libavutil/imgutils.h index 5b790ecf0a..b09ca11e90 100644 --- a/libavutil/imgutils.h +++ b/libavutil/imgutils.h @@ -67,6 +67,17 @@ int av_image_get_linesize(enum AVPixelFormat pix_fmt, int width, int plane); */ int av_image_fill_linesizes(int linesizes[4], enum AVPixelFormat pix_fmt, int width); +/** + * Fill plane sizes for an image with pixel format pix_fmt and height height. + * + * @param size the array to be filled with the size of each image plane + * @param linesizes the array containing the linesize for each + * plane, should be filled by av_image_fill_linesizes() + * @return >= 0 in case of success, a negative error code otherwise + */ +int av_image_fill_plane_sizes(size_t size[4], enum AVPixelFormat pix_fmt, + int height, const ptrdiff_t linesizes[4]); + /** * Fill plane data pointers for an image with pixel format pix_fmt and * height height. diff --git a/libavutil/version.h b/libavutil/version.h index 3ce9b1831e..a63f79feb1 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -79,7 +79,7 @@ */ #define LIBAVUTIL_VERSION_MAJOR 56 -#define LIBAVUTIL_VERSION_MINOR 55 +#define LIBAVUTIL_VERSION_MINOR 56 #define LIBAVUTIL_VERSION_MICRO 100 #define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \