From patchwork Thu Jul 9 01:51:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Kim X-Patchwork-Id: 20904 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 2BC69449579 for ; Thu, 9 Jul 2020 04:58:13 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0C15268B504; Thu, 9 Jul 2020 04:58:13 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com [209.85.208.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8C9E06881A5 for ; Thu, 9 Jul 2020 04:58:06 +0300 (EEST) Received: by mail-lj1-f175.google.com with SMTP id f5so495968ljj.10 for ; Wed, 08 Jul 2020 18:58:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=ZFCvR7iRJ5KEdoeS+BVaiHzdxmTq8vIgrTlBrosQnis=; b=aXvkYgkSBR8jUI0cUtDMHAb+oT18+tpDJlw+C1TnAdASZ8JvCZi50NkthZ/n7IEgW6 ebRO9glRpplEDSIe3Q7whu/oyBz7g8sSKI27pfliDYiZneTDb3dFmIC9+uYe0ZI2zAk7 qGfZ5DWri9rS85YSRT/La5bO3nAHiL8w2pKUZx7mmwizaIHFTV16zv/XT+oeU+rfW2/S ttN018Oq/6j95aFyMKcvewXoFz++JUv+iBdqbVLhVuJ5h8NC+SwDWiezHg0vGQcj047m pBKTP40PBMNqk2n/2lliZCVGVrMwyACAawt7qE8HVSV7jOP1CJxYxoNaJuzrRQjIYWlH Jrcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ZFCvR7iRJ5KEdoeS+BVaiHzdxmTq8vIgrTlBrosQnis=; b=L/7OWRlJYM3Qw5W4fDdAUDhxBxAvmI2yHlEHMx2uquBqrvnro5+khyctr8QIfy1jy0 Bw3HowjYfqVmdtMDSaofPGysSlpAeohpKUMzFruVkYw4dKuraOLcEvGMScrO9K72lsBr HIVMKRRaCTJ9/2iNM32y0Y4Giun0YB5qcshsEnKVrwVswB+aDPWFQkrG/8U9RhUpsBf7 QGksVToLHxqH8y5PRFMwAjVBKN0N0t3aHNQqgYiXeUcW1p189ZVynVNj4fYBuojDsluw pLun/cHI2X0i8eGLRcMnED6eYO57ML08D7kO1KK2IAZNfYl8NgIrhszJ20Pe7GtrPJ6T VCHg== X-Gm-Message-State: AOAM531spcCPYPzNEeslKQW/0nI6JcvL7+KNnJisDRuUWYnbSc0XttXj iz6nUdvmRRLPXiHVzNZR3hJWFMku8qhyVk+2VgdPm9CP72w= X-Google-Smtp-Source: ABdhPJzORa94Mmus0DwShonxOAyoFCVZAxaAlM71sMXecwddDCgSapHxmG9Oyp39k6dvTbglNlM7F0rd1sfkct1z+L4= X-Received: by 2002:a17:907:20ba:: with SMTP id pw26mr53039632ejb.425.1594259519982; Wed, 08 Jul 2020 18:51:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Brian Kim Date: Wed, 8 Jul 2020 18:51:49 -0700 Message-ID: To: FFmpeg development discussions and patches Subject: [FFmpeg-devel] [PATCH v2 2/3] libavcodec/decode: avoid UB when getting plane sizes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Patch attached Subject: [PATCH 2/3] libavcodec/decode: avoid UB when getting plane sizes This uses av_image_fill_plane_sizes instead of av_image_fill_pointers when we are getting plane sizes to avoid UB from adding offsets to NULL. Signed-off-by: Brian Kim --- libavcodec/decode.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/libavcodec/decode.c b/libavcodec/decode.c index de9c079f9d..c1753cfffb 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -1471,12 +1471,12 @@ static int update_frame_pool(AVCodecContext *avctx, AVFrame *frame) switch (avctx->codec_type) { case AVMEDIA_TYPE_VIDEO: { - uint8_t *data[4]; int linesize[4]; - int size[4] = { 0 }; int w = frame->width; int h = frame->height; - int tmpsize, unaligned; + int unaligned; + ptrdiff_t tmpsize, linesize1[4]; + size_t size[4]; avcodec_align_dimensions2(avctx, &w, &h, pool->stride_align); @@ -1494,20 +1494,22 @@ static int update_frame_pool(AVCodecContext *avctx, AVFrame *frame) unaligned |= linesize[i] % pool->stride_align[i]; } while (unaligned); - tmpsize = av_image_fill_pointers(data, avctx->pix_fmt, h, - NULL, linesize); + for (i = 0; i < 4; i++) + linesize1[i] = linesize[i]; + tmpsize = av_image_fill_plane_sizes(size, avctx->pix_fmt, h, + linesize1); if (tmpsize < 0) { ret = tmpsize; goto fail; } - for (i = 0; i < 3 && data[i + 1]; i++) - size[i] = data[i + 1] - data[i]; - size[i] = tmpsize - (data[i] - data[0]); - for (i = 0; i < 4; i++) { pool->linesize[i] = linesize[i]; if (size[i]) { + if (size[i] > INT_MAX - (16 + STRIDE_ALIGN - 1)) { + ret = AVERROR(EINVAL); + goto fail; + } pool->pools[i] = av_buffer_pool_init(size[i] + 16 + STRIDE_ALIGN - 1, CONFIG_MEMORY_POISONING ? NULL :