From patchwork Tue Feb 15 09:58:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiasheng Jiang X-Patchwork-Id: 34303 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6838:90eb:0:0:0:0 with SMTP id a11csp146657nkf; Tue, 15 Feb 2022 01:58:51 -0800 (PST) X-Google-Smtp-Source: ABdhPJz5yZp5ib8E+wTEOFuoTicyJi8cM3kPh2WM1lOob6Ut41gy1gelYxFmvePrJyy63pifIk1U X-Received: by 2002:a17:906:77cb:: with SMTP id m11mr2456425ejn.116.1644919131464; Tue, 15 Feb 2022 01:58:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1644919131; cv=none; d=google.com; s=arc-20160816; b=RidhscydXUu0ATP10RGFOQRTPwtr/kti+b6K9TckGnUUjYApvNm6Ilx+Q5WcraQpMJ V+APUTsWXtiMD/XDrotagbqLdh6buBVjHpIOjS5pc7jSPbTF/WwT+QG0Ehuybv4Z+Ipt N4vl5ing6qzpw3lBB0nMjM3BJfPOWyv+DggCP6EaGluSZT4VaemVt08sRpWXswA/Jjxs 5dP5QWnRpgkubzgkrIaepDowqWZIUCHci6AgtiEWLRasU2LER/OgZPluIxQI2trl60pg hiDsAA7zbTNGHKUdqCYbgF0mmcIGxlz98VedM7X3XxIsIEUYF7UQzAncUNuPlB46p9H5 Hr4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=VywIzPw89GfatLnyn4nYY/DnGR1exYeSrGszaPP7cVA=; b=CM6f6TO/Cn+yCYiQ+uGn4SjJ/fvb1jlQJfdhdHgW1/A0FqERy0apxMZQfQkvvB3YYu s+2Fk4Oeq+mgodYZAk3ef4nNe0j7e6TUqnMqFhVUUZ7QvikskwmmRvGWoRk4uOfaM1hK e/lEJzzPZMlRwmpPAUoJukQtD6gxQZwDT78NYcFlm7CmIGdUlNCU2BzsJKcfqtsU9p81 AohkA4p36/ZzbeHc5dupNwWclrkGKLBt4mD9Op+/qBfe/gHAYc5a3ZzK22SOQders7H7 1eJYcfzq3BJjnJc23KhwL/m7nLB3iG0b+/V96MOltgIEqQmQAoFhuEuhB46m7FItRW0n bwBw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a65si4159284edf.60.2022.02.15.01.58.51; Tue, 15 Feb 2022 01:58:51 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 02CE268B2D5; Tue, 15 Feb 2022 11:58:49 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from cstnet.cn (smtp21.cstnet.cn [159.226.251.21]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4865168B21E for ; Tue, 15 Feb 2022 11:58:41 +0200 (EET) Received: from localhost.localdomain (unknown [124.16.138.126]) by APP-01 (Coremail) with SMTP id qwCowACX3sZOeQtioszZAA--.64169S2; Tue, 15 Feb 2022 17:58:38 +0800 (CST) From: Jiasheng Jiang To: ffmpeg-devel@ffmpeg.org Date: Tue, 15 Feb 2022 17:58:35 +0800 Message-Id: <20220215095835.1672071-1-jiasheng@iscas.ac.cn> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-CM-TRANSID: qwCowACX3sZOeQtioszZAA--.64169S2 X-Coremail-Antispam: 1UD129KBjvJXoWxXFykCFy7Aw47Jr1kWFWfXwb_yoWrZr15pr 4rG3WUJ3s8JF47Cw1qgan5tr43J3W3uF4a9395KFyUta1Utrn5G3yIyFWS93WUCF1rJrWr ZF18tw1UA34vvrJanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUka14x267AKxVWUJVW8JwAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2ocxC64kIII0Yj41l84x0c7CEw4AK67xGY2AK02 1l84ACjcxK6xIIjxv20xvE14v26r4j6ryUM28EF7xvwVC0I7IYx2IY6xkF7I0E14v26r4j 6F4UM28EF7xvwVC2z280aVAFwI0_Cr1j6rxdM28EF7xvwVC2z280aVCY1x0267AKxVW0oV Cq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0 I7IYx2IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r 4UM4x0Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwCY02Avz4vE14v_GFWl 42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJV WUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1Y6r17MIIYrxkI7VAK I48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r1j6r 4UMIIF0xvE42xK8VAvwI8IcIk0rVW8JVW3JwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY 6I8E87Iv6xkF7I0E14v26r1j6r4UYxBIdaVFxhVjvjDU0xZFpf9x0JU6rWrUUUUU= X-Originating-IP: [124.16.138.126] X-CM-SenderInfo: pmld2xxhqjqxpvfd2hldfou0/ Subject: [FFmpeg-devel] [PATCH] swscale/utils: Add missing check for av_malloc X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Jiasheng Jiang Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: BjdhRSBadJ93 As the potential failure of the memory allocation, the return value of the av_malloc() could be NULL and be dereferenced on. Therefore it should be better to check it and return error if fails. Also, the callers of the ff_shuffle_filter_coefficients() should deal with the return value. Fixes: f900a19fa9 ("libswscale: Adds ff_hscale8to15_4_avx2 and ff_hscale8to15_X4_avx2 for all filter sizes.") Signed-off-by: Jiasheng Jiang --- libswscale/swscale_internal.h | 2 +- libswscale/utils.c | 13 ++++++++++--- tests/checkasm/sw_scale.c | 3 ++- 3 files changed, 13 insertions(+), 5 deletions(-) diff --git a/libswscale/swscale_internal.h b/libswscale/swscale_internal.h index 3a78d95ba6..26d28d42e6 100644 --- a/libswscale/swscale_internal.h +++ b/libswscale/swscale_internal.h @@ -1144,5 +1144,5 @@ void ff_sws_slice_worker(void *priv, int jobnr, int threadnr, #define MAX_LINES_AHEAD 4 //shuffle filter and filterPos for hyScale and hcScale filters in avx2 -void ff_shuffle_filter_coefficients(SwsContext *c, int* filterPos, int filterSize, int16_t *filter, int dstW); +int ff_shuffle_filter_coefficients(SwsContext *c, int* filterPos, int filterSize, int16_t *filter, int dstW); #endif /* SWSCALE_SWSCALE_INTERNAL_H */ diff --git a/libswscale/utils.c b/libswscale/utils.c index c5ea8853d5..7754a03e00 100644 --- a/libswscale/utils.c +++ b/libswscale/utils.c @@ -278,7 +278,7 @@ static const FormatEntry format_entries[] = { [AV_PIX_FMT_P416LE] = { 1, 1 }, }; -void ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSize, int16_t *filter, int dstW){ +int ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSize, int16_t *filter, int dstW){ #if ARCH_X86_64 int i, j, k, l; int cpu_flags = av_get_cpu_flags(); @@ -292,6 +292,9 @@ void ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSiz } if (filterSize > 4){ int16_t *tmp2 = av_malloc(dstW * filterSize * 2); + if (!tmp2) + return AVERROR(ENOMEM); + memcpy(tmp2, filter, dstW * filterSize * 2); for (i = 0; i < dstW; i += 16){//pixel for (k = 0; k < filterSize / 4; ++k){//fcoeff @@ -311,6 +314,8 @@ void ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSiz } } #endif + + return 0; } int sws_isSupportedInput(enum AVPixelFormat pix_fmt) @@ -1836,7 +1841,8 @@ av_cold int sws_init_context(SwsContext *c, SwsFilter *srcFilter, get_local_pos(c, 0, 0, 0), get_local_pos(c, 0, 0, 0))) < 0) goto fail; - ff_shuffle_filter_coefficients(c, c->hLumFilterPos, c->hLumFilterSize, c->hLumFilter, dstW); + if ((ret = ff_shuffle_filter_coefficients(c, c->hLumFilterPos, c->hLumFilterSize, c->hLumFilter, dstW)) < 0) + goto fail; if ((ret = initFilter(&c->hChrFilter, &c->hChrFilterPos, &c->hChrFilterSize, c->chrXInc, c->chrSrcW, c->chrDstW, filterAlign, 1 << 14, @@ -1846,7 +1852,8 @@ av_cold int sws_init_context(SwsContext *c, SwsFilter *srcFilter, get_local_pos(c, c->chrSrcHSubSample, c->src_h_chr_pos, 0), get_local_pos(c, c->chrDstHSubSample, c->dst_h_chr_pos, 0))) < 0) goto fail; - ff_shuffle_filter_coefficients(c, c->hChrFilterPos, c->hChrFilterSize, c->hChrFilter, c->chrDstW); + if ((ret = ff_shuffle_filter_coefficients(c, c->hChrFilterPos, c->hChrFilterSize, c->hChrFilter, c->chrDstW)) < 0) + goto fail; } } // initialize horizontal stuff diff --git a/tests/checkasm/sw_scale.c b/tests/checkasm/sw_scale.c index 3c0a083b42..0cb0ac4a4a 100644 --- a/tests/checkasm/sw_scale.c +++ b/tests/checkasm/sw_scale.c @@ -218,7 +218,8 @@ static void check_hscale(void) ff_sws_init_scale(ctx); memcpy(filterAvx2, filter, sizeof(uint16_t) * (SRC_PIXELS * MAX_FILTER_WIDTH + MAX_FILTER_WIDTH)); if ((cpu_flags & AV_CPU_FLAG_AVX2) && !(cpu_flags & AV_CPU_FLAG_SLOW_GATHER)) - ff_shuffle_filter_coefficients(ctx, filterPosAvx, width, filterAvx2, SRC_PIXELS); + if (ff_shuffle_filter_coefficients(ctx, filterPosAvx, width, filterAvx2, SRC_PIXELS) < 0) + fail(); if (check_func(ctx->hcScale, "hscale_%d_to_%d_width%d", ctx->srcBpc, ctx->dstBpc + 1, width)) { memset(dst0, 0, SRC_PIXELS * sizeof(dst0[0]));