From patchwork Thu Feb 17 10:03:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Kelly X-Patchwork-Id: 34355 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6838:d078:0:0:0:0 with SMTP id x24csp489785nkx; Thu, 17 Feb 2022 02:03:59 -0800 (PST) X-Google-Smtp-Source: ABdhPJwzeCH6RjzL61iRayDD5hT7srOZ8w5HVgLwY4DURTL/V50c1nb9jjBhePNsoPPJdcrfKOQQ X-Received: by 2002:a05:6402:cb:b0:410:8094:872b with SMTP id i11-20020a05640200cb00b004108094872bmr1742078edu.378.1645092239350; Thu, 17 Feb 2022 02:03:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645092239; cv=none; d=google.com; s=arc-20160816; b=Y1so8HKp7UYwLo5y4sv40VFL1D0fYlfTAX+CIvbbat0Gt665NkXZEHV4vFvbU9p8sh GUhkEFMThNQXCTD4cFrGMYWDDf8RBzq5LjBFrV57wXIEdYziw2Ytbewjc6HzMAO90+c0 WzQHPjHhcdJeqMGMY+Mg4cx7Cho6CkS30xumMIkprUmbiQ8MCvvq94ThOARr1oasHSYX B58me3VGQIN4DcdQ6WDi6FwLRxUYeOCUMDAZhr1h2de7AnsyaaF/BskZnW7P5W5ED70O vArFCtiwtxLM+xXOwr6DtuOmEBp4I6X6uHXzGOOGhYQUNi658xfWPJ1eXAEjdez2lMqq iVaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:from:mime-version:message-id:date :dkim-signature:delivered-to; bh=+NNplIaP0YmUZhzKdcDmapuWKat0wgdmai0RTCXV6EI=; b=sn8XZZRqqC4pvL65iNYKevKeDVWRJ3Q8trU/quuo01sv4jL52SLGKNuhF4KQd/Gevm RZivauotpOn5tNitwnptf4TZbzXdYoLKgNQWILOMqXVPcm2BW2YqfdVsL1Rnr62VsJvJ JaehdiPAe5zHjx1RUjorIjvitYFNyrUDotdgsxjNVzhIjQAlIHFl6PBiHtpbd6/n6jix 1A4H1b2HzYAB2/+VhMhfE16dNJbp8CSxY621r/R0eyt9Eg6IL6nKPfwfGUhORWocvQvu UI6HTHhs/9b9H4JywDTeJDIyZ3Uqj0E6e541oQAIV4IHnzZ9l6vJ59JQT19TStf5AN8e hY0A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20210112 header.b=cD+B5V4J; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b3si1365313ejl.768.2022.02.17.02.03.35; Thu, 17 Feb 2022 02:03:59 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20210112 header.b=cD+B5V4J; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 132FF68B1E0; Thu, 17 Feb 2022 12:03:33 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1821E68B07E for ; Thu, 17 Feb 2022 12:03:26 +0200 (EET) Received: by mail-wr1-f73.google.com with SMTP id j8-20020adfc688000000b001e3322ced69so2080175wrg.13 for ; Thu, 17 Feb 2022 02:03:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=z2bT2fPLvDXm891Tc56t27V5/fusQLjdKu3PLiPsC9I=; b=cD+B5V4Jlk8C0MotON8MDeKi3uPjVVU2yi2DKl0vaHkp/jis7o1dK0JYGGOLRJtvEv 9XYag0y4igO6A5pCtdEX9yKfJ8mE45g23DsMvTBGyVVhvc4dQ0Bp+nBkhVJReXhzMYi1 wzkeD0P+RW4NHIs1buG+MR4+wSfpWuiAf4PClF1a7eLV20z1iK21+1xLrXSiyg3u3xBr K5b3G2xdOwmza3QL0x/bla18BYVA+VCxm8bwYMe8XmqEfq/vYeqfWIKtaw+goRnj9W9N 9GzLzO1BDE+JszVRf1G+dfbNBGhiUyD1x/XhnK+eAMUNqTlIX7CkKvbNfdqKvYbl8L5q f7pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=z2bT2fPLvDXm891Tc56t27V5/fusQLjdKu3PLiPsC9I=; b=OCamkTELJti6DU7YRkPLdSX1hAoatVzARZpl/2mnHrXWRRnkPklNDt8E6QNJV87di2 8T0mQPWneOAcUJmTB5IvY00tlTbRnmWA/Dghev+2cPT8wcCiquDObM8Uz+vMXrfW2O8j 5fD/vuqV+FI2DMEoESt4KUrCrY5QRrEPM3W9l3B8E+peFQytvc/w6NHTXx8VMKeSh4B3 QBB0K18+xWPGAJ08rMZHScbVX5Pjp/VXjiCPu7VTHsJSU8o5ocbqJvaMnj+hcrffZ2lk apFY0AN+lJZRfjW1vGwlVgKF7m8Wnh+woak+/bPR6VQ7tgbjVyjExDFZ8Ua/0N/Tye7v dodw== X-Gm-Message-State: AOAM530RJAKz4kfiG598NpoLuSNPO8P2ejwY72uCdj6vZJ4bwo4zP3jn ne8V7J9cfvM2tmUgAnzhO8iyJI9qcTwvjMWQZJSg9FwiLA7ZEMa8AL7MHSj8tsGog5KPgKmYkRk 605prZz5slcXEeyoyazaBSHeijGf9DII+vVST0f9s9yGMvJAIL7w3MbR4wyIvu0kwaeEDZgI= X-Received: from alankelly0.zrh.corp.google.com ([2a00:79e0:61:301:b159:808d:943e:13ba]) (user=alankelly job=sendgmr) by 2002:a5d:6c68:0:b0:1e8:9827:b978 with SMTP id r8-20020a5d6c68000000b001e89827b978mr1568559wrz.633.1645092205341; Thu, 17 Feb 2022 02:03:25 -0800 (PST) Date: Thu, 17 Feb 2022 11:03:21 +0100 Message-Id: <20220217100321.1110443-1-alankelly@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.35.1.265.g69c8d7142f-goog From: Alan Kelly To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v2 1/5] libswscale: Check and propagate memory allocation errors from ff_shuffle_filter_coefficients. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Alan Kelly Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Nm9l1TNPK/2r --- libswscale/swscale_internal.h | 2 +- libswscale/utils.c | 11 ++++++++--- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/libswscale/swscale_internal.h b/libswscale/swscale_internal.h index 3a78d95ba6..26d28d42e6 100644 --- a/libswscale/swscale_internal.h +++ b/libswscale/swscale_internal.h @@ -1144,5 +1144,5 @@ void ff_sws_slice_worker(void *priv, int jobnr, int threadnr, #define MAX_LINES_AHEAD 4 //shuffle filter and filterPos for hyScale and hcScale filters in avx2 -void ff_shuffle_filter_coefficients(SwsContext *c, int* filterPos, int filterSize, int16_t *filter, int dstW); +int ff_shuffle_filter_coefficients(SwsContext *c, int* filterPos, int filterSize, int16_t *filter, int dstW); #endif /* SWSCALE_SWSCALE_INTERNAL_H */ diff --git a/libswscale/utils.c b/libswscale/utils.c index c5ea8853d5..344c87dfdf 100644 --- a/libswscale/utils.c +++ b/libswscale/utils.c @@ -278,7 +278,7 @@ static const FormatEntry format_entries[] = { [AV_PIX_FMT_P416LE] = { 1, 1 }, }; -void ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSize, int16_t *filter, int dstW){ +int ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSize, int16_t *filter, int dstW){ #if ARCH_X86_64 int i, j, k, l; int cpu_flags = av_get_cpu_flags(); @@ -292,6 +292,8 @@ void ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSiz } if (filterSize > 4){ int16_t *tmp2 = av_malloc(dstW * filterSize * 2); + if (!tmp2) + return AVERROR(ENOMEM); memcpy(tmp2, filter, dstW * filterSize * 2); for (i = 0; i < dstW; i += 16){//pixel for (k = 0; k < filterSize / 4; ++k){//fcoeff @@ -310,6 +312,7 @@ void ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSiz } } } + return 0; #endif } @@ -1836,7 +1839,8 @@ av_cold int sws_init_context(SwsContext *c, SwsFilter *srcFilter, get_local_pos(c, 0, 0, 0), get_local_pos(c, 0, 0, 0))) < 0) goto fail; - ff_shuffle_filter_coefficients(c, c->hLumFilterPos, c->hLumFilterSize, c->hLumFilter, dstW); + if (ff_shuffle_filter_coefficients(c, c->hLumFilterPos, c->hLumFilterSize, c->hLumFilter, dstW) < 0) + goto nomem; if ((ret = initFilter(&c->hChrFilter, &c->hChrFilterPos, &c->hChrFilterSize, c->chrXInc, c->chrSrcW, c->chrDstW, filterAlign, 1 << 14, @@ -1846,7 +1850,8 @@ av_cold int sws_init_context(SwsContext *c, SwsFilter *srcFilter, get_local_pos(c, c->chrSrcHSubSample, c->src_h_chr_pos, 0), get_local_pos(c, c->chrDstHSubSample, c->dst_h_chr_pos, 0))) < 0) goto fail; - ff_shuffle_filter_coefficients(c, c->hChrFilterPos, c->hChrFilterSize, c->hChrFilter, c->chrDstW); + if (ff_shuffle_filter_coefficients(c, c->hChrFilterPos, c->hChrFilterSize, c->hChrFilter, c->chrDstW) < 0) + goto nomem; } } // initialize horizontal stuff