From patchwork Sun Oct 3 09:48:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Rheinhardt X-Patchwork-Id: 30885 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp3289988iob; Sun, 3 Oct 2021 02:48:56 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzk/PjRxufwAg8RMdJg+4q3mVayxyk775HBE+vu6BSvjJyohgFPmzpicsJazkxlClI/mIUw X-Received: by 2002:a50:d841:: with SMTP id v1mr9754577edj.221.1633254536724; Sun, 03 Oct 2021 02:48:56 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id i19si7702317edc.236.2021.10.03.02.48.56; Sun, 03 Oct 2021 02:48:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@outlook.com header.s=selector1 header.b=jl74rW4b; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=outlook.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5EE6D68A75A; Sun, 3 Oct 2021 12:48:44 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-oln040092073037.outbound.protection.outlook.com [40.92.73.37]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B53C368A6CC for ; Sun, 3 Oct 2021 12:48:37 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=i0DYVvo18HnQRXOg5Kta+bwkL7LOxEmsIS0+R8XYWHACVtyRwtKgfx3Bbe5Wxy1yJg2DmFnBUR/mudnAhW871LukGyar/zvyHmn7Mo777bZ3JAJnnCwVm5luz2Fe1yrfk+gfhT6oxv9YWzN0e8Qh51WjK+uY+gD/oUEuvak1avHEjY3TK7aqrM6ZBdXVBmrw45mTWItwnnql5W7YOIW6SeH8KhjecTepTCaXLqbGL/2BS/gq/063jzdSZdeRn3OgRDRykq+2TFKkeazqpy0Dmk/PgMRZwcPD/rVQL4a3U56vms8p774RZT8yrJPLaOnuyyBsAlSfMKV5hRcCsk+VOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kiQjbRLsChE+DF0nf+m78ouRqd8+sie5TscvMLp/iHI=; b=nRRDxXOeJ+mjav/6Od+WOe+MEBo9n1rQHP5+COSYp3n0VHfqtATwZXKxv6GPZYS3emA1iAthHp6bwBTwisPp0oA5MQ8q1HD5jKAwadg0h2vk5oi/s96LwZ+rHhgLZaia3lUtsNlX+ppgw0KhQxBCIJ9EY+D+1cw7/fS9sMII6+XYu85Bl3FKJ70QD9cWQ7zachdvlTjIH8FZz4WGyE/I2Xt/n+xhBOmJb5SHZ/Grox4Yz/PYTFkAvzznDZxiGBQs2qJEJ07HwlQDVWL3B/sIgPm6T2VV/i2PEfRJRex6WC5Kj5ID7yWEq9ATbrXQ4ubmV9wM+zG2km6h3/E2jeoBdg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kiQjbRLsChE+DF0nf+m78ouRqd8+sie5TscvMLp/iHI=; b=jl74rW4b6H9rlgSZdMRB/ymlJYt3Bd+PY9mE+UYYRzeyA8N/wYrnL6nrjf7UmGSkUCPyDbtzTof3H53JEi0fl2PMWNAOEeFzdvSJBHLV+jR7znJt5VJkRdhoyrLvLInlc9a4ZE8h2txZgMmnliTTYVbJw3/Jn4nMp2990csp2kFd6cM72uG6wMII8+RyeABLNmEu2swErp8vO00Y9aisjQBwKLOJfSH0+uSX2aX3Yo9G7QMKGzLajYh4atFZUaA7vuYOLmm+FRHLaeWH7vElOn6pwmdjJZuPaP+SbQgTGFoTkr1q+3/QKOlLay19ucp6rUQR8JWx98qA6+TYphJrIA== Received: from AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) by AM7PR03MB6328.eurprd03.prod.outlook.com (2603:10a6:20b:141::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.16; Sun, 3 Oct 2021 09:48:27 +0000 Received: from AM7PR03MB6660.eurprd03.prod.outlook.com ([fe80::700f:d70b:3bb8:4d51]) by AM7PR03MB6660.eurprd03.prod.outlook.com ([fe80::700f:d70b:3bb8:4d51%7]) with mapi id 15.20.4566.019; Sun, 3 Oct 2021 09:48:27 +0000 From: Andreas Rheinhardt To: ffmpeg-devel@ffmpeg.org Date: Sun, 3 Oct 2021 11:48:17 +0200 Message-ID: X-Mailer: git-send-email 2.30.2 In-Reply-To: References: X-TMN: [8Zw+SYX9T4Gqkd2HjZ/5fUvj5DTjQPuu] X-ClientProxiedBy: AM3PR05CA0114.eurprd05.prod.outlook.com (2603:10a6:207:2::16) To AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) X-Microsoft-Original-Message-ID: <20211003094818.3416770-3-andreas.rheinhardt@outlook.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from sblaptop.fritz.box (188.192.142.38) by AM3PR05CA0114.eurprd05.prod.outlook.com (2603:10a6:207:2::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.14 via Frontend Transport; Sun, 3 Oct 2021 09:48:27 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ad7df8f4-61db-423f-4e87-08d98652f333 X-MS-Exchange-SLBlob-MailProps: S/btQ8cKWiQbzu686YDUKEY6maueAvXAJzCk9hM5hETbmpEI0FzLXrB7sUw0Yz7FMy+VZ3chhvd3/NBinrX6ptFo2NViEAEoSac65hlXhEi5MULdJZB3b0CM1y5ocnBF9mDOsw9LHtNvZC2aCUcxnGA5MvC50Fvr6wW6TNpE2ii+K+2NDln57FuPbkefVqBaNawwXRVI2Mz7FqChici/77vcqCFSd05tgSrsSabiiTbxbGxXl4DnqMYsfIWtrsILq0w95WQczBoHpBDmFersq9l8oPbQrrlIMH1S/PhKOkypCpERWipzzOkXst+2p3aCkU1O08HUDqbFct+SrfqSlReFbTUTA3xAdW3tc/1JuSaXtmhxpwT/a1ZSbwxPb+dtR/wW310G+sNtTiYjTV1SUQIbBsm9wQFYZ5KhyKZqkRJAbP7dViCP4hLTzg5YLEA4yUm8R64IKLwK/+oOwOGrhI7nvMlTmDCIz/8QsaTCa4/+gQUaj2Z4ig4+g8ZnSQvYAJELdDBCh5ab8gxQ3WElAgkoY1rJi5Vcn40cLBW3UKeaIAnbcH+LGSc5MruvMEccL6X7y23pj4jMnqIspjC1uWSkuJtIKugZp6ulU+kWDcUjiyuFeLtBTNbcN1D+XG6YLMEE6Z7tra98DqeAou4yczlGYr9GhEfxd3XPcblyXy6HdKOiYJ1kgAKaXaGTggyttKX72JDCvwWOGFOZ4u6c8MnPA0aJyqJFm3eVUjzQiXZOLYjtEml6qkia4xGvqMGsiz+j7u6MFTI= X-MS-TrafficTypeDiagnostic: AM7PR03MB6328: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oAtVv46DP7f0OCEIB/KEFsGX6K0q28CdujXV5gABpdIqxWCao9wjB73ZKixe6oRiD76uzntJmfTdBEIngEmEjNXW8VJUJYkO2obzZsu/SpVzd9uN0kgLk9Uh1F8diyNySwptJj9A7+62oABM0slLNJ5URevngPZW+0eu1edpuEdUKrDSv/O1vDf/MMEFEA//Fao+pvplTPXN1/kYcGVfXEOm2zpEkjNpPUJ0Vrt87GyHDz/TJ+iLtlEqP54ZekMQF+LItPwUSHVi5QtXW0YAJ/P8NrKH0mVoid3s/lln2yLHZ6MLnjJX2yHFJYHEUAFRF/TRzSwEE5e6QAG1U0QWidKqksKoM1OgO1EZogAJ5Um2HfcISjwFGnSbVavlR71k7w4q5Hi86qTOwP1oVBms/qQPhgUeZn6kkadEdRK9qs4mKaeyY7AVosQIish34/wN X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: BnHIuYNjscDjG1DBSGo9R+YIMIq9gEImVgE8shgsysge6MB25A4QUZwIwGXPkIU7p0RWM6bVgctiX2wOWZV0UKqIFL+hisYUGm+trUjvWRa7iaaLuncqQVXCvhpFqT304KLp9Q227hSPfN87D0liEQ== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: ad7df8f4-61db-423f-4e87-08d98652f333 X-MS-Exchange-CrossTenant-AuthSource: AM7PR03MB6660.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Oct 2021 09:48:27.7522 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7PR03MB6328 Subject: [FFmpeg-devel] [PATCH 4/5] avfilter/vf_morpho: Fix invalid frees on error X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: vTDxlM4NFYxZ The current code used a pointer to an array (of arrays) that is offset relative to the start of the actually allocated buffer. Yet offsetting the pointer is only done on success, whereas the freeing code believes it to have happened even on error. So if any of the subarrays (or the subarrays' subarrays) can't be successfully allocated, one gets a bad free in free_lut(). Furthermore, said offsetting is only permissible in case the offsetted pointer points in the allocated buffer (here: in case the LUT's min_r is <= 0), as pointer arithmetic is undefined in case it exceeds the allocated object. Moreover, in case one of the subarrays couldn't be allocated, the code nevertheless tried to free the subarray's subarrays; and in case one of the subarray's subarrays could not be allocated successfully, there will be an invalid free, too, because the pointers for the subarrays' subarrays are also offset compared to the base pointer. This commit fixes all of this, by using the actually allocated pointer for freeing and by adding appropriate checks before freeing the subarrays. The former also allows to distinguish the cases in which the lut is currently only half-allocated due to an error in an earlier allocation attempt from the success case. Signed-off-by: Andreas Rheinhardt --- libavfilter/vf_morpho.c | 49 +++++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 21 deletions(-) diff --git a/libavfilter/vf_morpho.c b/libavfilter/vf_morpho.c index 8c1e084e7e..818ebd6b9a 100644 --- a/libavfilter/vf_morpho.c +++ b/libavfilter/vf_morpho.c @@ -61,7 +61,10 @@ typedef struct IPlane { } IPlane; typedef struct LUT { + /* arr is shifted from base_arr by FFMAX(min_r, 0). + * arr != NULL means "lut completely allocated" */ uint8_t ***arr; + uint8_t ***base_arr; int min_r; int max_r; int I; @@ -262,7 +265,8 @@ static void maxinplace16_fun(uint8_t *aa, const uint8_t *bb, int x) static int alloc_lut(LUT *Ty, chord_set *SE, int type_size, int mode) { - const int size = Ty->max_r + 1 - Ty->min_r; + const int min = FFMAX(Ty->min_r, 0); + const int max = min + (Ty->max_r - Ty->min_r); int pre_pad_x = 0; if (SE->minX < 0) @@ -270,55 +274,58 @@ static int alloc_lut(LUT *Ty, chord_set *SE, int type_size, int mode) Ty->pre_pad_x = pre_pad_x; Ty->type_size = type_size; - Ty->arr = av_calloc(size, sizeof(*Ty->arr)); - if (!Ty->arr) + Ty->base_arr = av_calloc(max + 1, sizeof(*Ty->base_arr)); + if (!Ty->base_arr) return AVERROR(ENOMEM); - for (int r = 0; r < Ty->max_r - Ty->min_r + 1; r++) { - Ty->arr[r] = av_calloc(Ty->I, sizeof(uint8_t *)); - if (!Ty->arr[r]) + for (int r = min; r <= max; r++) { + uint8_t **arr = Ty->base_arr[r] = av_calloc(Ty->I, sizeof(uint8_t *)); + if (!Ty->base_arr[r]) return AVERROR(ENOMEM); for (int i = 0; i < Ty->I; i++) { - Ty->arr[r][i] = av_calloc(Ty->X + pre_pad_x, type_size); - if (!Ty->arr[r][i]) + arr[i] = av_calloc(Ty->X + pre_pad_x, type_size); + if (!arr[i]) return AVERROR(ENOMEM); if (mode == ERODE) - memset(Ty->arr[r][i], UINT8_MAX, pre_pad_x * type_size); + memset(arr[i], UINT8_MAX, pre_pad_x * type_size); /* Shifting the X index such that negative indices correspond to * the pre-padding. */ - Ty->arr[r][i] = &(Ty->arr[r][i][pre_pad_x * type_size]); + arr[i] = &(arr[i][pre_pad_x * type_size]); } } - Ty->arr = &(Ty->arr[0 - Ty->min_r]); + Ty->arr = &(Ty->base_arr[min - Ty->min_r]); return 0; } static void free_lut(LUT *table) { - uint8_t ***rp; + const int min = FFMAX(table->min_r, 0); + const int max = min + (table->max_r - table->min_r); - if (!table->arr) + if (!table->base_arr) return; - // The R index was shifted, create a pointer to the original array - rp = &(table->arr[table->min_r]); - - for (int r = table->min_r; r <= table->max_r; r++) { + for (int r = min; r <= max; r++) { + if (!table->base_arr[r]) + break; for (int i = 0; i < table->I; i++) { + if (!table->base_arr[r][i]) + break; // The X index was also shifted, for padding purposes. - av_free(table->arr[r][i] - table->pre_pad_x * table->type_size); + av_free(table->base_arr[r][i] - table->pre_pad_x * table->type_size); } - av_freep(&table->arr[r]); + av_freep(&table->base_arr[r]); } - av_freep(&rp); + av_freep(&table->base_arr); + table->arr = NULL; } static int alloc_lut_if_necessary(LUT *Ty, IPlane *f, chord_set *SE, int y, int num, enum MorphModes mode) { - if (Ty->I != SE->Lnum || + if (!Ty->arr || Ty->I != SE->Lnum || Ty->X != f->w || SE->minX < 0 && -SE->minX > Ty->pre_pad_x || Ty->min_r != SE->minY ||