From patchwork Fri Sep 17 02:08:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Rheinhardt X-Patchwork-Id: 30299 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp1785207iov; Thu, 16 Sep 2021 19:10:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwfphO6fe0aPbQ1/FHA2ea9E7OQPsFs9QEMadQYbeLB8k2Y8YrQ6V6E6r94tXqme3rUKTKq X-Received: by 2002:a17:907:2d0b:: with SMTP id gs11mr9421240ejc.151.1631844611559; Thu, 16 Sep 2021 19:10:11 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e11si5186615edj.557.2021.09.16.19.10.10; Thu, 16 Sep 2021 19:10:11 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@outlook.com header.s=selector1 header.b=n8Nl9OsH; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=outlook.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CF54A68B14F; Fri, 17 Sep 2021 05:08:43 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-oln040092075092.outbound.protection.outlook.com [40.92.75.92]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A305C68B12A for ; Fri, 17 Sep 2021 05:08:38 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H5JGBwQ8GmGt77NPhf/qbpsMPBQLoyZfjJ5HOgGljSX3GaWEyKvuNlPaFMwUOY71slaoFQVT/VR8Qpj5Yd1fdqJDWpE7unoYJVxwQSRwvb9/DQOXw4wb6ooSCTDbvC6bxfD3gWiLP4OkO7YwCf9BU8PPPfypCD0Sw6+u6AHr0TSYUC+s42XmTjbA4zhmdYyoRcvxTfFVAnV5JFbadJVtvNkZr24kAQ0XetUNfpRUvKa4TpF4c7MTZkO5lahxV/OqJvFdlendmLRQ8sk6nprndVz7XeGsQV4KP7WhcAAAU+XoJWQ91XRUItZRrZ0VtHsF52Mrz5wZ+RVVw5k1S27hEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=NsXxGbTbLDHbT5NZsyE6DCEc2ySYobLYMaunJp6fjGs=; b=hYoropcigGRR7g0eThDbbaDJS49SP5YYFFT50z8Hili21vuG4nz5hJuHQHChNT25cQCKBot1WB05gaorTUPUEelqg3U2bRY4xV/Bzfkja/8pECq16H1GxY2PCBKJrb8yBO1D3qNOrnEuadf/aISAztYHU3/wk1xQyXnC0eDPYcbhD7SKi7zTo8n+YHWcEXU6nQSW+maoBbJpPOBRmd5RWpqfvbk4aXqQL6ncGrJoLQG4vHsRV1i3BAbpaqKCMZQ41zKET8PVAJqXbI8p/2DcwiDllOZZNbL5LoRtkquiYwjwYvpYZ9Fx+LerI8dUWV28UeBWDf6tSK+P6yzlzT6lVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NsXxGbTbLDHbT5NZsyE6DCEc2ySYobLYMaunJp6fjGs=; b=n8Nl9OsHMjxFt2QvpJGyYvKj503MPlVS8Eqi3zZUh9JEnqmEv743Qbydu5n8EieFtXOM6JWAy1uZTHfV5gLCuOQhy+y3ZnOT4CSDmVabMVsuMJ4LNxJtOSNBbJwag+arDYGcAKi84mJ/HrGxR/uH5gIf9558JHRF+NxfDWAUzqfv89wswdQTwfzUk8oruLsC3PLMQc8QYJR8t3zDrc6P/7dmWiWo8DwnWsyB7Dl08MVdxqOujhyo9OOBD8SlHDCTA9YiG5yBDaHvNb5nqTD5lDfW8ddNk9PuB0ZPoSRptZJey9vcpftlqLbZE7sJ169Bda8ULgeeU8w0jC2A4G+9Mw== Received: from AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) by AM6PR03MB5297.eurprd03.prod.outlook.com (2603:10a6:20b:c5::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4523.16; Fri, 17 Sep 2021 02:08:31 +0000 Received: from AM7PR03MB6660.eurprd03.prod.outlook.com ([fe80::787b:2156:ca99:fe00]) by AM7PR03MB6660.eurprd03.prod.outlook.com ([fe80::787b:2156:ca99:fe00%3]) with mapi id 15.20.4523.016; Fri, 17 Sep 2021 02:08:31 +0000 From: Andreas Rheinhardt To: ffmpeg-devel@ffmpeg.org Date: Fri, 17 Sep 2021 04:08:05 +0200 Message-ID: X-Mailer: git-send-email 2.30.2 In-Reply-To: References: X-TMN: [5FSPaUfypsxmCIztDnurkBErox9ln7e8] X-ClientProxiedBy: AM8P191CA0018.EURP191.PROD.OUTLOOK.COM (2603:10a6:20b:21a::23) To AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) X-Microsoft-Original-Message-ID: <20210917020808.275498-9-andreas.rheinhardt@outlook.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from sblaptop.fritz.box (188.192.142.38) by AM8P191CA0018.EURP191.PROD.OUTLOOK.COM (2603:10a6:20b:21a::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4523.14 via Frontend Transport; Fri, 17 Sep 2021 02:08:31 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: af5cb649-5a3c-4a0b-698e-08d979800bdb X-MS-Exchange-SLBlob-MailProps: q+fD6XS3/ULHQVHO031XvYglgXuzPUZXAwLUo0PUPg0PZLbidF5Zr4Pkevs2xDgdWSQBrRHl8ID1Y+yZouMKdrhY2vu4E0mSZKxPr71ung+L1CEeLwW7McqW8VXJzp4J20/D9tT7PC3RpbhhVgB1dOgS498ukjm3cSN2zdYUAVIcyeYR85hnKpvNoUXzpN+FkEBQgYzcuTC7sN4hXpYZoB/pgVIMR3S5CyF4BdHTqT4as/AE/C5U/KIdIIsBbIClJzzS+evUuWRgdavVmjJi0+LgzfGc9HjRSaNLrnq49Nt86qE+k2zSfHyjqhkmKCDcK5kERthU3JuV85sUuWICjIsMlT1ScS3gsmtf+yidlRThUt1CzRugZsnNuzQp6WyfJRmVm6hiOA6BPds4eYjfSTVkid+874tcHRrxFlolMmu6s8J93cgLVn5W540cHijYy3W8kX1IEazvwtN02hYBFKeFD4YR+Am2A7KgBQmvvf2UD/typJbzFj60wVjcP33Ljak/rz2S3Z67q10VLznLeDYCPJ71RRnHEy1aGoazD9fjS59O9eH7C+wGDOnI49XaMY7UTauPHvkHflowQRvMWCXe8KdHNFNIc2/o5iLxdyPt/hZG7WniQAgW+EnzjroTmSCOoZ7jj8ZA3jidp7/Mn/jNWCjcPuEReh0wUSXF98DxH4S8ipDF2V/qmrGuR1pTFoUn0LVFzfWY66ONK2aSHA== X-MS-TrafficTypeDiagnostic: AM6PR03MB5297: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WkRskwuh43k8/aLVKmXerHyLgBo87E3+bmKvtOEKDdZf6/hy9SDMFP7SoLkoXnMjOdlfFSYVqRUo1cqPV+41f5IXqH/f6uoW4qbXpC9VgFtIhnhksdWTOs3UTOAZ7lT94nGHmCdqnX5xoyTuomQ/B8+1fJhrKdv1nMp6W6j+qxFgak/opGissYifSeXOpVyTuAkZg0XJjR9o7Ywdd0v8w2lGZquCVQ/m4HyV7OG+Kb+Zc/SpcEcWAQCN4gzosyQOpkEsqpw9eNr+6RV8GTllQqdwDqBY2tyycXThTI3Q66DSdm6NobUIc+hyEmX/IavqbADeY3504eKQOl11ZBe2AS2B61Xtnxt4xCDb6dOFbUS3IQibEI9ejBHK2tqpmUpc9qNCMZEL23VJT26OvO+pSA8TilDRoXDLe2rTdoBCnZN/DMMHlgG4iijsuiC3UqT0 X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: uhTdPvYJ/hPt7+9j/ryUx7SBuT+VRW8bemnIV1PV2fIYF9KjtUqtmeacgi02TIWrl763Xu3gmurW6INO1LOCZ2yI8e9jvWR+jM878/OlkD91Cn+DeRSt80iM6RpnELRxD1r3fdJ537DjT/XKKieXCQ== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: af5cb649-5a3c-4a0b-698e-08d979800bdb X-MS-Exchange-CrossTenant-AuthSource: AM7PR03MB6660.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Sep 2021 02:08:31.4874 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR03MB5297 Subject: [FFmpeg-devel] [PATCH 10/13] avcodec/cinepakenc: Check all calls to avpriv_elbg_do() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: TW5IMHZn9CG/ Signed-off-by: Andreas Rheinhardt --- libavcodec/cinepakenc.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/libavcodec/cinepakenc.c b/libavcodec/cinepakenc.c index 2984b93de3..d1bcf2b2d5 100644 --- a/libavcodec/cinepakenc.c +++ b/libavcodec/cinepakenc.c @@ -709,6 +709,7 @@ static int quantize(CinepakEncContext *s, int h, uint8_t *data[4], uint8_t vq_pict_buf[(MB_AREA * 3) / 2]; uint8_t *sub_data[4], *vq_data[4]; int sub_linesize[4], vq_linesize[4]; + int ret; for (mbn = i = y = 0; y < h; y += MB_SIZE) { for (x = 0; x < s->w; x += MB_SIZE, ++mbn) { @@ -762,8 +763,10 @@ static int quantize(CinepakEncContext *s, int h, uint8_t *data[4], if (i < size) size = i; - avpriv_elbg_do(&s->elbg, s->codebook_input, entry_size, i, codebook, - size, 1, s->codebook_closest, &s->randctx); + ret = avpriv_elbg_do(&s->elbg, s->codebook_input, entry_size, i, codebook, + size, 1, s->codebook_closest, &s->randctx); + if (ret < 0) + return ret; // set up vq_data, which contains a single MB vq_data[0] = vq_pict_buf; @@ -888,8 +891,10 @@ static int rd_strip(CinepakEncContext *s, int y, int h, int keyframe, if (mode == MODE_V1_ONLY) { info.v1_size = v1_size; // the size may shrink even before optimizations if the input is short: - info.v1_size = quantize(s, h, data, linesize, 1, - &info, ENC_UNCERTAIN); + if ((new_v1_size = quantize(s, h, data, linesize, 1, + &info, ENC_UNCERTAIN)) < 0) + return new_v1_size; + info.v1_size = new_v1_size; if (info.v1_size < v1_size) // too few eligible blocks, no sense in trying bigger sizes v1enough = 1; @@ -902,8 +907,11 @@ static int rd_strip(CinepakEncContext *s, int y, int h, int keyframe, if (mode == MODE_V1_V4) { info.v4_size = v4_size; - info.v4_size = quantize(s, h, data, linesize, 0, - &info, ENC_UNCERTAIN); + new_v4_size = quantize(s, h, data, linesize, 0, + &info, ENC_UNCERTAIN); + if (new_v4_size < 0) + return new_v4_size; + info.v4_size = new_v4_size; if (info.v4_size < v4_size) // too few eligible blocks, no sense in trying bigger sizes v4enough = 1; @@ -921,11 +929,15 @@ static int rd_strip(CinepakEncContext *s, int y, int h, int keyframe, // we assume we _may_ come here with more blocks to encode than before info.v1_size = v1_size; new_v1_size = quantize(s, h, data, linesize, 1, &info, ENC_V1); + if (new_v1_size < 0) + return new_v1_size; if (new_v1_size < info.v1_size) info.v1_size = new_v1_size; // we assume we _may_ come here with more blocks to encode than before info.v4_size = v4_size; new_v4_size = quantize(s, h, data, linesize, 0, &info, ENC_V4); + if (new_v4_size < 0) + return new_v4_size; if (new_v4_size < info.v4_size) info.v4_size = new_v4_size; // calculate the resulting score @@ -942,12 +954,16 @@ static int rd_strip(CinepakEncContext *s, int y, int h, int keyframe, if (v1shrunk) { info.v1_size = v1_size; new_v1_size = quantize(s, h, data, linesize, 1, &info, ENC_V1); + if (new_v1_size < 0) + return new_v1_size; if (new_v1_size < info.v1_size) info.v1_size = new_v1_size; } if (v4shrunk) { info.v4_size = v4_size; new_v4_size = quantize(s, h, data, linesize, 0, &info, ENC_V4); + if (new_v4_size < 0) + return new_v4_size; if (new_v4_size < info.v4_size) info.v4_size = new_v4_size; }