From patchwork Tue Nov 15 22:16:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 39292 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp23288pzb; Tue, 15 Nov 2022 14:17:10 -0800 (PST) X-Google-Smtp-Source: AA0mqf6w1XyEeJyQrMzV7JBf+JUym+RMegGhkdVdcs5/KbBtv7hpDsV+PcHSh+iRoCdx8WNE7uAS X-Received: by 2002:a17:906:3a55:b0:78d:f2d8:4623 with SMTP id a21-20020a1709063a5500b0078df2d84623mr15983506ejf.274.1668550630305; Tue, 15 Nov 2022 14:17:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668550630; cv=none; d=google.com; s=arc-20160816; b=lIs32cxCP+g1riDyaD7Swd8IuftWAOai7rdHwlawzGn2Y1vfZakCce/UKBAuzGpDoy OhB/Efw3sUiDpzgOQcLyE/CohVS67dIzqMnaeKdkjAJyARReVjMS9Zufy7lORvSkJrcX pbzsTUKILL8gbKcisdMDqNJV2lYKX0ujszsdoHq+u//pDXgc+uTYVOjfMIh6aHnr6chS a86spGcOQ0YipelZiiWIvuDn+1QkgoilQ+2JuenN8Pyef5JNgqoFLfUC55oFijK66S2c +MsuQF4RgYV3EEI6iKfcy49HnBxNMJeE+KAj4b6W1tVYHg7rXq1jZu3XbLn/1FtV3JXd 4Bog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:delivered-to; bh=JJ7+iVrMRKgStfrojtp9VfyiVHy4rA4W5dIGh20THd8=; b=ijhA709YPGdHbNUryrUwWG/Mg1T8RB5uGKEWU29AqxU2PV3HgFmUdpF2vTjIDXtmiT DKrmHchLemJvlqzXKjLqAbLMERAYs7htQtFQ+Vjkk37dehpSq8lg0RxFoVr3aD5VmO/D wHzPDaTJ49z0I5gmIDvkmHz/wax9Gsu4NKi+8ZWH0v39tiUg3pE4Cq3s/90bcgoqYR+3 Jl1mPvsF0xZFinCnXz6CjObpK5j/Hu1t9wc0sGecheHRBJwY1eZm+hwueN5Oc7TxbYeD DyLKM8Mlc9oFXiXByrOp8lOVYfADOth6As61PF6CSNQZsBqu/O9LnVlMvm+M+1J8x4NZ gSeA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id jz1-20020a170906bb0100b007815ca78c38si5609917ejb.235.2022.11.15.14.17.09; Tue, 15 Nov 2022 14:17:10 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7D27368BD5D; Wed, 16 Nov 2022 00:17:06 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6C37D68B695 for ; Wed, 16 Nov 2022 00:16:59 +0200 (EET) Received: (Authenticated sender: michael@niedermayer.cc) by mail.gandi.net (Postfix) with ESMTPSA id 69EC960003; Tue, 15 Nov 2022 22:16:58 +0000 (UTC) From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Tue, 15 Nov 2022 23:16:56 +0100 Message-Id: <20221115221657.23290-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH 1/2] swscale/input: Use unsigned intermediates in rgb64ToUV_c_template X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Jeremy Dorfman MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: YTlnHlHYAUpY From: Jeremy Dorfman Large rgb2yuv tables and high pixel values cause the intermediate int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined behavior. This causes libswscale built with LLVM -fsanitize=undefined to assert. Using unsigned integers instead has defined behavior and produces identical results, and makes rgb64ToUV_c_template match rgb64ToY_c_template. Fixes: signed integer overflow Signed-off-by: Michael Niedermayer --- libswscale/input.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libswscale/input.c b/libswscale/input.c index 7ff7bfaa01..d7dbedd82f 100644 --- a/libswscale/input.c +++ b/libswscale/input.c @@ -65,9 +65,9 @@ rgb64ToUV_c_template(uint16_t *dstU, uint16_t *dstV, int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX]; av_assert1(src1==src2); for (i = 0; i < width; i++) { - int r_b = input_pixel(&src1[i*4+0]); - int g = input_pixel(&src1[i*4+1]); - int b_r = input_pixel(&src1[i*4+2]); + unsigned int r_b = input_pixel(&src1[i*4+0]); + unsigned int g = input_pixel(&src1[i*4+1]); + unsigned int b_r = input_pixel(&src1[i*4+2]); dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; dstV[i] = (rv*r + gv*g + bv*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT;