From patchwork Tue Nov 15 22:16:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 39292 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp23288pzb; Tue, 15 Nov 2022 14:17:10 -0800 (PST) X-Google-Smtp-Source: AA0mqf6w1XyEeJyQrMzV7JBf+JUym+RMegGhkdVdcs5/KbBtv7hpDsV+PcHSh+iRoCdx8WNE7uAS X-Received: by 2002:a17:906:3a55:b0:78d:f2d8:4623 with SMTP id a21-20020a1709063a5500b0078df2d84623mr15983506ejf.274.1668550630305; Tue, 15 Nov 2022 14:17:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668550630; cv=none; d=google.com; s=arc-20160816; b=lIs32cxCP+g1riDyaD7Swd8IuftWAOai7rdHwlawzGn2Y1vfZakCce/UKBAuzGpDoy OhB/Efw3sUiDpzgOQcLyE/CohVS67dIzqMnaeKdkjAJyARReVjMS9Zufy7lORvSkJrcX pbzsTUKILL8gbKcisdMDqNJV2lYKX0ujszsdoHq+u//pDXgc+uTYVOjfMIh6aHnr6chS a86spGcOQ0YipelZiiWIvuDn+1QkgoilQ+2JuenN8Pyef5JNgqoFLfUC55oFijK66S2c +MsuQF4RgYV3EEI6iKfcy49HnBxNMJeE+KAj4b6W1tVYHg7rXq1jZu3XbLn/1FtV3JXd 4Bog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:delivered-to; bh=JJ7+iVrMRKgStfrojtp9VfyiVHy4rA4W5dIGh20THd8=; b=ijhA709YPGdHbNUryrUwWG/Mg1T8RB5uGKEWU29AqxU2PV3HgFmUdpF2vTjIDXtmiT DKrmHchLemJvlqzXKjLqAbLMERAYs7htQtFQ+Vjkk37dehpSq8lg0RxFoVr3aD5VmO/D wHzPDaTJ49z0I5gmIDvkmHz/wax9Gsu4NKi+8ZWH0v39tiUg3pE4Cq3s/90bcgoqYR+3 Jl1mPvsF0xZFinCnXz6CjObpK5j/Hu1t9wc0sGecheHRBJwY1eZm+hwueN5Oc7TxbYeD DyLKM8Mlc9oFXiXByrOp8lOVYfADOth6As61PF6CSNQZsBqu/O9LnVlMvm+M+1J8x4NZ gSeA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id jz1-20020a170906bb0100b007815ca78c38si5609917ejb.235.2022.11.15.14.17.09; Tue, 15 Nov 2022 14:17:10 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7D27368BD5D; Wed, 16 Nov 2022 00:17:06 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6C37D68B695 for ; Wed, 16 Nov 2022 00:16:59 +0200 (EET) Received: (Authenticated sender: michael@niedermayer.cc) by mail.gandi.net (Postfix) with ESMTPSA id 69EC960003; Tue, 15 Nov 2022 22:16:58 +0000 (UTC) From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Tue, 15 Nov 2022 23:16:56 +0100 Message-Id: <20221115221657.23290-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH 1/2] swscale/input: Use unsigned intermediates in rgb64ToUV_c_template X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Jeremy Dorfman MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: YTlnHlHYAUpY From: Jeremy Dorfman Large rgb2yuv tables and high pixel values cause the intermediate int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined behavior. This causes libswscale built with LLVM -fsanitize=undefined to assert. Using unsigned integers instead has defined behavior and produces identical results, and makes rgb64ToUV_c_template match rgb64ToY_c_template. Fixes: signed integer overflow Signed-off-by: Michael Niedermayer --- libswscale/input.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libswscale/input.c b/libswscale/input.c index 7ff7bfaa01..d7dbedd82f 100644 --- a/libswscale/input.c +++ b/libswscale/input.c @@ -65,9 +65,9 @@ rgb64ToUV_c_template(uint16_t *dstU, uint16_t *dstV, int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX]; av_assert1(src1==src2); for (i = 0; i < width; i++) { - int r_b = input_pixel(&src1[i*4+0]); - int g = input_pixel(&src1[i*4+1]); - int b_r = input_pixel(&src1[i*4+2]); + unsigned int r_b = input_pixel(&src1[i*4+0]); + unsigned int g = input_pixel(&src1[i*4+1]); + unsigned int b_r = input_pixel(&src1[i*4+2]); dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; dstV[i] = (rv*r + gv*g + bv*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; From patchwork Tue Nov 15 22:16:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 39293 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp23363pzb; Tue, 15 Nov 2022 14:17:18 -0800 (PST) X-Google-Smtp-Source: AA0mqf532vvSIPzFovX6Q+mvcNXtLvx1QF+C5EDFz31jd/s0g1/Xv1UcBRbAsYtGcBYgxVgc/7z7 X-Received: by 2002:a05:6402:4506:b0:467:8dd2:b4b with SMTP id ez6-20020a056402450600b004678dd20b4bmr12790300edb.112.1668550638528; Tue, 15 Nov 2022 14:17:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668550638; cv=none; d=google.com; s=arc-20160816; b=CBu2PzeOQHbpHPboxmduR/cbUm14vApDgbe1+3igpxGj6wEApnUDCQF/9S+KZQubcO 1mXbt6T9m4U8kF8+cOSEB40FeEMV+V1tgl3XyGmeynizPs2XdXCGpOm4piZo54JAlMMJ sGscyjuQxemojdFdn2LxgOrrV4BQZinTZ7+OJiYObbc/dyxJ6JgdqRJ3PYLlLqoCEMyA D1oi2+cJV6IF7irdBOkRCpsw5b3zNEApjry2EGK5IsqmyoBz/3bqdzQ7l7RuSGeI+ztM 0cCqWAPRPVl/WxthKSzOJv44NIdmsPTEreQye9IZ9cUiLl7cxfeulqcz6julTaFZPI0v CC9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:delivered-to; bh=lUlqn+eSACy2Of40H4RnXSw2YiAKpNDsZ57nDLvkg+k=; b=xVoH1zdfpj2yS94VfFBU01d5gub1LUV8n6ieXE5q+XTPmZ3fUZtGUnh95NLjk44qc6 8JUbBVLBGw7dpk7cO/ANyqtcsYkBg/ij0hgPhTLiyXPikJkCyBgqY9ehK6wBAn+gQtfl zHGVunm4jy4gqO0vpX6xIPINqs6FuqGhHRBbs4Z1Iid2FubFjuVkdj9orkkDYyjc+Z8V CtfjoEqtXl0z25nT6QMVlb1AqBmwXJQIL3RgarjS1YLJ8ZcPv9Y5bqrlBBWo454z/c2C yTuTV+QjehDfU10x/vnRtWNg25uizrfKw69tMjNk+EfLzTc1usz77WCjuiOExQT32Cq/ 1Sdw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id dm8-20020a170907948800b0078e063fc787si13068232ejc.433.2022.11.15.14.17.18; Tue, 15 Nov 2022 14:17:18 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 79DDA68BD8E; Wed, 16 Nov 2022 00:17:07 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5D06968BBD9 for ; Wed, 16 Nov 2022 00:17:00 +0200 (EET) Received: (Authenticated sender: michael@niedermayer.cc) by mail.gandi.net (Postfix) with ESMTPSA id 87BA5E0003 for ; Tue, 15 Nov 2022 22:16:59 +0000 (UTC) From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Tue, 15 Nov 2022 23:16:57 +0100 Message-Id: <20221115221657.23290-2-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221115221657.23290-1-michael@niedermayer.cc> References: <20221115221657.23290-1-michael@niedermayer.cc> Subject: [FFmpeg-devel] [PATCH 2/2] swscale/input: Use more unsigned intermediates X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: dAEWRURcq2sK Same principle as previous commit, with sufficiently huge rgb2yuv table values this produces wrong results and undefined behavior. The unsigned produces the same incorrect results. That is probably ok as these cases with huge values seem not to occur in any real use case. There are more cases but someone is refactoring them, so i didnt yet change them. Fixes: signed integer overflow Signed-off-by: Michael Niedermayer --- libswscale/input.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/libswscale/input.c b/libswscale/input.c index d7dbedd82f..d5676062a2 100644 --- a/libswscale/input.c +++ b/libswscale/input.c @@ -84,9 +84,9 @@ rgb64ToUV_half_c_template(uint16_t *dstU, uint16_t *dstV, int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX]; av_assert1(src1==src2); for (i = 0; i < width; i++) { - int r_b = (input_pixel(&src1[8 * i + 0]) + input_pixel(&src1[8 * i + 4]) + 1) >> 1; - int g = (input_pixel(&src1[8 * i + 1]) + input_pixel(&src1[8 * i + 5]) + 1) >> 1; - int b_r = (input_pixel(&src1[8 * i + 2]) + input_pixel(&src1[8 * i + 6]) + 1) >> 1; + unsigned r_b = (input_pixel(&src1[8 * i + 0]) + input_pixel(&src1[8 * i + 4]) + 1) >> 1; + unsigned g = (input_pixel(&src1[8 * i + 1]) + input_pixel(&src1[8 * i + 5]) + 1) >> 1; + unsigned b_r = (input_pixel(&src1[8 * i + 2]) + input_pixel(&src1[8 * i + 6]) + 1) >> 1; dstU[i]= (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; dstV[i]= (rv*r + gv*g + bv*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; @@ -158,9 +158,9 @@ static av_always_inline void rgb48ToUV_c_template(uint16_t *dstU, int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX]; av_assert1(src1 == src2); for (i = 0; i < width; i++) { - int r_b = input_pixel(&src1[i * 3 + 0]); - int g = input_pixel(&src1[i * 3 + 1]); - int b_r = input_pixel(&src1[i * 3 + 2]); + unsigned r_b = input_pixel(&src1[i * 3 + 0]); + unsigned g = input_pixel(&src1[i * 3 + 1]); + unsigned b_r = input_pixel(&src1[i * 3 + 2]); dstU[i] = (ru*r + gu*g + bu*b + (0x10001 << (RGB2YUV_SHIFT - 1))) >> RGB2YUV_SHIFT; dstV[i] = (rv*r + gv*g + bv*b + (0x10001 << (RGB2YUV_SHIFT - 1))) >> RGB2YUV_SHIFT; @@ -180,12 +180,12 @@ static av_always_inline void rgb48ToUV_half_c_template(uint16_t *dstU, int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX]; av_assert1(src1 == src2); for (i = 0; i < width; i++) { - int r_b = (input_pixel(&src1[6 * i + 0]) + - input_pixel(&src1[6 * i + 3]) + 1) >> 1; - int g = (input_pixel(&src1[6 * i + 1]) + - input_pixel(&src1[6 * i + 4]) + 1) >> 1; - int b_r = (input_pixel(&src1[6 * i + 2]) + - input_pixel(&src1[6 * i + 5]) + 1) >> 1; + unsigned r_b = (input_pixel(&src1[6 * i + 0]) + + input_pixel(&src1[6 * i + 3]) + 1) >> 1; + unsigned g = (input_pixel(&src1[6 * i + 1]) + + input_pixel(&src1[6 * i + 4]) + 1) >> 1; + unsigned b_r = (input_pixel(&src1[6 * i + 2]) + + input_pixel(&src1[6 * i + 5]) + 1) >> 1; dstU[i] = (ru*r + gu*g + bu*b + (0x10001 << (RGB2YUV_SHIFT - 1))) >> RGB2YUV_SHIFT; dstV[i] = (rv*r + gv*g + bv*b + (0x10001 << (RGB2YUV_SHIFT - 1))) >> RGB2YUV_SHIFT;