From patchwork Tue Nov 15 19:30:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeremy Dorfman X-Patchwork-Id: 39291 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp3310200pzb; Tue, 15 Nov 2022 11:31:04 -0800 (PST) X-Google-Smtp-Source: AA0mqf5HuNJadv2GtLcUkRtAs41iytKJbjvKi7Xt+V17/A5BpPapVEJP82y+4ETqBtFi9lzC3dH1 X-Received: by 2002:a17:906:4d58:b0:781:bc28:f455 with SMTP id b24-20020a1709064d5800b00781bc28f455mr14927154ejv.170.1668540664415; Tue, 15 Nov 2022 11:31:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668540664; cv=none; d=google.com; s=arc-20160816; b=hdCMSNcTXCyOeEG/BRTanIeFmSccItzSE2YvegxilK7hKTzJiMl6c23Ak0+WbgSDnO RRS4+4kURRZiBLQeFB9CHfV+XXJMtghjGitiEUYP9YL5+A0OfomzUCx5R1cwQsbRTLbE tzWKysw3qmNbAjEsneEX9od6ftTmUDmb1iLBp4AclboD9yrCvIZpbXNbs0RfJXJ89Vy+ QOLcwuNQTTGgCe83OvFR9IC3JzgYkc4dJ/g4UKOdtsKUooVxcECQaMJLODVs7ZnQvzor 43fubQXZK6pFK1nRi8iwxtAocbT5jcdxZ6ImSrHvATB72zelm9rZy87GXql5NAKndQa1 3yzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:from:message-id:mime-version:date :dkim-signature:delivered-to; bh=G/ktQ4KAn49Vt0czsKWcVrnZq25TZaCeJR+uUF5Y70c=; b=rgDYGPUG3FXXiilWkHyAe1rOKi1uHwXNhRWeLvbq7qIO36ptzLxNNMMFYrN39lagXZ dV3XGiWh8L+RtYviFttupMe4xgA5EwByI42OWSBemPKZTPcRYpOMvc/IDVFWQtai09cF uD1XLHpfJ3Y5RCiytLYKx/sj6Uk4XOHQSONAAfWYdj59sbXMvsU4+mnfOzzpqCc4xOsM fbUnR/tx2qdIhWv9ppTmkmdkRyphWi8I17dxrfhD8X6FwDPTYmjUQOaODKsgGQepaC7U nAg1kAGuGRMaqmNryV2yDMdkC8IjnSghs5VirtalfcCFiSWr9Fk1EHiVz0pmmgrpjhlR BJ9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20210112 header.b=UzwX8Aw4; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id h22-20020a056402281600b004661bfac392si15683040ede.311.2022.11.15.11.30.43; Tue, 15 Nov 2022 11:31:04 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20210112 header.b=UzwX8Aw4; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6926068BD2D; Tue, 15 Nov 2022 21:30:40 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6C45568B0D3 for ; Tue, 15 Nov 2022 21:30:34 +0200 (EET) Received: by mail-yb1-f201.google.com with SMTP id 204-20020a2510d5000000b006be7970889cso14221602ybq.21 for ; Tue, 15 Nov 2022 11:30:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=ArO73+w4I/cijdCz5T5iRIfHM3ZQ9eyTXB/F7iAreyI=; b=UzwX8Aw4e1f+SSCb6Tx64FmMGYbHbzIak9vdnpE4yJYknBubV3JdWiOLlKSdnEXQpF DnfcNr1yDi9mMaH8WZdImwDbmpeHo7YSsH202c/sVCo4aDUx3n6QcuTK++UlIsJyFYth kyT1IhllOGsi4xe4bAI/KySm/ikP33P0oj7U/H5RTSa4pc5gwvFcZKa0NSPtuK1nnOJu UCZhMjJbLCIkyPcjOFdiqJ+cQKOn3JtLBfbWPaPDj2dYCyjvrP7m0uVvkN7R4q/471NJ cbBWheKEhOlChEykuIUcLhdo0Ir7meO92j8S27AL3fOVm6o2/Yhlzf4j4s5CiStDGhzn RmFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ArO73+w4I/cijdCz5T5iRIfHM3ZQ9eyTXB/F7iAreyI=; b=tDP8rHBsVgqV4l5ctnKSuzgSRRAfFRJSJpL5WQLUv0X4q2v+f944FJPCu3rIZem5JD Op69pFnjsKn+4Uqkb8G/3bBmAmo2gcWlfJZIE6W4H+hupG+0sqbd/B7rf2x9xx0G/LPM J61UCwPqltm6U/4b9c1E0THDxxPSYqFJaJK5jnF7sjMQo2HQ+L4EJWErAXUUePPOcgzZ xRVCZSvHn1b3IR1VFzrq++lr/Ro3Qa/LwV2cgwTAvo5vFG1zlA5upJQeDDpfgb72Q5VZ pT3uLR5Z25x2Ujl7kEYSnV0stUGQz+1OFlzTir36V5pokVkhqeMz56e0b9rxNIVqCzHo KA1g== X-Gm-Message-State: ACrzQf02ynTmicdBrK9v4wOaEvFD6Xdfd4iEvhqyyVCv7ftQl6mJQ+zu a+ym3HpafuJRwx1xAwJQNpYylXu29Po6UOCXpCz3SJzwpxl0IuxbqD9r1uO7CI67fH7Oi+4R7Ob c4tdAeb3oP49XPz1LXavJT0MVTgAolvBZ+/9cTUcMKuV8kJZ+eogR8I7ZlMlhrDRMEejB X-Received: from jdorfman01.cam.corp.google.com ([2620:15c:93:a:72da:426f:3662:b649]) (user=jdorfman job=sendgmr) by 2002:a25:1041:0:b0:6bd:6c15:b8f7 with SMTP id 62-20020a251041000000b006bd6c15b8f7mr66096569ybq.325.1668540631783; Tue, 15 Nov 2022 11:30:31 -0800 (PST) Date: Tue, 15 Nov 2022 14:30:23 -0500 Mime-Version: 1.0 X-Mailer: git-send-email 2.38.1.493.g58b659f92b-goog Message-ID: <20221115193022.2362787-1-jdorfman@google.com> From: Jeremy Dorfman To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH] swscale/input: Use unsigned intermediates in rgb64ToUV_c_template X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Jeremy Dorfman Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Tz8aNtNVP6Pa Large rgb2yuv tables and high pixel values cause the intermediate int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined behavior. This causes libswscale built with LLVM -fsanitize=undefined to assert. Using unsigned integers instead has defined behavior and produces identical results, and makes rgb64ToUV_c_template match rgb64ToY_c_template. Fixes: signed integer overflow --- libswscale/input.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libswscale/input.c b/libswscale/input.c index 7ff7bfaa01..d7dbedd82f 100644 --- a/libswscale/input.c +++ b/libswscale/input.c @@ -65,9 +65,9 @@ rgb64ToUV_c_template(uint16_t *dstU, uint16_t *dstV, int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX]; av_assert1(src1==src2); for (i = 0; i < width; i++) { - int r_b = input_pixel(&src1[i*4+0]); - int g = input_pixel(&src1[i*4+1]); - int b_r = input_pixel(&src1[i*4+2]); + unsigned int r_b = input_pixel(&src1[i*4+0]); + unsigned int g = input_pixel(&src1[i*4+1]); + unsigned int b_r = input_pixel(&src1[i*4+2]); dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; dstV[i] = (rv*r + gv*g + bv*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT;