From patchwork Thu Sep 12 23:33:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 51559 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:c541:0:b0:48e:c0f8:d0de with SMTP id f1csp774vqr; Thu, 12 Sep 2024 16:34:43 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUa8/LtPUwVpHDLxgX51tqWZW5PQIlMrKdU2tkMhpvlUxeq0lErMICFvo7DzaS5+FhXpJhl+4NZPRlycWhAIefB@gmail.com X-Google-Smtp-Source: AGHT+IEYCI2UbfXJELCmk3hjZ94rFORdlRO00Mm12q145baTOAf2QtAKnJjMEnpNfZyTWfiLvPp0 X-Received: by 2002:a2e:d1a:0:b0:2f7:5980:78ca with SMTP id 38308e7fff4ca-2f787f1cc18mr20082111fa.32.1726184083623; Thu, 12 Sep 2024 16:34:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1726184083; cv=none; d=google.com; s=arc-20240605; b=gYbsUP2SQCDTSWe38SDG/b/QJlnrb0nV20/c9VXSUPAYyz2SnZ0DUd3EGzbOPlSflm TsTRPYquyB4FmU6J/NHVhpV5koQwBGSO/oPupgT6Sh38eBRS8+BR5WHpwNcArPYVeEjE NwrAdSQ8XZHk/cXxqp9cKXa3zWC3Fw3DNpUdl3vzw5D/d5eahaCyPJ7rL62UONixKLuM 4ezsJu2ejtormonqxcI0UPKKxaw03Z4EYM7P94NqVuRDnt0Fd+jGc+JrRAqXDm+Nhf77 OoUUgjh1O0ep8AJcfn7ryMeXKe39fJdSU8jRTrN4f0YWAjx/OxrIuTAizn71gJmmu0PQ 04Bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=yxvrRd6WkOPoclJGePB5FcljEyQl/akF7AQ3lZMLnYQ=; fh=e5zN9xSzcxLA6bGo3lF+CqTbY/oLwzApV03EO/RBfgQ=; b=UiaZg7YbFpVGtUcnZrLxsNnezk8x4ngeP9MpJp/kQ3/LjCcxsGIFONwX5mgSJwn912 cf5CJG9yQLNWCY25sQjgZdaIXXQFB12pv3vWCBJtg8iUDFs2Jkyx1lJw5GUd6EfVfry7 JXP2cWvKlEMU7mKlQnqvqjL19wqwdxgkusLlYfxwV299zm53/Eo8WyrrJGLJasoeNt3g KvAYF6irsY339zcjOF4BtmyMdxjtPTmhC5S9RZH9eTaTp+EXEalsv9VQ52Utgedyvhf8 9y/qPD1gg1zzSjC0aD0AjwS3OOYZfoKcor8hLX07lLuCXKiEyVt1OBwl6H67scLQztm+ 60/w==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@niedermayer.cc header.s=gm1 header.b=fR4omc70; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5c3ebd9f602si8901698a12.425.2024.09.12.16.34.43; Thu, 12 Sep 2024 16:34:43 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@niedermayer.cc header.s=gm1 header.b=fR4omc70; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4334E68DDB4; Fri, 13 Sep 2024 02:33:53 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from relay2-d.mail.gandi.net (relay2-d.mail.gandi.net [217.70.183.194]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3860168DD12 for ; Fri, 13 Sep 2024 02:33:44 +0300 (EEST) Received: by mail.gandi.net (Postfix) with ESMTPSA id 804A740002 for ; Thu, 12 Sep 2024 23:33:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=niedermayer.cc; s=gm1; t=1726184023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ojjnUqR3k0UJsjueCwBS/lX1F24XD+tpGndFn6XahNg=; b=fR4omc70jQKRxvGfYynp1C7ur/QYH2HS9wFN2X3jGpO1S2nMfsFW85+T9x+DTfLouk9j2U vGty0nqputL47P5RM3rbjAy9GFo3t3QqIpZY3joDbIEMaBNRuvD28o+X+yCfqeAXj73845 ZXwU+YoF9ZpbwyF5lM8LZcMLYrMDLjdSyS4WprJ1rTFxSDguUr6vM27YBxNLjxXLLzY433 GjypHfWJuwxYrWtxucYGBuxzAnTVQ+gQeOPH5629XeeFdfz5nLJP8gFKDHdCDq9KqyjrOr SERsPjayl+OJGwyeKTbIlVak9cYnCP4Oe5kqYEJB12ujnBlvfnIaJg+QjNcGew== From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Fri, 13 Sep 2024 01:33:36 +0200 Message-ID: <20240912233337.2444412-6-michael@niedermayer.cc> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20240912233337.2444412-1-michael@niedermayer.cc> References: <20240912233337.2444412-1-michael@niedermayer.cc> MIME-Version: 1.0 X-GND-Sasl: michael@niedermayer.cc Subject: [FFmpeg-devel] [PATCH 6/7] swscale/output: Fix undefined integer overflow in yuv2rgba64_2_c_template() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Yxntwp+rvsoa Fixes: signed integer overflow: -1082982400 + -1083218484 cannot be represented in type 'int' Fixes: 70657/clusterfuzz-testcase-minimized-ffmpeg_SWS_fuzzer-6707819712675840 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer --- libswscale/output.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/libswscale/output.c b/libswscale/output.c index abfb0fd1cee..31921a3ccec 100644 --- a/libswscale/output.c +++ b/libswscale/output.c @@ -1150,8 +1150,8 @@ yuv2rgba64_2_c_template(SwsContext *c, const int32_t *buf[2], av_assert2(uvalpha <= 4096U); for (i = 0; i < ((dstW + 1) >> 1); i++) { - int Y1 = (buf0[i * 2] * yalpha1 + buf1[i * 2] * yalpha) >> 14; - int Y2 = (buf0[i * 2 + 1] * yalpha1 + buf1[i * 2 + 1] * yalpha) >> 14; + unsigned Y1 = (buf0[i * 2] * yalpha1 + buf1[i * 2] * yalpha) >> 14; + unsigned Y2 = (buf0[i * 2 + 1] * yalpha1 + buf1[i * 2 + 1] * yalpha) >> 14; int U = (ubuf0[i] * uvalpha1 + ubuf1[i] * uvalpha - (128 << 23)) >> 14; int V = (vbuf0[i] * uvalpha1 + vbuf1[i] * uvalpha - (128 << 23)) >> 14; int R, G, B; @@ -1175,20 +1175,20 @@ yuv2rgba64_2_c_template(SwsContext *c, const int32_t *buf[2], A2 += 1 << 13; } - output_pixel(&dest[0], av_clip_uintp2(((R_B + Y1) >> 14) + (1<<15), 16)); - output_pixel(&dest[1], av_clip_uintp2((( G + Y1) >> 14) + (1<<15), 16)); - output_pixel(&dest[2], av_clip_uintp2(((B_R + Y1) >> 14) + (1<<15), 16)); + output_pixel(&dest[0], av_clip_uintp2(((int)(R_B + Y1) >> 14) + (1<<15), 16)); + output_pixel(&dest[1], av_clip_uintp2(((int)( G + Y1) >> 14) + (1<<15), 16)); + output_pixel(&dest[2], av_clip_uintp2(((int)(B_R + Y1) >> 14) + (1<<15), 16)); if (eightbytes) { output_pixel(&dest[3], av_clip_uintp2(A1 , 30) >> 14); - output_pixel(&dest[4], av_clip_uintp2(((R_B + Y2) >> 14) + (1<<15), 16)); - output_pixel(&dest[5], av_clip_uintp2((( G + Y2) >> 14) + (1<<15), 16)); - output_pixel(&dest[6], av_clip_uintp2(((B_R + Y2) >> 14) + (1<<15), 16)); + output_pixel(&dest[4], av_clip_uintp2(((int)(R_B + Y2) >> 14) + (1<<15), 16)); + output_pixel(&dest[5], av_clip_uintp2(((int)( G + Y2) >> 14) + (1<<15), 16)); + output_pixel(&dest[6], av_clip_uintp2(((int)(B_R + Y2) >> 14) + (1<<15), 16)); output_pixel(&dest[7], av_clip_uintp2(A2 , 30) >> 14); dest += 8; } else { - output_pixel(&dest[3], av_clip_uintp2(((R_B + Y2) >> 14) + (1<<15), 16)); - output_pixel(&dest[4], av_clip_uintp2((( G + Y2) >> 14) + (1<<15), 16)); - output_pixel(&dest[5], av_clip_uintp2(((B_R + Y2) >> 14) + (1<<15), 16)); + output_pixel(&dest[3], av_clip_uintp2(((int)(R_B + Y2) >> 14) + (1<<15), 16)); + output_pixel(&dest[4], av_clip_uintp2(((int)( G + Y2) >> 14) + (1<<15), 16)); + output_pixel(&dest[5], av_clip_uintp2(((int)(B_R + Y2) >> 14) + (1<<15), 16)); dest += 6; } }