From patchwork Thu Sep 1 19:28:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Carl Eugen Hoyos X-Patchwork-Id: 37600 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:139a:b0:8f:1db5:eae2 with SMTP id w26csp491663pzh; Thu, 1 Sep 2022 12:28:31 -0700 (PDT) X-Google-Smtp-Source: AA6agR6N1U4DcEjrQu2pP6FrRkrqQbPswO5GUjNOCSB1fA60C40LoKDPoeoK3MrDa0kDEpriznSW X-Received: by 2002:a05:6402:1e8f:b0:440:eb20:7a05 with SMTP id f15-20020a0564021e8f00b00440eb207a05mr29464513edf.169.1662060510905; Thu, 01 Sep 2022 12:28:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1662060510; cv=none; d=google.com; s=arc-20160816; b=P8ghy9n0DopiqmDbOh0ocqck8zXOD39s3Mycijg3WD0nY7rChXBgHQ/YB9O/4NN4dH XXUmtZGJupdZQ3B/tSGc5AspwJSqrnj4hUDG/bOIq+SEU+NP98BXRmgIUxQlgzsaxVRw IepIBzS2oMChnIr81SNIHjXXRDgfIdudZNCARKMYbzwbKXpGgpyocjAGczDRGuz8R/Tz 3Tf1KTskKcSLJxQzcmnqqaq6p0Kf0ze9LVMko93D9gQxuZbslx2nKkyCeqywjR6zNYqj RzHMcw+0y5FU+kqQu599w6mNk+GXEoEjP4RO1gkHrBWWyu9ou43lGvWUZliN2LiJS4dV dGUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject:to :message-id:date:from:mime-version:dkim-signature:delivered-to; bh=8NkkwTWtILBRqlY5vky7fUVogryNB6G6wk573jE3kZ0=; b=b3rYEEHvTzJzLZLDyJvSkvWE9vYJs6ZNI7kBGqO3wICHW51M6eD6Oxa88crAKDcL/p tCUs1z9GhUh1nmsp24sA9V1W+q64tIT1HXOjGA0+lj7l5jE9NBfT6wDTLEl7EqRdp1WX mGvZTwh0oJfAPKXj+QQh7pn8lbpbUY5+13HT+NNTdZbhrltWSW9HilGzvDHqelcMxmzA ZcSj00/P2pwEpfuio7PvvtEzRl6eIkZc0M/wIOHW59KuU20hL34d7YUBwq/c1s2oRUcP LD/N3//kE30tRfLKg1zxKaDMz/pv+jKU9Alc+enBKIdXIUaW+pwGkZH5AU0eMR3ku3MZ 7uBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=B08nlf+p; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x3-20020a056402414300b0043dcd3625e4si2249459eda.328.2022.09.01.12.28.29; Thu, 01 Sep 2022 12:28:30 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=B08nlf+p; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 989F568B9D0; Thu, 1 Sep 2022 22:28:27 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B21A568B9D0 for ; Thu, 1 Sep 2022 22:28:20 +0300 (EEST) Received: by mail-yb1-f175.google.com with SMTP id 130so171182ybw.8 for ; Thu, 01 Sep 2022 12:28:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date; bh=buRZgRxmY1mXxr2H6tG4sYwvAvbxbIk9LZiSiP/l4uM=; b=B08nlf+pxW5gF8A106h61qybAujbj91raAPqqeDkVGKDW5GUNlJX1JN5Fc2KwJ+x3c KKHv5zZPQ8STya44Cce8Rn4OZ9n4IkOATbxQqmxocBsA1yA99+lL42hYPIYyYcFuXvzd BtNWi/pkKx7ozjeNRFHpODvv7Qvzq5/YjkwkucENH9jRA8HZwJG9dceftyT41P9DwrX0 ggUFN06Nv/mEkmRFnovCG47Xv3WPpCoVYVjuQJoBCZMo2+j5njeMLTY8MC3VpkudniYC o1BIK5O775A+tz/zSEHVO3Ul3/D45AxrtqqBVzwGQp9MRtm4fOlaKsLGD/3D4VSkVF7D MhBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date; bh=buRZgRxmY1mXxr2H6tG4sYwvAvbxbIk9LZiSiP/l4uM=; b=I00Of39Df5GGHo5y3pbEKkBVXPpl+u5DqLqNWW31o+t+gWgJbiz5dORwNximV5CnwT wKVwkjA9z3ExQyRpIw0LkOuF/nQgqoOESFQNvIixwdKzt1nZSDZUNiIG5GCiAJNB248h fQ6S02DNUJQHL3U1diHi9vV3ziwCeIsA/QpHQiAOg0NQA4KUoZp5WUyvjZ97zzJzmeuB Jp2bUYXWcQ+a/CPt2rilFCbBvIdQ3mT7FubCAyxsTAoxg0rVwe6EVMg6Yef/v1ay3aCs ZKWNdwNMAXRTXMXpHlwxxDdHGkManOBNyzCfi+ISLW9Z0hiEerPIYkUkid5MVtTehAbb lK1w== X-Gm-Message-State: ACgBeo3PBCdupd4/4OoEZZxAAvJ7bOxAKk/k/ER5p6BqJVoF4gyTUO34 F3xcOYjzKDgyZHZis95rQSfK2SU+x7Vdf7w0gcpPhr3m X-Received: by 2002:a25:874d:0:b0:695:9b0d:abfe with SMTP id e13-20020a25874d000000b006959b0dabfemr20621467ybn.88.1662060499255; Thu, 01 Sep 2022 12:28:19 -0700 (PDT) MIME-Version: 1.0 From: Carl Eugen Hoyos Date: Thu, 1 Sep 2022 21:28:08 +0200 Message-ID: To: FFmpeg development discussions and patches Subject: [FFmpeg-devel] [PATCH]lavfi/rotate: Fix undefined behaviour X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: uoJQl877Yvh9 Hi! Attached patch fixes ticket #9799. Please comment, Carl Eugen From 2cce687961c3b56a92d88184269bf9fa075ae297 Mon Sep 17 00:00:00 2001 From: Carl Eugen Hoyos Date: Thu, 1 Sep 2022 20:55:54 +0200 Subject: [PATCH] lavfi/rotate: Avoid undefined behaviour. Fixes the following integer overflows: libavfilter/vf_rotate.c:273:13: runtime error: signed integer overflow: 92951468 + 2058533568 cannot be represented in type 'int' libavfilter/vf_rotate.c:273:37: runtime error: signed integer overflow: 39684 * 54149 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:13: runtime error: signed integer overflow: 247587320 + 1900985032 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:37: runtime error: signed integer overflow: 42584 * 50430 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:50: runtime error: signed integer overflow: 65083 * 52912 cannot be represented in type 'int' libavfilter/vf_rotate.c:273:50: runtime error: signed integer overflow: 65286 * 38044 cannot be represented in type 'int' Fixes ticket #9799, different output with different compilers. --- libavfilter/vf_rotate.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavfilter/vf_rotate.c b/libavfilter/vf_rotate.c index 4429e3d543..d319dfe3d9 100644 --- a/libavfilter/vf_rotate.c +++ b/libavfilter/vf_rotate.c @@ -269,8 +269,8 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, int s01 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y ]); int s10 = AV_RL16(&src[src_linestep * int_x + i + src_linesize * int_y1]); int s11 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y1]); - int s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); - int s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); + int64_t s0 = (((int64_t)(1<<16) - frac_x)*s00 + (int64_t)frac_x*s01); + int64_t s1 = (((int64_t)(1<<16) - frac_x)*s10 + (int64_t)frac_x*s11); AV_WL16(&dst_color[i], ((int64_t)((1<<16) - frac_y)*s0 + (int64_t)frac_y*s1) >> 32); } -- 2.30.1