From patchwork Thu Dec 15 00:32:18 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 1792 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.65.86 with SMTP id o83csp465591vsa; Wed, 14 Dec 2016 16:32:30 -0800 (PST) X-Received: by 10.194.222.132 with SMTP id qm4mr106758879wjc.150.1481761950078; Wed, 14 Dec 2016 16:32:30 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 203si9909307wme.104.2016.12.14.16.32.29; Wed, 14 Dec 2016 16:32:30 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DA6D5689A5C; Thu, 15 Dec 2016 02:32:21 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe04-3.mx.upcmail.net (vie01a-dmta-pe04-3.mx.upcmail.net [62.179.121.165]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9BBD56891F6 for ; Thu, 15 Dec 2016 02:32:15 +0200 (EET) Received: from [172.31.216.43] (helo=vie01a-pemc-psmtp-pe01) by vie01a-dmta-pe04.mx.upcmail.net with esmtp (Exim 4.87) (envelope-from ) id 1cHJy8-0001Kx-RC for ffmpeg-devel@ffmpeg.org; Thu, 15 Dec 2016 01:32:20 +0100 Received: from localhost ([213.47.41.20]) by vie01a-pemc-psmtp-pe01 with SMTP @ mailcloud.upcmail.net id L0YK1u00k0S5wYM010YLZ3; Thu, 15 Dec 2016 01:32:20 +0100 X-SourceIP: 213.47.41.20 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Thu, 15 Dec 2016 01:32:18 +0100 Message-Id: <20161215003218.20579-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.11.0 Subject: [FFmpeg-devel] [PATCH] avcodec/flacdsp: Avoid undefined operations in non debug builds X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This fixes ubsan warnings in non debug builds by using unsigned operations in debug builds the correct signed operations are retained so that overflows (which should not occur in valid files and may indicate problems in the DSP code or decoder) can be detected. Alternatively they can be changed to unsigned unconditionally, then its not possible though to detect overflows easily if someone wants to test the DSP code for overflows. The 2nd alternative would be to leave the code as it is and accept that there are undefined operations in the DSP code and that ubsan output is full of them in some cases. Similar changes would be needed in some other DSP routines Suggested-by: Matt Wolenetz Signed-off-by: Michael Niedermayer --- libavcodec/flacdsp.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/libavcodec/flacdsp.c b/libavcodec/flacdsp.c index 30b66484e8..180d4e7ba6 100644 --- a/libavcodec/flacdsp.c +++ b/libavcodec/flacdsp.c @@ -43,14 +43,22 @@ #define PLANAR 1 #include "flacdsp_template.c" +// For debuging we use signed operations so overflows can be detected (by ubsan) +// For production we use unsigned so there are no undefined operations +#ifdef DEBUG +#define SUINT int +#else +#define SUINT unsigned +#endif + static void flac_lpc_16_c(int32_t *decoded, const int coeffs[32], int pred_order, int qlevel, int len) { int i, j; for (i = pred_order; i < len - 1; i += 2, decoded += 2) { - int c = coeffs[0]; - int d = decoded[0]; + SUINT c = coeffs[0]; + SUINT d = decoded[0]; int s0 = 0, s1 = 0; for (j = 1; j < pred_order; j++) { s0 += c*d; @@ -66,7 +74,7 @@ static void flac_lpc_16_c(int32_t *decoded, const int coeffs[32], if (i < len) { int sum = 0; for (j = 0; j < pred_order; j++) - sum += coeffs[j] * decoded[j]; + sum += coeffs[j] * (SUINT)decoded[j]; decoded[j] += sum >> qlevel; } }