From patchwork Sun Jan 24 21:32:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Rheinhardt X-Patchwork-Id: 25150 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id AC08844AF6B for ; Sun, 24 Jan 2021 23:33:16 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7E6A86881BD; Sun, 24 Jan 2021 23:33:16 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EA3F86800C9 for ; Sun, 24 Jan 2021 23:33:09 +0200 (EET) Received: by mail-wr1-f54.google.com with SMTP id d16so9671123wro.11 for ; Sun, 24 Jan 2021 13:33:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=QcJVRNbpwzFmo9+KhrpqiAEBrkYOUohZuiZoXVu5ac4=; b=jBnk6R5iewYDL8G7bsW35J4L7U9xiKLm4/EEya052hFK4HIg/v2190to7WaxSZlrAk 8j7j+jvNpLOZSy2kWj2mHyGpPul1RohNHcmxyc6juyscLZAgX3wI10hFyWYIQAtA2HM4 tzoSECm2QWXrCttniE3YffFy86mjLpE94FJC5dHxK1TqzNFOltyDo8YMZUYUQHowBJLk IZcLWW3Jrxm7Jk0Pt/YF5sfr7SfYpdmdV1bRpxzISfcC/vsGuwxvf8VBWzBde9JWWHi+ pkyGRoiDfb5YVi+3uw3rET54HHKf/4EXrW2dkmvHZxU8TJvKx7GlbCK+zW4TmNKBCWLH MS+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=QcJVRNbpwzFmo9+KhrpqiAEBrkYOUohZuiZoXVu5ac4=; b=UAiF0AVsIA6n1Z8u/sfV55nT77SnRieolJ0pe1p0G5dsb90qGzjM380vL9NoyFy671 NjTCRpqdPZ1PiYeanhGqyMBr2VixsuhLVXl3mWf3XBCYvsJpC2V5q56sU8GwQWs8ocAD unAwO0qqkHGwdNXtrXqcLe78+pMajIlp5WFc1dzUXdhJe3791uD8pMFO92Vh3OVssQ8J 9ub4pKj3L2BPnlgTobRfQ25EwnVBfuKsUsJCW3udnX/ZXvpHHkIiV4gqe1vxDA0IqdvM DXRnvQn22kdmxQTiSrFZgckoz3NPhxx6v22+Y/v1o6IbDJzn+uxOG1BM1BshNPwMNr54 6VNA== X-Gm-Message-State: AOAM5310FSQVlSwZKlbneLof+B6/nHyeyOJ/hNR6/hPaErbzb0mMkDwM Qn50SdQ4XxYbciYcQcHt0SwhcQn/zSI= X-Google-Smtp-Source: ABdhPJwd6HPoKWlLLTBIqjfN1QYPfmjQRPRcfIm8GeA7KrYFSWDq188XTHfWLjUbcWVOkgKmypkuzQ== X-Received: by 2002:a5d:4e92:: with SMTP id e18mr4282843wru.66.1611523989144; Sun, 24 Jan 2021 13:33:09 -0800 (PST) Received: from sblaptop.fritz.box (ipbcc1aa4b.dynamic.kabel-deutschland.de. [188.193.170.75]) by smtp.gmail.com with ESMTPSA id v6sm19247837wrx.32.2021.01.24.13.33.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 13:33:08 -0800 (PST) From: Andreas Rheinhardt To: ffmpeg-devel@ffmpeg.org Date: Sun, 24 Jan 2021 22:32:52 +0100 Message-Id: <20210124213253.1344753-1-andreas.rheinhardt@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/2] avfilter/vf_nnedi: Fix segfault when prescreening is disabled X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Since c737f6edcef74a64f4d0ebcefa970bd31266d512 prescreening is nevertheless run because of a wrong check: "if (s->prescreen > 0)". s->prescreen is an array of two function pointers that is contained in the context and comparing it with 0 (i.e. NULL) is actually undefined behaviour, because NULL and s->prescreen do not point to the same object (NULL after all never points to any object). Nevertheless both Clang as well as GCC compile this to code that treat s->prescreen > 0 as true, leading to segfaults, because the code then tries to access the -1th member of an array. This commit fixes the check as well as another such check a few lines below. (Found via compiler warnings enabled by -pedantic: "ordered comparison between pointer and zero is an extension".) Signed-off-by: Andreas Rheinhardt --- libavfilter/vf_nnedi.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavfilter/vf_nnedi.c b/libavfilter/vf_nnedi.c index 1462ce1042..4229150668 100644 --- a/libavfilter/vf_nnedi.c +++ b/libavfilter/vf_nnedi.c @@ -637,7 +637,7 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) width, 1, in_scale); for (int y = 0; y < slice_end - slice_start; y += 2) { - if (s->prescreen > 0) + if (s->pscrn > 0) s->prescreen[s->pscrn > 1](ctx, srcbuf + (y / 2) * srcbuf_stride + 32, srcbuf_stride, prescreen_buf, width, &s->prescreener[s->pscrn - 1]); @@ -649,7 +649,7 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) prescreen_buf, width, &s->coeffs[s->etype][s->nnsparam][s->nsize], s->qual == 2); - if (s->prescreen > 0) + if (s->pscrn > 0) interpolation(srcbuf + (y / 2) * srcbuf_stride + 32, srcbuf_stride, dstbuf + (y / 2) * dstbuf_stride,