From patchwork Sun Nov 12 15:11:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Ramacher X-Patchwork-Id: 35010 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:92a5:b0:181:818d:5e7f with SMTP id q37csp836597pzg; Sun, 12 Nov 2023 07:12:37 -0800 (PST) X-Google-Smtp-Source: AGHT+IHZ17xZffowZDSARGeNLWzaG6z3T59+CPGQLjeU6Xkzg35S2e/qPbHVnmYyEw+Im42erpIA X-Received: by 2002:a17:907:e89:b0:9de:6d57:c005 with SMTP id ho9-20020a1709070e8900b009de6d57c005mr9035274ejc.9.1699801957206; Sun, 12 Nov 2023 07:12:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699801957; cv=none; d=google.com; s=arc-20160816; b=UTNQeEBjACYRLYtQ70wrAYgdpBKxNX81cMiCeiUWeeLNmAULPRyp9VrfJIq3Cbl6Hu 5nbHuHnE9fLh85Piip8pg7aN6wPuuOYZf1zHWHT/qoHH23tCWr8j4YfKOoW2AD9oHlFs nW1CVBbNl3URu36RGDzTbN6chmcts7v0D6+pTX3qRdTMdjvrzQU7kGgSMzzs9FK5TjeM wyXnHHAfdXq61xL/EmQnmTh1hnwNBm4gs3fHMsfkcT9xyYdIWzXjq8XrOFsG2vLFhXZM ynpXF++tWXCTBP2Cmbe0Lv6gN0g3Mm9q+iWQKSefe+EP7F9eO59kfhIZNvUZCMHyArGx YmPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=6LA74XVv+NAyAPVojWeJee9x6LiVpI6rue+5JvNiKMk=; fh=w39lni1P4BnnoxhBEex6CszENd6FI9kbu5bpw99hC34=; b=xbG2yjdvootadxuT/Rv7SEGbrUNjmchMDWJ3y8mUIO34//CfEXWOdUIDYwLR7Om414 4ddhr6tD+7lmgJGfEt1TcK1qW4thdZuq/JEtMv/n0bjBo6gNV3+/jhkv9jJpWxIKv+bo 0/L2OP7fvGnP7lXrhIxNjVgKWYe8ZMwGJvFL15+fcw1Ss7OCljSlMT58Btcv5SSMTHpo 3676n7qX6OhRSQNdwXY30XUukTtQ/BNR80bCJGUd+fVFw+9M8FnzRBf+ZUOkoRWDnoNm fbHvinC9V/EXmx2k6g0zx2fSPsaoet77dtE/oL4geMXAk5IdkQnUZlmqF1L8F8wFIaOb dQCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@ramacher.at header.s=mail header.b=3KMNq+rq; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id vi12-20020a170907d40c00b009d8007870d8si1784729ejc.265.2023.11.12.07.12.20; Sun, 12 Nov 2023 07:12:37 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@ramacher.at header.s=mail header.b=3KMNq+rq; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DFAC268CC4E; Sun, 12 Nov 2023 17:12:08 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from alphacentauri.ramacher.at (alphacentauri.ramacher.at [195.201.139.148]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 55E3768CC4E for ; Sun, 12 Nov 2023 17:12:02 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ramacher.at ; s=mail; h=MIME-Version:Message-ID:Date:Subject:Cc:To:From:References: In-Reply-To:Content-Type; bh=3gUv6zuCPuKoz/tQ3OpiXh3X/aAimop1rhJhIxv3eVI=; b= 3KMNq+rq5+Scxtn/UGbv8ccjZvWfEfdBH69/o2oM56tURGL5bx7utszlxRXK3UpZ2apW8qzhl2nFs PnjrpsvSD7GdvFerW5IRN8MxpCoWk37rYMlzkoHtIukfuy+5hSetdRwrk1k9U7VZKh0cFewn4Cu9N SmHNWL81PAwd2Hq1s=; From: Sebastian Ramacher To: ffmpeg-devel@ffmpeg.org Date: Sun, 12 Nov 2023 16:11:41 +0100 Message-ID: <20231112151144.2307049-1-sramacher@debian.org> X-Mailer: git-send-email 2.42.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 0/3] Fix invalid frees, segfaults and memory leaks in avcodec/fft wrappers X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Sebastian Ramacher Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: tNV38Tdgcqwl The wrappers in avcodec/fft which were introduced in 6.1 may lead to invalid frees, segfaults and memory leaks. Consider the following example program: #include int main() { FFTContext* fft = av_fft_init(11, 0); av_fft_end(fft); FFTContext* mdct = av_mdct_init(11, 0, 1.0); av_mdct_end(mdct); mdct = av_mdct_init(11, 1, 1.0); av_mdct_end(mdct); RDFTContext* rdft = av_rdft_init(11, DFT_R2C); av_rdft_end(rdft); DCTContext* dct = av_dct_init(11, DCT_II); av_dct_end(dct); } When executed under valgrind, one obtains: ==2300086== Memcheck, a memory error detector ==2300086== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al. ==2300086== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info ==2300086== Command: ./a.out ==2300086== ==2300086== Conditional jump or move depends on uninitialised value(s) ==2300086== at 0x5FB6CBE: av_tx_uninit (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x490B3AA: av_fft_end (in /usr/lib/x86_64-linux-gnu/libavcodec.so.60.31.102) ==2300086== by 0x1090D7: main (test.c:5) ==2300086== Uninitialised value was created by a heap allocation ==2300086== at 0x4845990: memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2300086== by 0x4845AED: posix_memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2300086== by 0x5FFAC14: av_malloc (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x4A4B4A5: av_fft_init (in /usr/lib/x86_64-linux-gnu/libavcodec.so.60.31.102) ==2300086== by 0x1090CF: main (test.c:4) ==2300086== ==2300086== Conditional jump or move depends on uninitialised value(s) ==2300086== at 0x4843131: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2300086== by 0x490B4A5: av_dct_end (in /usr/lib/x86_64-linux-gnu/libavcodec.so.60.31.102) ==2300086== by 0x10913A: main (test.c:17) ==2300086== Uninitialised value was created by a heap allocation ==2300086== at 0x4845990: memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2300086== by 0x4845AED: posix_memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2300086== by 0x5FFAC14: av_malloc (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x4A4B96F: av_dct_init (in /usr/lib/x86_64-linux-gnu/libavcodec.so.60.31.102) ==2300086== by 0x109132: main (test.c:16) ==2300086== ==2300086== ==2300086== HEAP SUMMARY: ==2300086== in use at exit: 66,528 bytes in 270 blocks ==2300086== total heap usage: 1,353 allocs, 1,083 frees, 386,566 bytes allocated ==2300086== ==2300086== 8,064 (640 direct, 7,424 indirect) bytes in 1 blocks are definitely lost in loss record 247 of 249 ==2300086== at 0x4845990: memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2300086== by 0x4845AED: posix_memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2300086== by 0x5FFAC14: av_malloc (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x5FFAF80: av_mallocz (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x5FB732D: ??? (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x5FB7616: av_tx_init (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x4A4B678: av_mdct_init (in /usr/lib/x86_64-linux-gnu/libavcodec.so.60.31.102) ==2300086== by 0x10910A: main (test.c:10) ==2300086== ==2300086== 8,192 bytes in 1 blocks are possibly lost in loss record 248 of 249 ==2300086== at 0x4845990: memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2300086== by 0x4845AED: posix_memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2300086== by 0x5FFAC14: av_malloc (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x6030F60: ??? (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x5FBC968: ??? (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x5FB73C8: ??? (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x5FB73C8: ??? (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x5FB7616: av_tx_init (in /usr/lib/x86_64-linux-gnu/libavutil.so.58.29.100) ==2300086== by 0x4A4B678: av_mdct_init (in /usr/lib/x86_64-linux-gnu/libavcodec.so.60.31.102) ==2300086== by 0x10910A: main (test.c:10) ==2300086== ==2300086== LEAK SUMMARY: ==2300086== definitely lost: 640 bytes in 1 blocks ==2300086== indirectly lost: 7,424 bytes in 4 blocks ==2300086== possibly lost: 8,192 bytes in 1 blocks ==2300086== still reachable: 48,256 bytes in 243 blocks ==2300086== suppressed: 0 bytes in 0 blocks ==2300086== Reachable blocks (those to which a pointer was found) are not shown. ==2300086== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==2300086== ==2300086== For lists of detected and suppressed errors, rerun with: -s ==2300086== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0) This patch series fixes the above issues. The initial issue in av_fft_end was discuvered via the test suite of r-cran-av. Sebastian Ramacher (3): avcodec/fft: Do not uninit never initialized ctx2 avcodec/fft: Set potentially unused wrapper variables to avoid invalid free/uninit avcoded/fft: Fix memory leak if ctx2 is used libavcodec/avfft.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)