From patchwork Tue Mar 21 12:37:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Martin_Storsj=C3=B6?= X-Patchwork-Id: 40751 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:d046:b0:cd:afd7:272c with SMTP id hv6csp2546813pzb; Tue, 21 Mar 2023 05:37:48 -0700 (PDT) X-Google-Smtp-Source: AK7set/6RO9SW1D/O1DVlWP/rJueo1tcHlOuai09DXSaUyEVznmB7Ig2pmJJE7eAqE90fncawN2G X-Received: by 2002:a17:906:f8c2:b0:930:3916:df17 with SMTP id lh2-20020a170906f8c200b009303916df17mr2971975ejb.0.1679402268317; Tue, 21 Mar 2023 05:37:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679402268; cv=none; d=google.com; s=arc-20160816; b=MnQUv96QuMbxqlJ/4BlTbcIa/zqcY1kWCJG26cinpx4iF3DkGMPL/QQhvt+XkdSJIC w3snVyidL9lQWzbc7Pru48W6MDqxRLzqgi+4pyPHojbfnrsztz5NyVNrJSQfsn4lJXvl oDlHB/oWXEy9IOCzZLV1/70u7HKNGHcGCQ4iGFJw+5pSgDmxpOEXNOR2XqRm0ZaeCbHz 5j8hibE9grN7HkxSGI82GmXVSJZ1Z6INUp8sqI7fSH8TM+h5v9W6a5+Rg139gY4MpRf+ 1Kybo6KE6epNGELMXT3rXL7ynPaWbKvE/YgB03oM/CYQKIFnph9lI9E/smTjY7XX3JiU q4bg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=XSF5FqCpvoou/AfQnsmGc94766pk8vXse4h6BJOf/Hc=; b=L1OL62haC/yxgAInQoTFSBM4yDQRwLTPMehet5Br1K29eonYB0TOHbOt+FoTAoC5x9 X5Jcq4sLdh4P4RWgrnjJ5YQ+loxf+ii/mFBYqFld7UDpJajZ/zwxi2uHiotqhFSNBKEd sZLPcpwmT2QOGX6CqXubVRSZvteSmxW/d1lcPRjx348wZAapCZWIbddH8NzrFG4x13+b T7nR9ZW3UJKeMDj3DwrP6wWPg7T9aIYbUPZ+ou0f0UcMdnTLlK64WSZ5Y0IsR4y2wE+K ZX2R9zN5TWBXFQhVrIo3rNcnG+RXkfejKTcM1vgh9pzqUkNlqXyerZ35SHg27agsWJ5F NDEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@martin-st.20210112.gappssmtp.com header.s=20210112 header.b=fxnQ0YJc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s5-20020a170906960500b00930f5ecbd94si13153706ejx.346.2023.03.21.05.37.47; Tue, 21 Mar 2023 05:37:48 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@martin-st.20210112.gappssmtp.com header.s=20210112 header.b=fxnQ0YJc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2779468C68B; Tue, 21 Mar 2023 14:37:38 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0C31D68C67A for ; Tue, 21 Mar 2023 14:37:32 +0200 (EET) Received: by mail-lf1-f47.google.com with SMTP id k37so5967013lfv.0 for ; Tue, 21 Mar 2023 05:37:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=martin-st.20210112.gappssmtp.com; s=20210112; t=1679402251; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=ZkxvFnMHZ8BJjnl6pVvRX6gu0RAXmBtTQJcyjvZPF7E=; b=fxnQ0YJcLivrISNJDmPGIwKI3WdLiG/niUg1oywuWt0i+xf81nkgoQd38vMSi0GWhK DWEa7rNqFYW7kqfLIiwXEk6zIQ6gOmPb4MMNK0jnzMhmHWraG1gtzuz/9OtBmupv5M2u oqEzi0yVaD4KO60nw5l7uG1dIrsz3P0shFCpC45Twu9JXv6WfyNUGP3I/JJ3BmlLLIZr z0zgBOSchgA6lzJHJUdmnLD1P88OJDF6LjfSH50NRO8AmrkIDh+L9TSAVzcYD5Ree4l3 Ilg8UakQe73j5lbz4lq7UOCqPPwUTM/r+RWkOi1YErqO5Cu1WzdT8euVROGylPffccSA 6seA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679402251; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZkxvFnMHZ8BJjnl6pVvRX6gu0RAXmBtTQJcyjvZPF7E=; b=k0y12BBKXAHqyRFpZrirW1iCdZnRg4aGE6twnFzXinj1Wf6torftIkcTOFqAjWJ0Hf LZ+O7i4McpZw2mp/a/lsTfjmCIeVjSK9DnKaF/QggFQtoRehnBLWqaWaP1PRcI9E7f+Y aymmNpzEFfrcvHiiiaiL8a7DE3nXI+A5jCDVtRqDRm6PlrAcWltSec33becEOYkrZ7xa tCgFBy9OZL4eaVNmTiuK8x2Hqm/xno/zVvMyMN1VwGkwzbnR3QO0nBHbMLgIzdH6jvPO p/FzInL45kvBwKhYuZ3HMXKf8KCLWrDwG17DW8yOOA3XHKktW8+cAtrjUX3nP0ae58Tw T6Dg== X-Gm-Message-State: AO0yUKV1pAyx+C09caGQsaiqIDn25bCFlX99yDX6CiesYi6E6LV3DPtQ IDKnwjKpybTp19t/3lS3G4FXNdnymdJ5hkHurHEBaQ== X-Received: by 2002:ac2:5231:0:b0:4e0:ee54:fa23 with SMTP id i17-20020ac25231000000b004e0ee54fa23mr769136lfl.8.1679402251203; Tue, 21 Mar 2023 05:37:31 -0700 (PDT) Received: from localhost (dsl-tkubng21-58c01c-243.dhcp.inet.fi. [88.192.28.243]) by smtp.gmail.com with ESMTPSA id r28-20020ac252bc000000b004e9cad1cd7csm1040471lfm.229.2023.03.21.05.37.30 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Mar 2023 05:37:30 -0700 (PDT) From: =?utf-8?q?Martin_Storsj=C3=B6?= To: ffmpeg-devel@ffmpeg.org Date: Tue, 21 Mar 2023 14:37:29 +0200 Message-Id: <20230321123729.74124-2-martin@martin.st> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230321123729.74124-1-martin@martin.st> References: <20230321123729.74124-1-martin@martin.st> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] aviobuf: Avoid clearing the whole buffer in fill_buffer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 8VQYmg5BEaZO Normally, fill_buffer reads in one max_packet_size/IO_BUFFER_SIZE worth of data into the buffer, slowly filling the buffer until it is full. Previously, when the buffer was full, fill_buffer would start over from the start, effectively discarding all the previously buffered data. For files that are read linearly, the previous behaviour was fine. For files that exhibit some amount of nonlinear read patterns, especially mov files (where ff_configure_buffers_for_index increases the buffer size to accomodate for the nonlinear reading!) we would mostly be able to seek within the buffer - but whenever we've hit the maximum buffer size, we'd discard most of the buffer and start over with a very small buffer, so the next seek backwards would end up outside of the buffer. Keep one fourth of the buffered data, moving it to the start of the buffer, freeing the rest to be refilled with future data. For mov files with nonlinear read patterns, this almost entirely avoids doing seeks on the lower IO level, where we previously would end up doing seeks occasionally. Signed-off-by: Martin Storsjö --- I'm open to suggestions on whether 1/4 of the buffer is a reasonable amount to keep. It does of course incur some amount of overhead for well behaved linear files, but is a decent improvement for nonlinear mov files. Alternatively we could trigger this behaviour only after we've observed a couple seeks backwards? --- libavformat/aviobuf.c | 46 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/libavformat/aviobuf.c b/libavformat/aviobuf.c index 4ad734a3c3..dfc3e77016 100644 --- a/libavformat/aviobuf.c +++ b/libavformat/aviobuf.c @@ -534,8 +534,7 @@ static void fill_buffer(AVIOContext *s) FFIOContext *const ctx = (FFIOContext *)s; int max_buffer_size = s->max_packet_size ? s->max_packet_size : IO_BUFFER_SIZE; - uint8_t *dst = s->buf_end - s->buffer + max_buffer_size <= s->buffer_size ? - s->buf_end : s->buffer; + uint8_t *dst = s->buf_end; int len = s->buffer_size - (dst - s->buffer); /* can't fill the buffer without read_packet, just set EOF if appropriate */ @@ -546,11 +545,46 @@ static void fill_buffer(AVIOContext *s) if (s->eof_reached) return; - if (s->update_checksum && dst == s->buffer) { - if (s->buf_end > s->checksum_ptr) + if (len < max_buffer_size && s->buffer_size > max_buffer_size) { + /* If the buffer is almost full and we're not trying to read + one whole buffer worth of data at once; keep some amount of + the currently buffered data, but move it to the start of the + buffer, to allow filling the buffer with more data. */ + int keep = (s->buf_end - s->buffer)/4; + int shift = s->buf_end - keep - s->buffer; + + if (s->update_checksum && s->checksum_ptr - s->buffer < shift) { + /* Checksum up to the buffer + shift position (that we're + shifting out of the buffer. */ s->checksum = s->update_checksum(s->checksum, s->checksum_ptr, - s->buf_end - s->checksum_ptr); - s->checksum_ptr = s->buffer; + s->buffer + shift - s->checksum_ptr); + } + + memmove(s->buffer, s->buf_end - keep, keep); + s->buf_end -= shift; + s->buf_ptr -= shift; + if (s->update_checksum) { + if (s->checksum_ptr - s->buffer < shift) + s->checksum_ptr = s->buffer; + else + s->checksum_ptr -= shift; + } + + dst = s->buf_end; + len = s->buffer_size - (dst - s->buffer); + } else if (len < max_buffer_size) { + /* If the buffer is full so we can't fit a whole write of max_buffer_size, + just restart the pointers from the start of the buffer. */ + dst = s->buffer; + len = s->buffer_size; + + if (s->update_checksum) { + /* Checksum all data that gets shifted out of the buffer. */ + if (s->buf_end > s->checksum_ptr) + s->checksum = s->update_checksum(s->checksum, s->checksum_ptr, + s->buf_end - s->checksum_ptr); + s->checksum_ptr = s->buffer; + } } /* make buffer smaller in case it ended up large after probing */