From patchwork Tue Sep 29 21:10:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marton Balint X-Patchwork-Id: 22658 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 8A14D449925 for ; Wed, 30 Sep 2020 00:10:37 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 670C268ABFD; Wed, 30 Sep 2020 00:10:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from iq.passwd.hu (iq.passwd.hu [217.27.212.140]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0BF266881B9 for ; Wed, 30 Sep 2020 00:10:31 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 78A51E45AC; Tue, 29 Sep 2020 23:10:30 +0200 (CEST) X-Virus-Scanned: amavisd-new at passwd.hu Received: from iq.passwd.hu ([127.0.0.1]) by localhost (iq.passwd.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6QgHbyime2x8; Tue, 29 Sep 2020 23:10:28 +0200 (CEST) Received: from bluegene.passwd.hu (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 7DFC4E4333; Tue, 29 Sep 2020 23:10:28 +0200 (CEST) From: Marton Balint To: ffmpeg-devel@ffmpeg.org Date: Tue, 29 Sep 2020 23:10:13 +0200 Message-Id: <20200929211021.25030-1-cus@passwd.hu> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/9] avformat/aviobuf: write data into the IO buffer till the very end of the buffer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marton Balint Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" There was an off-by-one error when checking if the IO buffer still has enough space till the end. One more byte can be safely written. Signed-off-by: Marton Balint --- libavformat/aviobuf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavformat/aviobuf.c b/libavformat/aviobuf.c index a77517d712..9675425349 100644 --- a/libavformat/aviobuf.c +++ b/libavformat/aviobuf.c @@ -540,7 +540,7 @@ static void fill_buffer(AVIOContext *s) { int max_buffer_size = s->max_packet_size ? s->max_packet_size : IO_BUFFER_SIZE; - uint8_t *dst = s->buf_end - s->buffer + max_buffer_size < s->buffer_size ? + uint8_t *dst = s->buf_end - s->buffer + max_buffer_size <= s->buffer_size ? s->buf_end : s->buffer; int len = s->buffer_size - (dst - s->buffer); From patchwork Tue Sep 29 21:10:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marton Balint X-Patchwork-Id: 22659 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id A6408449925 for ; Wed, 30 Sep 2020 00:10:39 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 91E8E68B01D; Wed, 30 Sep 2020 00:10:39 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from iq.passwd.hu (iq.passwd.hu [217.27.212.140]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 75D2F68B4D9 for ; Wed, 30 Sep 2020 00:10:33 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id D4EADE4612; Tue, 29 Sep 2020 23:10:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at passwd.hu Received: from iq.passwd.hu ([127.0.0.1]) by localhost (iq.passwd.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7pHLsPPAJ7Zn; Tue, 29 Sep 2020 23:10:31 +0200 (CEST) Received: from bluegene.passwd.hu (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 01423E4333; Tue, 29 Sep 2020 23:10:30 +0200 (CEST) From: Marton Balint To: ffmpeg-devel@ffmpeg.org Date: Tue, 29 Sep 2020 23:10:14 +0200 Message-Id: <20200929211021.25030-2-cus@passwd.hu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200929211021.25030-1-cus@passwd.hu> References: <20200929211021.25030-1-cus@passwd.hu> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/9] avformat/aviobuf: check if requested seekback buffer is already read X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marton Balint Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Existing code did not check if the requested seekback buffer is already read entirely. In this case, nothing has to be done to guarantee seekback. Signed-off-by: Marton Balint --- libavformat/aviobuf.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/libavformat/aviobuf.c b/libavformat/aviobuf.c index 9675425349..d94be478ac 100644 --- a/libavformat/aviobuf.c +++ b/libavformat/aviobuf.c @@ -999,6 +999,9 @@ int ffio_ensure_seekback(AVIOContext *s, int64_t buf_size) int filled = s->buf_end - s->buffer; ptrdiff_t checksum_ptr_offset = s->checksum_ptr ? s->checksum_ptr - s->buffer : -1; + if (buf_size <= s->buf_end - s->buf_ptr) + return 0; + buf_size += s->buf_ptr - s->buffer + max_buffer_size; if (buf_size < filled || s->seekable || !s->read_packet) From patchwork Tue Sep 29 21:10:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marton Balint X-Patchwork-Id: 22660 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id E98DB449925 for ; Wed, 30 Sep 2020 00:10:41 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D1D2B68B4DA; Wed, 30 Sep 2020 00:10:41 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from iq.passwd.hu (iq.passwd.hu [217.27.212.140]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 186866881B9 for ; Wed, 30 Sep 2020 00:10:36 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id C0806E4333; Tue, 29 Sep 2020 23:10:35 +0200 (CEST) X-Virus-Scanned: amavisd-new at passwd.hu Received: from iq.passwd.hu ([127.0.0.1]) by localhost (iq.passwd.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FYmxcoKSppLT; Tue, 29 Sep 2020 23:10:34 +0200 (CEST) Received: from bluegene.passwd.hu (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 34B32E46A2; Tue, 29 Sep 2020 23:10:34 +0200 (CEST) From: Marton Balint To: ffmpeg-devel@ffmpeg.org Date: Tue, 29 Sep 2020 23:10:15 +0200 Message-Id: <20200929211021.25030-3-cus@passwd.hu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200929211021.25030-1-cus@passwd.hu> References: <20200929211021.25030-1-cus@passwd.hu> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/9] avformat/aviobuf: fix checks in ffio_ensure_seekback X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marton Balint Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" The new buf_size was detemined too conservatively, maybe because of the off-by-one issue which was fixed recently in fill_buffer. We can safely substract 1 more from the new buffer size, because max_buffer_size space must only be guaranteed when we are reading the last byte of the requested window. Comparing the new buf_size against filled did not make a lot of sense, what makes sense is that we want to reallocate the buffer if the new buf_size is bigger than the old, therefore the change in the check. Signed-off-by: Marton Balint --- libavformat/aviobuf.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavformat/aviobuf.c b/libavformat/aviobuf.c index d94be478ac..aa1d6c0830 100644 --- a/libavformat/aviobuf.c +++ b/libavformat/aviobuf.c @@ -1002,9 +1002,9 @@ int ffio_ensure_seekback(AVIOContext *s, int64_t buf_size) if (buf_size <= s->buf_end - s->buf_ptr) return 0; - buf_size += s->buf_ptr - s->buffer + max_buffer_size; + buf_size += s->buf_ptr - s->buffer + max_buffer_size - 1; - if (buf_size < filled || s->seekable || !s->read_packet) + if (buf_size <= s->buffer_size || s->seekable || !s->read_packet) return 0; av_assert0(!s->write_flag); From patchwork Tue Sep 29 21:10:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marton Balint X-Patchwork-Id: 22661 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 492E3449925 for ; Wed, 30 Sep 2020 00:10:46 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2E8C668B551; Wed, 30 Sep 2020 00:10:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from iq.passwd.hu (iq.passwd.hu [217.27.212.140]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 50F4A68B534 for ; Wed, 30 Sep 2020 00:10:37 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 221EEE4648; Tue, 29 Sep 2020 23:10:37 +0200 (CEST) X-Virus-Scanned: amavisd-new at passwd.hu Received: from iq.passwd.hu ([127.0.0.1]) by localhost (iq.passwd.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 83CvTaNFC-iH; Tue, 29 Sep 2020 23:10:35 +0200 (CEST) Received: from bluegene.passwd.hu (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id B9E9BE45E4; Tue, 29 Sep 2020 23:10:35 +0200 (CEST) From: Marton Balint To: ffmpeg-devel@ffmpeg.org Date: Tue, 29 Sep 2020 23:10:16 +0200 Message-Id: <20200929211021.25030-4-cus@passwd.hu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200929211021.25030-1-cus@passwd.hu> References: <20200929211021.25030-1-cus@passwd.hu> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 4/9] avformat/mpegts: make sure mpegts_resync do not run out of the initially requested probe buffer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marton Balint Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Marton Balint --- libavformat/mpegts.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/libavformat/mpegts.c b/libavformat/mpegts.c index 50d4d5e9bc..432b1c3ea2 100644 --- a/libavformat/mpegts.c +++ b/libavformat/mpegts.c @@ -3051,10 +3051,11 @@ static int mpegts_read_header(AVFormatContext *s) MpegTSContext *ts = s->priv_data; AVIOContext *pb = s->pb; int64_t pos, probesize = s->probesize; + int64_t seekback = FFMAX(s->probesize, (int64_t)ts->resync_size + PROBE_PACKET_MAX_BUF); s->internal->prefer_codec_framerate = 1; - if (ffio_ensure_seekback(pb, probesize) < 0) + if (ffio_ensure_seekback(pb, seekback) < 0) av_log(s, AV_LOG_WARNING, "Failed to allocate buffers for seekback\n"); pos = avio_tell(pb); From patchwork Tue Sep 29 21:10:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marton Balint X-Patchwork-Id: 22663 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id D7348449925 for ; Wed, 30 Sep 2020 00:10:47 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C450E68B5BB; Wed, 30 Sep 2020 00:10:47 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from iq.passwd.hu (iq.passwd.hu [217.27.212.140]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1AC5C68B31D for ; Wed, 30 Sep 2020 00:10:39 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 8C139E45AC; Tue, 29 Sep 2020 23:10:39 +0200 (CEST) X-Virus-Scanned: amavisd-new at passwd.hu Received: from iq.passwd.hu ([127.0.0.1]) by localhost (iq.passwd.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8FtLcNmafC8E; Tue, 29 Sep 2020 23:10:38 +0200 (CEST) Received: from bluegene.passwd.hu (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 9D5A2E45E4; Tue, 29 Sep 2020 23:10:37 +0200 (CEST) From: Marton Balint To: ffmpeg-devel@ffmpeg.org Date: Tue, 29 Sep 2020 23:10:17 +0200 Message-Id: <20200929211021.25030-5-cus@passwd.hu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200929211021.25030-1-cus@passwd.hu> References: <20200929211021.25030-1-cus@passwd.hu> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 5/9] avformat/mpjpegdec: make sure we seek back to the ensured buffer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marton Balint Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" It was possible for the old code to seek back before the most recently read data if start of a new multipart was across read boundaries. Now we read some small sections multiple times to avoid this, but that is OK. Signed-off-by: Marton Balint --- libavformat/mpjpegdec.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/libavformat/mpjpegdec.c b/libavformat/mpjpegdec.c index df2880412d..5fe00fd162 100644 --- a/libavformat/mpjpegdec.c +++ b/libavformat/mpjpegdec.c @@ -328,18 +328,18 @@ static int mpjpeg_read_packet(AVFormatContext *s, AVPacket *pkt) ret = av_get_packet(s->pb, pkt, size); } else { /* no size was given -- we read until the next boundary or end-of-file */ - int remaining = 0, len; + int len; const int read_chunk = 2048; pkt->pos = avio_tell(s->pb); - while ((ret = ffio_ensure_seekback(s->pb, read_chunk - remaining)) >= 0 && /* we may need to return as much as all we've read back to the buffer */ - (ret = av_append_packet(s->pb, pkt, read_chunk - remaining)) >= 0) { + while ((ret = ffio_ensure_seekback(s->pb, read_chunk)) >= 0 && /* we may need to return as much as all we've read back to the buffer */ + (ret = av_append_packet(s->pb, pkt, read_chunk)) >= 0) { /* scan the new data */ char *start; - len = ret + remaining; + len = ret; start = pkt->data + pkt->size - len; do { if (!memcmp(start, mpjpeg->searchstr, mpjpeg->searchstr_len)) { @@ -351,7 +351,8 @@ static int mpjpeg_read_packet(AVFormatContext *s, AVPacket *pkt) len--; start++; } while (len >= mpjpeg->searchstr_len); - remaining = len; + avio_seek(s->pb, -len, SEEK_CUR); + pkt->size -= len; } /* error or EOF occurred */ From patchwork Tue Sep 29 21:10:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marton Balint X-Patchwork-Id: 22666 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id E2683449925 for ; Wed, 30 Sep 2020 00:10:53 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C75FE68B5F6; Wed, 30 Sep 2020 00:10:53 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from iq.passwd.hu (iq.passwd.hu [217.27.212.140]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9623D68B546 for ; Wed, 30 Sep 2020 00:10:44 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 61ACCE45E4; Tue, 29 Sep 2020 23:10:44 +0200 (CEST) X-Virus-Scanned: amavisd-new at passwd.hu Received: from iq.passwd.hu ([127.0.0.1]) by localhost (iq.passwd.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yaCq-ht5Kbzv; Tue, 29 Sep 2020 23:10:41 +0200 (CEST) Received: from bluegene.passwd.hu (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 94917E4612; Tue, 29 Sep 2020 23:10:39 +0200 (CEST) From: Marton Balint To: ffmpeg-devel@ffmpeg.org Date: Tue, 29 Sep 2020 23:10:18 +0200 Message-Id: <20200929211021.25030-6-cus@passwd.hu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200929211021.25030-1-cus@passwd.hu> References: <20200929211021.25030-1-cus@passwd.hu> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 6/9] avformat/aviobuf: discard part of the IO buffer in ffio_ensure_seekback if needed X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marton Balint Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Previously ffio_ensure_seekback never flushed the buffer, so successive ffio_ensure_seekback calls were all respected. This could eventually cause unlimited memory and CPU usage if a demuxer called ffio_ensure_seekback on all it's read data. Most demuxers however only rely on being able to seek back till the position of the last ffio_ensure_seekback call, therefore we change the semantics of ffio_ensure_seekback so that a new call can invalidate seek guarantees of the old. In order to support some level of "nested" ffio_ensure_seekback calls, we document that the function only invalidates the old window (and potentially discards the already read data from the IO buffer), if the newly requested window does not fit into the old one. This way we limit the memory usage for ffio_ensure_seekback calls requesting consecutive data windows. Signed-off-by: Marton Balint --- libavformat/avio_internal.h | 4 +++- libavformat/aviobuf.c | 27 +++++++++++++++++---------- 2 files changed, 20 insertions(+), 11 deletions(-) diff --git a/libavformat/avio_internal.h b/libavformat/avio_internal.h index c575df8035..fe87f2a288 100644 --- a/libavformat/avio_internal.h +++ b/libavformat/avio_internal.h @@ -100,7 +100,9 @@ int ffio_realloc_buf(AVIOContext *s, int buf_size); * * Will ensure that when reading sequentially up to buf_size, seeking * within the current pos and pos+buf_size is possible. - * Once the stream position moves outside this window this guarantee is lost. + * Once the stream position moves outside this window or another + * ffio_ensure_seekback call requests a buffer outside this window this + * guarantee is lost. */ int ffio_ensure_seekback(AVIOContext *s, int64_t buf_size); diff --git a/libavformat/aviobuf.c b/libavformat/aviobuf.c index aa1d6c0830..75ddf52efc 100644 --- a/libavformat/aviobuf.c +++ b/libavformat/aviobuf.c @@ -991,35 +991,42 @@ URLContext* ffio_geturlcontext(AVIOContext *s) return NULL; } +static void update_checksum(AVIOContext *s) +{ + if (s->update_checksum && s->buf_ptr > s->checksum_ptr) { + s->checksum = s->update_checksum(s->checksum, s->checksum_ptr, + s->buf_ptr - s->checksum_ptr); + } +} + int ffio_ensure_seekback(AVIOContext *s, int64_t buf_size) { uint8_t *buffer; int max_buffer_size = s->max_packet_size ? s->max_packet_size : IO_BUFFER_SIZE; - int filled = s->buf_end - s->buffer; - ptrdiff_t checksum_ptr_offset = s->checksum_ptr ? s->checksum_ptr - s->buffer : -1; + ptrdiff_t filled = s->buf_end - s->buf_ptr; if (buf_size <= s->buf_end - s->buf_ptr) return 0; - buf_size += s->buf_ptr - s->buffer + max_buffer_size - 1; + buf_size += max_buffer_size - 1; - if (buf_size <= s->buffer_size || s->seekable || !s->read_packet) + if (buf_size + s->buf_ptr - s->buffer <= s->buffer_size || s->seekable || !s->read_packet) return 0; av_assert0(!s->write_flag); + buf_size = FFMAX(buf_size, s->buffer_size); buffer = av_malloc(buf_size); if (!buffer) return AVERROR(ENOMEM); - - memcpy(buffer, s->buffer, filled); + update_checksum(s); + memcpy(buffer, s->buf_ptr, filled); av_free(s->buffer); - s->buf_ptr = buffer + (s->buf_ptr - s->buffer); - s->buf_end = buffer + (s->buf_end - s->buffer); s->buffer = buffer; s->buffer_size = buf_size; - if (checksum_ptr_offset >= 0) - s->checksum_ptr = s->buffer + checksum_ptr_offset; + s->buf_ptr = s->buffer; + s->buf_end = s->buffer + filled; + s->checksum_ptr = s->buffer; return 0; } From patchwork Tue Sep 29 21:10:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marton Balint X-Patchwork-Id: 22665 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id E41FE449925 for ; Wed, 30 Sep 2020 00:10:52 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C932268B5EE; Wed, 30 Sep 2020 00:10:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from iq.passwd.hu (iq.passwd.hu [217.27.212.140]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 86A5C68B5C8 for ; Wed, 30 Sep 2020 00:10:43 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 56557E4333; Tue, 29 Sep 2020 23:10:43 +0200 (CEST) X-Virus-Scanned: amavisd-new at passwd.hu Received: from iq.passwd.hu ([127.0.0.1]) by localhost (iq.passwd.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id P3q8zJBguWti; Tue, 29 Sep 2020 23:10:41 +0200 (CEST) Received: from bluegene.passwd.hu (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 5B889E45E4; Tue, 29 Sep 2020 23:10:41 +0200 (CEST) From: Marton Balint To: ffmpeg-devel@ffmpeg.org Date: Tue, 29 Sep 2020 23:10:19 +0200 Message-Id: <20200929211021.25030-7-cus@passwd.hu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200929211021.25030-1-cus@passwd.hu> References: <20200929211021.25030-1-cus@passwd.hu> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 7/9] avformat/aviobuf: do not allocate a new buffer in ffio_ensure_seekback if not needed X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marton Balint Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Let's move unread data to the start of the old buffer instead. Signed-off-by: Marton Balint --- libavformat/aviobuf.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/libavformat/aviobuf.c b/libavformat/aviobuf.c index 75ddf52efc..2f1abe7d81 100644 --- a/libavformat/aviobuf.c +++ b/libavformat/aviobuf.c @@ -1015,7 +1015,10 @@ int ffio_ensure_seekback(AVIOContext *s, int64_t buf_size) return 0; av_assert0(!s->write_flag); - buf_size = FFMAX(buf_size, s->buffer_size); + if (buf_size <= s->buffer_size) { + update_checksum(s); + memmove(s->buffer, s->buf_ptr, filled); + } else { buffer = av_malloc(buf_size); if (!buffer) return AVERROR(ENOMEM); @@ -1024,6 +1027,7 @@ int ffio_ensure_seekback(AVIOContext *s, int64_t buf_size) av_free(s->buffer); s->buffer = buffer; s->buffer_size = buf_size; + } s->buf_ptr = s->buffer; s->buf_end = s->buffer + filled; s->checksum_ptr = s->buffer; From patchwork Tue Sep 29 21:10:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marton Balint X-Patchwork-Id: 22662 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 0AF31449925 for ; Wed, 30 Sep 2020 00:10:47 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E709B68B545; Wed, 30 Sep 2020 00:10:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from iq.passwd.hu (iq.passwd.hu [217.27.212.140]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1AD1268B53F for ; Wed, 30 Sep 2020 00:10:44 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id BBEF6E4612; Tue, 29 Sep 2020 23:10:44 +0200 (CEST) X-Virus-Scanned: amavisd-new at passwd.hu Received: from iq.passwd.hu ([127.0.0.1]) by localhost (iq.passwd.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M30UCBbSD--l; Tue, 29 Sep 2020 23:10:43 +0200 (CEST) Received: from bluegene.passwd.hu (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 12919E46A2; Tue, 29 Sep 2020 23:10:43 +0200 (CEST) From: Marton Balint To: ffmpeg-devel@ffmpeg.org Date: Tue, 29 Sep 2020 23:10:20 +0200 Message-Id: <20200929211021.25030-8-cus@passwd.hu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200929211021.25030-1-cus@passwd.hu> References: <20200929211021.25030-1-cus@passwd.hu> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 8/9] avformat/aviobuf: fix indentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marton Balint Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Marton Balint --- libavformat/aviobuf.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/libavformat/aviobuf.c b/libavformat/aviobuf.c index 2f1abe7d81..c4195e310b 100644 --- a/libavformat/aviobuf.c +++ b/libavformat/aviobuf.c @@ -1019,14 +1019,14 @@ int ffio_ensure_seekback(AVIOContext *s, int64_t buf_size) update_checksum(s); memmove(s->buffer, s->buf_ptr, filled); } else { - buffer = av_malloc(buf_size); - if (!buffer) - return AVERROR(ENOMEM); - update_checksum(s); - memcpy(buffer, s->buf_ptr, filled); - av_free(s->buffer); - s->buffer = buffer; - s->buffer_size = buf_size; + buffer = av_malloc(buf_size); + if (!buffer) + return AVERROR(ENOMEM); + update_checksum(s); + memcpy(buffer, s->buf_ptr, filled); + av_free(s->buffer); + s->buffer = buffer; + s->buffer_size = buf_size; } s->buf_ptr = s->buffer; s->buf_end = s->buffer + filled; From patchwork Tue Sep 29 21:10:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marton Balint X-Patchwork-Id: 22664 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id D0DA6449925 for ; Wed, 30 Sep 2020 00:10:51 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B94CE68B5D7; Wed, 30 Sep 2020 00:10:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from iq.passwd.hu (iq.passwd.hu [217.27.212.140]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3FEF268B5C8 for ; Wed, 30 Sep 2020 00:10:48 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 07A64E45AC; Tue, 29 Sep 2020 23:10:48 +0200 (CEST) X-Virus-Scanned: amavisd-new at passwd.hu Received: from iq.passwd.hu ([127.0.0.1]) by localhost (iq.passwd.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DL4hYETTEzgv; Tue, 29 Sep 2020 23:10:46 +0200 (CEST) Received: from bluegene.passwd.hu (localhost [127.0.0.1]) by iq.passwd.hu (Postfix) with ESMTP id 37AA4E46D0; Tue, 29 Sep 2020 23:10:45 +0200 (CEST) From: Marton Balint To: ffmpeg-devel@ffmpeg.org Date: Tue, 29 Sep 2020 23:10:21 +0200 Message-Id: <20200929211021.25030-9-cus@passwd.hu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200929211021.25030-1-cus@passwd.hu> References: <20200929211021.25030-1-cus@passwd.hu> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 9/9] avformat/aviobuf: increase default read buffer size to 2*max_buffer_size for streamed data X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Marton Balint Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This should increase the effectiveness of ffio_ensure_seekback by reducing the number of buffer reallocations and memmoves/memcpys because even a small seekback window requires max_buffer_size+window_size buffer space. Signed-off-by: Marton Balint --- libavformat/aviobuf.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/libavformat/aviobuf.c b/libavformat/aviobuf.c index c4195e310b..15d91f91bc 100644 --- a/libavformat/aviobuf.c +++ b/libavformat/aviobuf.c @@ -938,6 +938,11 @@ int ffio_fdopen(AVIOContext **s, URLContext *h) } else { buffer_size = IO_BUFFER_SIZE; } + if (!(h->flags & AVIO_FLAG_WRITE) && h->is_streamed) { + if (buffer_size > INT_MAX/2) + return AVERROR(EINVAL); + buffer_size *= 2; + } buffer = av_malloc(buffer_size); if (!buffer) return AVERROR(ENOMEM);