From patchwork Wed Mar 28 02:11:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 8203 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.1.70 with SMTP id c67csp143121jad; Tue, 27 Mar 2018 19:12:11 -0700 (PDT) X-Google-Smtp-Source: AIpwx48FVCTyGijaAaoNjd5kfWJRG5AUHECJt2RAzSPY6dzBPox7k72WAL9UJ6dUv4f0ncHlsdX0 X-Received: by 10.223.225.17 with SMTP id d17mr1183834wri.51.1522203131575; Tue, 27 Mar 2018 19:12:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1522203131; cv=none; d=google.com; s=arc-20160816; b=MukcBzeHWUHdNpW1TQCXZauYspXdTCPu7mZqgYXHGBuwKajPgg1CAeN7ErgX2CM8NI SkEuXmYcKoMfBZS1OQNt8qIhAB9Jf475jVE4fzeVIrfVZjSv/HV9wTKi9ODrMBEtlWUa G78Lopwd/ZGM77jsuN6nQw/KIawC9jcMNStUvKH4x8w3osiwELGopU46NrKRVD6pvwNn Y3IiYUwgeqE0ygzkrWorXSHYRXkJBMN1i1FvGYhWnY1JiJcjbmKvEnA28weOvZI5vWF8 zONq3tKZqL0fcO35N5pSPilPbrLStAzEZF9d92o5kzXAH4I508olwtBUXKoNT3lwEyQ2 Wdhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=gnw4xh4SrxGK94gX9qfRvLpvGFmTyD+c6Z/RYMe9rqw=; b=AWfOEqnZJS4S5x7P6Bnfeu/lR1dGei2PQ6qhpbEHDwdCs3tw8nSurLnZNZf8W/tUrQ DxgP1FiZpp2I9L7cwPSkXu1E4ZW5i3yAJP1wSjWz71eT9qTM9uIUshCwgQDUKaNAxf1G ETYmf+UJ9SYvnnetdxp3/lzLykH1PS2qKoL7yS5g+avNuzbjYNieP57dX1jh/vXoAH3E iYWsAWtArLgQHDUvwVeNtukLqdLivUd5JrPF/DcRuTso3ySGIyHfh8Nn2oLT1l6/9brd PkPLqezZkl4KBfcC1SbQuUiw1Sz5KbRUM7hAxetUxlOulGFCzw2Kv9Ix61wx2M0LKEJy OB2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=S7jWpP8y; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r94si2079718wrb.399.2018.03.27.19.12.10; Tue, 27 Mar 2018 19:12:11 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=S7jWpP8y; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0D94A689A03; Wed, 28 Mar 2018 05:11:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk0-f194.google.com (mail-qk0-f194.google.com [209.85.220.194]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BBBAB688293 for ; Wed, 28 Mar 2018 05:11:44 +0300 (EEST) Received: by mail-qk0-f194.google.com with SMTP id o64so926053qkl.7 for ; Tue, 27 Mar 2018 19:12:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=XmgZPvwmjMpAN7ZoK98ZQ8HQNxLTX9mGSjyacIQkTIg=; b=S7jWpP8yx2zMyPgmdQ1e4TNqP5zK2elSCAcFG/2hmUPpsREAewYNEMDXGpuIFtMSzs 0DbNqXzxgz4Aq6njQcaulUJlnl164xboOyxlF1i1zDvEBxfs1AZfcUmuFaTfxtsbooIl QvniNA45+1EeHad0wUYuvCoGcsmhqWWgYjmiVmChNLhtS8P2ZqLxq0UttB7J2IoEEzt8 hPe07PuzQ1unwbVzk2vmuRQ/FnBU4pptFuEuGtkCyD3uL9KfuwU8igOFKuZaCW44avL8 BfEgwX+IvEzcKM1gSUho1Jbc4eyzzS1Bz4dSum2rYvZ+qiUTa1p9F1k6C4iOCMSZiH07 Z59w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=XmgZPvwmjMpAN7ZoK98ZQ8HQNxLTX9mGSjyacIQkTIg=; b=TaGe+Uwpc45TXmcKqXEOeloBDgMQcWUNXUFgmGQZbFweMEQ8xP0U5zpcFOA2E4GD9j HfwvpLZZbdHgKQNpolFRmTeorlTQ2696SoU2OUGP4M2KrswCyUxJN2WCCBkxUK0EHTE3 ImElkkxh4f6MyvG9qi3RNGmQy2OPWy3v6pZKf9ncSmsBw3wqX83Ctcx9xyoOnTsk9t0t dAHklbu95mHbVIA7+OMOv7nXlRo/ycsDwf2QfpN64bzjRy6GAbmX00+xVE+aP4JoY/+h pJk2q20pNCh6BStDYBggTiA9kmn5igR5NTDmdcKY1fyX7axU37jdc+X4yVrleXboKpxO mKAA== X-Gm-Message-State: ALQs6tCuOTFZ1fh1tVbUE460qyookowQHAtPoMv0xT/nRxSF5zsJjiAJ udstsvI1iZsa6Op/VL5GB9CZbQ== X-Received: by 10.233.230.75 with SMTP id x11mr2612831qkl.316.1522203121072; Tue, 27 Mar 2018 19:12:01 -0700 (PDT) Received: from localhost.localdomain ([190.188.171.140]) by smtp.gmail.com with ESMTPSA id c25sm1804521qtn.9.2018.03.27.19.11.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Mar 2018 19:12:00 -0700 (PDT) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Tue, 27 Mar 2018 23:11:27 -0300 Message-Id: <20180328021127.7296-1-jamrial@gmail.com> X-Mailer: git-send-email 2.16.2 Subject: [FFmpeg-devel] [PATCH] avutil/buffer: add a dynamic size buffer pool API X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: James Almer --- Implemented as a completely separate API as suggested. Missing Changelog, APIChanges and version bump as usual. libavutil/buffer.c | 159 ++++++++++++++++++++++++++++++++++++++++++++ libavutil/buffer.h | 53 +++++++++++++++ libavutil/buffer_internal.h | 14 ++++ 3 files changed, 226 insertions(+) diff --git a/libavutil/buffer.c b/libavutil/buffer.c index 8d1aa5fa84..c39a14c3c7 100644 --- a/libavutil/buffer.c +++ b/libavutil/buffer.c @@ -24,6 +24,7 @@ #include "common.h" #include "mem.h" #include "thread.h" +#include "tree.h" AVBufferRef *av_buffer_create(uint8_t *data, int size, void (*free)(void *opaque, uint8_t *data), @@ -355,3 +356,161 @@ AVBufferRef *av_buffer_pool_get(AVBufferPool *pool) return ret; } + +AVBufferDynPool *av_buffer_dyn_pool_init(AVBufferRef* (*alloc)(int size)) +{ + AVBufferDynPool *pool = av_mallocz(sizeof(*pool)); + if (!pool) + return NULL; + + ff_mutex_init(&pool->mutex, NULL); + + pool->alloc = alloc ? alloc : av_buffer_alloc; + + atomic_init(&pool->refcount, 1); + + return pool; +} + +static int free_node(void *opaque, void *elem) +{ + BufferPoolEntry *buf = elem; + + buf->free(buf->opaque, buf->data); + av_free(buf); + + return 0; +} + +static void buffer_dyn_pool_free(AVBufferDynPool *pool) +{ + av_tree_enumerate(pool->root, NULL, NULL, free_node); + av_tree_destroy(pool->root); + + ff_mutex_destroy(&pool->mutex); + + av_freep(&pool); +} + +void av_buffer_dyn_pool_uninit(AVBufferDynPool **ppool) +{ + AVBufferDynPool *pool; + + if (!ppool || !*ppool) + return; + pool = *ppool; + *ppool = NULL; + + if (atomic_fetch_add_explicit(&pool->refcount, -1, memory_order_acq_rel) == 1) + buffer_dyn_pool_free(pool); +} + +static int cmp_insert(const void *key, const void *node) +{ + int ret = ((const BufferPoolEntry *) key)->size - ((const BufferPoolEntry *) node)->size; + + if (!ret) + ret = ((const BufferPoolEntry *) key)->data - ((const BufferPoolEntry *) node)->data; + return ret; +} + +static void pool_release_dyn_buffer(void *opaque, uint8_t *data) +{ + BufferPoolEntry *buf = opaque; + AVBufferDynPool *pool = buf->dynpool; + + if(CONFIG_MEMORY_POISONING) + memset(buf->data, FF_MEMORY_POISON, buf->size); + + ff_mutex_lock(&pool->mutex); + /* Add the buffer into the pool, using the preallocated + * AVTreeNode stored in buf->node */ + av_tree_insert(&pool->root, buf, cmp_insert, &buf->node); + ff_mutex_unlock(&pool->mutex); + + if (atomic_fetch_add_explicit(&pool->refcount, -1, memory_order_acq_rel) == 1) + buffer_dyn_pool_free(pool); +} + +static AVBufferRef *pool_alloc_dyn_buffer(AVBufferDynPool *pool, int size) +{ + BufferPoolEntry *buf; + AVBufferRef *ret; + + ret = pool->alloc(size); + if (!ret) + return NULL; + + buf = av_mallocz(sizeof(*buf)); + if (!buf) { + av_buffer_unref(&ret); + return NULL; + } + + buf->node = av_tree_node_alloc(); + if (!buf->node) { + av_free(buf); + av_buffer_unref(&ret); + return NULL; + } + + buf->data = ret->buffer->data; + buf->opaque = ret->buffer->opaque; + buf->free = ret->buffer->free; + buf->size = size; + buf->dynpool = pool; + + ret->buffer->opaque = buf; + ret->buffer->free = pool_release_dyn_buffer; + + return ret; +} + +static int cmp_find(const void *key, const void *node) +{ + return *(const int *)key - ((const BufferPoolEntry *) node)->size; +} + +AVBufferRef *av_buffer_dyn_pool_get(AVBufferDynPool *pool, int size) +{ + AVBufferRef *ret; + BufferPoolEntry *buf, *next[2] = { NULL, NULL }; + + ff_mutex_lock(&pool->mutex); + /* Find a big enough buffer in the pool. */ + buf = av_tree_find(pool->root, &size, cmp_find, (void **)next); + + if (!buf) + /* If none of the requested size exists, use a bigger one. */ + buf = next[1]; + if (!buf && (buf = next[0])) { + /* If the pool also doesn't have a bigger buffer, but does + * have a smaller one, then replace it with a new buffer of + * the requested size. */ + av_tree_insert(&pool->root, buf, cmp_insert, &buf->node); + buf->free(buf->opaque, buf->data); + av_free(buf->node); + av_freep(&buf); + } + + if (buf) { + ret = av_buffer_create(buf->data, buf->size, pool_release_dyn_buffer, + buf, 0); + if (ret) { + /* Remove the buffer from the pool. Zero and store the + * AVTreeNode used for it in buf->node so we can use it + * again once the buffer is put back in the pool. */ + av_tree_insert(&pool->root, buf, cmp_insert, &buf->node); + memset(buf->node, 0, av_tree_node_size); + ret->size = size; + } + } else { + ret = pool_alloc_dyn_buffer(pool, size); + } + ff_mutex_unlock(&pool->mutex); + + if (ret) + atomic_fetch_add_explicit(&pool->refcount, 1, memory_order_relaxed); + + return ret; +} diff --git a/libavutil/buffer.h b/libavutil/buffer.h index 73b6bd0b14..d06b301fe5 100644 --- a/libavutil/buffer.h +++ b/libavutil/buffer.h @@ -284,6 +284,59 @@ void av_buffer_pool_uninit(AVBufferPool **pool); */ AVBufferRef *av_buffer_pool_get(AVBufferPool *pool); +/** + * @} + */ + +/** + * @defgroup lavu_bufferdynpool AVBufferDynPool + * @ingroup lavu_data + * + * @{ + * AVBufferDynPool is an API for a lock-free thread-safe pool of AVBuffers. + * + * Unlike AVBufferPool, AVBufferDynPool allows the user to request buffers + * of any arbitrary size. It is functionally the same otherwise. + */ + +/** + * The buffer pool. This structure is opaque and not meant to be accessed + * directly. It is allocated with av_buffer_dyn_pool_init() and freed with + * av_buffer_dyn_pool_uninit(). + */ +typedef struct AVBufferDynPool AVBufferDynPool; + +/** + * Allocate and initialize a buffer pool. + * + * @param alloc a function that will be used to allocate new buffers when the + * pool is empty. May be NULL, then the default allocator will be used + * (av_buffer_alloc()). + * @return newly created buffer pool on success, NULL on error. + */ +AVBufferDynPool *av_buffer_dyn_pool_init(AVBufferRef* (*alloc)(int size)); + +/** + * Mark the pool as being available for freeing. It will actually be freed only + * once all the allocated buffers associated with the pool are released. Thus it + * is safe to call this function while some of the allocated buffers are still + * in use. + * + * @param pool pointer to the pool to be freed. It will be set to NULL. + */ +void av_buffer_dyn_pool_uninit(AVBufferDynPool **pool); + +/** + * Allocate a new AVBuffer, reusing an old buffer from the pool when available. + * This function may be called simultaneously from multiple threads. + * + * @param pool pointer to an initialized pool. + * @param size Required buffer size in bytes. + * + * @return a reference to the new buffer on success, NULL on error. + */ +AVBufferRef *av_buffer_dyn_pool_get(AVBufferDynPool *pool, int size); + /** * @} */ diff --git a/libavutil/buffer_internal.h b/libavutil/buffer_internal.h index 54b67047e5..2c0e9ea063 100644 --- a/libavutil/buffer_internal.h +++ b/libavutil/buffer_internal.h @@ -61,6 +61,7 @@ struct AVBuffer { typedef struct BufferPoolEntry { uint8_t *data; + size_t size; /* * Backups of the original opaque/free of the AVBuffer corresponding to @@ -71,6 +72,9 @@ typedef struct BufferPoolEntry { AVBufferPool *pool; struct BufferPoolEntry *next; + + AVBufferDynPool *dynpool; + struct AVTreeNode *node; } BufferPoolEntry; struct AVBufferPool { @@ -95,4 +99,14 @@ struct AVBufferPool { void (*pool_free)(void *opaque); }; +struct AVBufferDynPool { + AVMutex mutex; + struct AVTreeNode *root; + + atomic_uint refcount; + + int size; + AVBufferRef* (*alloc)(int size); +}; + #endif /* AVUTIL_BUFFER_INTERNAL_H */