From patchwork Thu May 23 22:10:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Connor Worley X-Patchwork-Id: 49193 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:542:0:b0:460:55fa:d5ed with SMTP id 63csp1350316vqf; Thu, 23 May 2024 15:10:54 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWzIkJadi63nE3KcHxypjDiQypZKLjv5uIkkCfLtxSiCcGU53AXqRgX7S4cNN6bXLiqkkTSMB2ZtgYT32AQ5roQIojBig2wDMLhYA== X-Google-Smtp-Source: AGHT+IGe1agHboNCkjfLPok8V0pjbuGCF0zHq5fRY/+ejVBP4Ai6kdsVtbCxE4FjR43PwyIjrI8G X-Received: by 2002:a2e:b6c2:0:b0:2d9:f7f8:3e87 with SMTP id 38308e7fff4ca-2e95b25ec19mr2245701fa.32.1716502254587; Thu, 23 May 2024 15:10:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716502254; cv=none; d=google.com; s=arc-20160816; b=KfnxmOt2XkLdZMavB6sgcPHs+c3V3YdrQNb299/1qgT/6NsMAREqPprV6zn+ys5dod bQc2hJQGjs2+JvcKcI44EviYWQQaus3Witimyijq6LjaI2dig3wWv0J6lfFjY9OvFn9H t05tMOgtBrGK/4LFEhJnlnP/R1cayR+zxciQ3pUD/q/cvdAufI4e0PsAwfN6Yp74AhpR jpQS+oz6oOg+2YP3+T1EFAWfW9NX886i+u10eQAFhpPkVWl93eP+qtDigz3MIxXfSARn qAGgKvJA4usnxFPd3G1XD+Rc2yo3i4Ou//I+8vyRI4Y1K2566isQpLtv3Xs/ZpwhUllU mmGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=a0PMEgdGFLvX/65ZBWltpP2qz98Byj1Tc08/HYgHGHA=; fh=4o9WedBEbOrq7GvNs0s+TOKeyzmDZRM3O7oe6ZP6vYM=; b=KvE0foCL++a/6XMnAugKcK5aVci6NNYwOBGi6rMouv45dPDlVolxdDoTZ5tXJcPVbi sLEvNwTaIbbnawNhd/A3A5zTiissX/M9LkyQrycw1Jyyh5HDc9eZFsDeAS37nYcuB7Uq zTaOP7cYi1JhLcMen3OabZ74Byc2PCgaCOu1/mhhKI6tRNMS5tulFn0CGRHXYQb68U0H 3oepAchGtQlQm17MXuZW/LLoT2EXFdGf/KXG0jN8lnXo921iJXV+tHGxDvFNM/PKkw0V UH3CibkaGUzUU9aFOZZoyGyT53Uee3GXCli6eQsbCFMwgGpGmG+xRNE/fp53EO/ssDoZ O+wA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b="UxRjqd/U"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5785247b1edsi136452a12.345.2024.05.23.15.10.54; Thu, 23 May 2024 15:10:54 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b="UxRjqd/U"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3A3E468D50D; Fri, 24 May 2024 01:10:50 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E285768D2B5 for ; Fri, 24 May 2024 01:10:42 +0300 (EEST) Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1f44b51b367so1079725ad.1 for ; Thu, 23 May 2024 15:10:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716502240; x=1717107040; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=mzJfHvDqBfZA282nm7NqXFezAmOtrI3/fN+1YLm+aW8=; b=UxRjqd/UExxokXoPrSpepGEKoI2f+z+NGc0bNwcxp8jKzRdHwRsPBCVmVE4donz4e8 TnXcJkzesmRs0uZazxy9cp3VMWY9esYrx12B2FMfkcK7831njYj7Er7fUOrwwGRKayW2 bTwM1y2/ph8EIF02iyEi68rvfLxXbau8mxu8oS2UndLrzfoMZHUyLrem5g64Zan0P4c/ t6f34qfU/WhBk113GlML4nGLNzp9MP1ySYFU+D9vyQWwa85mh/8PqHyT6SRKL1QgUZFK m8Ad+rapZM0Iuc+dQpFET81GUW7eVmHs3QazYIInva7nJTGDnhu+KvpqZzDSVJ+rSE4u D+Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716502240; x=1717107040; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mzJfHvDqBfZA282nm7NqXFezAmOtrI3/fN+1YLm+aW8=; b=XrC+fkDn1jejBw5BTVQr9oi6Eoyg3Q6tbAd8eKzLvdlxu185HIPSM5Y3bYzqL8pAXf QN8Uzh2wVUFXwSogCcejSO2g72wz8vKGSgb6UCQ4nlu1tBQ1JgxckRZG56B9I+VBVFRl 9KP/En5lmo9IzxEnH+XUACNFaxIN4i/6Ac31deUUYZ87xOGssA7V2pMNadlYcQPtdAAp Xvkx2ElZ4i8nHJ0dFBCXC7+teJAlbNCj8qCL6bvexHz2UI7O5+c7mIq6P/nxyHbwOWUR sXEnx2mTp5ZynXLXrharjlc+jTA4Ohk/Mcv4EoNdGmJsuMowCa5OVl9kJKqGyvx4WezW eNMw== X-Gm-Message-State: AOJu0YxCYMduU7p1GHXV6U8+YkCdgk0z7+nRhX87zbgOITF16xnhg5/E bwtRK3JULGQgjcb1D/w+8Psvl2Maavob5ZVln/f93EVvfXr4bQMq68oxvQ== X-Received: by 2002:a17:903:32ce:b0:1f3:488a:bc54 with SMTP id d9443c01a7336-1f448d3510bmr6921485ad.36.1716502239911; Thu, 23 May 2024 15:10:39 -0700 (PDT) Received: from localhost ([2607:fb10:7011:1::7d4]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-1f44c9988besm683915ad.198.2024.05.23.15.10.38 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 23 May 2024 15:10:39 -0700 (PDT) From: Connor Worley To: ffmpeg-devel@ffmpeg.org Date: Thu, 23 May 2024 15:10:04 -0700 Message-ID: <20240523221005.20659-1-connorbworley@gmail.com> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v5 1/2] lavu/hashtable: create generic robin hood hash table X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Connor Worley Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: MH9Jlu/i58aK Signed-off-by: Connor Worley --- libavutil/Makefile | 2 + libavutil/hashtable.c | 192 ++++++++++++++++++++++++++++++++++++ libavutil/hashtable.h | 91 +++++++++++++++++ libavutil/tests/hashtable.c | 108 ++++++++++++++++++++ 4 files changed, 393 insertions(+) create mode 100644 libavutil/hashtable.c create mode 100644 libavutil/hashtable.h create mode 100644 libavutil/tests/hashtable.c diff --git a/libavutil/Makefile b/libavutil/Makefile index 6e6fa8d800..8c7b4452b4 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -138,6 +138,7 @@ OBJS = adler32.o \ fixed_dsp.o \ frame.o \ hash.o \ + hashtable.o \ hdr_dynamic_metadata.o \ hdr_dynamic_vivid_metadata.o \ hmac.o \ @@ -252,6 +253,7 @@ TESTPROGS = adler32 \ file \ fifo \ hash \ + hashtable \ hmac \ hwdevice \ integer \ diff --git a/libavutil/hashtable.c b/libavutil/hashtable.c new file mode 100644 index 0000000000..155a264665 --- /dev/null +++ b/libavutil/hashtable.c @@ -0,0 +1,192 @@ +/* + * Generic hashtable + * Copyright (C) 2024 Connor Worley + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include +#include + +#include "crc.h" +#include "error.h" +#include "mem.h" +#include "hashtable.h" + +#define ALIGN _Alignof(size_t) + +struct AVHashtableContext { + size_t key_size; + size_t key_size_aligned; + size_t val_size; + size_t val_size_aligned; + size_t entry_size; + size_t max_entries; + size_t utilization; + const AVCRC *crc; + uint8_t *table; + uint8_t *swapbuf; +}; + +#define ENTRY_PSL(entry) (entry) +#define ENTRY_OCC(entry) (ENTRY_PSL(entry) + FFALIGN(sizeof(size_t), ALIGN)) +#define ENTRY_KEY(entry) (ENTRY_OCC(entry) + FFALIGN(sizeof(size_t), ALIGN)) +#define ENTRY_VAL(entry) (ENTRY_KEY(entry) + ctx->key_size_aligned) + +#define KEYS_EQUAL(k1, k2) !memcmp(k1, k2, ctx->key_size) + +int av_hashtable_alloc(struct AVHashtableContext **ctx, size_t key_size, size_t val_size, size_t max_entries) +{ + struct AVHashtableContext *res = av_malloc(sizeof(struct AVHashtableContext)); + if (!res) + return AVERROR(ENOMEM); + res->key_size = key_size; + res->key_size_aligned = FFALIGN(key_size, ALIGN); + res->val_size = val_size; + res->val_size_aligned = FFALIGN(val_size, ALIGN); + res->entry_size = FFALIGN(sizeof(size_t), ALIGN) + + FFALIGN(sizeof(size_t), ALIGN) + + res->key_size_aligned + + res->val_size_aligned; + res->max_entries = max_entries; + res->utilization = 0; + res->crc = av_crc_get_table(AV_CRC_32_IEEE); + if (!res->crc) { + av_hashtable_freep(&res); + return AVERROR_BUG; + } + res->table = av_calloc(res->max_entries, res->entry_size); + if (!res->table) { + av_hashtable_freep(&res); + return AVERROR(ENOMEM); + } + res->swapbuf = av_calloc(2, res->key_size_aligned + res->val_size_aligned); + if (!res->swapbuf) { + av_hashtable_freep(&res); + return AVERROR(ENOMEM); + } + *ctx = res; + return 0; +} + +static size_t hash_key(const struct AVHashtableContext *ctx, const void *key) +{ + return av_crc(ctx->crc, 0, key, ctx->key_size) % ctx->max_entries; +} + +int av_hashtable_get(const struct AVHashtableContext *ctx, const void *key, void *val) +{ + size_t hash = hash_key(ctx, key); + + if (!ctx->utilization) + return 0; + + for (size_t psl = 0; psl < ctx->max_entries; psl++) { + size_t wrapped_index = (hash + psl) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || *(size_t*)ENTRY_PSL(entry) < psl) + return 0; + if (KEYS_EQUAL(ENTRY_KEY(entry), key)) { + memcpy(val, ENTRY_VAL(entry), ctx->val_size); + return 1; + } + } + return 0; +} + +int av_hashtable_set(struct AVHashtableContext *ctx, const void *key, const void *val) +{ + int swapping = 0; + size_t psl = 0; + size_t hash = hash_key(ctx, key); + uint8_t *set = ctx->swapbuf; + uint8_t *tmp = ctx->swapbuf + ctx->key_size_aligned + ctx->val_size_aligned; + + memcpy(set, key, ctx->key_size); + memcpy(set + ctx->key_size_aligned, val, ctx->val_size); + + for (size_t i = 0; i < ctx->max_entries; i++) { + size_t wrapped_index = (hash + i) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || (!swapping && KEYS_EQUAL(ENTRY_KEY(entry), set))) { + if (!*ENTRY_OCC(entry)) + ctx->utilization++; + *(size_t*)ENTRY_PSL(entry) = psl; + *ENTRY_OCC(entry) = 1; + memcpy(ENTRY_KEY(entry), set, ctx->key_size_aligned + ctx->val_size); + return 1; + } + if (*(size_t*)ENTRY_PSL(entry) < psl) { + if (ctx->utilization == ctx->max_entries) + return 0; + swapping = 1; + // set needs to swap with entry + memcpy(tmp, ENTRY_KEY(entry), ctx->key_size_aligned + ctx->val_size); + memcpy(ENTRY_KEY(entry), set, ctx->key_size_aligned + ctx->val_size); + FFSWAP(uint8_t*, set, tmp); + FFSWAP(size_t, psl, *(size_t*)ENTRY_PSL(entry)); + } + psl++; + } + return 0; +} + +int av_hashtable_delete(struct AVHashtableContext *ctx, const void *key) +{ + uint8_t *next_entry; + size_t hash = hash_key(ctx, key); + + if (!ctx->utilization) + return 0; + + for (size_t psl = 0; psl < ctx->max_entries; psl++) { + size_t wrapped_index = (hash + psl) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || *(size_t*)ENTRY_PSL(entry) < psl) + return 0; + if (KEYS_EQUAL(ENTRY_KEY(entry), key)) { + *ENTRY_OCC(entry) = 0; + for (psl++; psl < ctx->max_entries; psl++) { + wrapped_index = (hash + psl) % ctx->max_entries; + next_entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(next_entry) || !*(size_t*)ENTRY_PSL(next_entry)) { + ctx->utilization--; + return 1; + } + memcpy(entry, next_entry, ctx->entry_size); + (*(size_t*)ENTRY_PSL(entry))--; + *ENTRY_OCC(next_entry) = 0; + entry = next_entry; + } + } + }; + return 0; +} + +void av_hashtable_clear(struct AVHashtableContext *ctx) +{ + memset(ctx->table, 0, ctx->entry_size * ctx->max_entries); +} + +void av_hashtable_freep(struct AVHashtableContext **ctx) +{ + if (*ctx) { + av_freep(&(*ctx)->table); + av_freep(&(*ctx)->swapbuf); + } + av_freep(ctx); +} diff --git a/libavutil/hashtable.h b/libavutil/hashtable.h new file mode 100644 index 0000000000..8ce6faafe3 --- /dev/null +++ b/libavutil/hashtable.h @@ -0,0 +1,91 @@ +/* + * Generic hashtable + * Copyright (C) 2024 Connor Worley + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_HASHTABLE_H +#define AVUTIL_HASHTABLE_H + +#include + +/* Implements a hash table using Robin Hood open addressing. + * See: https://cs.uwaterloo.ca/research/tr/1986/CS-86-14.pdf + */ + +struct AVHashtableContext; + +/** + * Creates a fixed-sized Robin Hood hash table. + * + * @param ctx context to allocate and initialize + * @param key_size size of key type in bytes + * @param val_size size of value type in bytes + * @param max_entries maximum number of key-value pairs to store + * + * @return zero on success, nonzero on error + */ +int av_hashtable_alloc(struct AVHashtableContext **ctx, size_t key_size, size_t val_size, size_t max_entries); + +/** + * Looks up a value from a hash table given a key. + * + * @param ctx hash table context + * @param key pointer to key data + * @param val destination pointer for value data + * + * @return 1 if the key is found, zero if the key is not found + */ +int av_hashtable_get(const struct AVHashtableContext *ctx, const void *key, void *val); + +/** + * Stores a value in a hash table given a key. + * + * @param ctx hash table context + * @param key pointer to key data + * @param val pointer for value data + * + * @return 1 if the key is written, zero if the key is not written due to the hash table reaching max capacity + */ +int av_hashtable_set(struct AVHashtableContext *ctx, const void *key, const void *val); + +/** + * Deletes a value from a hash table given a key. + * + * @param ctx hash table context + * @param key pointer to key data + * + * @return 1 if the key is deleted, zero if the key is not deleted due to not being found + */ +int av_hashtable_delete(struct AVHashtableContext *ctx, const void *key); + +/** + * Deletes all values from a hash table. + * + * @param ctx hash table context + */ +void av_hashtable_clear(struct AVHashtableContext *ctx); + +/** + * Frees a hash table. + * + * @param ctx hash table context + */ +void av_hashtable_freep(struct AVHashtableContext **ctx); + +#endif diff --git a/libavutil/tests/hashtable.c b/libavutil/tests/hashtable.c new file mode 100644 index 0000000000..0b38b34c5c --- /dev/null +++ b/libavutil/tests/hashtable.c @@ -0,0 +1,108 @@ +/* + * Generic hashtable tests + * Copyright (C) 2024 Connor Worley + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/avassert.h" +#include "libavutil/hashtable.h" + +int main(void) +{ + struct AVHashtableContext *ctx; + uint8_t k; + uint64_t v; + + // impossibly large allocation should fail gracefully + av_assert0(av_hashtable_alloc(&ctx, -1, -1, -1) < 0); + + // hashtable can store up to 3 uint8_t->uint64_t entries + av_assert0(!av_hashtable_alloc(&ctx, sizeof(k), sizeof(v), 3)); + + // unsuccessful deletes return 0 + k = 1; + av_assert0(!av_hashtable_delete(ctx, &k)); + + // unsuccessful gets return 0 + k = 1; + av_assert0(!av_hashtable_get(ctx, &k, &v)); + + // successful sets returns 1 + k = 1; + v = 1; + av_assert0(av_hashtable_set(ctx, &k, &v)); + + // get should now contain 1 + k = 1; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 1); + + // updating sets should return 1 + k = 1; + v = 2; + av_assert0(av_hashtable_set(ctx, &k, &v)); + + // get should now contain 2 + k = 1; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 2); + + // fill the table + k = 2; + v = 2; + av_assert0(av_hashtable_set(ctx, &k, &v)); + k = 3; + v = 3; + av_assert0(av_hashtable_set(ctx, &k, &v)); + + // inserting sets on a full table should return 0 + k = 4; + v = 4; + av_assert0(!av_hashtable_set(ctx, &k, &v)); + + // updating sets on a full table should return 1 + k = 1; + v = 4; + av_assert0(av_hashtable_set(ctx, &k, &v)); + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 4); + + // successful deletes should return 1 + k = 1; + av_assert0(av_hashtable_delete(ctx, &k)); + + // get should now return 0 + av_assert0(!av_hashtable_get(ctx, &k, &v)); + + // sanity check remaining keys + k = 2; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 2); + k = 3; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 3); + + av_hashtable_freep(&ctx); + + return 0; +} From patchwork Thu May 23 22:10:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Connor Worley X-Patchwork-Id: 49194 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:542:0:b0:460:55fa:d5ed with SMTP id 63csp1350503vqf; Thu, 23 May 2024 15:11:16 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCU1nMzZJFvua05StgLeDsC8qCtqvVCiSy45lDn2F9qEwRXkTM6n9Ble8owdSOTaruSQHZth/HfXXgL1cPofn3MM7nnMwuX3QKHrwg== X-Google-Smtp-Source: AGHT+IGLjMLci55F+lVwQ2A6iMc1lrUCCoVCtNKNeUhUUSrI7hwGyJv4QLMak6fnm1pW14LANYgt X-Received: by 2002:a2e:b385:0:b0:2e6:be3c:9d4d with SMTP id 38308e7fff4ca-2e95b096b68mr3142851fa.12.1716502276027; Thu, 23 May 2024 15:11:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716502275; cv=none; d=google.com; s=arc-20160816; b=OZJe/gZf+wiy9MU61rRYtYmo5xY4vbTq+wNqvHnqGnzbiWjgMxNrl79C32kIt/a6cM Yae6+1ATGS3LNiFtJRY7y2J7JS2X5bEn34X4jvhPJpHqtVmHBn9LTb7JU+fxTom31dG5 fE5AEz9S0qaE8BuesfpT2ran8Jroqg2vm96PtWinqyFkr5vgoRxhXqj/hDME8RwcS4Ad KNMl0XfkWAMgG0dUAy6HaxSvmwbQo6smaSfk/11JFbhs3tbb1cCJ7kGtDeCfp9gjB5mH 8ewDeVAoTru2RxL1MS96su9Gzx2stHNgjkgWUCBQn31EY4yU0xNrSjkb+3RQqW72y2Qc rV2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=679Q+YNT/0WfM5NKVIkwZYatBtUktxK43s/tNIR7zsw=; fh=4o9WedBEbOrq7GvNs0s+TOKeyzmDZRM3O7oe6ZP6vYM=; b=Y91Cle7EwsMPvt6KjadH1u8E8ZaWtA6v9WEruK8tV3ZMMqVll8owbw+YWKnqQWaLG9 Bog6rwfcUfCpOfPQb1nY2C86wa1Do/ZvFFZ48+jLhQde/tVIgVXgJpaI8Bm3HcWi6jPB Eao7SpbmKoj07+jPYDdC8r5griIaj2FtGn9ZuJPI3RgCJSwZtaHHd1SMC+i4mKBglZ46 r/kPPBc+XfZUj8T9UNgR5Xqwurqe6vRCdMDGHnlAQ5zBl4XUf0boc1YE3qltL2gYL4X7 c7Q8N7puuCJNURJ1dF6Y9IuhFbkECpuXJic4vv4t8zXrUCj7U5wIBH/DUzCEyUmbMNJG yLwA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=RkCUJGC8; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-578524b03ecsi133946a12.480.2024.05.23.15.11.15; Thu, 23 May 2024 15:11:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=RkCUJGC8; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9B10668D51F; Fri, 24 May 2024 01:11:13 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2B6CA68D29F for ; Fri, 24 May 2024 01:11:07 +0300 (EEST) Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6f8e9870e72so288217b3a.1 for ; Thu, 23 May 2024 15:11:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716502264; x=1717107064; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H7xfGCdQLUa6tGdp8AhlUzDShD/84uH5wRKBb+er4v8=; b=RkCUJGC89Y1gJYr3qNNzThWnP4zvAJODkQUuTuGG85sgFhWWThVnU7NtHTiQtyK2St KZygkGApootGwy0+enS21lsjgBzs8oFgA5pwaIly3gixjacQDHCjP8ANJBxGdyY/vpW+ eLCljPomKtO6MmJGQ6ct9MyJKqVvTD8xiDHyOnZKo041zvVXgfr3tprpU3fFa/LZ2Kt1 +umGD43MZO0A/iqWDHXLmOSHK6MqSQvgw9tel/mEAJ7AVnXRX6cy5OV6yPSIcyFCmp5A NLznQhQsqXAHkLkwCDHYdmooUEpXjFgeCJr/jD+tWiMqjkXgwL8JgiXfBDdgc48bcIsp 1tZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716502264; x=1717107064; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H7xfGCdQLUa6tGdp8AhlUzDShD/84uH5wRKBb+er4v8=; b=uyk35U9DUFDZosv4mtdFsftsOCZ2Wu4uqhRLZnspPeny+VweiEgjwbTtbjFPi/a4LB nVn7QVYGebT7M2h8NoW4KrAoabEBlgphNMly5sigJGm9G2alDFlwqIpNPmkpNq4ivVrI R4S2CsPxUUVWXHcNBE5BiQ7UYoGKRSmxmVbazGIfACMxqRTlLtiRMg3dDWP12zrjjZpe P+9T4WxehmOB1X5rQ4JHuT434YoiUrrMA4fCSQclKPgNuUr8S6CFbIgyMoTENJlGL8ad /xO7h/jpQ+BRmpnYtYVCinNYpoIxRsKTNmz0dHPiWI8KfWL6XKCT7z7SwDaO3aLJdJp3 CSFA== X-Gm-Message-State: AOJu0Yy4ZwmCEyf2dnf7w7TqzXeJGPzNLkS9QgJuBYL56qIX48MKJvzP slL6GmFjJtceF/eX1rbhcoLRlz1yrar1nT6MbCmsBr+lkuJ7wFAPL7uHPQ== X-Received: by 2002:a05:6a21:186:b0:1a7:3b4a:3e8 with SMTP id adf61e73a8af0-1b212ccf7eamr737494637.7.1716502264235; Thu, 23 May 2024 15:11:04 -0700 (PDT) Received: from localhost ([2607:fb10:7011:1::7d4]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-6f8fcbea96fsm92689b3a.108.2024.05.23.15.11.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 23 May 2024 15:11:03 -0700 (PDT) From: Connor Worley To: ffmpeg-devel@ffmpeg.org Date: Thu, 23 May 2024 15:10:05 -0700 Message-ID: <20240523221005.20659-2-connorbworley@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240523221005.20659-1-connorbworley@gmail.com> References: <20240523221005.20659-1-connorbworley@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v5 2/2] lavc/dxvenc: migrate DXT1 encoder to lavu hashtable X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Connor Worley Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: QU4jJPLwxZo3 Offers a modest performance gain due to the switch from naive linear probling to robin hood. Signed-off-by: Connor Worley --- libavcodec/dxvenc.c | 119 ++++++++++++-------------------------------- 1 file changed, 32 insertions(+), 87 deletions(-) diff --git a/libavcodec/dxvenc.c b/libavcodec/dxvenc.c index 8229438373..8962be25a1 100644 --- a/libavcodec/dxvenc.c +++ b/libavcodec/dxvenc.c @@ -21,7 +21,7 @@ #include -#include "libavutil/crc.h" +#include "libavutil/hashtable.h" #include "libavutil/imgutils.h" #include "libavutil/mem.h" #include "libavutil/opt.h" @@ -39,72 +39,9 @@ * appeared in the decompressed stream. Using a simple hash table (HT) * significantly speeds up the lookback process while encoding. */ -#define LOOKBACK_HT_ELEMS 0x40000 +#define LOOKBACK_HT_ELEMS 0x20202 #define LOOKBACK_WORDS 0x20202 -typedef struct HTEntry { - uint32_t key; - uint32_t pos; -} HTEntry; - -static void ht_init(HTEntry *ht) -{ - for (size_t i = 0; i < LOOKBACK_HT_ELEMS; i++) { - ht[i].pos = -1; - } -} - -static uint32_t ht_lookup_and_upsert(HTEntry *ht, const AVCRC *hash_ctx, - uint32_t key, uint32_t pos) -{ - uint32_t ret = -1; - size_t hash = av_crc(hash_ctx, 0, (uint8_t*)&key, 4) % LOOKBACK_HT_ELEMS; - for (size_t i = hash; i < hash + LOOKBACK_HT_ELEMS; i++) { - size_t wrapped_index = i % LOOKBACK_HT_ELEMS; - HTEntry *entry = &ht[wrapped_index]; - if (entry->key == key || entry->pos == -1) { - ret = entry->pos; - entry->key = key; - entry->pos = pos; - break; - } - } - return ret; -} - -static void ht_delete(HTEntry *ht, const AVCRC *hash_ctx, - uint32_t key, uint32_t pos) -{ - HTEntry *removed_entry = NULL; - size_t removed_hash; - size_t hash = av_crc(hash_ctx, 0, (uint8_t*)&key, 4) % LOOKBACK_HT_ELEMS; - - for (size_t i = hash; i < hash + LOOKBACK_HT_ELEMS; i++) { - size_t wrapped_index = i % LOOKBACK_HT_ELEMS; - HTEntry *entry = &ht[wrapped_index]; - if (entry->pos == -1) - return; - if (removed_entry) { - size_t candidate_hash = av_crc(hash_ctx, 0, (uint8_t*)&entry->key, 4) % LOOKBACK_HT_ELEMS; - if ((wrapped_index > removed_hash && (candidate_hash <= removed_hash || candidate_hash > wrapped_index)) || - (wrapped_index < removed_hash && (candidate_hash <= removed_hash && candidate_hash > wrapped_index))) { - *removed_entry = *entry; - entry->pos = -1; - removed_entry = entry; - removed_hash = wrapped_index; - } - } else if (entry->key == key) { - if (entry->pos <= pos) { - entry->pos = -1; - removed_entry = entry; - removed_hash = wrapped_index; - } else { - return; - } - } - } -} - typedef struct DXVEncContext { AVClass *class; @@ -121,10 +58,8 @@ typedef struct DXVEncContext { DXVTextureFormat tex_fmt; int (*compress_tex)(AVCodecContext *avctx); - const AVCRC *crc_ctx; - - HTEntry color_lookback_ht[LOOKBACK_HT_ELEMS]; - HTEntry lut_lookback_ht[LOOKBACK_HT_ELEMS]; + struct AVHashtableContext *color_ht; + struct AVHashtableContext *lut_ht; } DXVEncContext; /* Converts an index offset value to a 2-bit opcode and pushes it to a stream. @@ -161,25 +96,30 @@ static int dxv_compress_dxt1(AVCodecContext *avctx) void *value; uint32_t color, lut, idx, color_idx, lut_idx, prev_pos, state = 16, pos = 2, op = 0; - ht_init(ctx->color_lookback_ht); - ht_init(ctx->lut_lookback_ht); + av_hashtable_clear(ctx->color_ht); + av_hashtable_clear(ctx->lut_ht); bytestream2_put_le32(pbc, AV_RL32(ctx->tex_data)); + av_hashtable_set(ctx->color_ht, ctx->tex_data, &pos); + pos++; bytestream2_put_le32(pbc, AV_RL32(ctx->tex_data + 4)); - - ht_lookup_and_upsert(ctx->color_lookback_ht, ctx->crc_ctx, AV_RL32(ctx->tex_data), 0); - ht_lookup_and_upsert(ctx->lut_lookback_ht, ctx->crc_ctx, AV_RL32(ctx->tex_data + 4), 1); + av_hashtable_set(ctx->lut_ht, ctx->tex_data + 4, &pos); + pos++; while (pos + 2 <= ctx->tex_size / 4) { idx = 0; + color_idx = 0; + lut_idx = 0; color = AV_RL32(ctx->tex_data + pos * 4); - prev_pos = ht_lookup_and_upsert(ctx->color_lookback_ht, ctx->crc_ctx, color, pos); - color_idx = prev_pos != -1 ? pos - prev_pos : 0; + if (av_hashtable_get(ctx->color_ht, &color, &prev_pos)) + color_idx = pos - prev_pos; + av_hashtable_set(ctx->color_ht, &color, &pos); + if (pos >= LOOKBACK_WORDS) { uint32_t old_pos = pos - LOOKBACK_WORDS; - uint32_t old_color = AV_RL32(ctx->tex_data + old_pos * 4); - ht_delete(ctx->color_lookback_ht, ctx->crc_ctx, old_color, old_pos); + if (av_hashtable_get(ctx->color_ht, ctx->tex_data + old_pos * 4, &prev_pos) && prev_pos <= old_pos) + av_hashtable_delete(ctx->color_ht, ctx->tex_data + old_pos * 4); } pos++; @@ -188,13 +128,14 @@ static int dxv_compress_dxt1(AVCodecContext *avctx) idx = color_idx; } else { idx = 0; - prev_pos = ht_lookup_and_upsert(ctx->lut_lookback_ht, ctx->crc_ctx, lut, pos); - lut_idx = prev_pos != -1 ? pos - prev_pos : 0; + if (av_hashtable_get(ctx->lut_ht, &lut, &prev_pos)) + lut_idx = pos - prev_pos; + av_hashtable_set(ctx->lut_ht, &lut, &pos); } if (pos >= LOOKBACK_WORDS) { uint32_t old_pos = pos - LOOKBACK_WORDS; - uint32_t old_lut = AV_RL32(ctx->tex_data + old_pos * 4); - ht_delete(ctx->lut_lookback_ht, ctx->crc_ctx, old_lut, old_pos); + if (av_hashtable_get(ctx->lut_ht, ctx->tex_data + old_pos * 4, &prev_pos) && prev_pos <= old_pos) + av_hashtable_delete(ctx->lut_ht, ctx->tex_data + old_pos * 4); } pos++; @@ -306,11 +247,12 @@ static av_cold int dxv_init(AVCodecContext *avctx) return AVERROR(ENOMEM); } - ctx->crc_ctx = av_crc_get_table(AV_CRC_32_IEEE); - if (!ctx->crc_ctx) { - av_log(avctx, AV_LOG_ERROR, "Could not initialize CRC table.\n"); - return AVERROR_BUG; - } + ret = av_hashtable_alloc(&ctx->color_ht, sizeof(uint32_t), sizeof(uint32_t), LOOKBACK_HT_ELEMS); + if (ret < 0) + return ret; + ret = av_hashtable_alloc(&ctx->lut_ht, sizeof(uint32_t), sizeof(uint32_t), LOOKBACK_HT_ELEMS); + if (ret < 0) + return ret; return 0; } @@ -321,6 +263,9 @@ static av_cold int dxv_close(AVCodecContext *avctx) av_freep(&ctx->tex_data); + av_hashtable_freep(&ctx->color_ht); + av_hashtable_freep(&ctx->lut_ht); + return 0; }