From patchwork Sat Feb 24 20:06:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Connor Worley X-Patchwork-Id: 46504 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:c51b:b0:19e:cdac:8cce with SMTP id gm27csp598317pzb; Sat, 24 Feb 2024 12:06:22 -0800 (PST) X-Forwarded-Encrypted: i=2; AJvYcCWkAvEWSYgGfCdINz0QqrLdyoSapIdFwDXwWm+F39zdNObEk/rejgXbKBji1zAjaQt747+gzIdc2BM4k3mB8V/MuQBRAMF2HuVd9g== X-Google-Smtp-Source: AGHT+IGb0tGV/AUXf/HACJEFxoYkKG2rPAj6d7Av3fw+1PPMLFC1ufWP51cZMuG1hPRgdPgQaYgM X-Received: by 2002:a17:906:3c18:b0:a3e:d5ac:9995 with SMTP id h24-20020a1709063c1800b00a3ed5ac9995mr1804101ejg.59.1708805181849; Sat, 24 Feb 2024 12:06:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1708805181; cv=none; d=google.com; s=arc-20160816; b=L4zf7w+oMVFjjG86ny5Bcqk+dLKcz1HrIkORGGuD8qT9w2b+TCPnT+IWworcl7uEnf ZkasmGsrhyJg9juF1r1BdHnrMS0Xir8z9odODQEiPzi0EZbXknaWv2ah4JtGVGChoKC6 sCtf6uAqESHZxGO+wYM0OwB4LPnGYzWap4NIeAxaX/ZEgJ7vmRUdsaoU8Mi/oMwyZFFg bgmhxEId6SHmFjTYCzS389p5C9+9vErtmrCVyfJwKpQAcKYe/3YHMsTjuYJ6ATwyQRQm aiMoaS5/ThISh0+o55NTGcE/IeuNb6K94oCiU1ISQPvmWgZRHRWKACtFTNHBvUWAFSHA j3hA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=3uRxXlR8SW/tkNz43IbhnCVU2QEDmjZTAvoGn0AKNSE=; fh=4o9WedBEbOrq7GvNs0s+TOKeyzmDZRM3O7oe6ZP6vYM=; b=a7UeWzknA1z5+8/7+yhBZK6UIML8aS3l0Pq8kLtxMMqTtXnxD3IWfww0B/g8Mdg64c +tlCt9VZQFBFcDjXO/fHq1dmpH3go5n4kF/fehPmiy+5Qvmr33fsO/fXFSgEFIsJs8b8 01AA3fKIpHnjSW1ifZoxSmeMYUNKZGjJi1DHScvs9YEiY6vjaK8MPrbkz+O1PGwDJYCH /oW3e6mCukHQL895VqPDIjtb55TVr2zE6V+ZWcdTIJNlPQ2Fi9JZdJ1JFtJx36nwB4Hz s3vLJmJF0YwLQfbXQPE2J/ZkJIBtTRwjDIcIWkQmMIAV8zNYP9RAz5GJliwg0pIYDIGu lArQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=FUzQ8NB+; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bu15-20020a170906a14f00b00a3f707a45f0si691501ejb.783.2024.02.24.12.06.19; Sat, 24 Feb 2024 12:06:21 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=FUzQ8NB+; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EB86A68C694; Sat, 24 Feb 2024 22:06:15 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 00E1768C59F for ; Sat, 24 Feb 2024 22:06:08 +0200 (EET) Received: by mail-pg1-f175.google.com with SMTP id 41be03b00d2f7-5ca29c131ebso1536094a12.0 for ; Sat, 24 Feb 2024 12:06:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708805166; x=1709409966; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=TAZs5ARzVfyWkPQJInV5FAxctXrAjNgJ+8b2mZXqgoA=; b=FUzQ8NB+5srKdm7aiQzlpqC8TUKRK4WlKuGY9MPw6EiuGkuDfKLgJe+mI9leqTWOSN bYqAJAsRk7jGx+ls9K0egpnASnEONAu1Pw5FMMRpVsPxzdzf5tX1U6tbnQIwdSwhNADB F3lZOvwIGQ2TMPeSl8Ja/C+4iiQWMO6KJ8BMQBanBxbPvQZoxHXZI8IeEfvnW8c7FnXc c231TPK1l4N01wptHounnlw8nvyn0T5JsJ0E12cjwAX5vOCHPCLG8E+10Z+6ZmYdn6tQ 4UGcI0JbESlAu092XDOa5GfReLougK6w+D1cCd5KegcpCza79P7weF+NgSRJmxCfG5en FRCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708805166; x=1709409966; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TAZs5ARzVfyWkPQJInV5FAxctXrAjNgJ+8b2mZXqgoA=; b=J2D/ukLKFU3COQNOoOyG666Hvb+Ik5nuiHY2BqQUU35da3JojxP5DYpy+lrXOgDS2o 9YAus3cMnFBdhFX+E9Ldr0Sr1OoSC7KJDJWkHzNFivsdnQnlKfY5mblKtgIQlYlse7c8 UeodXw5tXhi1qI3rt4IM5+v/DnFn2JFDv0b7rTuQGu/u44i0Jnz2J8Ex/7HEgARJvhfC f8dIEbf4ocPGS8U81xi6C0faHN3KBvtWuhIRm+iyZ2Ezcqp6qakKWQ6qlnczBnBjBfI6 Q6GYI+TyxEmwj/hjnv2anrrOg0cJULcEarZybZTfklO4VAf4hSbrFvIuJUwmy0oGdpZT Zn2g== X-Gm-Message-State: AOJu0YzUQLQuUJIqtE9z4QFh3YYxfiku4USqp1tAgQgXXtUMDdRwgZFd bBSDZKKxCrZqEGU8myQYr1EztLVHassWX8I/6BSwLMlGi51SGus1u5p0dj7ROsI= X-Received: by 2002:a17:90a:bd0a:b0:299:3550:9fda with SMTP id y10-20020a17090abd0a00b0029935509fdamr2688770pjr.6.1708805166400; Sat, 24 Feb 2024 12:06:06 -0800 (PST) Received: from localhost ([2601:647:4600:84e0:21ab:155b:fbb:f2b0]) by smtp.gmail.com with UTF8SMTPSA id p16-20020a17090a429000b0029967638141sm1569859pjg.37.2024.02.24.12.06.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 24 Feb 2024 12:06:05 -0800 (PST) From: Connor Worley To: ffmpeg-devel@ffmpeg.org Date: Sat, 24 Feb 2024 12:06:02 -0800 Message-ID: <20240224200603.6613-1-connorbworley@gmail.com> X-Mailer: git-send-email 2.43.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 1/2] lavu/hashtable: create generic robin hood hash table X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Connor Worley Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 1wvt6m/hgbGT Signed-off-by: Connor Worley --- libavutil/Makefile | 2 + libavutil/hashtable.c | 192 ++++++++++++++++++++++++++++++++++++ libavutil/hashtable.h | 40 ++++++++ libavutil/tests/hashtable.c | 110 +++++++++++++++++++++ 4 files changed, 344 insertions(+) create mode 100644 libavutil/hashtable.c create mode 100644 libavutil/hashtable.h create mode 100644 libavutil/tests/hashtable.c diff --git a/libavutil/Makefile b/libavutil/Makefile index e7709b97d0..be75d464fc 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -138,6 +138,7 @@ OBJS = adler32.o \ fixed_dsp.o \ frame.o \ hash.o \ + hashtable.o \ hdr_dynamic_metadata.o \ hdr_dynamic_vivid_metadata.o \ hmac.o \ @@ -251,6 +252,7 @@ TESTPROGS = adler32 \ file \ fifo \ hash \ + hashtable \ hmac \ hwdevice \ integer \ diff --git a/libavutil/hashtable.c b/libavutil/hashtable.c new file mode 100644 index 0000000000..155a264665 --- /dev/null +++ b/libavutil/hashtable.c @@ -0,0 +1,192 @@ +/* + * Generic hashtable + * Copyright (C) 2024 Connor Worley + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include +#include + +#include "crc.h" +#include "error.h" +#include "mem.h" +#include "hashtable.h" + +#define ALIGN _Alignof(size_t) + +struct AVHashtableContext { + size_t key_size; + size_t key_size_aligned; + size_t val_size; + size_t val_size_aligned; + size_t entry_size; + size_t max_entries; + size_t utilization; + const AVCRC *crc; + uint8_t *table; + uint8_t *swapbuf; +}; + +#define ENTRY_PSL(entry) (entry) +#define ENTRY_OCC(entry) (ENTRY_PSL(entry) + FFALIGN(sizeof(size_t), ALIGN)) +#define ENTRY_KEY(entry) (ENTRY_OCC(entry) + FFALIGN(sizeof(size_t), ALIGN)) +#define ENTRY_VAL(entry) (ENTRY_KEY(entry) + ctx->key_size_aligned) + +#define KEYS_EQUAL(k1, k2) !memcmp(k1, k2, ctx->key_size) + +int av_hashtable_alloc(struct AVHashtableContext **ctx, size_t key_size, size_t val_size, size_t max_entries) +{ + struct AVHashtableContext *res = av_malloc(sizeof(struct AVHashtableContext)); + if (!res) + return AVERROR(ENOMEM); + res->key_size = key_size; + res->key_size_aligned = FFALIGN(key_size, ALIGN); + res->val_size = val_size; + res->val_size_aligned = FFALIGN(val_size, ALIGN); + res->entry_size = FFALIGN(sizeof(size_t), ALIGN) + + FFALIGN(sizeof(size_t), ALIGN) + + res->key_size_aligned + + res->val_size_aligned; + res->max_entries = max_entries; + res->utilization = 0; + res->crc = av_crc_get_table(AV_CRC_32_IEEE); + if (!res->crc) { + av_hashtable_freep(&res); + return AVERROR_BUG; + } + res->table = av_calloc(res->max_entries, res->entry_size); + if (!res->table) { + av_hashtable_freep(&res); + return AVERROR(ENOMEM); + } + res->swapbuf = av_calloc(2, res->key_size_aligned + res->val_size_aligned); + if (!res->swapbuf) { + av_hashtable_freep(&res); + return AVERROR(ENOMEM); + } + *ctx = res; + return 0; +} + +static size_t hash_key(const struct AVHashtableContext *ctx, const void *key) +{ + return av_crc(ctx->crc, 0, key, ctx->key_size) % ctx->max_entries; +} + +int av_hashtable_get(const struct AVHashtableContext *ctx, const void *key, void *val) +{ + size_t hash = hash_key(ctx, key); + + if (!ctx->utilization) + return 0; + + for (size_t psl = 0; psl < ctx->max_entries; psl++) { + size_t wrapped_index = (hash + psl) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || *(size_t*)ENTRY_PSL(entry) < psl) + return 0; + if (KEYS_EQUAL(ENTRY_KEY(entry), key)) { + memcpy(val, ENTRY_VAL(entry), ctx->val_size); + return 1; + } + } + return 0; +} + +int av_hashtable_set(struct AVHashtableContext *ctx, const void *key, const void *val) +{ + int swapping = 0; + size_t psl = 0; + size_t hash = hash_key(ctx, key); + uint8_t *set = ctx->swapbuf; + uint8_t *tmp = ctx->swapbuf + ctx->key_size_aligned + ctx->val_size_aligned; + + memcpy(set, key, ctx->key_size); + memcpy(set + ctx->key_size_aligned, val, ctx->val_size); + + for (size_t i = 0; i < ctx->max_entries; i++) { + size_t wrapped_index = (hash + i) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || (!swapping && KEYS_EQUAL(ENTRY_KEY(entry), set))) { + if (!*ENTRY_OCC(entry)) + ctx->utilization++; + *(size_t*)ENTRY_PSL(entry) = psl; + *ENTRY_OCC(entry) = 1; + memcpy(ENTRY_KEY(entry), set, ctx->key_size_aligned + ctx->val_size); + return 1; + } + if (*(size_t*)ENTRY_PSL(entry) < psl) { + if (ctx->utilization == ctx->max_entries) + return 0; + swapping = 1; + // set needs to swap with entry + memcpy(tmp, ENTRY_KEY(entry), ctx->key_size_aligned + ctx->val_size); + memcpy(ENTRY_KEY(entry), set, ctx->key_size_aligned + ctx->val_size); + FFSWAP(uint8_t*, set, tmp); + FFSWAP(size_t, psl, *(size_t*)ENTRY_PSL(entry)); + } + psl++; + } + return 0; +} + +int av_hashtable_delete(struct AVHashtableContext *ctx, const void *key) +{ + uint8_t *next_entry; + size_t hash = hash_key(ctx, key); + + if (!ctx->utilization) + return 0; + + for (size_t psl = 0; psl < ctx->max_entries; psl++) { + size_t wrapped_index = (hash + psl) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || *(size_t*)ENTRY_PSL(entry) < psl) + return 0; + if (KEYS_EQUAL(ENTRY_KEY(entry), key)) { + *ENTRY_OCC(entry) = 0; + for (psl++; psl < ctx->max_entries; psl++) { + wrapped_index = (hash + psl) % ctx->max_entries; + next_entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(next_entry) || !*(size_t*)ENTRY_PSL(next_entry)) { + ctx->utilization--; + return 1; + } + memcpy(entry, next_entry, ctx->entry_size); + (*(size_t*)ENTRY_PSL(entry))--; + *ENTRY_OCC(next_entry) = 0; + entry = next_entry; + } + } + }; + return 0; +} + +void av_hashtable_clear(struct AVHashtableContext *ctx) +{ + memset(ctx->table, 0, ctx->entry_size * ctx->max_entries); +} + +void av_hashtable_freep(struct AVHashtableContext **ctx) +{ + if (*ctx) { + av_freep(&(*ctx)->table); + av_freep(&(*ctx)->swapbuf); + } + av_freep(ctx); +} diff --git a/libavutil/hashtable.h b/libavutil/hashtable.h new file mode 100644 index 0000000000..f14086867e --- /dev/null +++ b/libavutil/hashtable.h @@ -0,0 +1,40 @@ +/* + * Generic hashtable + * Copyright (C) 2024 Connor Worley + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_HASHTABLE_H +#define AVUTIL_HASHTABLE_H + +#include + +/* Implements a hash table using Robin Hood open addressing. + * See: https://cs.uwaterloo.ca/research/tr/1986/CS-86-14.pdf + */ + +struct AVHashtableContext; + +int av_hashtable_alloc(struct AVHashtableContext **ctx, size_t key_size, size_t val_size, size_t max_entries); +int av_hashtable_get(const struct AVHashtableContext *ctx, const void *key, void *val); +int av_hashtable_set(struct AVHashtableContext *ctx, const void *key, const void *val); +int av_hashtable_delete(struct AVHashtableContext *ctx, const void *key); +void av_hashtable_clear(struct AVHashtableContext *ctx); +void av_hashtable_freep(struct AVHashtableContext **ctx); + +#endif diff --git a/libavutil/tests/hashtable.c b/libavutil/tests/hashtable.c new file mode 100644 index 0000000000..1e37d7663b --- /dev/null +++ b/libavutil/tests/hashtable.c @@ -0,0 +1,110 @@ +/* + * Generic hashtable tests + * Copyright (C) 2024 Connor Worley + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include + +#include "libavutil/avassert.h" +#include "libavutil/hashtable.h" + +int main(void) +{ + struct AVHashtableContext *ctx; + uint8_t k; + uint64_t v; + + // impossibly large allocation should fail gracefully + av_assert0(av_hashtable_alloc(&ctx, -1, -1, -1) < 0); + + // hashtable can store up to 3 uint8_t->uint64_t entries + av_assert0(!av_hashtable_alloc(&ctx, sizeof(k), sizeof(v), 3)); + + // unsuccessful deletes return 0 + k = 1; + av_assert0(!av_hashtable_delete(ctx, &k)); + + // unsuccessful gets return 0 + k = 1; + av_assert0(!av_hashtable_get(ctx, &k, &v)); + + // successful sets returns 1 + k = 1; + v = 1; + av_assert0(av_hashtable_set(ctx, &k, &v)); + + // get should now contain 1 + k = 1; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 1); + + // updating sets should return 1 + k = 1; + v = 2; + av_assert0(av_hashtable_set(ctx, &k, &v)); + + // get should now contain 2 + k = 1; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 2); + + // fill the table + k = 2; + v = 2; + av_assert0(av_hashtable_set(ctx, &k, &v)); + k = 3; + v = 3; + av_assert0(av_hashtable_set(ctx, &k, &v)); + + // inserting sets on a full table should return 0 + k = 4; + v = 4; + av_assert0(!av_hashtable_set(ctx, &k, &v)); + + // updating sets on a full table should return 1 + k = 1; + v = 4; + av_assert0(av_hashtable_set(ctx, &k, &v)); + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 4); + + // successful deletes should return 1 + k = 1; + av_assert0(av_hashtable_delete(ctx, &k)); + + // get should now return 0 + av_assert0(!av_hashtable_get(ctx, &k, &v)); + + // sanity check remaining keys + k = 2; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 2); + k = 3; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 3); + + av_hashtable_freep(&ctx); + + return 0; +}