From patchwork Tue Feb 6 10:57:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Connor Worley X-Patchwork-Id: 46070 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:7b08:b0:19e:8a94:b663 with SMTP id s8csp1320529pzh; Tue, 6 Feb 2024 02:58:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IEgCsL9x4WBavQLN1H99/b2S9bOrGc/bdRA5EFzfagsqNcvLLsDl+xzEA/Ue8SYwlMC2mon X-Received: by 2002:a05:6402:b18:b0:560:5031:589e with SMTP id bm24-20020a0564020b1800b005605031589emr1164589edb.35.1707217093715; Tue, 06 Feb 2024 02:58:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1707217093; cv=none; d=google.com; s=arc-20160816; b=QQ/bDIa5XVybQ0hUD1sRmhIB5mABChSUkWaXZ+SKQRvbbdJ+l3gUbmAfk87FJemeqw k1q8RQH4bDz/wN30okBnPKcCFBY7QN2TRM8fiW7HLAHJmgaQ+OWvgJFKiYs0I9IeDyMN BNlaSNLdFp124OIL93EYCmE6bKDOXWpeQa6Jqzb4KESPQ/B0XWqbHEptGBzOe5mj2TR+ BXtEaHqZO3g3d+ylhkJqOra51D2Lg3Lj9i6Nl6aCFJK02GerN5w18aiTzMQ+1vsn6cLV FjLbNeXBaT/G37AjZcejvsv6DtSaay7+zsy+JTbzWB3B/NLIGJn2Un/50IQKgZRC+fiK kIcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=YOKRzvy+/GzA49Wg/f8xJREecOEKQG5BoyxyXOBOGM4=; fh=zNaiYXkL31z7a7UeB8/g2Ef568eo5d4VyhaIdg7yV94=; b=wodp4BAyzWnbAxIUFdg8K/vQiJeabBa+N5jo6/+Rk2YAfcwCFtSu9s2N50wooftsSo /T0/llkrougWYS4aW/HEUykoB/0AkEuQ8eSZtUuPe6LCXoJpEscrd6cogK5X+3ynliTL ykdPtBxi6atqO3i+Wu3N5MaW7rs5l0ZnerAPvsoRJ0QxeHO4MVvVPuKimLYH/+tbekHP 6dNxNj1zHipeRLSeUigmFojMoisdFUSXC7oeHyul3i+0NszvFYUPxeTNOGIFXN9z+T3l FQ/Ve6aEUIOHljOirm3l4cDoIR7T7av5pndZ6dncTFQjnJdNdQsaF8z26Wex1+aKWIHy JMDA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=BBUkMEVr; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=0; AJvYcCU/8zu2r3zoxA/fkOyPG27OAyghyFy+CB+bd3h5+x1m2+R5jzDjsans2CFAcELcIaGKxqGDmk4whTpZ0KFBNo4OlkP8VtvlrPTay/nOnw2mLiIDocbTUYpRb8SIisdHUbEEZ8MHQ/qtDBPLZ2W45Mxpv+asRL16/40F9zkbfTnsIyFmfrMP5d4hZfuQxFb9OsKG65r5Y//RJJFJZRiO/jMzeA2kIkKYhIt8bghUHRwmDr+WEc/392QTrPQZn+ReRQrj8ZkU5M3OTRH0e43GaOp/0Rrz+ocCdZli2m58T/YHD4JvD70umRhr1B9xG1dBJCjQg+WVh7JfGQZhl3xF6CDEoM1haboCYLrcIDWT7x+Smtd6G2I0fH530O5gYe74P8gvRsNiqHQgZBB/rF64FXZY8iW1Rll2ILN+TpT38cckrzyOS3h30u4Swgv80SMZ5Nvq879r7OrAnN841UrKG2/eRwZ37hiJT3EANg27VgHPfJwjRF32ziXcKLxJWFxcf96OwwrLfQcP55jYuS9IaM118Oh8BLr2IUxXHPz7o2kjwoCusxT8UeCkXKVMQlHmXfQtuu7vxG0P/E+sY0edLYgissRRY1sFDaC2L/OgMO/352sv97LNswGFGIK5GjAVrFqz7Lz/QIjwNQSA75i4mvuW3VoMdh7MYelXXRcXoimjqgClQSOhJEvQrONDj1ES9uRLfmXfuIWM+nWj6IVSMt+MM0REMXKNAR5c/ZlCyyCjAQ/BwKTw3e8Dvu8vprPyBFNd5S0m8VB3iw/8ScAj99n2q5VFS4+G8WZsmDtrPgyDO7cMlcrOhKHtSBYk2l/BAEJb3SVI41IXYwqW9CN1jXWs6dCdGXEmCAPZZdvM/wfS2XzFX7D5fpbvsij1rtqoMDb1LClQwcDl409Vbrec0MTkYqVCqyhC4SLWx6cer76Ye7hT8LOe2osudaQTczxxOW/Ae7 t8hSJgVkxCCBROadgHLXTYs09nTeRqnTZz41yZQXzfKg2zmQNurQ60m0sdcJefa3o8e+yV9Evx03MavrM8MyX8HYPx4iBrJipJXFUJt8+fl8zvaVEQJvMk+gz+ysypAFJb8/DNCc9laH2lVoNtBECXY0OrrwZp+LkEVIa/46VoaGNv9WJa1pi9HruConQeeYDQAGfOhunV1jCtDe1KDVB4+bbxIn7nng+kGVEoti4+L+OBFq/Pp+g2f0OOUjHG1r4a5R1Ly1DR2/IaUgNh4t1DHLpIPkBp8sSVH9uvFlwU6Nlt3oea4g9/Ywvn+bisbRcz7McNVLAKHY0Guvxri4qPrAYWQw+preuwmbFQc3K+r/CYSCXVmguwmJJ+Tl7jhicmJ578ncTVyKV7Mg22qSh5UokBToXfc3HnzMPo19APLc23JOu03xC/Ug4fOTEDOcKYY4DXZ5rAIdlhumHHL0vrFGGoLSUzdsQ97vDQqTfQ2+xevLbA8/Ic0hRovD0l9Y5f3A2rO+AgrQggVmceQWGhdbQmJ3cj/DhfRN5J+6azxyPWZIh9W46IIlWajZR7LqB4mDaYmRtruPJqIcAjDeSmRX8hyN+cmJs4MuwZbztqVYpDSVLrMQPnPoAkdLu1k+/0q34= Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e9-20020a056402190900b00560a1525fc1si521964edz.303.2024.02.06.02.58.13; Tue, 06 Feb 2024 02:58:13 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=BBUkMEVr; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4C15068D14F; Tue, 6 Feb 2024 12:58:10 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6B68268CBBD for ; Tue, 6 Feb 2024 12:58:03 +0200 (EET) Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-6dc36e501e1so3838466a34.1 for ; Tue, 06 Feb 2024 02:58:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707217080; x=1707821880; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=x1gT+5NPNbQyFXc8abzBxXrAZerRHgJbp9mdPmX+dOk=; b=BBUkMEVrmqyCCV4vxy/yP6HjZVJpaDBglolsVqBiixVJmLI7apVie6/PjeLl27msJQ XTfwjBoQksGsJCOQM8rlNIrbZexD794FDBlkJTKYdPSc+tLZsxEVI+r5Hkulkb8edzJh KYJ71nRRSqGRuwVbO8As79vb2A1T2vE2yg58pv3q6a+ckqBO0rbm/5A3MPeyxRTpuwg2 Jp5qZ79+F4xZnm8omlQUTKiYeNiL6xG4KlqFP6iShaZ45Kw2i1dlOuLXZSXdiTeJpEw/ lz0fQHHNUCXdPk9hmB2ct/s61nDTgOdNQpidHNKeqbzKelmU/Nx1hXVg1ePKbtbZkCRv 4Xfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707217080; x=1707821880; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=x1gT+5NPNbQyFXc8abzBxXrAZerRHgJbp9mdPmX+dOk=; b=JJM7RhfF3FZGcJtDv6f4NB1QIaUwrwB/8QpxM0uQkAcz+syN4q4H8/YYsuxNQuhq0M YJLHGDWchPgnorjr93ylNyCefEHCZ23xaM2M/ByPH/MaU84qr36hU26d+E/Qoo/XD48d bpZfzKY09F7KZQgP0dn+jbrhAxAPgvpjI8QhSvxUHiM+XkC+r53fEVatDMqkyps2K/vU BgzEfrxQ9uT2B0gaskbv3QlcEeEnSZgkR6qvr7N4nno9IZ0tm6RKDxFSyIzLzbXxr2hD clud6GILuWH5dALciqMZxbX9kgS2J8+s6KMThRFDNASLkyEjiEi5rfshayxC0V3cTmH5 dxcQ== X-Gm-Message-State: AOJu0Yxq7+YPrfv/+kJEXe5CTxAKF2aPc1ZRSle5yqVr+v5ezfsrI/AV wwGB7bRDVk7aMjmGyvo4UYG4lGoxCUCJg7O7ZJ7qCBwTTy2jObX2wwyahhZpuFw= X-Received: by 2002:a05:6358:7247:b0:176:91bb:9600 with SMTP id i7-20020a056358724700b0017691bb9600mr1744387rwa.3.1707217080329; Tue, 06 Feb 2024 02:58:00 -0800 (PST) Received: from localhost ([2601:647:4600:84e0:11ee:353:31ea:3350]) by smtp.gmail.com with UTF8SMTPSA id ei27-20020a056a0080db00b006e04c3b3b5asm1620627pfb.175.2024.02.06.02.57.59 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 06 Feb 2024 02:57:59 -0800 (PST) From: Connor Worley To: ffmpeg-devel@ffmpeg.org Date: Tue, 6 Feb 2024 02:57:53 -0800 Message-Id: <20240206105754.94877-1-connorbworley@gmail.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 1/2] lavu/hashtable: create generic robin hood hash table X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Connor Worley Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: L4jHFbKBKdLM Signed-off-by: Connor Worley --- libavutil/Makefile | 2 + libavutil/hashtable.c | 183 ++++++++++++++++++++++++++++++++++++ libavutil/hashtable.h | 47 +++++++++ libavutil/tests/hashtable.c | 108 +++++++++++++++++++++ 4 files changed, 340 insertions(+) create mode 100644 libavutil/hashtable.c create mode 100644 libavutil/hashtable.h create mode 100644 libavutil/tests/hashtable.c diff --git a/libavutil/Makefile b/libavutil/Makefile index e7709b97d0..be75d464fc 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -138,6 +138,7 @@ OBJS = adler32.o \ fixed_dsp.o \ frame.o \ hash.o \ + hashtable.o \ hdr_dynamic_metadata.o \ hdr_dynamic_vivid_metadata.o \ hmac.o \ @@ -251,6 +252,7 @@ TESTPROGS = adler32 \ file \ fifo \ hash \ + hashtable \ hmac \ hwdevice \ integer \ diff --git a/libavutil/hashtable.c b/libavutil/hashtable.c new file mode 100644 index 0000000000..6fa2af8af3 --- /dev/null +++ b/libavutil/hashtable.c @@ -0,0 +1,183 @@ +/* + * Generic hashtable + * Copyright (C) 2024 Connor Worley + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include +#include + +#include "crc.h" +#include "error.h" +#include "mem.h" +#include "hashtable.h" + +struct AVHashtableContext { + size_t key_size; + size_t val_size; + size_t entry_size; + size_t max_entries; + size_t utilization; + const AVCRC *crc; + uint8_t *table; + uint8_t *swapbuf; +}; + +#define ENTRY_PSL(entry) (entry) +#define ENTRY_OCC(entry) (ENTRY_PSL(entry) + sizeof(size_t)) +#define ENTRY_KEY(entry) (ENTRY_OCC(entry) + 4) +#define ENTRY_VAL(entry) (ENTRY_KEY(entry) + FFALIGN(ctx->key_size, 4)) + +#define KEYS_EQUAL(k1, k2) !memcmp(k1, k2, ctx->key_size) + +int av_hashtable_alloc(struct AVHashtableContext** ctx, size_t key_size, size_t val_size, size_t max_entries) +{ + struct AVHashtableContext *res = av_malloc(sizeof(struct AVHashtableContext)); + if (!res) + return AVERROR(ENOMEM); + res->key_size = key_size; + res->val_size = val_size; + res->entry_size = sizeof(size_t) + 4 + FFALIGN(key_size, 4) + FFALIGN(val_size, 4); + res->max_entries = max_entries; + res->utilization = 0; + res->crc = av_crc_get_table(AV_CRC_32_IEEE); + if (!res->crc) { + av_hashtable_freep(&res); + return AVERROR_BUG; + } + res->table = av_calloc(res->max_entries, res->entry_size); + if (!res->table) { + av_hashtable_freep(&res); + return AVERROR(ENOMEM); + } + res->swapbuf = av_calloc(2, FFALIGN(key_size, 4) + FFALIGN(val_size, 4)); + if (!res->swapbuf) { + av_hashtable_freep(&res); + return AVERROR(ENOMEM); + } + *ctx = res; + return 0; +} + +static size_t hash_key(const struct AVHashtableContext* ctx, const void* key) +{ + return av_crc(ctx->crc, 0, key, ctx->key_size) % ctx->max_entries; +} + +int av_hashtable_get(const struct AVHashtableContext* ctx, const void* key, void* val) +{ + size_t hash = hash_key(ctx, key); + + if (!ctx->utilization) + return 0; + + for (size_t psl = 0; psl < ctx->max_entries; psl++) { + size_t wrapped_index = (hash + psl) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || *(size_t*)ENTRY_PSL(entry) < psl) + return 0; + if (KEYS_EQUAL(ENTRY_KEY(entry), key)) { + memcpy(val, ENTRY_VAL(entry), ctx->val_size); + return 1; + } + } + return 0; +} + +int av_hashtable_set(struct AVHashtableContext* ctx, const void* key, const void* val) +{ + int swapping = 0; + size_t psl = 0; + size_t hash = hash_key(ctx, key); + uint8_t *set = ctx->swapbuf; + uint8_t *tmp = ctx->swapbuf + FFALIGN(ctx->key_size, 4) + FFALIGN(ctx->val_size, 4); + + memcpy(set, key, ctx->key_size); + memcpy(set + FFALIGN(ctx->key_size, 4), val, ctx->val_size); + + for (size_t i = 0; i < ctx->max_entries; i++) { + size_t wrapped_index = (hash + i) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || (!swapping && KEYS_EQUAL(ENTRY_KEY(entry), set))) { + if (!*ENTRY_OCC(entry)) + ctx->utilization++; + *(size_t*)ENTRY_PSL(entry) = psl; + *ENTRY_OCC(entry) = 1; + memcpy(ENTRY_KEY(entry), set, FFALIGN(ctx->key_size, 4) + ctx->val_size); + return 1; + } + if (*(size_t*)ENTRY_PSL(entry) < psl) { + if (ctx->utilization == ctx->max_entries) + return 0; + swapping = 1; + // set needs to swap with entry + memcpy(tmp, ENTRY_KEY(entry), FFALIGN(ctx->key_size, 4) + ctx->val_size); + memcpy(ENTRY_KEY(entry), set, FFALIGN(ctx->key_size, 4) + ctx->val_size); + FFSWAP(uint8_t*, set, tmp); + FFSWAP(size_t, psl, *(size_t*)ENTRY_PSL(entry)); + } + psl++; + } + return 0; +} + +int av_hashtable_delete(struct AVHashtableContext* ctx, const void* key) +{ + uint8_t *next_entry; + size_t hash = hash_key(ctx, key); + + if (!ctx->utilization) + return 0; + + for (size_t psl = 0; psl < ctx->max_entries; psl++) { + size_t wrapped_index = (hash + psl) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || *(size_t*)ENTRY_PSL(entry) < psl) + return 0; + if (KEYS_EQUAL(ENTRY_KEY(entry), key)) { + *ENTRY_OCC(entry) = 0; + for (psl++; psl < ctx->max_entries; psl++) { + wrapped_index = (hash + psl) % ctx->max_entries; + next_entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(next_entry) || !*(size_t*)ENTRY_PSL(next_entry)) { + ctx->utilization--; + return 1; + } + memcpy(ENTRY_OCC(entry), ENTRY_OCC(next_entry), ctx->entry_size); + *(size_t*)ENTRY_PSL(entry)--; + *ENTRY_OCC(next_entry) = 0; + entry = next_entry; + } + } + }; + return 0; +} + +void av_hashtable_clear(struct AVHashtableContext* ctx) +{ + memset(ctx->table, 0, ctx->entry_size * ctx->max_entries); +} + +void av_hashtable_freep(struct AVHashtableContext** ctx) +{ + if (*ctx) { + av_freep(&(*ctx)->table); + av_freep(&(*ctx)->swapbuf); + } + av_freep(ctx); +} diff --git a/libavutil/hashtable.h b/libavutil/hashtable.h new file mode 100644 index 0000000000..90c6cad68a --- /dev/null +++ b/libavutil/hashtable.h @@ -0,0 +1,47 @@ +/* + * Generic hashtable + * Copyright (C) 2024 Connor Worley + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_HASHTABLE_H +#define AVUTIL_HASHTABLE_H + +#include + +/* Implements a hash table using Robin Hood open addressing. + * See: https://cs.uwaterloo.ca/research/tr/1986/CS-86-14.pdf + * + * Entry slots are layed out in memory as follows: + * - [1 byte] occupancy flag that indicates whether a slot if occupied. + * - [key_size bytes] entry key. + * - [val_size bytes] entry value. + * - [sizeof(size_t) bytes] PSL (probe sequence length), used by the Robin Hood + * algorithm. + */ + +struct AVHashtableContext; + +int av_hashtable_alloc(struct AVHashtableContext** ctx, size_t key_size, size_t val_size, size_t max_entries); +int av_hashtable_get(const struct AVHashtableContext* ctx, const void* key, void* val); +int av_hashtable_set(struct AVHashtableContext* ctx, const void* key, const void* val); +int av_hashtable_delete(struct AVHashtableContext* ctx, const void* key); +void av_hashtable_clear(struct AVHashtableContext* ctx); +void av_hashtable_freep(struct AVHashtableContext** ctx); + +#endif diff --git a/libavutil/tests/hashtable.c b/libavutil/tests/hashtable.c new file mode 100644 index 0000000000..0b38b34c5c --- /dev/null +++ b/libavutil/tests/hashtable.c @@ -0,0 +1,108 @@ +/* + * Generic hashtable tests + * Copyright (C) 2024 Connor Worley + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/avassert.h" +#include "libavutil/hashtable.h" + +int main(void) +{ + struct AVHashtableContext *ctx; + uint8_t k; + uint64_t v; + + // impossibly large allocation should fail gracefully + av_assert0(av_hashtable_alloc(&ctx, -1, -1, -1) < 0); + + // hashtable can store up to 3 uint8_t->uint64_t entries + av_assert0(!av_hashtable_alloc(&ctx, sizeof(k), sizeof(v), 3)); + + // unsuccessful deletes return 0 + k = 1; + av_assert0(!av_hashtable_delete(ctx, &k)); + + // unsuccessful gets return 0 + k = 1; + av_assert0(!av_hashtable_get(ctx, &k, &v)); + + // successful sets returns 1 + k = 1; + v = 1; + av_assert0(av_hashtable_set(ctx, &k, &v)); + + // get should now contain 1 + k = 1; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 1); + + // updating sets should return 1 + k = 1; + v = 2; + av_assert0(av_hashtable_set(ctx, &k, &v)); + + // get should now contain 2 + k = 1; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 2); + + // fill the table + k = 2; + v = 2; + av_assert0(av_hashtable_set(ctx, &k, &v)); + k = 3; + v = 3; + av_assert0(av_hashtable_set(ctx, &k, &v)); + + // inserting sets on a full table should return 0 + k = 4; + v = 4; + av_assert0(!av_hashtable_set(ctx, &k, &v)); + + // updating sets on a full table should return 1 + k = 1; + v = 4; + av_assert0(av_hashtable_set(ctx, &k, &v)); + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 4); + + // successful deletes should return 1 + k = 1; + av_assert0(av_hashtable_delete(ctx, &k)); + + // get should now return 0 + av_assert0(!av_hashtable_get(ctx, &k, &v)); + + // sanity check remaining keys + k = 2; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 2); + k = 3; + v = 0; + av_assert0(av_hashtable_get(ctx, &k, &v)); + av_assert0(v == 3); + + av_hashtable_freep(&ctx); + + return 0; +} From patchwork Tue Feb 6 10:57:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Connor Worley X-Patchwork-Id: 46071 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:7b08:b0:19e:8a94:b663 with SMTP id s8csp1320578pzh; Tue, 6 Feb 2024 02:58:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IHrn1VrnIMKh5RbrUJY2rUI0Aa5mElahQnTb6ebMy8Ei38ANf0RDbeDtlsdea0sXDTAbtwf X-Received: by 2002:a17:906:d18b:b0:a38:2ad:e458 with SMTP id c11-20020a170906d18b00b00a3802ade458mr1341935ejz.29.1707217101907; Tue, 06 Feb 2024 02:58:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1707217101; cv=none; d=google.com; s=arc-20160816; b=hIzBU657/FgQqXA3j4IxfFB+eQWRMY+C+/u3ORcjislbO+k/4RUZ+LyLEVgj+DWfqp yxaNaL9gQxmb4wq2bTXCmF65rP2cOu6eqdze0QZUxNtHFxybOPrgDcgfyyn/Rc7HeI27 2mca0HuUWNcGWmfu+YeX5+0gsVWpS/Z90soB32OSxji8tM2wFTTAmxbMhxCTrJtA18NZ 6dQ3312Iq4ijSlFsgN1yYSjQPn2WkmZwLvDXiFhgD9/ht5PFjwAtONK4pvfQQ5tYVsTO MgKCVq1snVnky8NtHLFfqdQJwdaUBurB1ePXPZ+7N/bdFVNh/p/kh7cLMdHVsEH9m5ps oPWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=Yl+KiG1+ZPUpP69q+ev1a0D1TjSG8e7jHkFMl9nSEAw=; fh=+lhFRxN09WE0v5u3t4IIqWsrGrpfKUfQxw53WhQJ1J0=; b=sS9PytfSVqIbceEUT42Tk44cXbD0mp2tfHbyAkOrtIOarWYWK51kLhsp80VzRJK+9Y gc4D2Hc5VuYGuf/UrMuUN3N4mtSwfu4XUh0CkD/8N1WfQNCFU3qPo0b5d74vw7Rpk1TN FbMAh4LXKbhWq2JWSPg+RWlT18AIu/m4JQOdPWFYfGWEUaJKpbFeRnRCVQmyx3/OAJAT 9Z2IVYfqyYdDg0SAKwveteRZlh2ww3MKIHq6EcPAmOLjCGY+8Ho+yQ+a13gVMS7m2NYy fcewxutofehjAG+z1Z1zkbICKZxXSa39PdVoCj64TNqzAoGB0QEK3GlaEfjXBmP03CL3 cOlQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=Gwx6thyP; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=0; AJvYcCUgcn1TvKa8naIS3HctAI7qU1yAj0GOT0cjWtQvvjCR/klC26u+kuBYL/dPR/9K8FCOGxTYXlgNQhhYY3zv5qhIgEl+IYo9EoEQotV65sPJw8V4pxUOhYgyuee4FKPOXAZllKOGsl6qpJEAFrwiPxyNbcfRgeBYKzq5teI3Vg3NFJYIWtDHKnblqgTZY5LlChugNoS2amT690wg9riwdjBSx6v5KGklviaAv2jAt20+wLrgdkcPqcIG7vFuWZ+EE08qVKECJQzSy/Le9qK2dwmP7FOlh7UkRoo1qZwr1TSvUZtL2tEyYnqZQ0dUByKCTC2bwYsoUWXuAL/Ppwbzih74MTW2nuVcPYsZFdHZUXlnwxV+WNOkO+yHf+vuVA0TBCW5xUJqXdU/l67xV9dRqEyzLXInlm0o9ixemvW6TTEA4/GhVAjz25NpC2fN0gnm8VU1ZgHIFzUUy8Pn1AAx7wfvA+WKDOrvNFswxkSz0FCwcz5rUibkTiLLlQbLLGTlLyWNqSydyz7ADfdafuNwqLGlBGGO0p304i8ujebNDF8haeAgixOQjFoum0rKjRlKmAO9+hKi+ahV7j8dRNNGgyXeepkxy1sxC7zDWvTkqacT/V5p6g4gX9S83WiDSgH0jjp3/fZtGdG5cEAiWuzYmpdBwww3e/f9MXmVaOsVxn9OCwsQndA4zDHplXHOaRXaikJ7UZmC3poVW9DnDcCQDe4RS1JCPwxa0C4j3MfPiz+sQTbaXiAHZ9FKPOfR5WUklWxdTBFPeRhZRGF8wHbDddbBQmuuw7TW1u10nFclvNjhp1I+cqz6d/x/9EU/CN/ljbYnPW/undHV19LI7d8VQnO0YQPn5qd/G8st5bKcWcSbg6n08m1R3e2V0avE3BqrnkEfIrWjuG9WhaCaVwL5eE8/v9x4dpyNMQ6zSi1D0Nm1n5gR3wxeZsrGMNUZQW5vLv1jf4 liD9Bb137kIiT6L6/1dYyki417QTd2EdDfPclqnlXVegMkv42TpsiVnmn3TtYHpXcnZkN2VzZPGnpL5PPpg7IMa5x3tloYlFGKIed1N5AOaGLPsbbVhVHWVbNh0Fl2beVO3lEa5HSaHSV3/aoYE9uTPC99+apeUvIgiilNRlyfAFwRfovlHVhO0kYWwAkzFWecNGKWxCfU2FFm76WweyeoD35/XOR7G3UBtxsWB8RFPX8g8I70ItWNncCDVFuHApQlmUh2U8I4mbQq/9vSQ9UCApi6vd9qTFNxXbXh+adjvUXWs6waXzhCICrVcCw7FezIclBWRtHHRe7o2PwIIGIjJHc8sZeu5k2WlWd1591PqAzFTIVGoabK+EI3N+zf9U1NZOKeiHtzKsAiBYUtdKPimdl5bAaSmumwJtylReAJfgCcnk10yHIlLHI5qGge5utFGlY3C5HhL1Izi6h3J7VEroRydGs402jVgFkmqg7ydXXdOfOKo/LdqYqtFYV9uWRTBJi9CHTeT115/cVCmEjSRXNsS2+7wBZSUZL3i72Was5nYkOc9uZGvL/jabxWfNcDWcaKv5wFuffIbAv47gghkrfyyhqSuIpgXgxRMOQvsYaKg8m5CBQ5Hkl4Kqw= Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id u24-20020a170906409800b00a37d9853c39si942642ejj.738.2024.02.06.02.58.21; Tue, 06 Feb 2024 02:58:21 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=Gwx6thyP; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 467D568D164; Tue, 6 Feb 2024 12:58:12 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-il1-f177.google.com (mail-il1-f177.google.com [209.85.166.177]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CE4FB68D01B for ; Tue, 6 Feb 2024 12:58:05 +0200 (EET) Received: by mail-il1-f177.google.com with SMTP id e9e14a558f8ab-363b5ce4260so13010745ab.0 for ; Tue, 06 Feb 2024 02:58:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707217083; x=1707821883; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=npj2AOyecchkLxk7f0JZvuGQ0rxxasL2JuuM4/IT3S0=; b=Gwx6thyPbFIYYXERGTMx2R6gbrkMNybnzEvmR5qDBm5xojscmqfdzDrwAo+7kYGMIg I3TWDA7RRUdRfW3tvdZXC2TmdhYHkx0i6JzWM8SJYqWwoVtWt2hMDFDuvlI++VvnBcaI wWPooDeQB6oF+YEk1rmjEqtmTsKfygORqVY6wisbtd/ptmOAlruyMBCfiPY0pMAL53KC QTioL8eSjXcMyeS4Sv69pL3jyqG+gpkWfpfg1/Ri6LbqvhCB8hnWWrD/KzAPbfAE+03A fbns960FFmcRCeGUq/DA6+JfSFQL0CqC0f2zbv3ACjaaYPdSAJesr1mNmcivhFWZjBxT NVGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707217083; x=1707821883; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=npj2AOyecchkLxk7f0JZvuGQ0rxxasL2JuuM4/IT3S0=; b=J+7F0PiZZTS/ojbLnXYHpQH1syLzLjlTV9p4UNJ7ts9wrxgDXbgPXjfppnxS7n+Gru 3ysWf1Y8ai7lISab8olWgBZ464fX5nNv1PCnKnp1X6eeP6lgStw2J+ZycqLi24esMR0x vZsGCBWheXf03tMdedmVvGCdnB/YMOYWasAQ7QoBZFj1jgRRcsEqXYADNKcsZso6KkyJ DrjVno8cx1tXvtYfK9FTscLLF7/dD9P7xJS1v3Wsf2LWRnX7Mj46v+SLRoR7ZUacojak QpfNlH9Cd7N8WU6WiGEGdDUNJwm5J/XNU9NeM84bTh6buS4LWmoRUU1YgUwnhH4eBbSq Nhnw== X-Gm-Message-State: AOJu0YzRWmgYcZz6iD56ROW0tzWfE9627S9BivTSdSyZlHkuSGYqzcZV G+Mk932K3M7SzNUhJRsughbA6HTYyqq5agWm8iKJPmNbClgW2zXP3wkaW2g/dzM= X-Received: by 2002:a92:d312:0:b0:363:9ce7:42ed with SMTP id x18-20020a92d312000000b003639ce742edmr2194656ila.22.1707217083134; Tue, 06 Feb 2024 02:58:03 -0800 (PST) Received: from localhost ([2601:647:4600:84e0:11ee:353:31ea:3350]) by smtp.gmail.com with UTF8SMTPSA id h17-20020a656391000000b005cfbdf71baasm1562362pgv.47.2024.02.06.02.58.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 06 Feb 2024 02:58:02 -0800 (PST) From: Connor Worley To: ffmpeg-devel@ffmpeg.org Date: Tue, 6 Feb 2024 02:57:54 -0800 Message-Id: <20240206105754.94877-2-connorbworley@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240206105754.94877-1-connorbworley@gmail.com> References: <20240206105754.94877-1-connorbworley@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 2/2] lavc/dxvenc: migrate DXT1 encoder to lavu hashtable X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Connor Worley Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: FpQpDPXdm8Kp Offers a modest performance gain due to the switch from naive linear probling to robin hood. Signed-off-by: Connor Worley --- libavcodec/dxvenc.c | 121 ++++++++++++-------------------------------- 1 file changed, 33 insertions(+), 88 deletions(-) diff --git a/libavcodec/dxvenc.c b/libavcodec/dxvenc.c index b274175689..ebc48aace3 100644 --- a/libavcodec/dxvenc.c +++ b/libavcodec/dxvenc.c @@ -21,7 +21,7 @@ #include -#include "libavutil/crc.h" +#include "libavutil/hashtable.h" #include "libavutil/imgutils.h" #include "libavutil/opt.h" @@ -37,76 +37,13 @@ * appeared in the decompressed stream. Using a simple hash table (HT) * significantly speeds up the lookback process while encoding. */ -#define LOOKBACK_HT_ELEMS 0x40000 +#define LOOKBACK_HT_ELEMS 0x20202 #define LOOKBACK_WORDS 0x20202 enum DXVTextureFormat { DXV_FMT_DXT1 = MKBETAG('D', 'X', 'T', '1'), }; -typedef struct HTEntry { - uint32_t key; - uint32_t pos; -} HTEntry; - -static void ht_init(HTEntry *ht) -{ - for (size_t i = 0; i < LOOKBACK_HT_ELEMS; i++) { - ht[i].pos = -1; - } -} - -static uint32_t ht_lookup_and_upsert(HTEntry *ht, const AVCRC *hash_ctx, - uint32_t key, uint32_t pos) -{ - uint32_t ret = -1; - size_t hash = av_crc(hash_ctx, 0, (uint8_t*)&key, 4) % LOOKBACK_HT_ELEMS; - for (size_t i = hash; i < hash + LOOKBACK_HT_ELEMS; i++) { - size_t wrapped_index = i % LOOKBACK_HT_ELEMS; - HTEntry *entry = &ht[wrapped_index]; - if (entry->key == key || entry->pos == -1) { - ret = entry->pos; - entry->key = key; - entry->pos = pos; - break; - } - } - return ret; -} - -static void ht_delete(HTEntry *ht, const AVCRC *hash_ctx, - uint32_t key, uint32_t pos) -{ - HTEntry *removed_entry = NULL; - size_t removed_hash; - size_t hash = av_crc(hash_ctx, 0, (uint8_t*)&key, 4) % LOOKBACK_HT_ELEMS; - - for (size_t i = hash; i < hash + LOOKBACK_HT_ELEMS; i++) { - size_t wrapped_index = i % LOOKBACK_HT_ELEMS; - HTEntry *entry = &ht[wrapped_index]; - if (entry->pos == -1) - return; - if (removed_entry) { - size_t candidate_hash = av_crc(hash_ctx, 0, (uint8_t*)&entry->key, 4) % LOOKBACK_HT_ELEMS; - if ((wrapped_index > removed_hash && (candidate_hash <= removed_hash || candidate_hash > wrapped_index)) || - (wrapped_index < removed_hash && (candidate_hash <= removed_hash && candidate_hash > wrapped_index))) { - *removed_entry = *entry; - entry->pos = -1; - removed_entry = entry; - removed_hash = wrapped_index; - } - } else if (entry->key == key) { - if (entry->pos <= pos) { - entry->pos = -1; - removed_entry = entry; - removed_hash = wrapped_index; - } else { - return; - } - } - } -} - typedef struct DXVEncContext { AVClass *class; @@ -123,10 +60,8 @@ typedef struct DXVEncContext { enum DXVTextureFormat tex_fmt; int (*compress_tex)(AVCodecContext *avctx); - const AVCRC *crc_ctx; - - HTEntry color_lookback_ht[LOOKBACK_HT_ELEMS]; - HTEntry lut_lookback_ht[LOOKBACK_HT_ELEMS]; + struct AVHashtableContext *color_ht; + struct AVHashtableContext *lut_ht; } DXVEncContext; /* Converts an index offset value to a 2-bit opcode and pushes it to a stream. @@ -161,27 +96,32 @@ static int dxv_compress_dxt1(AVCodecContext *avctx) DXVEncContext *ctx = avctx->priv_data; PutByteContext *pbc = &ctx->pbc; uint32_t *value; - uint32_t color, lut, idx, color_idx, lut_idx, prev_pos, state = 16, pos = 2, op = 0; + uint32_t color, lut, idx, color_idx, lut_idx, prev_pos, state = 16, pos = 0, op = 0; - ht_init(ctx->color_lookback_ht); - ht_init(ctx->lut_lookback_ht); + av_hashtable_clear(ctx->color_ht); + av_hashtable_clear(ctx->lut_ht); bytestream2_put_le32(pbc, AV_RL32(ctx->tex_data)); + av_hashtable_set(ctx->color_ht, ctx->tex_data, &pos); + pos++; bytestream2_put_le32(pbc, AV_RL32(ctx->tex_data + 4)); - - ht_lookup_and_upsert(ctx->color_lookback_ht, ctx->crc_ctx, AV_RL32(ctx->tex_data), 0); - ht_lookup_and_upsert(ctx->lut_lookback_ht, ctx->crc_ctx, AV_RL32(ctx->tex_data + 4), 1); + av_hashtable_set(ctx->lut_ht, ctx->tex_data + 4, &pos); + pos++; while (pos + 2 <= ctx->tex_size / 4) { idx = 0; + color_idx = 0; + lut_idx = 0; color = AV_RL32(ctx->tex_data + pos * 4); - prev_pos = ht_lookup_and_upsert(ctx->color_lookback_ht, ctx->crc_ctx, color, pos); - color_idx = prev_pos != -1 ? pos - prev_pos : 0; + if (av_hashtable_get(ctx->color_ht, &color, &prev_pos)) + color_idx = pos - prev_pos; + av_hashtable_set(ctx->color_ht, &color, &pos); + if (pos >= LOOKBACK_WORDS) { uint32_t old_pos = pos - LOOKBACK_WORDS; - uint32_t old_color = AV_RL32(ctx->tex_data + old_pos * 4); - ht_delete(ctx->color_lookback_ht, ctx->crc_ctx, old_color, old_pos); + if (av_hashtable_get(ctx->color_ht, ctx->tex_data + old_pos * 4, &prev_pos) && prev_pos <= old_pos) + av_hashtable_delete(ctx->color_ht, ctx->tex_data + old_pos * 4); } pos++; @@ -190,13 +130,14 @@ static int dxv_compress_dxt1(AVCodecContext *avctx) idx = color_idx; } else { idx = 0; - prev_pos = ht_lookup_and_upsert(ctx->lut_lookback_ht, ctx->crc_ctx, lut, pos); - lut_idx = prev_pos != -1 ? pos - prev_pos : 0; + if (av_hashtable_get(ctx->lut_ht, &lut, &prev_pos)) + lut_idx = pos - prev_pos; + av_hashtable_set(ctx->lut_ht, &lut, &pos); } if (pos >= LOOKBACK_WORDS) { uint32_t old_pos = pos - LOOKBACK_WORDS; - uint32_t old_lut = AV_RL32(ctx->tex_data + old_pos * 4); - ht_delete(ctx->lut_lookback_ht, ctx->crc_ctx, old_lut, old_pos); + if (av_hashtable_get(ctx->lut_ht, ctx->tex_data + old_pos * 4, &prev_pos) && prev_pos <= old_pos) + av_hashtable_delete(ctx->lut_ht, ctx->tex_data + old_pos * 4); } pos++; @@ -298,11 +239,12 @@ static av_cold int dxv_init(AVCodecContext *avctx) return AVERROR(ENOMEM); } - ctx->crc_ctx = av_crc_get_table(AV_CRC_32_IEEE); - if (!ctx->crc_ctx) { - av_log(avctx, AV_LOG_ERROR, "Could not initialize CRC table.\n"); - return AVERROR_BUG; - } + ret = av_hashtable_alloc(&ctx->color_ht, 4, 4, LOOKBACK_HT_ELEMS); + if (ret < 0) + return ret; + ret = av_hashtable_alloc(&ctx->lut_ht, 4, 4, LOOKBACK_HT_ELEMS); + if (ret < 0) + return ret; return 0; } @@ -313,6 +255,9 @@ static av_cold int dxv_close(AVCodecContext *avctx) av_freep(&ctx->tex_data); + av_hashtable_freep(&ctx->color_ht); + av_hashtable_freep(&ctx->lut_ht); + return 0; }