From patchwork Thu May 23 12:27:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 49177 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:542:0:b0:460:55fa:d5ed with SMTP id 63csp1005153vqf; Thu, 23 May 2024 05:28:20 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUgk42W0yI1jfa9LFNsKmhBvJ9mKLjjPkgnOCl9iIK+u46o08MrhhQ0VEtdnQPxEyHebDg+Hi2UZBecDxQXP8iLUoiOkaJ+K1geyg== X-Google-Smtp-Source: AGHT+IFDriXpnoWyZvlpr+fsBALCEtZ2zHXEFLfY9w3+JIL3M1bCvrHGr/W4+ktkU4Ed5gwrXz4u X-Received: by 2002:a50:c308:0:b0:575:a7e:4f82 with SMTP id 4fb4d7f45d1cf-57843c4349fmr1981892a12.4.1716467300486; Thu, 23 May 2024 05:28:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716467300; cv=none; d=google.com; s=arc-20160816; b=DoGkbZ8ojv8NjlG65DCGsTNl92rSjSZDW8LfrxlY00cq5vJQCNxINYlfLSoOeet0os bqWT5TrRC5uk4gjI4fe3A4Lz+okf3NQ7TwC7FVOVuqB1HCu3yMDM0GjlyEI/x8dqyy1V 1DEwgzJZBT1XxbBI1XdH/yifnEsgE7Qx7zHML/N1EjKQ5sFG6Hu3qhl678Ln21rKjnli 4igYNLNSbfWJlGIKJGN6xt/6bOwP0W5d15F0MhgYxzxfbLJZb4RJdBh9zH9ljR0h8nqG 0fSMCrDRhf3GiThTYpDsVKD1jnforz5wToaD80LJ5DsE+hTUjgophHMrVrDmaJYrAfeP gnlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=kcf7H6fi3ezoCWor2PVwIz0uhpiVI1AkdxzQxLl3ziA=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=JfbSL73FXedt+/PTlGk2Gb9IMOi894hNoiXMMp6Ur5XeN6aTKzTJGuMP1qchFvEuF8 pcFpCw2tVPyG9rfT7/HyU/Ypeq7QrNLDfZhQXammgUnTNa9ZumTCt4y89XG2xXLb7mtZ 6uV8Y4Ziac26bMDTpB+kSurbKZNdbLWP6W04K1It8vXYYzkUgPlbCGttYCigiZmr3Zbi XehLhAILLMQuLPTjr+Vd9STWUPkZbwq0iQO2yBlXfZirntTsMlaaY11AP9Wov2CfTzkb GILlY71QAMdRjXN5tyDko2b9CIEfTW51wbWN/yvEOsDij/z7IdlB1kgH+FTs95Mv/NXx cGag==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=cNtX4pxu; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5733bec496dsi17290296a12.197.2024.05.23.05.28.20; Thu, 23 May 2024 05:28:20 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=cNtX4pxu; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 665BD68D3CA; Thu, 23 May 2024 15:28:01 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CCD3468D47C for ; Thu, 23 May 2024 15:27:52 +0300 (EEST) Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1f333e7a669so10980575ad.3 for ; Thu, 23 May 2024 05:27:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716467270; x=1717072070; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=ekMMY5zLfAjfdBlCPOpvb6u3f2PYonkYg6dLAUmAveQ=; b=cNtX4pxuC0b0t8xNV1dCnNe9scGRxNnpJQhvZc6rgvEuk15+54mmyJM9TlUE16sMRn nVNj0F+xh1Vwf1eizcqrpTPFaoQUcavfwSP17socc22FHDPOIpQTwXnsgcKXhRZwp3a4 eYex0jBKlqp4B6eOapcXYS1WEUrqnrus07YScfeYiOH986aJXsfM+jlGzF3OolcE6Xkx tBYd0ETcSa3ac5F5FASzdiylBAA3dWpUPRUul1fwYAtDVwwQI6vBGMy9ketvXC5A7foY f9QxyOehLm/TAUHJmEyEAIE4Vwk7N6rv2S8tWFBCMsR3bVvpRUtVCpZPOnMAnAmG4YO8 hOHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716467270; x=1717072070; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ekMMY5zLfAjfdBlCPOpvb6u3f2PYonkYg6dLAUmAveQ=; b=Nj7RciNweQZd0MYShJWJi/pmptgLnvCO+//lqAhNE2Y1tRXcWrbbivPR5/eVyVQoFd cla36xJMIWO8X5TtViROqJIqHhubm6sSr36DOjFZqlWYkdWHVa0GgvXLXMnM7gpbxAf+ Qu3pM6VIAgC7GIyhh0kX2+lRShsAs0P05kZFHsHk51IIR7cbvVIbUBnrmoYg4770lQ3Z Yq4odynePnv+u5KbnY02AMBHCQsGYalvRK7YQJSePZfS8nVeD7X0DQ6SxX7LPSeCWvvk Ast0t9kOP18VcM3iWKuCLILopeZ07DWl31gtyN52Tnn9V7ZkzZkmU8e6BbhfcUgYnhu9 BXBg== X-Gm-Message-State: AOJu0Yw3EbWgE+Gbyil07fzpZ1M2vM0pIiK3fgw3eAohf5QAKdqsZdB2 gQHIG5BEE5R+uoGRhIuB6fo4hjTljm0TESl9cUy1UL/AGG7pCo3fRm6YMA== X-Received: by 2002:a17:902:ce08:b0:1f2:fd5d:ced0 with SMTP id d9443c01a7336-1f31c94bdefmr58985265ad.12.1716467270428; Thu, 23 May 2024 05:27:50 -0700 (PDT) Received: from localhost.localdomain ([190.194.167.233]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1f335da4b87sm16158405ad.100.2024.05.23.05.27.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 05:27:50 -0700 (PDT) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Thu, 23 May 2024 09:27:14 -0300 Message-ID: <20240523122716.2158-3-jamrial@gmail.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240523122716.2158-1-jamrial@gmail.com> References: <20240523122716.2158-1-jamrial@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/5] x86/vvc_sad: add sse4 versions of all functions X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 6q052lauZXQz And remove sad_8x8_avx2, as it's not faster than sad_8x8_sse4. sad_8x8_c: 54.8 sad_8x8_sse4: 14.3 sad_16x16_c: 200.8 sad_16x16_sse4: 34.8 sad_16x16_avx2: 29.8 sad_32x32_c: 826.3 sad_32x32_sse4: 113.8 sad_32x32_avx2: 69.3 sad_64x64_c: 3679.8 sad_64x64_sse4: 392.8 sad_64x64_avx2: 257.3 sad_128x128_c: 12581.3 sad_128x128_sse4: 1560.8 sad_128x128_avx2: 1151.8 Signed-off-by: James Almer --- libavcodec/x86/vvc/vvc_sad.asm | 53 +++++++++++++++++++++----------- libavcodec/x86/vvc/vvcdsp_init.c | 42 +++++++++++++++++-------- 2 files changed, 65 insertions(+), 30 deletions(-) diff --git a/libavcodec/x86/vvc/vvc_sad.asm b/libavcodec/x86/vvc/vvc_sad.asm index 829dbce489..26df25ec66 100644 --- a/libavcodec/x86/vvc/vvc_sad.asm +++ b/libavcodec/x86/vvc/vvc_sad.asm @@ -26,7 +26,7 @@ SECTION_RODATA -pw_1: times 2 dw 1 +cextern pw_1 ; DMVR SAD is only calculated on even rows to reduce complexity SECTION .text @@ -38,20 +38,21 @@ SECTION .text %endmacro %macro HORIZ_ADD 3 ; xm0, xm1, m1 +%if mmsize == 32 vextracti128 %1, %3, q0001 ; 3 2 1 0 - paddd %1, %2 ; xm0 (7 + 3) (6 + 2) (5 + 1) (4 + 0) - pshufd %2, %1, q0032 ; xm1 - - (7 + 3) (6 + 2) + paddd %2, %1 ; xm1 (7 + 3) (6 + 2) (5 + 1) (4 + 0) +%endif + pshufd %1, %2, q0032 ; xm0 - - (7 + 3) (6 + 2) paddd %1, %1, %2 ; xm0 _ _ (5 1 7 3) (4 0 6 2) pshufd %2, %1, q0001 ; xm1 _ _ (5 1 7 3) (5 1 7 3) paddd %1, %1, %2 ; (01234567) %endmacro -%if ARCH_X86_64 -%if HAVE_AVX2_EXTERNAL - -INIT_YMM avx2 - -cglobal vvc_sad_8, 6, 9, 5, src1, src2, dx, dy, block_w, block_h, off1, off2, row_idx +%macro VVC_SAD 1 +cglobal vvc_sad_%1, 4, 7, 5, src1, src2, dx, dy, off1, block_h, off2 +%if UNIX64 == 0 + mov block_hd, dword r5m +%endif movsxdifnidn dxq, dxd movsxdifnidn dyq, dyd @@ -74,29 +75,32 @@ cglobal vvc_sad_8, 6, 9, 5, src1, src2, dx, dy, block_w, block_h, off1, off2, ro lea src2q, [src2q + off2q * 2 + 2 * 2] pxor m3, m3 +%if mmsize == 32 vpbroadcastd m4, [pw_1] +%else + mova m4, [pw_1] +%endif .loop_height: - movu xm0, [src1q] - vinserti128 m0, m0, [src1q + MAX_PB_SIZE * ROWS * 2], 1 - movu xm1, [src2q] - vinserti128 m1, m1, [src2q + MAX_PB_SIZE * ROWS * 2], 1 - + movu m0, [src1q] + movu m1, [src2q] MIN_MAX_SAD m1, m0, m2 pmaddwd m1, m4 paddd m3, m1 - add src1q, 2 * MAX_PB_SIZE * ROWS * 2 - add src2q, 2 * MAX_PB_SIZE * ROWS * 2 + add src1q, ROWS * MAX_PB_SIZE * 2 + add src2q, ROWS * MAX_PB_SIZE * 2 - sub block_hd, 4 + sub block_hd, 2 jg .loop_height HORIZ_ADD xm0, xm3, m3 movd eax, xm0 RET +%endmacro -cglobal vvc_sad_16, 6, 8, 5, src1, src2, dx, dy, block_w, block_h, off1, off2 +%macro VVC_SAD_LOOP 1 +cglobal vvc_sad_%1, 6, 8, 5, src1, src2, dx, dy, block_w, block_h, off1, off2 movsxdifnidn dxq, dxd movsxdifnidn dyq, dyd @@ -119,7 +123,11 @@ cglobal vvc_sad_16, 6, 8, 5, src1, src2, dx, dy, block_w, block_h, off1, off2 lea src2q, [src2q + off2q * 2 + 2 * 2] pxor m3, m3 +%if mmsize == 32 vpbroadcastd m4, [pw_1] +%else + mova m4, [pw_1] +%endif shl block_wd, 1 add src1q, block_wq @@ -149,6 +157,15 @@ DEFINE_ARGS src1, src2, dx, dy, block_w, block_h, row_idx HORIZ_ADD xm0, xm3, m3 movd eax, xm0 RET +%endmacro +%if ARCH_X86_64 +INIT_XMM sse4 +VVC_SAD 8 +VVC_SAD_LOOP 16 +%if HAVE_AVX2_EXTERNAL +INIT_YMM avx2 +VVC_SAD 16 +VVC_SAD_LOOP 32 %endif %endif diff --git a/libavcodec/x86/vvc/vvcdsp_init.c b/libavcodec/x86/vvc/vvcdsp_init.c index bd60963432..cdf0e36b62 100644 --- a/libavcodec/x86/vvc/vvcdsp_init.c +++ b/libavcodec/x86/vvc/vvcdsp_init.c @@ -316,16 +316,10 @@ ALF_FUNCS(16, 12, avx2) int bf(ff_vvc_sad, w, opt)(const int16_t *src0, const int16_t *src1, \ int dx, int dy, int block_w, int block_h) \ -SAD_PROTOTYPE(8, avx2); +SAD_PROTOTYPE(8, sse4); +SAD_PROTOTYPE(16, sse4); SAD_PROTOTYPE(16, avx2); - -#define SAD_INIT(opt) do { \ - c->inter.sad[0] = ff_vvc_sad_8_##opt; \ - c->inter.sad[1] = \ - c->inter.sad[2] = \ - c->inter.sad[3] = \ - c->inter.sad[4] = ff_vvc_sad_16_##opt; \ -} while (0) +SAD_PROTOTYPE(32, avx2); #endif void ff_vvc_dsp_init_x86(VVCDSPContext *const c, const int bd) @@ -337,36 +331,60 @@ void ff_vvc_dsp_init_x86(VVCDSPContext *const c, const int bd) case 8: if (EXTERNAL_SSE4(cpu_flags)) { MC_LINK_SSE4(8); + c->inter.sad[0] = ff_vvc_sad_8_sse4; + c->inter.sad[1] = + c->inter.sad[2] = + c->inter.sad[3] = + c->inter.sad[4] = ff_vvc_sad_16_sse4; } if (EXTERNAL_AVX2_FAST(cpu_flags)) { ALF_INIT(8); AVG_INIT(8, avx2); MC_LINKS_AVX2(8); - SAD_INIT(avx2); + c->inter.sad[1] = ff_vvc_sad_16_avx2; + c->inter.sad[2] = + c->inter.sad[3] = + c->inter.sad[4] = ff_vvc_sad_32_avx2; } break; case 10: if (EXTERNAL_SSE4(cpu_flags)) { MC_LINK_SSE4(10); + c->inter.sad[0] = ff_vvc_sad_8_sse4; + c->inter.sad[1] = + c->inter.sad[2] = + c->inter.sad[3] = + c->inter.sad[4] = ff_vvc_sad_16_sse4; } if (EXTERNAL_AVX2_FAST(cpu_flags)) { ALF_INIT(10); AVG_INIT(10, avx2); MC_LINKS_AVX2(10); MC_LINKS_16BPC_AVX2(10); - SAD_INIT(avx2); + c->inter.sad[1] = ff_vvc_sad_16_avx2; + c->inter.sad[2] = + c->inter.sad[3] = + c->inter.sad[4] = ff_vvc_sad_32_avx2; } break; case 12: if (EXTERNAL_SSE4(cpu_flags)) { MC_LINK_SSE4(12); + c->inter.sad[0] = ff_vvc_sad_8_sse4; + c->inter.sad[1] = + c->inter.sad[2] = + c->inter.sad[3] = + c->inter.sad[4] = ff_vvc_sad_16_sse4; } if (EXTERNAL_AVX2_FAST(cpu_flags)) { ALF_INIT(12); AVG_INIT(12, avx2); MC_LINKS_AVX2(12); MC_LINKS_16BPC_AVX2(12); - SAD_INIT(avx2); + c->inter.sad[1] = ff_vvc_sad_16_avx2; + c->inter.sad[2] = + c->inter.sad[3] = + c->inter.sad[4] = ff_vvc_sad_32_avx2; } break; default: