From patchwork Wed Jun 29 08:24:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hubert Mazur X-Patchwork-Id: 36521 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp161162pzh; Wed, 29 Jun 2022 01:25:51 -0700 (PDT) X-Google-Smtp-Source: AGRyM1u82UqMoeMTp+WRRhXvQYOQMv+IoQJuoiAHcZL2ajle2Ogh7pc1qot8WwgS1RZxIGjSa9xW X-Received: by 2002:a17:906:a0cf:b0:726:cbdd:466b with SMTP id bh15-20020a170906a0cf00b00726cbdd466bmr2042562ejb.410.1656491150871; Wed, 29 Jun 2022 01:25:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656491150; cv=none; d=google.com; s=arc-20160816; b=Yr6RgbteORy8deHen1duLWBq+SJSjtpMmo0cpzgG0zo2GSz5ZNEhBfuJPjBbPRwJ6K yaIHBwbd1CPFA2WJOCyDSYoh6bANnmU32NxnCRoYBHPwxn6BeEO6eiDIcG9oWr5vLDK4 kwX82Yv4fsoPl+Tgkmt99R8dcUT5xD+uUBZUXS2JfGKpeyzhy14636NXeg02UVHTLVKB NFxCmsPuBsxJuIojJKpiBf9Z+EJ3xZBdhflBhKa2NnjfuquQhy0P9+JzcgfRIX772mVz KNR+hrFzbTrmgyIb31RAAG0YhyRxte1ffReWrUa+gSYsya7WhR4KPZSccm5HFsIic/On 2h2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=ebg5MTxHMedsBgRNX8DanijWPCe6rW/oNstn/TpuCbs=; b=iejZGgikfk0nl+s1nuUxZWQmCIrUyRUX3Ur4kUEiSKeFbctHqIvau6+Fah4r1lyQKM cx84xrUWSkmjgdyknrUAFUrGhluyGrIy6/Aba01b47mz0JC/pzVE5O12SFpPVUQGcP5V LGknSxboE66Dect3XTwz6OtckkjFShDUGn7JeeRPdm/aro8xC4xB05/JQn1+9mEifM5u cgsrWhNBu7vQBLjwBr6SgWIqYyjIJbrNx2Jgaac+U9pdgqgiAVwaldWibXUNh/zrY5Wm VaP8DzQZWHsO78PSKWXnwJH6QeOz8frZdlHEDsls4GfsxlIX6c35C8FJiN4aywino95W g+6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@semihalf.com header.s=google header.b=VW9+qd+8; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id gt17-20020a1709072d9100b0071d3b6e50fbsi3366434ejc.402.2022.06.29.01.25.50; Wed, 29 Jun 2022 01:25:50 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@semihalf.com header.s=google header.b=VW9+qd+8; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1847F68ACBA; Wed, 29 Jun 2022 11:25:41 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 348F368B7A1 for ; Wed, 29 Jun 2022 11:25:35 +0300 (EEST) Received: by mail-lf1-f46.google.com with SMTP id j21so26734517lfe.1 for ; Wed, 29 Jun 2022 01:25:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BAjxTH+7HECID9NtyZQJQ2aFfBMfGQzblYe+V0+0VCk=; b=VW9+qd+8booS91kZsuHqqlr1KQfpiD1sLrPATdaIH4Q81MxdUR8VB3bu8cELpQCl0G hcZpRYKx1mOvFgEibh27vwC/eTXoD09hycJwLIAjdltsAQYzFYxROrpB4+Bqqc1z4WjW YmJA1N9z1OtDhkKx7JcDtwWBWWzMq4B2VmjWr87xnPEEJ+vXaMFAi+xK6SyV7BmO3WmN 5SRARxRnjUfU8Fj7Fr8BNBts2nVMXtV9rd3wqLCKR7UPxfaFoGh0Ssj2JoJFN/LpQWpw CBcxekKWmI4QQFBBXll/9ytqqxcKbMzZbVpkos507szS7PQCXsczl6KOsKR5mF/JH75Y sYZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BAjxTH+7HECID9NtyZQJQ2aFfBMfGQzblYe+V0+0VCk=; b=1+yYvzQshB18g4QoilyDAdUaWsd3Zwau5hJez+ScpipxxKxLwvVs5wXJdDUhWuNMbe fAwMpjcz8lmL/2qn2dq+kpyi+l4t10Etai5yfYxpUYo4VBvjTDQm4DeogBK7g51ju1tp mYVTssTAi3vBlEu752WhTAMd4dLWZD44C/A8O+uDu5wHZjyofg3BAW/XvPSJvVgxM5Zf yAa8xIzHrE8H7+bm4qBJVU0Pynd1s0sM550LVeSNl398GsSRS3nmsAII55a/Fs+/hs/n Bn3ZwE9+jFRe+U3/zaJIgaIZ+lRiv6FDYft+JnigLnAE3cLE57uctTN+TxfFUzkcWNpw lLig== X-Gm-Message-State: AJIora8ruD4IFiu0TTBnu5l489rehcRoOF4Bj9rIfoVQD0oxO31Wr0fe CjT77Dwj002tpyPd2zVRvzFl9wfwAXCJcw== X-Received: by 2002:a05:6512:3b2c:b0:47f:6fc2:a9de with SMTP id f44-20020a0565123b2c00b0047f6fc2a9demr1369934lfv.352.1656491134159; Wed, 29 Jun 2022 01:25:34 -0700 (PDT) Received: from hum-HP-ProBook-440-G7.office.semihalf.net ([83.142.187.84]) by smtp.gmail.com with ESMTPSA id 8-20020ac25f08000000b00477cab33759sm2504451lfq.256.2022.06.29.01.25.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Jun 2022 01:25:33 -0700 (PDT) From: Hubert Mazur To: ffmpeg-devel@ffmpeg.org Date: Wed, 29 Jun 2022 10:24:39 +0200 Message-Id: <20220629082440.119841-2-hum@semihalf.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220629082440.119841-1-hum@semihalf.com> References: <20220629082440.119841-1-hum@semihalf.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/2] lavc/aarch64: Assign callback with function X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: gjb@semihalf.com, jswinney@amazon.com, Hubert Mazur , martin@martin.st, mw@semihalf.com, spop@amazon.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: o4edMQLkHseW Assign c->sad[0] callback with already existing neon implementation of pix_abs16 function. Signed-off-by: Hubert Mazur --- libavcodec/aarch64/me_cmp_init_aarch64.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/libavcodec/aarch64/me_cmp_init_aarch64.c b/libavcodec/aarch64/me_cmp_init_aarch64.c index 9fb63e9973..bec9148a1a 100644 --- a/libavcodec/aarch64/me_cmp_init_aarch64.c +++ b/libavcodec/aarch64/me_cmp_init_aarch64.c @@ -35,5 +35,7 @@ av_cold void ff_me_cmp_init_aarch64(MECmpContext *c, AVCodecContext *avctx) if (have_neon(cpu_flags)) { c->pix_abs[0][0] = ff_pix_abs16_neon; c->pix_abs[0][3] = ff_pix_abs16_xy2_neon; + + c->sad[0] = ff_pix_abs16_neon; } } From patchwork Wed Jun 29 08:24:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hubert Mazur X-Patchwork-Id: 36522 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp161196pzh; Wed, 29 Jun 2022 01:26:00 -0700 (PDT) X-Google-Smtp-Source: AGRyM1s61Mm064vW4nPZgcWVD4mkfolPc4aZFFpM29OOBzMmVUeBuVh2qiGfiXqRSQ3qWAI13ipm X-Received: by 2002:a17:907:970b:b0:726:9af5:c7c7 with SMTP id jg11-20020a170907970b00b007269af5c7c7mr1964646ejc.417.1656491159833; Wed, 29 Jun 2022 01:25:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656491159; cv=none; d=google.com; s=arc-20160816; b=JWTBGWR8qOQheyzJR/mN9Pvf89WhNG6FTAJRjNxidFLjKS2M5mhRSG22DHkuxaUIev 7J9/I/xNvCDmpAEjMY0wYQ6/tw3cUdX+SiYlz72e3u92KYtEsi1rss76A+QlPDwsfT+M qWRei30rpTkLzk22kWHRYNmBmH0PSh+MFuZYGUYac97mgVnL/wNQoRq11XqVS/i8OFKW NqpdhanEl1mz/I5dSSfGgac6pjQIYokipvrpZEWEVUrKn0zFgUoZmJMIN/gJI33r4nAR P/a5fnKOLGd5AuzeATayj7P0T/iDzkTvN5XFt8NMmLg+6QEgkvUopeRpe0KCL+3VBUmo WD8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=S/co52t0WbHnVoL1fU7u3QAzeHC/Ja+nmdKncDuSBoI=; b=HNYIWy81l3RdvzmfxlTFqtx8mHE3ZdY0opIVdyx+i6aFqeERj6q/Yy/4ySOUSIN5Hv U1BMDR/v7VnobHs3y1KCTkCfsm/vCeoqSjQ6PEtd/mY2ZvYBmD2+qmhZ/aPPvdPYu4T5 cL0FHtIJm7ark/zSpOjjlwEmqyx44ClRB1Q2peh/Om2Usl5nBEMwkL4Q3J+4TTcEOGo3 RcLKW4vyvERQ3M9ScQ8YxtUKq010Yy7CgRVv2K49CVtFdpkCa6ea7+r681YOGbOrGSRF cRqT9LIfh6cB8vVRbQT18/48z6K9QbBHmkwEGpI3szAhuEX99tHjFpWMyk0pJIztUYtX 3Vkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@semihalf.com header.s=google header.b=IqkLfuKt; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bx21-20020a0564020b5500b0043580fc2e3dsi17295878edb.472.2022.06.29.01.25.59; Wed, 29 Jun 2022 01:25:59 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@semihalf.com header.s=google header.b=IqkLfuKt; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 09A6968B7B9; Wed, 29 Jun 2022 11:25:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1BAB168B7A1 for ; Wed, 29 Jun 2022 11:25:42 +0300 (EEST) Received: by mail-lj1-f179.google.com with SMTP id c15so5015619ljr.0 for ; Wed, 29 Jun 2022 01:25:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=W+ornb0raI1GFmUqf18jt6aU8W2E8hb6eiQ4NuAvnU8=; b=IqkLfuKtYQd8//vNlfPk2b23dCFah6lITIEl3GSU9WXbQGEFD1FzrZ5FdVlVE4o3SG hQlRTlxEKUf5e4NPN4HWOdjsiLLMNjqHBzz1K+a+AnS+s9l8UKMzIsBzOgVorcGkNZSJ kboWlOMO1vEVMtI49oLGesHp1JYqapZKgVDqB5Jd5FaG2R3okZD+kuR/66RuLg4vwpuI 7Vnj4jvUd4181JjsSTBevGettFDFip0cfb1JGFjFcO3HH4LsKj5NjuAIZw6WKDCUcwb7 u3ZKoYN+NR81ls850KVS7C7VNDiYfD2DwjhCM7FEYBcmKPTVGv+y1QSDQFLAYjrjPcoe qSFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=W+ornb0raI1GFmUqf18jt6aU8W2E8hb6eiQ4NuAvnU8=; b=7SFcO13huKeN5HkrFs2eebtVfxdMfKPWPwgo6onPuJxpsBJfLqCrUR8amWILFZoAbG axvJGTTrNKOrqezzIDGFGPrX27tpcH8soRPaCHsfzMgDsz46absEhIILe8cq6WNI4+Zn PVkCfqb2qSnxEHJ6xpqMH3AYrOfraTuZzd0hXH7PpB1Zza9TMnDrGT7qz/HUIbegm2AF DLmJ1xiSTLuT0MxdCObd//7/ZkebNWZbqTburUcLCnFZg6bCdUPAUv8b64KRuNyXwkqy 1xQ1q6Vm/89cDXDKotMVxMKSt1YTZ/RmRVV15Lsnf9gy3P5zuZgOPnZ4cwDdFm/fN/AV 0rUA== X-Gm-Message-State: AJIora/5x0tpuQx2rq8AxO6vb5znfhIfp1vzGZVgrL2umMh+BnrqfmGj Gc3M2bfB/r3fjgze1DFNrp7guj5f4IWv6A== X-Received: by 2002:a05:651c:2050:b0:25b:84a7:32b0 with SMTP id t16-20020a05651c205000b0025b84a732b0mr1062704ljo.118.1656491141119; Wed, 29 Jun 2022 01:25:41 -0700 (PDT) Received: from hum-HP-ProBook-440-G7.office.semihalf.net ([83.142.187.84]) by smtp.gmail.com with ESMTPSA id 8-20020ac25f08000000b00477cab33759sm2504451lfq.256.2022.06.29.01.25.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Jun 2022 01:25:40 -0700 (PDT) From: Hubert Mazur To: ffmpeg-devel@ffmpeg.org Date: Wed, 29 Jun 2022 10:24:40 +0200 Message-Id: <20220629082440.119841-3-hum@semihalf.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220629082440.119841-1-hum@semihalf.com> References: <20220629082440.119841-1-hum@semihalf.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] lavc/aarch64: Add pix_abs16_x2 neon implementation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: gjb@semihalf.com, jswinney@amazon.com, Hubert Mazur , martin@martin.st, mw@semihalf.com, spop@amazon.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 1m8ldXaY6lGi Provide neon implementation for pix_abs16_x2 function. Performance tests of implementation are below. - pix_abs_0_1_c: 291.9 - pix_abs_0_1_neon: 73.7 Benchmarks and tests run with checkasm tool on AWS Graviton 3. Signed-off-by: Hubert Mazur Signed-off-by: Hubert Mazur --- libavcodec/aarch64/me_cmp_init_aarch64.c | 3 + libavcodec/aarch64/me_cmp_neon.S | 134 +++++++++++++++++++++++ 2 files changed, 137 insertions(+) diff --git a/libavcodec/aarch64/me_cmp_init_aarch64.c b/libavcodec/aarch64/me_cmp_init_aarch64.c index bec9148a1a..136b008eb7 100644 --- a/libavcodec/aarch64/me_cmp_init_aarch64.c +++ b/libavcodec/aarch64/me_cmp_init_aarch64.c @@ -27,6 +27,8 @@ int ff_pix_abs16_neon(MpegEncContext *s, uint8_t *blk1, uint8_t *blk2, ptrdiff_t stride, int h); int ff_pix_abs16_xy2_neon(MpegEncContext *s, uint8_t *blk1, uint8_t *blk2, ptrdiff_t stride, int h); +int ff_pix_abs16_x2_neon(MpegEncContext *v, uint8_t *pix1, uint8_t *pix2, + ptrdiff_t stride, int h); av_cold void ff_me_cmp_init_aarch64(MECmpContext *c, AVCodecContext *avctx) { @@ -34,6 +36,7 @@ av_cold void ff_me_cmp_init_aarch64(MECmpContext *c, AVCodecContext *avctx) if (have_neon(cpu_flags)) { c->pix_abs[0][0] = ff_pix_abs16_neon; + c->pix_abs[0][1] = ff_pix_abs16_x2_neon; c->pix_abs[0][3] = ff_pix_abs16_xy2_neon; c->sad[0] = ff_pix_abs16_neon; diff --git a/libavcodec/aarch64/me_cmp_neon.S b/libavcodec/aarch64/me_cmp_neon.S index a7937bd8be..c2fd94f4b3 100644 --- a/libavcodec/aarch64/me_cmp_neon.S +++ b/libavcodec/aarch64/me_cmp_neon.S @@ -203,3 +203,137 @@ function ff_pix_abs16_xy2_neon, export=1 fmov w0, s0 // copy result to general purpose register ret endfunc + +function ff_pix_abs16_x2_neon, export=1 + // x0 unused + // x1 uint8_t *pix1 + // x2 uint8_t *pix2 + // x3 ptrdiff_t stride + // x4 int h + + // preserve value of v8-v12 registers + stp d10, d11, [sp, #-0x10]! + stp d8, d9, [sp, #-0x10]! + + // initialize buffers + movi d18, #0 + movi v20.8h, #1 + add x5, x2, #1 // pix2 + 1 + cmp w4, #4 + b.lt 2f + +// make 4 iterations at once +1: + // v0 - pix1 + // v1 - pix2 + // v2 - pix2 + 1 + ld1 {v0.16b}, [x1], x3 + ld1 {v1.16b}, [x2], x3 + ld1 {v2.16b}, [x5], x3 + + ld1 {v3.16b}, [x1], x3 + ld1 {v4.16b}, [x2], x3 + ld1 {v5.16b}, [x5], x3 + + ld1 {v6.16b}, [x1], x3 + ld1 {v7.16b}, [x2], x3 + ld1 {v8.16b}, [x5], x3 + + ld1 {v9.16b}, [x1], x3 + ld1 {v10.16b}, [x2], x3 + ld1 {v11.16b}, [x5], x3 + + // abs(pix1[0] - avg2(pix2[0], pix2[1])) + // avg2(a,b) = (((a) + (b) + 1) >> 1) + // abs(x) = (x < 0 ? -x : x) + + // pix2[0] + pix2[1] + uaddl v30.8h, v1.8b, v2.8b + uaddl2 v29.8h, v1.16b, v2.16b + // add one to each element + add v30.8h, v30.8h, v20.8h + add v29.8h, v29.8h, v20.8h + // divide by 2, narrow width and store in v30 + uqshrn v30.8b, v30.8h, #1 + uqshrn2 v30.16b, v29.8h, #1 + + // abs(pix1[0] - avg2(pix2[0], pix2[1])) + uabd v16.16b, v0.16b, v30.16b + uaddlv h16, v16.16b + + // 2nd iteration + uaddl v28.8h, v4.8b, v5.8b + uaddl2 v27.8h, v4.16b, v5.16b + add v28.8h, v28.8h, v20.8h + add v27.8h, v27.8h, v20.8h + + uqshrn v28.8b, v28.8h, #1 + uqshrn2 v28.16b, v27.8h, #1 + + uabd v17.16b, v3.16b, v28.16b + uaddlv h17, v17.16b + + // 3rd iteration + uaddl v26.8h, v7.8b, v8.8b + uaddl2 v25.8h, v7.16b, v8.16b + add v26.8h, v26.8h, v20.8h + add v25.8h, v25.8h, v20.8h + + uqshrn v26.8b, v26.8h, #1 + uqshrn2 v26.16b, v25.8h, #1 + + uabd v19.16b, v6.16b, v26.16b + uaddlv h19, v19.16b + + // 4th iteration + uaddl v24.8h, v10.8b, v11.8b + uaddl2 v23.8h, v10.16b, v11.16b + add v24.8h, v24.8h, v20.8h + add v23.8h, v23.8h, v20.8h + + uqshrn v24.8b, v24.8h, #1 + uqshrn2 v24.16b, v23.8h, #1 + + uabd v21.16b, v9.16b, v24.16b + uaddlv h21, v21.16b + + sub w4, w4, #4 + + // accumulate the result in d18 + add d18, d18, d16 + add d18, d18, d17 + add d18, d18, d19 + add d18, d18, d21 + + cmp w4, #4 + b.ge 1b + cbz w4, 3f + +// iterate by one +2: + ld1 {v0.16b}, [x1], x3 + ld1 {v1.16b}, [x2], x3 + ld1 {v2.16b}, [x5], x3 + + uaddl v30.8h, v1.8b, v2.8b + uaddl2 v29.8h, v1.16b, v2.16b + add v30.8h, v30.8h, v20.8h + add v29.8h, v29.8h, v20.8h + + uqshrn v30.8b, v30.8h, #1 + uqshrn2 v30.16b, v20.8h, #1 + + uabd v28.16b, v0.16b, v30.16b + uaddlv h28, v28.16b + + add d18, d18, d28 + subs w4, w4, #1 + b.ne 2b + +3: + fmov w0, s18 + ldp d8, d9, [sp], 0x10 + ldp d10, d11, [sp], 0x10 + + ret +endfunc