From patchwork Sat Oct 14 08:45:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Logan.Lyu" X-Patchwork-Id: 44256 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4b15:b0:15d:8365:d4b8 with SMTP id fp21csp1476638pzb; Sat, 14 Oct 2023 01:45:54 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEGYlwR3dfon0EwNq75JmC9WaeeQpnmBvYtCoYBvdXalPP/0+lH9MY0ao1BBowjt5Q/7RxM X-Received: by 2002:a50:ee02:0:b0:533:20ab:5d7c with SMTP id g2-20020a50ee02000000b0053320ab5d7cmr25379064eds.22.1697273153845; Sat, 14 Oct 2023 01:45:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697273153; cv=none; d=google.com; s=arc-20160816; b=HZSwt1jq4eOq2OpjA0Y22fxpagx6pBadtszlssp0oVM39PdfeX+qnOXdt2KaDUqdB5 XEBMFPAPYsdh+TuVp5+7y7IGSjwOkbSbdXHgZF/kVqfcEOL/wrOsjMZqfUi/6lCag5w9 NmnuHkqWX1JVn92JdR6xNOmDLSU7j81oHU3/TDJykyxqZnqQtiL0ZN5TXhTSTxIxPyX8 trUBUM7tgvbKc+CtFMZi+FGeeynbVpx4lawkXdwSmZcup2uYLrBjKYOyYwH+f5OvJPdS ykoJyEMFsiR1NhMvy6+UspUTilL0D68yLOEMnOw6+hIToe1urTXttN/E4zoQhkAebHXM JRSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :organization:to:from:user-agent:mime-version:date:message-id :delivered-to; bh=B7dBoz/9nHClw4NmYw/teMxJQReTpr/quDI6pzKwZnk=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=dhvQOVo6tx+VXUMm5aQwwysPaG93BEGHyuRta/AJR4PLK3Up6aCjl9TBEZaqdNAlu+ HpW9YG+FgK6iXf5bcJCDsU2GJvU+7fRe3TJt4kht4L5ITrwM3m8wiirmlEnha76noBC3 E4zsaNs9qnjV0liTxiVC6qzzwnbCedGiWP3YJiXepvHVGERzdlSYzwXARKlgoUR3xOhR LEUAViPxjaUHeSooYmZM5DcRtst9OC5ixbK1XSaJdFhGk8t3iNAJkfQh3vmZybAXuC1y yj87Hq3x3na5JL5lPA/7x0tlaKGYrSu06bRwJ3tf6BDICxiU+cgbZ0L4a/JocI9LY92t gz/g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c13-20020a05640227cd00b0053e88c4d006si34533ede.25.2023.10.14.01.45.53; Sat, 14 Oct 2023 01:45:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C6A5A68C9D2; Sat, 14 Oct 2023 11:45:49 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-my3-08p62.yunyou.top (smtp-my3-08p62.yunyou.top [60.247.169.62]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 38E2B68C9BC for ; Sat, 14 Oct 2023 11:45:42 +0300 (EEST) Received: from [192.168.15.106] (unknown [125.121.27.199]) by smtp-my-08.yunyou.top (WestCloudMail) with ESMTPA id 6595EAE442 for ; Sat, 14 Oct 2023 16:45:39 +0800 (CST) Message-ID: <646d7216-e68d-4a49-821b-f358337797ef@myais.com.cn> Date: Sat, 14 Oct 2023 16:45:39 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Logan.Lyu" To: ffmpeg-devel@ffmpeg.org Organization: myais Subject: [FFmpeg-devel] [PATCH 1/4] lavc/aarch64: new optimization for 8-bit hevc_epel_v X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: fv6Ukx/ET9Ay checkasm bench: put_hevc_epel_v4_8_c: 79.9 put_hevc_epel_v4_8_neon: 25.7 put_hevc_epel_v6_8_c: 151.4 put_hevc_epel_v6_8_neon: 46.4 put_hevc_epel_v8_8_c: 250.9 put_hevc_epel_v8_8_neon: 41.7 put_hevc_epel_v12_8_c: 542.7 put_hevc_epel_v12_8_neon: 108.7 put_hevc_epel_v16_8_c: 939.4 put_hevc_epel_v16_8_neon: 169.2 put_hevc_epel_v24_8_c: 2104.9 put_hevc_epel_v24_8_neon: 307.9 put_hevc_epel_v32_8_c: 3713.9 put_hevc_epel_v32_8_neon: 524.2 put_hevc_epel_v48_8_c: 8175.2 put_hevc_epel_v48_8_neon: 1197.2 put_hevc_epel_v64_8_c: 16049.4 put_hevc_epel_v64_8_neon: 2094.9 Co-Authored-By: J. Dekker Signed-off-by: Logan Lyu --- libavcodec/aarch64/hevcdsp_epel_neon.S | 223 ++++++++++++++++++++++ libavcodec/aarch64/hevcdsp_init_aarch64.c | 5 + 2 files changed, 228 insertions(+) *c, const int bit_depth) c->put_hevc_qpel_bi[9][0][1] = ff_hevc_put_hevc_qpel_bi_h16_8_neon; NEON8_FNASSIGN(c->put_hevc_epel, 0, 0, pel_pixels,); + NEON8_FNASSIGN(c->put_hevc_epel, 1, 0, epel_v,); NEON8_FNASSIGN(c->put_hevc_qpel, 0, 0, pel_pixels,); NEON8_FNASSIGN(c->put_hevc_epel_uni, 0, 0, pel_uni_pixels,); NEON8_FNASSIGN(c->put_hevc_epel_uni, 1, 0, epel_uni_v,); diff --git a/libavcodec/aarch64/hevcdsp_epel_neon.S b/libavcodec/aarch64/hevcdsp_epel_neon.S index b4ca1e4c20..e541db5430 100644 --- a/libavcodec/aarch64/hevcdsp_epel_neon.S +++ b/libavcodec/aarch64/hevcdsp_epel_neon.S @@ -243,6 +243,229 @@ function ff_hevc_put_hevc_pel_pixels64_8_neon, export=1 ret endfunc + +function ff_hevc_put_hevc_epel_v4_8_neon, export=1 + load_epel_filterb x5, x4 + sub x1, x1, x2 + mov x10, #(MAX_PB_SIZE * 2) + ldr s16, [x1] + ldr s17, [x1 ,x2] + add x1, x1, x2, lsl #1 + ld1 {v18.s}[0], [x1], x2 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().s}[0], [x1], x2 + movi v4.8h, #0 + calc_epelb v4, \src0, \src1, \src2, \src3 + subs w3, w3, #1 + st1 {v4.4h}, [x0], x10 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_v6_8_neon, export=1 + load_epel_filterb x5, x4 + sub x1, x1, x2 + mov x10, #(MAX_PB_SIZE * 2 - 8) + ldr d16, [x1] + ldr d17, [x1, x2] + add x1, x1, x2, lsl #1 + ld1 {v18.8b}, [x1], x2 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().8b}, [x1], x2 + movi v4.8h, #0 + calc_epelb v4, \src0, \src1, \src2, \src3 + st1 {v4.d}[0], [x0], #8 + subs w3, w3, #1 + st1 {v4.s}[2], [x0], x10 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_v8_8_neon, export=1 + load_epel_filterb x5, x4 + sub x1, x1, x2 + mov x10, #(MAX_PB_SIZE * 2) + ldr d16, [x1] + ldr d17, [x1, x2] + add x1, x1, x2, lsl #1 + ld1 {v18.8b}, [x1], x2 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().8b}, [x1], x2 + movi v4.8h, #0 + calc_epelb v4, \src0, \src1, \src2, \src3 + subs w3, w3, #1 + st1 {v4.8h}, [x0], x10 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_v12_8_neon, export=1 + load_epel_filterb x5, x4 + sub x1, x1, x2 + mov x10, #(MAX_PB_SIZE * 2) + ldr q16, [x1] + ldr q17, [x1, x2] + add x1, x1, x2, lsl #1 + ld1 {v18.16b}, [x1], x2 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().16b}, [x1], x2 + movi v4.8h, #0 + movi v5.8h, #0 + calc_epelb v4, \src0, \src1, \src2, \src3 + calc_epelb2 v5, \src0, \src1, \src2, \src3 + str q4, [x0] + subs w3, w3, #1 + str d5, [x0, #16] + add x0, x0, x10 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_v16_8_neon, export=1 + load_epel_filterb x5, x4 + sub x1, x1, x2 + mov x10, #(MAX_PB_SIZE * 2) + ldr q16, [x1] + ldr q17, [x1, x2] + add x1, x1, x2, lsl #1 + ld1 {v18.16b}, [x1], x2 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().16b}, [x1], x2 + movi v4.8h, #0 + movi v5.8h, #0 + calc_epelb v4, \src0, \src1, \src2, \src3 + calc_epelb2 v5, \src0, \src1, \src2, \src3 + subs w3, w3, #1 + st1 {v4.8h, v5.8h}, [x0], x10 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_v24_8_neon, export=1 + load_epel_filterb x5, x4 + sub x1, x1, x2 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8b, v17.8b, v18.8b}, [x1], x2 + ld1 {v19.8b, v20.8b, v21.8b}, [x1], x2 + ld1 {v22.8b, v23.8b, v24.8b}, [x1], x2 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11 + ld1 {\src9\().8b, \src10\().8b, \src11\().8b}, [x1], x2 + movi v4.8h, #0 + movi v5.8h, #0 + movi v6.8h, #0 + calc_epelb v4, \src0, \src3, \src6, \src9 + calc_epelb v5, \src1, \src4, \src7, \src10 + calc_epelb v6, \src2, \src5, \src8, \src11 + subs w3, w3, #1 + st1 {v4.8h-v6.8h}, [x0], x10 +.endm +1: calc_all12 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_v32_8_neon, export=1 + load_epel_filterb x5, x4 + sub x1, x1, x2 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.16b, v17.16b}, [x1], x2 + ld1 {v18.16b, v19.16b}, [x1], x2 + ld1 {v20.16b, v21.16b}, [x1], x2 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\src6\().16b, \src7\().16b}, [x1], x2 + movi v4.8h, #0 + movi v5.8h, #0 + movi v6.8h, #0 + movi v7.8h, #0 + calc_epelb v4, \src0, \src2, \src4, \src6 + calc_epelb2 v5, \src0, \src2, \src4, \src6 + calc_epelb v6, \src1, \src3, \src5, \src7 + calc_epelb2 v7, \src1, \src3, \src5, \src7 + subs w3, w3, #1 + st1 {v4.8h-v7.8h}, [x0], x10 +.endm +1: calc_all8 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_v48_8_neon, export=1 + load_epel_filterb x5, x4 + sub x1, x1, x2 + mov x10, #64 + ld1 {v16.16b, v17.16b, v18.16b}, [x1], x2 + ld1 {v19.16b, v20.16b, v21.16b}, [x1], x2 + ld1 {v22.16b, v23.16b, v24.16b}, [x1], x2 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11 + ld1 {\src9\().16b, \src10\().16b, \src11\().16b}, [x1], x2 + movi v4.8h, #0 + movi v5.8h, #0 + movi v6.8h, #0 + movi v7.8h, #0 + movi v28.8h, #0 + movi v29.8h, #0 + calc_epelb v4, \src0, \src3, \src6, \src9 + calc_epelb2 v5, \src0, \src3, \src6, \src9 + calc_epelb v6, \src1, \src4, \src7, \src10 + calc_epelb2 v7, \src1, \src4, \src7, \src10 + calc_epelb v28, \src2, \src5, \src8, \src11 + calc_epelb2 v29, \src2, \src5, \src8, \src11 + st1 {v4.8h-v7.8h}, [x0], #64 + subs w3, w3, #1 + st1 {v28.8h-v29.8h}, [x0], x10 +.endm +1: calc_all12 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_v64_8_neon, export=1 + load_epel_filterb x5, x4 + sub sp, sp, #32 + st1 {v8.8b-v11.8b}, [sp] + sub x1, x1, x2 + ld1 {v16.16b, v17.16b, v18.16b, v19.16b}, [x1], x2 + ld1 {v20.16b, v21.16b, v22.16b, v23.16b}, [x1], x2 + ld1 {v24.16b, v25.16b, v26.16b, v27.16b}, [x1], x2 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11, src12, src13, src14, src15 + ld1 {\src12\().16b-\src15\().16b}, [x1], x2 + movi v4.8h, #0 + movi v5.8h, #0 + movi v6.8h, #0 + movi v7.8h, #0 + movi v8.8h, #0 + movi v9.8h, #0 + movi v10.8h, #0 + movi v11.8h, #0 + calc_epelb v4, \src0, \src4, \src8, \src12 + calc_epelb2 v5, \src0, \src4, \src8, \src12 + calc_epelb v6, \src1, \src5, \src9, \src13 + calc_epelb2 v7, \src1, \src5, \src9, \src13 + calc_epelb v8, \src2, \src6, \src10, \src14 + calc_epelb2 v9, \src2, \src6, \src10, \src14 + calc_epelb v10, \src3, \src7, \src11, \src15 + calc_epelb2 v11, \src3, \src7, \src11, \src15 + st1 {v4.8h-v7.8h}, [x0], #64 + subs w3, w3, #1 + st1 {v8.8h-v11.8h}, [x0], #64 +.endm +1: calc_all16 +.purgem calc +2: ld1 {v8.8b-v11.8b}, [sp] + add sp, sp, #32 + ret +endfunc + function ff_hevc_put_hevc_epel_uni_v4_8_neon, export=1 load_epel_filterb x6, x5 sub x2, x2, x3 diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index 4c377a7940..82e1623a67 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -156,6 +156,10 @@ NEON8_FNPROTO(pel_pixels, (int16_t *dst, const uint8_t *src, ptrdiff_t srcstride, int height, intptr_t mx, intptr_t my, int width),); +NEON8_FNPROTO(epel_v, (int16_t *dst, + const uint8_t *src, ptrdiff_t srcstride, + int height, intptr_t mx, intptr_t my, int width),); + NEON8_FNPROTO(pel_uni_pixels, (uint8_t *_dst, ptrdiff_t _dststride, const uint8_t *_src, ptrdiff_t _srcstride, int height, intptr_t mx, intptr_t my, int width),); @@ -305,6 +309,7 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext From patchwork Sat Oct 14 08:45:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Logan.Lyu" X-Patchwork-Id: 44257 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4b15:b0:15d:8365:d4b8 with SMTP id fp21csp1476676pzb; Sat, 14 Oct 2023 01:46:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEanVr1WXW9gkQcP0/R3NU4NGGqL0CE2p81gQAOvaZsJnl5BVrAJ5PMu9HEBxUZ2zrXKPBq X-Received: by 2002:a17:906:d54f:b0:9be:7de2:927c with SMTP id cr15-20020a170906d54f00b009be7de2927cmr1206723ejc.70.1697273164398; Sat, 14 Oct 2023 01:46:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697273164; cv=none; d=google.com; s=arc-20160816; b=Oq+6Umz1ll6KjwRmgA6MVqkAmeRdOKzse9shJdBY0ZLA0YOlwpkaRO+RKMI4tmtV72 K392c6RTAMPw9mVyqYgjcLtm3vRM1kxaPDrjKklVfTXv1a9EMxtPCzgFf3T4/ndNOZrG dOGNgB+ABquc5stnetLtF2Kx8IRYNYvuzhtRtGpgIp6f2nUKBNQ/mlAgiqirbYM1J/US BR6h4OJYXfxv8aNz2kqjdkqrSNqUzHBBJGP98iGS+nW41ptRM8wLB80VTKwQgZa7fiaO DLyqbClx24Kxsuvv42hTdj+LGSErxCeQzLuMehMmVF9JJE5zZyKSUpDKGz97maTIMcX6 qedw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :organization:to:from:user-agent:mime-version:date:message-id :delivered-to; bh=gcKLN2hYjgOxIviON8D1ilEe7HCiGluBKflUjTD3Uqs=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=iZOl9wrMmxgce+0swkYIovQr2xOwxXR1kxbMkIhA6zGWZDp3dwwjY+CXRWpyayBjxj BNO9p3MIkHcg9hBlInyLCAVxIXhAIfrhVNhg2C9vjSRliDlpl6z0fjJU3b8XhMKLSWsQ 5vGoq3rQFlXiucJZD5TgURpANkTje0DWS8KmNKKtEkIn1YQxo/tGEeXinHydqrL4IOTM EiJ/TO/lGUMPJ8ZK86tlcsUcx39ZBE8JmF+rBGOsHwMHb1nD7WE9EOezDLfx1rqdLWnq 5zPThSSQnH0zFtNVcacr5RV6qewzztjTZD+Ju+o0rfxCft9KfffMnlyxp4apNzwj98v7 GO/A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id fy10-20020a170906b7ca00b009b281fb0812si580742ejb.952.2023.10.14.01.46.04; Sat, 14 Oct 2023 01:46:04 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DEB3B68C9A0; Sat, 14 Oct 2023 11:45:55 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-my3-08p62.yunyou.top (smtp-my3-08p62.yunyou.top [60.247.169.62]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 39EBD68C9DE for ; Sat, 14 Oct 2023 11:45:49 +0300 (EEST) Received: from [192.168.15.106] (unknown [125.121.27.199]) by smtp-my-08.yunyou.top (WestCloudMail) with ESMTPA id F0FF3AE485 for ; Sat, 14 Oct 2023 16:45:46 +0800 (CST) Message-ID: Date: Sat, 14 Oct 2023 16:45:46 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Logan.Lyu" To: ffmpeg-devel@ffmpeg.org Organization: myais Subject: [FFmpeg-devel] [PATCH 2/4] lavc/aarch64: new optimization for 8-bit hevc_epel_hv X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Zo55gpItWtTP checkasm bench: put_hevc_epel_hv4_8_c: 213.7 put_hevc_epel_hv4_8_i8mm: 59.4 put_hevc_epel_hv6_8_c: 350.9 put_hevc_epel_hv6_8_i8mm: 130.2 put_hevc_epel_hv8_8_c: 548.7 put_hevc_epel_hv8_8_i8mm: 136.9 put_hevc_epel_hv12_8_c: 1126.7 put_hevc_epel_hv12_8_i8mm: 302.2 put_hevc_epel_hv16_8_c: 1925.2 put_hevc_epel_hv16_8_i8mm: 459.9 put_hevc_epel_hv24_8_c: 4301.9 put_hevc_epel_hv24_8_i8mm: 1024.9 put_hevc_epel_hv32_8_c: 7509.2 put_hevc_epel_hv32_8_i8mm: 1680.4 put_hevc_epel_hv48_8_c: 16566.9 put_hevc_epel_hv48_8_i8mm: 3945.4 put_hevc_epel_hv64_8_c: 29134.2 put_hevc_epel_hv64_8_i8mm: 6567.7 Co-Authored-By: J. Dekker Signed-off-by: Logan Lyu --- libavcodec/aarch64/hevcdsp_epel_neon.S | 265 ++++++++++++++++++++++ libavcodec/aarch64/hevcdsp_init_aarch64.c | 5 + 2 files changed, 270 insertions(+) *c, const int bit_depth) if (have_i8mm(cpu_flags)) { NEON8_FNASSIGN(c->put_hevc_epel, 0, 1, epel_h, _i8mm); + NEON8_FNASSIGN(c->put_hevc_epel, 1, 1, epel_hv, _i8mm); NEON8_FNASSIGN(c->put_hevc_epel_uni, 1, 1, epel_uni_hv, _i8mm); NEON8_FNASSIGN(c->put_hevc_epel_uni_w, 0, 1, epel_uni_w_h ,_i8mm); NEON8_FNASSIGN(c->put_hevc_qpel, 0, 1, qpel_h, _i8mm); diff --git a/libavcodec/aarch64/hevcdsp_epel_neon.S b/libavcodec/aarch64/hevcdsp_epel_neon.S index e541db5430..ebc16da5b6 100644 --- a/libavcodec/aarch64/hevcdsp_epel_neon.S +++ b/libavcodec/aarch64/hevcdsp_epel_neon.S @@ -1018,6 +1018,271 @@ function ff_hevc_put_hevc_epel_h64_8_neon_i8mm, export=1 ret endfunc + +function ff_hevc_put_hevc_epel_hv4_8_neon_i8mm, export=1 + add w10, w3, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + sub x1, x1, x2 + add w3, w3, #3 + bl X(ff_hevc_put_hevc_epel_h4_8_neon_i8mm) + ldp x5, x30, [sp] + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_epel_filterh x5, x4 + mov x10, #(MAX_PB_SIZE * 2) + ldr d16, [sp] + ldr d17, [sp, x10] + add sp, sp, x10, lsl #1 + ld1 {v18.4h}, [sp], x10 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().4h}, [sp], x10 + calc_epelh v4, \src0, \src1, \src2, \src3 + subs w3, w3, #1 + st1 {v4.4h}, [x0], x10 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_hv6_8_neon_i8mm, export=1 + add w10, w3, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + sub x1, x1, x2 + add w3, w3, #3 + bl X(ff_hevc_put_hevc_epel_h6_8_neon_i8mm) + ldp x5, x30, [sp] + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_epel_filterh x5, x4 + mov x5, #120 + mov x10, #(MAX_PB_SIZE * 2) + ldr q16, [sp] + ldr q17, [sp, x10] + add sp, sp, x10, lsl #1 + ld1 {v18.8h}, [sp], x10 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().8h}, [sp], x10 + calc_epelh v4, \src0, \src1, \src2, \src3 + calc_epelh2 v4, v5, \src0, \src1, \src2, \src3 + st1 {v4.d}[0], [x0], #8 + subs w3, w3, #1 + st1 {v4.s}[2], [x0], x5 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_hv8_8_neon_i8mm, export=1 + add w10, w3, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + sub x1, x1, x2 + add w3, w3, #3 + bl X(ff_hevc_put_hevc_epel_h8_8_neon_i8mm) + ldp x5, x30, [sp] + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_epel_filterh x5, x4 + mov x10, #(MAX_PB_SIZE * 2) + ldr q16, [sp] + ldr q17, [sp, x10] + add sp, sp, x10, lsl #1 + ld1 {v18.8h}, [sp], x10 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().8h}, [sp], x10 + calc_epelh v4, \src0, \src1, \src2, \src3 + calc_epelh2 v4, v5, \src0, \src1, \src2, \src3 + subs w3, w3, #1 + st1 {v4.8h}, [x0], x10 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_hv12_8_neon_i8mm, export=1 + add w10, w3, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + sub x1, x1, x2 + add w3, w3, #3 + bl X(ff_hevc_put_hevc_epel_h12_8_neon_i8mm) + ldp x5, x30, [sp] + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_epel_filterh x5, x4 + mov x5, #112 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8h, v17.8h}, [sp], x10 + ld1 {v18.8h, v19.8h}, [sp], x10 + ld1 {v20.8h, v21.8h}, [sp], x10 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\src6\().8h, \src7\().8h}, [sp], x10 + calc_epelh v4, \src0, \src2, \src4, \src6 + calc_epelh2 v4, v5, \src0, \src2, \src4, \src6 + calc_epelh v5, \src1, \src3, \src5, \src7 + st1 {v4.8h}, [x0], #16 + subs w3, w3, #1 + st1 {v5.4h}, [x0], x5 +.endm +1: calc_all8 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_hv16_8_neon_i8mm, export=1 + add w10, w3, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + sub x1, x1, x2 + add w3, w3, #3 + bl X(ff_hevc_put_hevc_epel_h16_8_neon_i8mm) + ldp x5, x30, [sp] + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_epel_filterh x5, x4 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8h, v17.8h}, [sp], x10 + ld1 {v18.8h, v19.8h}, [sp], x10 + ld1 {v20.8h, v21.8h}, [sp], x10 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\src6\().8h, \src7\().8h}, [sp], x10 + calc_epelh v4, \src0, \src2, \src4, \src6 + calc_epelh2 v4, v5, \src0, \src2, \src4, \src6 + calc_epelh v5, \src1, \src3, \src5, \src7 + calc_epelh2 v5, v6, \src1, \src3, \src5, \src7 + subs w3, w3, #1 + st1 {v4.8h, v5.8h}, [x0], x10 +.endm +1: calc_all8 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_hv24_8_neon_i8mm, export=1 + add w10, w3, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + sub x1, x1, x2 + add w3, w3, #3 + bl X(ff_hevc_put_hevc_epel_h24_8_neon_i8mm) + ldp x5, x30, [sp] + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_epel_filterh x5, x4 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8h, v17.8h, v18.8h}, [sp], x10 + ld1 {v19.8h, v20.8h, v21.8h}, [sp], x10 + ld1 {v22.8h, v23.8h, v24.8h}, [sp], x10 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11 + ld1 {\src9\().8h-\src11\().8h}, [sp], x10 + calc_epelh v4, \src0, \src3, \src6, \src9 + calc_epelh2 v4, v5, \src0, \src3, \src6, \src9 + calc_epelh v5, \src1, \src4, \src7, \src10 + calc_epelh2 v5, v6, \src1, \src4, \src7, \src10 + calc_epelh v6, \src2, \src5, \src8, \src11 + calc_epelh2 v6, v7, \src2, \src5, \src8, \src11 + subs w3, w3, #1 + st1 {v4.8h-v6.8h}, [x0], x10 +.endm +1: calc_all12 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_hv32_8_neon_i8mm, export=1 + stp x4, x5, [sp, #-64]! + stp x2, x3, [sp, #16] + stp x0, x1, [sp, #32] + str x30, [sp, #48] + mov x6, #16 + bl X(ff_hevc_put_hevc_epel_hv16_8_neon_i8mm) + ldp x4, x5, [sp] + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp, #32] + add sp, sp, #48 + add x0, x0, #32 + add x1, x1, #16 + mov x6, #16 + bl X(ff_hevc_put_hevc_epel_hv16_8_neon_i8mm) + ldr x30, [sp], #16 + ret +endfunc + +function ff_hevc_put_hevc_epel_hv48_8_neon_i8mm, export=1 + stp x4, x5, [sp, #-64]! + stp x2, x3, [sp, #16] + stp x0, x1, [sp, #32] + str x30, [sp, #48] + mov x6, #24 + bl X(ff_hevc_put_hevc_epel_hv24_8_neon_i8mm) + ldp x4, x5, [sp] + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp, #32] + add sp, sp, #48 + add x0, x0, #48 + add x1, x1, #24 + mov x6, #24 + bl X(ff_hevc_put_hevc_epel_hv24_8_neon_i8mm) + ldr x30, [sp], #16 + ret +endfunc + +function ff_hevc_put_hevc_epel_hv64_8_neon_i8mm, export=1 + stp x4, x5, [sp, #-64]! + stp x2, x3, [sp, #16] + stp x0, x1, [sp, #32] + str x30, [sp, #48] + mov x6, #16 + bl X(ff_hevc_put_hevc_epel_hv16_8_neon_i8mm) + ldp x4, x5, [sp] + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp, #32] + add x0, x0, #32 + add x1, x1, #16 + mov x6, #16 + bl X(ff_hevc_put_hevc_epel_hv16_8_neon_i8mm) + ldp x4, x5, [sp] + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp, #32] + add x0, x0, #64 + add x1, x1, #32 + mov x6, #16 + bl X(ff_hevc_put_hevc_epel_hv16_8_neon_i8mm) + ldp x4, x5, [sp] + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp, #32] + add sp, sp, #48 + add x0, x0, #96 + add x1, x1, #48 + mov x6, #16 + bl X(ff_hevc_put_hevc_epel_hv16_8_neon_i8mm) + ldr x30, [sp], #16 + ret +endfunc + function ff_hevc_put_hevc_epel_uni_hv4_8_neon_i8mm, export=1 add w10, w4, #3 lsl x10, x10, #7 diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index 82e1623a67..e9a341ecb9 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -191,6 +191,10 @@ NEON8_FNPROTO(epel_h, (int16_t *dst, const uint8_t *_src, ptrdiff_t _srcstride, int height, intptr_t mx, intptr_t my, int width), _i8mm); +NEON8_FNPROTO(epel_hv, (int16_t *dst, + const uint8_t *src, ptrdiff_t srcstride, + int height, intptr_t mx, intptr_t my, int width), _i8mm); + NEON8_FNPROTO(epel_uni_w_h, (uint8_t *_dst, ptrdiff_t _dststride, const uint8_t *_src, ptrdiff_t _srcstride, int height, int denom, int wx, int ox, @@ -322,6 +326,7 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext From patchwork Sat Oct 14 08:45:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Logan.Lyu" X-Patchwork-Id: 44258 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4b15:b0:15d:8365:d4b8 with SMTP id fp21csp1476740pzb; Sat, 14 Oct 2023 01:46:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEvFwqV0zPEwuXsBaez1SCsN5CZs4C7EY1sijJGb4lvVQEqghahnJlTdvr4no2IDw0FDUbr X-Received: by 2002:a17:907:1c94:b0:9bd:d405:4e8a with SMTP id nb20-20020a1709071c9400b009bdd4054e8amr1885779ejc.17.1697273176441; Sat, 14 Oct 2023 01:46:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697273176; cv=none; d=google.com; s=arc-20160816; b=SJ+WtGLCSI2I7dZ8t4WsYQ/Jlhxkp8b6cVPkc3Uea48UnYZwVbkH1QpQBNsEq1uYsg uyqssaA8B0voBun/gYvQKeSuO9spviqFyvWM+d65o9fOPLuLsfjhJF5cMlcsF3OJDTAN oyW2FQ58CDpN4w4wyRIOb360OaRPuMIfMgUi8Yqr0FfnZXlkbykMUsk7Y3Yy0i9tW62S HflBW2EhC+3tmSnZZvduSjOkuyFp6ygVcfFFYJfZMtF6LarBwUuVuSWWHO1WPUxHv56b B0JJqTGRXLRukrDwHj+dShNDZM8MdK3SI1oriqw1EGLGzmT270wS+T7e1Owhib0e6xq9 8pbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :organization:to:from:user-agent:mime-version:date:message-id :delivered-to; bh=MFNvBFkC3TA/jBsjIeqmDl+AJ7TyXPRzIP2YzdycSGk=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=uZv0dXcfqMhu+7eXrp+eNnlecykvvNSE/dnJisXpIxc6PZ8AoEHE+c2dnKBKPApjkK Lk9oPVp9c+ceiAPd0oBRK20xZJiRhcsT0Xz5UQb4z+cAPeLh6xtR9GMxXd+GnhCMPPzQ H7nx9mb25vVMR5wUEeefDoMpUt2vyOGuQJ8rmtQ/EmAvoEi5Ebgiu8Gv6oGRvYvXn+yy xX0FqXnqpu/X+HCRNwF3FzJAZhBLfXPW75D4assh41WHI+JHT4clyUNruF0GikFZMVMQ vwZwjB9dTz5UH+tDyjlT5dZk2AMq/7NiBpZiB3PzkUD6iwcFsqxJJ9dPCIWi5vdjr2wt +tNQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id g5-20020a17090670c500b00992d753205csi546231ejk.658.2023.10.14.01.46.16; Sat, 14 Oct 2023 01:46:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0C1E468C9F7; Sat, 14 Oct 2023 11:46:01 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-my3-08p62.yunyou.top (smtp-my3-08p62.yunyou.top [60.247.169.62]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 030DC68C9E0 for ; Sat, 14 Oct 2023 11:45:53 +0300 (EEST) Received: from [192.168.15.106] (unknown [125.121.27.199]) by smtp-my-08.yunyou.top (WestCloudMail) with ESMTPA id C23DAAE4A9 for ; Sat, 14 Oct 2023 16:45:50 +0800 (CST) Message-ID: Date: Sat, 14 Oct 2023 16:45:50 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Logan.Lyu" To: ffmpeg-devel@ffmpeg.org Organization: myais Subject: [FFmpeg-devel] [PATCH 3/4] lavc/aarch64: new optimization for 8-bit hevc_qpel_v X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: +hhnB5v0He5h checkasm bench: put_hevc_qpel_v4_8_c: 138.1 put_hevc_qpel_v4_8_neon: 41.1 put_hevc_qpel_v6_8_c: 276.6 put_hevc_qpel_v6_8_neon: 60.9 put_hevc_qpel_v8_8_c: 478.9 put_hevc_qpel_v8_8_neon: 72.9 put_hevc_qpel_v12_8_c: 1072.6 put_hevc_qpel_v12_8_neon: 203.9 put_hevc_qpel_v16_8_c: 1852.1 put_hevc_qpel_v16_8_neon: 264.1 put_hevc_qpel_v24_8_c: 4137.6 put_hevc_qpel_v24_8_neon: 586.9 put_hevc_qpel_v32_8_c: 7579.1 put_hevc_qpel_v32_8_neon: 1036.6 put_hevc_qpel_v48_8_c: 16355.6 put_hevc_qpel_v48_8_neon: 2326.4 put_hevc_qpel_v64_8_c: 33545.1 put_hevc_qpel_v64_8_neon: 4126.4 Co-Authored-By: J. Dekker Signed-off-by: Logan Lyu --- libavcodec/aarch64/hevcdsp_init_aarch64.c | 5 + libavcodec/aarch64/hevcdsp_qpel_neon.S | 347 +++++++++++++++++++--- 2 files changed, 314 insertions(+), 38 deletions(-) v17, v19, v21, v23, v25, v27, v29 + b.hi 1b +.endm + .macro put_hevc type .ifc \type, qpel // void put_hevc_qpel_h(int16_t *dst, @@ -558,6 +596,277 @@ put_hevc qpel put_hevc qpel_uni put_hevc qpel_bi +function ff_hevc_put_hevc_qpel_v4_8_neon, export=1 + load_qpel_filterb x5, x4 + sub x1, x1, x2, lsl #1 + mov x9, #(MAX_PB_SIZE * 2) + sub x1, x1, x2 + ldr s16, [x1] + ldr s17, [x1, x2] + add x1, x1, x2, lsl #1 + ldr s18, [x1] + ldr s19, [x1, x2] + add x1, x1, x2, lsl #1 + ldr s20, [x1] + ldr s21, [x1, x2] + add x1, x1, x2, lsl #1 + ldr s22, [x1] + add x1, x1, x2 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().s}[0], [x1], x2 + movi v24.8h, #0 + calc_qpelb v24, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + st1 {v24.4h}, [x0], x9 + subs w3, w3, #1 + b.eq 2f +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_v6_8_neon, export=1 + load_qpel_filterb x5, x4 + sub x1, x1, x2, lsl #1 + mov x9, #(MAX_PB_SIZE * 2 - 8) + sub x1, x1, x2 + ldr d16, [x1] + ldr d17, [x1, x2] + add x1, x1, x2, lsl #1 + ldr d18, [x1] + ldr d19, [x1, x2] + add x1, x1, x2, lsl #1 + ldr d20, [x1] + ldr d21, [x1, x2] + add x1, x1, x2, lsl #1 + ldr d22, [x1] + add x1, x1, x2 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().8b}, [x1], x2 + movi v24.8h, #0 + calc_qpelb v24, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + st1 {v24.4h}, [x0], #8 + st1 {v24.s}[2], [x0], x9 + subs w3, w3, #1 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_v8_8_neon, export=1 + load_qpel_filterb x5, x4 + sub x1, x1, x2, lsl #1 + mov x9, #(MAX_PB_SIZE * 2) + sub x1, x1, x2 + ldr d16, [x1] + ldr d17, [x1, x2] + add x1, x1, x2, lsl #1 + ldr d18, [x1] + ldr d19, [x1, x2] + add x1, x1, x2, lsl #1 + ldr d20, [x1] + ldr d21, [x1, x2] + add x1, x1, x2, lsl #1 + ldr d22, [x1] + add x1, x1, x2 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().8b}, [x1], x2 + movi v24.8h, #0 + calc_qpelb v24, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + st1 {v24.8h}, [x0], x9 + subs w3, w3, #1 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_v12_8_neon, export=1 + load_qpel_filterb x5, x4 + sub x1, x1, x2, lsl #1 + mov x9, #(MAX_PB_SIZE * 2 - 16) + sub x1, x1, x2 + ldr q16, [x1] + ldr q17, [x1, x2] + add x1, x1, x2, lsl #1 + ldr q18, [x1] + ldr q19, [x1, x2] + add x1, x1, x2, lsl #1 + ldr q20, [x1] + ldr q21, [x1, x2] + add x1, x1, x2, lsl #1 + ldr q22, [x1] + add x1, x1, x2 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().16b}, [x1], x2 + movi v24.8h, #0 + movi v25.8h, #0 + calc_qpelb v24, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb2 v25, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + st1 {v24.8h}, [x0], #16 + subs w3, w3, #1 + st1 {v25.4h}, [x0], x9 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_v16_8_neon, export=1 + load_qpel_filterb x5, x4 + sub x1, x1, x2, lsl #1 + mov x9, #(MAX_PB_SIZE * 2) + sub x1, x1, x2 + ldr q16, [x1] + ldr q17, [x1, x2] + add x1, x1, x2, lsl #1 + ldr q18, [x1] + ldr q19, [x1, x2] + add x1, x1, x2, lsl #1 + ldr q20, [x1] + ldr q21, [x1, x2] + add x1, x1, x2, lsl #1 + ldr q22, [x1] + add x1, x1, x2 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().16b}, [x1], x2 + movi v24.8h, #0 + movi v25.8h, #0 + calc_qpelb v24, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb2 v25, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + subs w3, w3, #1 + st1 {v24.8h, v25.8h}, [x0], x9 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +// todo: reads #32 bytes +function ff_hevc_put_hevc_qpel_v24_8_neon, export=1 + sub sp, sp, #32 + st1 {v8.8b, v9.8b, v10.8b}, [sp] + load_qpel_filterb x5, x4 + sub x1, x1, x2, lsl #1 + sub x1, x1, x2 + mov x9, #(MAX_PB_SIZE * 2) + ld1 {v16.16b, v17.16b}, [x1], x2 + ld1 {v18.16b, v19.16b}, [x1], x2 + ld1 {v20.16b, v21.16b}, [x1], x2 + ld1 {v22.16b, v23.16b}, [x1], x2 + ld1 {v24.16b, v25.16b}, [x1], x2 + ld1 {v26.16b, v27.16b}, [x1], x2 + ld1 {v28.16b, v29.16b}, [x1], x2 +.macro calc tmp0, tmp1, src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11, src12, src13, src14, src15 + ld1 {\tmp0\().16b, \tmp1\().16b}, [x1], x2 + movi v8.8h, #0 + movi v9.8h, #0 + movi v10.8h, #0 + calc_qpelb v8, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb2 v9, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb v10, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15 + subs w3, w3, #1 + st1 {v8.8h, v9.8h, v10.8h}, [x0], x9 +.endm +1: calc_all2 +.purgem calc +2: ld1 {v8.8b, v9.8b, v10.8b}, [sp] + add sp, sp, #32 + ret +endfunc + +function ff_hevc_put_hevc_qpel_v32_8_neon, export=1 + sub sp, sp, #32 + st1 {v8.8b-v11.8b}, [sp] + load_qpel_filterb x5, x4 + sub x1, x1, x2, lsl #1 + mov x9, #(MAX_PB_SIZE * 2) + sub x1, x1, x2 + ld1 {v16.16b, v17.16b}, [x1], x2 + ld1 {v18.16b, v19.16b}, [x1], x2 + ld1 {v20.16b, v21.16b}, [x1], x2 + ld1 {v22.16b, v23.16b}, [x1], x2 + ld1 {v24.16b, v25.16b}, [x1], x2 + ld1 {v26.16b, v27.16b}, [x1], x2 + ld1 {v28.16b, v29.16b}, [x1], x2 +.macro calc tmp0, tmp1, src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11, src12, src13, src14, src15 + ld1 {\tmp0\().16b, \tmp1\().16b}, [x1], x2 + movi v8.8h, #0 + movi v9.8h, #0 + movi v10.8h, #0 + movi v11.8h, #0 + calc_qpelb v8, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb2 v9, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb v10, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15 + calc_qpelb2 v11, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15 + subs w3, w3, #1 + st1 {v8.8h-v11.8h}, [x0], x9 +.endm +1: calc_all2 +.purgem calc +2: ld1 {v8.8b-v11.8b}, [sp], #32 + ret +endfunc + +function ff_hevc_put_hevc_qpel_v48_8_neon, export=1 + stp x2, x3, [sp, #-48]! + stp x0, x1, [sp, #16] + stp x5, x30, [sp, #32] + bl X(ff_hevc_put_hevc_qpel_v24_8_neon) + ldp x2, x3, [sp] + ldp x0, x1, [sp, #16] + ldr x5, [sp, #32] + add sp, sp, #32 + add x0, x0, #48 + add x1, x1, #24 + bl X(ff_hevc_put_hevc_qpel_v24_8_neon) + ldr x30, [sp, #8] + add sp, sp, #16 + ret +endfunc + +function ff_hevc_put_hevc_qpel_v64_8_neon, export=1 + sub sp, sp, #32 + st1 {v8.8b-v11.8b}, [sp] + load_qpel_filterb x5, x4 + sub x1, x1, x2, lsl #1 + sub x1, x1, x2 + mov x9, #(MAX_PB_SIZE * 2) +0: mov x8, x1 // src + ld1 {v16.16b, v17.16b}, [x8], x2 + mov w11, w3 // height + ld1 {v18.16b, v19.16b}, [x8], x2 + mov x10, x0 // dst + ld1 {v20.16b, v21.16b}, [x8], x2 + ld1 {v22.16b, v23.16b}, [x8], x2 + ld1 {v24.16b, v25.16b}, [x8], x2 + ld1 {v26.16b, v27.16b}, [x8], x2 + ld1 {v28.16b, v29.16b}, [x8], x2 +.macro calc tmp0, tmp1, src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11, src12, src13, src14, src15 + ld1 {\tmp0\().16b, \tmp1\().16b}, [x8], x2 + movi v8.8h, #0 + movi v9.8h, #0 + movi v10.8h, #0 + movi v11.8h, #0 + calc_qpelb v8, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb2 v9, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb v10, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15 + calc_qpelb2 v11, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15 + subs x11, x11, #1 + st1 {v8.8h-v11.8h}, [x10], x9 +.endm +1: calc_all2 +.purgem calc +2: add x0, x0, #64 + add x1, x1, #32 + subs w6, w6, #32 + b.hi 0b + ld1 {v8.8b-v11.8b}, [sp], #32 + ret +endfunc + + function ff_hevc_put_hevc_pel_uni_pixels4_8_neon, export=1 1: ldr s0, [x2] @@ -663,25 +972,6 @@ function ff_hevc_put_hevc_pel_uni_pixels64_8_neon, export=1 ret endfunc -.macro calc_all - calc v23, v16, v17, v18, v19, v20, v21, v22, v23 - b.eq 2f - calc v16, v17, v18, v19, v20, v21, v22, v23, v16 - b.eq 2f - calc v17, v18, v19, v20, v21, v22, v23, v16, v17 - b.eq 2f - calc v18, v19, v20, v21, v22, v23, v16, v17, v18 - b.eq 2f - calc v19, v20, v21, v22, v23, v16, v17, v18, v19 - b.eq 2f - calc v20, v21, v22, v23, v16, v17, v18, v19, v20 - b.eq 2f - calc v21, v22, v23, v16, v17, v18, v19, v20, v21 - b.eq 2f - calc v22, v23, v16, v17, v18, v19, v20, v21, v22 - b.hi 1b -.endm - function ff_hevc_put_hevc_qpel_uni_v4_8_neon, export=1 load_qpel_filterb x6, x5 sub x2, x2, x3, lsl #1 @@ -1559,25 +1849,6 @@ endfunc #if HAVE_I8MM -.macro calc_all2 - calc v30, v31, v16, v18, v20, v22, v24, v26, v28, v30, v17, v19, v21, v23, v25, v27, v29, v31 - b.eq 2f - calc v16, v17, v18, v20, v22, v24, v26, v28, v30, v16, v19, v21, v23, v25, v27, v29, v31, v17 - b.eq 2f - calc v18, v19, v20, v22, v24, v26, v28, v30, v16, v18, v21, v23, v25, v27, v29, v31, v17, v19 - b.eq 2f - calc v20, v21, v22, v24, v26, v28, v30, v16, v18, v20, v23, v25, v27, v29, v31, v17, v19, v21 - b.eq 2f - calc v22, v23, v24, v26, v28, v30, v16, v18, v20, v22, v25, v27, v29, v31, v17, v19, v21, v23 - b.eq 2f - calc v24, v25, v26, v28, v30, v16, v18, v20, v22, v24, v27, v29, v31, v17, v19, v21, v23, v25 - b.eq 2f - calc v26, v27, v28, v30, v16, v18, v20, v22, v24, v26, v29, v31, v17, v19, v21, v23, v25, v27 - b.eq 2f - calc v28, v29, v30, v16, v18, v20, v22, v24, v26, v28, v31, v17, v19, v21, v23, v25, v27, v29 - b.hi 1b -.endm - function ff_hevc_put_hevc_qpel_uni_hv4_8_neon_i8mm, export=1 add w10, w4, #7 lsl x10, x10, #7 diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index e9a341ecb9..f6b4c31d17 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -204,6 +204,10 @@ NEON8_FNPROTO(qpel_h, (int16_t *dst, const uint8_t *_src, ptrdiff_t _srcstride, int height, intptr_t mx, intptr_t my, int width), _i8mm); +NEON8_FNPROTO(qpel_v, (int16_t *dst, + const uint8_t *src, ptrdiff_t srcstride, + int height, intptr_t mx, intptr_t my, int width),); + NEON8_FNPROTO(qpel_uni_v, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *src, ptrdiff_t srcstride, int height, intptr_t mx, intptr_t my, int width),); @@ -315,6 +319,7 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) NEON8_FNASSIGN(c->put_hevc_epel, 0, 0, pel_pixels,); NEON8_FNASSIGN(c->put_hevc_epel, 1, 0, epel_v,); NEON8_FNASSIGN(c->put_hevc_qpel, 0, 0, pel_pixels,); + NEON8_FNASSIGN(c->put_hevc_qpel, 1, 0, qpel_v,); NEON8_FNASSIGN(c->put_hevc_epel_uni, 0, 0, pel_uni_pixels,); NEON8_FNASSIGN(c->put_hevc_epel_uni, 1, 0, epel_uni_v,); NEON8_FNASSIGN(c->put_hevc_qpel_uni, 0, 0, pel_uni_pixels,); diff --git a/libavcodec/aarch64/hevcdsp_qpel_neon.S b/libavcodec/aarch64/hevcdsp_qpel_neon.S index 4132d7a8a9..eff70d70a4 100644 --- a/libavcodec/aarch64/hevcdsp_qpel_neon.S +++ b/libavcodec/aarch64/hevcdsp_qpel_neon.S @@ -112,6 +112,44 @@ endconst .endif .endm +.macro calc_all + calc v23, v16, v17, v18, v19, v20, v21, v22, v23 + b.eq 2f + calc v16, v17, v18, v19, v20, v21, v22, v23, v16 + b.eq 2f + calc v17, v18, v19, v20, v21, v22, v23, v16, v17 + b.eq 2f + calc v18, v19, v20, v21, v22, v23, v16, v17, v18 + b.eq 2f + calc v19, v20, v21, v22, v23, v16, v17, v18, v19 + b.eq 2f + calc v20, v21, v22, v23, v16, v17, v18, v19, v20 + b.eq 2f + calc v21, v22, v23, v16, v17, v18, v19, v20, v21 + b.eq 2f + calc v22, v23, v16, v17, v18, v19, v20, v21, v22 + b.hi 1b +.endm + +.macro calc_all2 + calc v30, v31, v16, v18, v20, v22, v24, v26, v28, v30, v17, v19, v21, v23, v25, v27, v29, v31 + b.eq 2f + calc v16, v17, v18, v20, v22, v24, v26, v28, v30, v16, v19, v21, v23, v25, v27, v29, v31, v17 + b.eq 2f + calc v18, v19, v20, v22, v24, v26, v28, v30, v16, v18, v21, v23, v25, v27, v29, v31, v17, v19 + b.eq 2f + calc v20, v21, v22, v24, v26, v28, v30, v16, v18, v20, v23, v25, v27, v29, v31, v17, v19, v21 + b.eq 2f + calc v22, v23, v24, v26, v28, v30, v16, v18, v20, v22, v25, v27, v29, v31, v17, v19, v21, v23 + b.eq 2f + calc v24, v25, v26, v28, v30, v16, v18, v20, v22, v24, v27, v29, v31, v17, v19, v21, v23, v25 + b.eq 2f + calc v26, v27, v28, v30, v16, v18, v20, v22, v24, v26, v29, v31, v17, v19, v21, v23, v25, v27 + b.eq 2f + calc v28, v29, v30, v16, v18, v20, v22, v24, v26, v28, v31, From patchwork Sat Oct 14 08:45:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Logan.Lyu" X-Patchwork-Id: 44259 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4b15:b0:15d:8365:d4b8 with SMTP id fp21csp1476802pzb; Sat, 14 Oct 2023 01:46:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEu/DJllXObPSx0BHpyEcWjIryHcAO2fDX4cijzjjk3oYTM8xyYlwQ211tSi5FVQ4QHU2J4 X-Received: by 2002:a17:907:801:b0:9ba:8ed:ea58 with SMTP id wv1-20020a170907080100b009ba08edea58mr2239186ejb.30.1697273190477; Sat, 14 Oct 2023 01:46:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697273190; cv=none; d=google.com; s=arc-20160816; b=zyEH+2j7YgNCxvkoj+7UKRHvQI2tGGdyqYGK2vwscRMG7fK//mfzi3SVheWxve3F0Y TU+u6yZ9cx8o9eXbcfB3/UdRG/yHZi8NiX8wk5CgVrzQVoYRIioAAlgWQtCzPnVh2/HO er8ltjQ10JkQUihJeuQ4U/SsHhez/zqtuIA0vhxtU5gw6jfzkuixX2QSjuNyNr1N9MAQ r8IS21/EesC2qnoajmQtrdd8aNhLFx9OeZurw7vGEiIAYl5zHQG6zSWxRbDy1TvcXpfJ iR+Dfp806SF/qM/K8Cu52MHV3p7hDQvfwlAqcqyhqw7TAR3qJv/JShUgZesyF5cBrpLG BZ0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :organization:to:from:user-agent:mime-version:date:message-id :delivered-to; bh=D4uUSYbFwGAIwxNGJEIX4zjOQPzplsZ+Z/CY5UiOG0g=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=Ap8cn+NBPVKeqyUpcId+yRDPX0o/55AliAD7XPcrmFwjeO+7sRZ2xEqjJLUiAAeG+K 4QPBctrB8rlQ2+wR2EoS/LL4RrAB8xAQWuUY/f0CPpiGsOGKviwyuvM29RnCXUaanAAn HWDxqeTNo/cD67rp1+gZ7ZwOq60orcQ3How+T9LLrdE1X0iwpt9ZCMBxjUzyAKjLx8Lt vvWR1PM2561stT3biNkn5dmNLJvu8k/lMBF47KwbJ/Ah52pPsye9QCpiqI0higVxqvAS 3l/Uj25VzYu34fxYelgxxtB+pZu+REpAeq2eM1AzmojzHlFl5oBsMPQ0pskCz+xHFLWG YBHQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cf15-20020a170906b2cf00b009a9f767b62fsi602382ejb.114.2023.10.14.01.46.30; Sat, 14 Oct 2023 01:46:30 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 142DF68C9FD; Sat, 14 Oct 2023 11:46:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-my3-08p62.yunyou.top (smtp-my3-08p62.yunyou.top [60.247.169.62]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2085868C9F2 for ; Sat, 14 Oct 2023 11:45:57 +0300 (EEST) Received: from [192.168.15.106] (unknown [125.121.27.199]) by smtp-my-08.yunyou.top (WestCloudMail) with ESMTPA id DD482AE4A0 for ; Sat, 14 Oct 2023 16:45:54 +0800 (CST) Message-ID: Date: Sat, 14 Oct 2023 16:45:54 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Logan.Lyu" To: ffmpeg-devel@ffmpeg.org Organization: myais Subject: [FFmpeg-devel] [PATCH 4/4] lavc/aarch64: new optimization for 8-bit hevc_qpel_hv X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Gw5HG7nsNwjC checkasm bench: put_hevc_qpel_hv4_8_c: 422.1 put_hevc_qpel_hv4_8_i8mm: 101.6 put_hevc_qpel_hv6_8_c: 756.4 put_hevc_qpel_hv6_8_i8mm: 225.9 put_hevc_qpel_hv8_8_c: 1189.9 put_hevc_qpel_hv8_8_i8mm: 296.6 put_hevc_qpel_hv12_8_c: 2407.4 put_hevc_qpel_hv12_8_i8mm: 552.4 put_hevc_qpel_hv16_8_c: 4021.4 put_hevc_qpel_hv16_8_i8mm: 886.6 put_hevc_qpel_hv24_8_c: 8992.1 put_hevc_qpel_hv24_8_i8mm: 1968.9 put_hevc_qpel_hv32_8_c: 15197.9 put_hevc_qpel_hv32_8_i8mm: 3209.4 put_hevc_qpel_hv48_8_c: 32811.1 put_hevc_qpel_hv48_8_i8mm: 7442.1 put_hevc_qpel_hv64_8_c: 58106.1 put_hevc_qpel_hv64_8_i8mm: 12423.9 Co-Authored-By: J. Dekker Signed-off-by: Logan Lyu --- libavcodec/aarch64/hevcdsp_init_aarch64.c | 5 + libavcodec/aarch64/hevcdsp_qpel_neon.S | 397 ++++++++++++++++++++++ 2 files changed, 402 insertions(+) + add sp, sp, x10 // tmp_array without first line + ret +endfunc + +function ff_hevc_put_hevc_qpel_hv48_8_neon_i8mm, export=1 + stp x4, x5, [sp, #-64]! + stp x2, x3, [sp, #16] + stp x0, x1, [sp, #32] + str x30, [sp, #48] + bl X(ff_hevc_put_hevc_qpel_hv24_8_neon_i8mm) + ldp x4, x5, [sp] + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp, #32] + add sp, sp, #48 + add x1, x1, #24 + add x0, x0, #48 + bl X(ff_hevc_put_hevc_qpel_hv24_8_neon_i8mm) + ldr x30, [sp], #16 + ret +endfunc + +function ff_hevc_put_hevc_qpel_hv64_8_neon_i8mm, export=1 + stp x4, x5, [sp, #-64]! + stp x2, x3, [sp, #16] + stp x0, x1, [sp, #32] + str x30, [sp, #48] + mov x6, #32 + bl X(ff_hevc_put_hevc_qpel_hv32_8_neon_i8mm) + ldp x4, x5, [sp] + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp, #32] + add sp, sp, #48 + add x1, x1, #32 + add x0, x0, #64 + mov x6, #32 + bl X(ff_hevc_put_hevc_qpel_hv32_8_neon_i8mm) + ldr x30, [sp], #16 + ret +endfunc + .macro QPEL_UNI_W_HV_HEADER width ldp x14, x15, [sp] // mx, my ldr w13, [sp, #16] // width diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index f6b4c31d17..7d889efe68 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -208,6 +208,10 @@ NEON8_FNPROTO(qpel_v, (int16_t *dst, const uint8_t *src, ptrdiff_t srcstride, int height, intptr_t mx, intptr_t my, int width),); +NEON8_FNPROTO(qpel_hv, (int16_t *dst, + const uint8_t *src, ptrdiff_t srcstride, + int height, intptr_t mx, intptr_t my, int width), _i8mm); + NEON8_FNPROTO(qpel_uni_v, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *src, ptrdiff_t srcstride, int height, intptr_t mx, intptr_t my, int width),); @@ -335,6 +339,7 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) NEON8_FNASSIGN(c->put_hevc_epel_uni, 1, 1, epel_uni_hv, _i8mm); NEON8_FNASSIGN(c->put_hevc_epel_uni_w, 0, 1, epel_uni_w_h ,_i8mm); NEON8_FNASSIGN(c->put_hevc_qpel, 0, 1, qpel_h, _i8mm); + NEON8_FNASSIGN(c->put_hevc_qpel, 1, 1, qpel_hv, _i8mm); NEON8_FNASSIGN(c->put_hevc_qpel_uni, 1, 1, qpel_uni_hv, _i8mm); NEON8_FNASSIGN(c->put_hevc_qpel_uni_w, 0, 1, qpel_uni_w_h, _i8mm); NEON8_FNASSIGN(c->put_hevc_epel_uni_w, 1, 1, epel_uni_w_hv, _i8mm); diff --git a/libavcodec/aarch64/hevcdsp_qpel_neon.S b/libavcodec/aarch64/hevcdsp_qpel_neon.S index eff70d70a4..e4475ba920 100644 --- a/libavcodec/aarch64/hevcdsp_qpel_neon.S +++ b/libavcodec/aarch64/hevcdsp_qpel_neon.S @@ -3070,6 +3070,403 @@ function ff_hevc_put_hevc_qpel_h64_8_neon_i8mm, export=1 ret endfunc + +function ff_hevc_put_hevc_qpel_hv4_8_neon_i8mm, export=1 + add w10, w3, #7 + mov x7, #128 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + sub x1, x1, x2, lsl #1 + add x3, x3, #7 + sub x1, x1, x2 + bl X(ff_hevc_put_hevc_qpel_h4_8_neon_i8mm) + ldp x5, x30, [sp] + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_qpel_filterh x5, x4 + ldr d16, [sp] + ldr d17, [sp, x7] + add sp, sp, x7, lsl #1 + ldr d18, [sp] + ldr d19, [sp, x7] + add sp, sp, x7, lsl #1 + ldr d20, [sp] + ldr d21, [sp, x7] + add sp, sp, x7, lsl #1 + ldr d22, [sp] + add sp, sp, x7 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().4h}, [sp], x7 + calc_qpelh v1, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn + subs w3, w3, #1 + st1 {v1.4h}, [x0], x7 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_hv6_8_neon_i8mm, export=1 + add w10, w3, #7 + mov x7, #128 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + sub x1, x1, x2, lsl #1 + add x3, x3, #7 + sub x1, x1, x2 + bl X(ff_hevc_put_hevc_qpel_h6_8_neon_i8mm) + ldp x5, x30, [sp] + mov x8, #120 + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_qpel_filterh x5, x4 + ldr q16, [sp] + ldr q17, [sp, x7] + add sp, sp, x7, lsl #1 + ldr q18, [sp] + ldr q19, [sp, x7] + add sp, sp, x7, lsl #1 + ldr q20, [sp] + ldr q21, [sp, x7] + add sp, sp, x7, lsl #1 + ldr q22, [sp] + add sp, sp, x7 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().8h}, [sp], x7 + calc_qpelh v1, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn + calc_qpelh2 v1, v2, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn2 + st1 {v1.4h}, [x0], #8 + subs w3, w3, #1 + st1 {v1.s}[2], [x0], x8 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_hv8_8_neon_i8mm, export=1 + add w10, w3, #7 + lsl x10, x10, #7 + sub x1, x1, x2, lsl #1 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + add x3, x3, #7 + sub x1, x1, x2 + bl X(ff_hevc_put_hevc_qpel_h8_8_neon_i8mm) + ldp x5, x30, [sp] + mov x7, #128 + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_qpel_filterh x5, x4 + ldr q16, [sp] + ldr q17, [sp, x7] + add sp, sp, x7, lsl #1 + ldr q18, [sp] + ldr q19, [sp, x7] + add sp, sp, x7, lsl #1 + ldr q20, [sp] + ldr q21, [sp, x7] + add sp, sp, x7, lsl #1 + ldr q22, [sp] + add sp, sp, x7 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().8h}, [sp], x7 + calc_qpelh v1, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn + calc_qpelh2 v1, v2, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn2 + subs w3, w3, #1 + st1 {v1.8h}, [x0], x7 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_hv12_8_neon_i8mm, export=1 + add w10, w3, #7 + lsl x10, x10, #7 + sub x1, x1, x2, lsl #1 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + add x3, x3, #7 + sub x1, x1, x2 + bl X(ff_hevc_put_hevc_qpel_h12_8_neon_i8mm) + ldp x5, x30, [sp] + mov x7, #128 + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_qpel_filterh x5, x4 + mov x8, #112 + ld1 {v16.8h, v17.8h}, [sp], x7 + ld1 {v18.8h, v19.8h}, [sp], x7 + ld1 {v20.8h, v21.8h}, [sp], x7 + ld1 {v22.8h, v23.8h}, [sp], x7 + ld1 {v24.8h, v25.8h}, [sp], x7 + ld1 {v26.8h, v27.8h}, [sp], x7 + ld1 {v28.8h, v29.8h}, [sp], x7 +.macro calc tmp0, tmp1, src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11, src12, src13, src14, src15 + ld1 {\tmp0\().8h, \tmp1\().8h}, [sp], x7 + calc_qpelh v1, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn + calc_qpelh2 v1, v2, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn2 + calc_qpelh v2, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15, sqshrn + st1 {v1.8h}, [x0], #16 + subs w3, w3, #1 + st1 {v2.4h}, [x0], x8 +.endm +1: calc_all2 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_hv16_8_neon_i8mm, export=1 + add w10, w3, #7 + lsl x10, x10, #7 + sub x1, x1, x2, lsl #1 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x3, x3, #7 + add x0, sp, #32 + sub x1, x1, x2 + bl X(ff_hevc_put_hevc_qpel_h16_8_neon_i8mm) + ldp x5, x30, [sp] + mov x7, #128 + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_qpel_filterh x5, x4 + ld1 {v16.8h, v17.8h}, [sp], x7 + ld1 {v18.8h, v19.8h}, [sp], x7 + ld1 {v20.8h, v21.8h}, [sp], x7 + ld1 {v22.8h, v23.8h}, [sp], x7 + ld1 {v24.8h, v25.8h}, [sp], x7 + ld1 {v26.8h, v27.8h}, [sp], x7 + ld1 {v28.8h, v29.8h}, [sp], x7 +.macro calc tmp0, tmp1, src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11, src12, src13, src14, src15 + ld1 {\tmp0\().8h, \tmp1\().8h}, [sp], x7 + calc_qpelh v1, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn + calc_qpelh2 v1, v2, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn2 + calc_qpelh v2, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15, sqshrn + calc_qpelh2 v2, v3, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15, sqshrn2 + subs w3, w3, #1 + st1 {v1.8h, v2.8h}, [x0], x7 +.endm +1: calc_all2 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_hv24_8_neon_i8mm, export=1 + sub sp, sp, #32 + st1 {v8.8b-v11.8b}, [sp] + sub x1, x1, x2, lsl #1 + sub sp, sp, #32 + add w10, w3, #7 + st1 {v12.8b-v15.8b}, [sp] + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x0, sp, #32 + add x3, x3, #7 + sub x1, x1, x2 + bl X(ff_hevc_put_hevc_qpel_h24_8_neon_i8mm) + ldp x5, x30, [sp] + mov x7, #128 + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_qpel_filterh x5, x4 + ld1 {v8.8h-v10.8h}, [sp], x7 + ld1 {v11.8h-v13.8h}, [sp], x7 + ld1 {v14.8h-v16.8h}, [sp], x7 + ld1 {v17.8h-v19.8h}, [sp], x7 + ld1 {v20.8h-v22.8h}, [sp], x7 + ld1 {v23.8h-v25.8h}, [sp], x7 + ld1 {v26.8h-v28.8h}, [sp], x7 +1: ld1 {v29.8h-v31.8h}, [sp], x7 + calc_qpelh v1, v8, v11, v14, v17, v20, v23, v26, v29, sqshrn + calc_qpelh2 v1, v2, v8, v11, v14, v17, v20, v23, v26, v29, sqshrn2 + calc_qpelh v2, v9, v12, v15, v18, v21, v24, v27, v30, sqshrn + calc_qpelh2 v2, v3, v9, v12, v15, v18, v21, v24, v27, v30, sqshrn2 + calc_qpelh v3, v10, v13, v16, v19, v22, v25, v28, v31, sqshrn + calc_qpelh2 v3, v4, v10, v13, v16, v19, v22, v25, v28, v31, sqshrn2 + subs w3, w3, #1 + st1 {v1.8h-v3.8h}, [x0], x7 + b.eq 2f + + ld1 {v8.8h-v10.8h}, [sp], x7 + calc_qpelh v1, v11, v14, v17, v20, v23, v26, v29, v8, sqshrn + calc_qpelh2 v1, v2, v11, v14, v17, v20, v23, v26, v29, v8, sqshrn2 + calc_qpelh v2, v12, v15, v18, v21, v24, v27, v30, v9, sqshrn + calc_qpelh2 v2, v3, v12, v15, v18, v21, v24, v27, v30, v9, sqshrn2 + calc_qpelh v3, v13, v16, v19, v22, v25, v28, v31, v10, sqshrn + calc_qpelh2 v3, v4, v13, v16, v19, v22, v25, v28, v31, v10, sqshrn2 + subs w3, w3, #1 + st1 {v1.8h-v3.8h}, [x0], x7 + b.eq 2f + + ld1 {v11.8h-v13.8h}, [sp], x7 + calc_qpelh v1, v14, v17, v20, v23, v26, v29, v8, v11, sqshrn + calc_qpelh2 v1, v2, v14, v17, v20, v23, v26, v29, v8, v11, sqshrn2 + calc_qpelh v2, v15, v18, v21, v24, v27, v30, v9, v12, sqshrn + calc_qpelh2 v2, v3, v15, v18, v21, v24, v27, v30, v9, v12, sqshrn2 + calc_qpelh v3, v16, v19, v22, v25, v28, v31, v10, v13, sqshrn + calc_qpelh2 v3, v4, v16, v19, v22, v25, v28, v31, v10, v13, sqshrn2 + subs w3, w3, #1 + st1 {v1.8h-v3.8h}, [x0], x7 + b.eq 2f + + ld1 {v14.8h-v16.8h}, [sp], x7 + calc_qpelh v1, v17, v20, v23, v26, v29, v8, v11, v14, sqshrn + calc_qpelh2 v1, v2, v17, v20, v23, v26, v29, v8, v11, v14, sqshrn2 + calc_qpelh v2, v18, v21, v24, v27, v30, v9, v12, v15, sqshrn + calc_qpelh2 v2, v3, v18, v21, v24, v27, v30, v9, v12, v15, sqshrn2 + calc_qpelh v3, v19, v22, v25, v28, v31, v10, v13, v16, sqshrn + calc_qpelh2 v3, v4, v19, v22, v25, v28, v31, v10, v13, v16, sqshrn2 + subs w3, w3, #1 + st1 {v1.8h-v3.8h}, [x0], x7 + b.eq 2f + + ld1 {v17.8h-v19.8h}, [sp], x7 + calc_qpelh v1, v20, v23, v26, v29, v8, v11, v14, v17, sqshrn + calc_qpelh2 v1, v2, v20, v23, v26, v29, v8, v11, v14, v17, sqshrn2 + calc_qpelh v2, v21, v24, v27, v30, v9, v12, v15, v18, sqshrn + calc_qpelh2 v2, v3, v21, v24, v27, v30, v9, v12, v15, v18, sqshrn2 + calc_qpelh v3, v22, v25, v28, v31, v10, v13, v16, v19, sqshrn + calc_qpelh2 v3, v4, v22, v25, v28, v31, v10, v13, v16, v19, sqshrn2 + subs w3, w3, #1 + st1 {v1.8h-v3.8h}, [x0], x7 + b.eq 2f + + ld1 {v20.8h-v22.8h}, [sp], x7 + calc_qpelh v1, v23, v26, v29, v8, v11, v14, v17, v20, sqshrn + calc_qpelh2 v1, v2, v23, v26, v29, v8, v11, v14, v17, v20, sqshrn2 + calc_qpelh v2, v24, v27, v30, v9, v12, v15, v18, v21, sqshrn + calc_qpelh2 v2, v3, v24, v27, v30, v9, v12, v15, v18, v21, sqshrn2 + calc_qpelh v3, v25, v28, v31, v10, v13, v16, v19, v22, sqshrn + calc_qpelh2 v3, v4, v25, v28, v31, v10, v13, v16, v19, v22, sqshrn2 + subs w3, w3, #1 + st1 {v1.8h-v3.8h}, [x0], x7 + b.eq 2f + + ld1 {v23.8h-v25.8h}, [sp], x7 + calc_qpelh v1, v26, v29, v8, v11, v14, v17, v20, v23, sqshrn + calc_qpelh2 v1, v2, v26, v29, v8, v11, v14, v17, v20, v23, sqshrn2 + calc_qpelh v2, v27, v30, v9, v12, v15, v18, v21, v24, sqshrn + calc_qpelh2 v2, v3, v27, v30, v9, v12, v15, v18, v21, v24, sqshrn2 + calc_qpelh v3, v28, v31, v10, v13, v16, v19, v22, v25, sqshrn + calc_qpelh2 v3, v4, v28, v31, v10, v13, v16, v19, v22, v25, sqshrn2 + subs w3, w3, #1 + st1 {v1.8h-v3.8h}, [x0], x7 + b.eq 2f + + ld1 {v26.8h-v28.8h}, [sp], x7 + calc_qpelh v1, v29, v8, v11, v14, v17, v20, v23, v26, sqshrn + calc_qpelh2 v1, v2, v29, v8, v11, v14, v17, v20, v23, v26, sqshrn2 + calc_qpelh v2, v30, v9, v12, v15, v18, v21, v24, v27, sqshrn + calc_qpelh2 v2, v3, v30, v9, v12, v15, v18, v21, v24, v27, sqshrn2 + calc_qpelh v3, v31, v10, v13, v16, v19, v22, v25, v28, sqshrn + calc_qpelh2 v3, v4, v31, v10, v13, v16, v19, v22, v25, v28, sqshrn2 + subs w3, w3, #1 + st1 {v1.8h-v3.8h}, [x0], x7 + b.hi 1b +2: ld1 {v12.8b-v15.8b}, [sp], #32 + ld1 {v8.8b-v11.8b}, [sp], #32 + ret +endfunc + +function ff_hevc_put_hevc_qpel_hv32_8_neon_i8mm, export=1 + add w10, w3, #7 + sub x1, x1, x2, lsl #1 + lsl x10, x10, #7 + sub x1, x1, x2 + sub sp, sp, x10 // tmp_array + stp x5, x30, [sp, #-32]! + stp x0, x3, [sp, #16] + add x3, x3, #7 + add x0, sp, #32 + bl X(ff_hevc_put_hevc_qpel_h32_8_neon_i8mm) + ldp x5, x30, [sp] + mov x7, #128 + ldp x0, x3, [sp, #16] + add sp, sp, #32 + load_qpel_filterh x5, x4 +0: mov x8, sp // src + ld1 {v16.8h, v17.8h}, [x8], x7 + mov w9, w3 // height + ld1 {v18.8h, v19.8h}, [x8], x7 + mov x5, x0 // dst + ld1 {v20.8h, v21.8h}, [x8], x7 + ld1 {v22.8h, v23.8h}, [x8], x7 + ld1 {v24.8h, v25.8h}, [x8], x7 + ld1 {v26.8h, v27.8h}, [x8], x7 + ld1 {v28.8h, v29.8h}, [x8], x7 +.macro calc tmp0, tmp1, src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11, src12, src13, src14, src15 + ld1 {\tmp0\().8h, \tmp1\().8h}, [x8], x7 + calc_qpelh v1, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn + calc_qpelh2 v1, v2, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sqshrn2 + calc_qpelh v2, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15, sqshrn + calc_qpelh2 v2, v3, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15, sqshrn2 + subs x9, x9, #1 + st1 {v1.8h, v2.8h}, [x5], x7 +.endm +1: calc_all2 +.purgem calc +2: add x0, x0, #32 + add sp, sp, #32 + subs w6, w6, #16 + b.hi 0b + add w10, w3, #6 + add sp, sp, #64 // discard rest of first line + lsl x10, x10, #7