From patchwork Sat Nov 18 02:06:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Logan.Lyu" X-Patchwork-Id: 44702 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6a89:b0:181:818d:5e7f with SMTP id bi9csp293110pzb; Fri, 17 Nov 2023 18:07:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IHCsZ9lNgT4svlrtEXgXGyHBKBKG/2wgn+gGgX8wd52IYgkfjx00nDMH82FiJt03PjDxese X-Received: by 2002:aa7:d755:0:b0:547:da7:9c10 with SMTP id a21-20020aa7d755000000b005470da79c10mr569823eds.2.1700273219605; Fri, 17 Nov 2023 18:06:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700273219; cv=none; d=google.com; s=arc-20160816; b=sDMXyQIv7wmIPZ+HkiZ6TKtJy82ZvdaFxqzH5c1pkiQZ94EMhJNSMyrZ3ByepkgztV AJiTtxK85CQBhJCuVMBR7Gey0LuEgbm6SR03Ths3wdDZ2gaY1o0JQySx0u6dIQnvq8dS Ybep8xTe2/BI6jYai274J0SfABRpm6lSQnHNbJu1Yt2MRPl1WiA3D1ugWolFgjwTHrkA olXKU9nDn9WTXWKiBBXFnmCDBTNXu0fEasqE7Yl9gAc66hwQIT3VoMyFxkWlS0oSWwib 8r0CdEG/36wGykLOwewTbLQ7ayrahXrDzgdQMeinVL+PGx12d/0fmcju8mJZ3xoQOqsm j9jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :organization:to:from:user-agent:mime-version:date:message-id :delivered-to; bh=iOjdIKyIojTt/X2Wa/1VwtQdSHHBCz9m1/toAWJ0Je8=; fh=4VBelKDE4DH3L7jF6H/1Jmu78FdN+YP76yfdJCQTJ30=; b=K0/94tAn6Z/Sr3hiPS6aOyHgU2XqosZ2Prtjo0ndwfXuNxDGF5RfGU/ykVt3Ayuoxf Ine0kcd8q/UhmXHpXtYedKfcFoS3o1rFC8X0pm69bcOtp/Z2p9bqJ5WDRUTBm+tGAUPz s21dB9DQPkq9QD9obhvEqwvj7a+7KRp1FhYC25mNQ6xDCZZxbbYJQjJvFPsVlcnsVLNs 3BZ8DGDZb/KZqQENoHDgpPW1V+cdH+KqljtzXalTssKeM4mjOQGw3LgccQ42NAxIMcSL zjPQUgDkVQja/XuYJl4n4qELcZJkg+bMSePV2KvkyuYCaVRb8D/+A5js6MYRDAzE5tbu WYtg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y11-20020a50eb0b000000b0053073f22103si1790960edp.8.2023.11.17.18.06.58; Fri, 17 Nov 2023 18:06:59 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BFA7568CD19; Sat, 18 Nov 2023 04:06:53 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-my3-01p11.yunyou.top (unknown [60.247.169.11]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2294568C96C for ; Sat, 18 Nov 2023 04:06:44 +0200 (EET) Received: from [192.168.15.105] (unknown [122.233.146.151]) by smtp-my-01.yunyou.top (WestCloudMail) with ESMTPA id 2FCF1141F27; Sat, 18 Nov 2023 10:06:38 +0800 (CST) Message-ID: <01e3c77f-56a3-4191-9637-df9999df694c@myais.com.cn> Date: Sat, 18 Nov 2023 10:06:37 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Logan.Lyu" To: ffmpeg-devel@ffmpeg.org Organization: myais Subject: [FFmpeg-devel] [PATCH 1/6] lavc/aarch64: new optimization for 8-bit hevc_pel_bi_pixels X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: jdek@itanimul.li Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: rBp3NJ3hQMwP put_hevc_pel_bi_pixels4_8_c: 54.7 put_hevc_pel_bi_pixels4_8_neon: 43.0 put_hevc_pel_bi_pixels6_8_c: 94.7 put_hevc_pel_bi_pixels6_8_neon: 37.0 put_hevc_pel_bi_pixels8_8_c: 171.0 put_hevc_pel_bi_pixels8_8_neon: 24.0 put_hevc_pel_bi_pixels12_8_c: 354.0 put_hevc_pel_bi_pixels12_8_neon: 68.7 put_hevc_pel_bi_pixels16_8_c: 588.2 put_hevc_pel_bi_pixels16_8_neon: 77.5 put_hevc_pel_bi_pixels24_8_c: 1670.7 put_hevc_pel_bi_pixels24_8_neon: 173.0 put_hevc_pel_bi_pixels32_8_c: 2267.7 put_hevc_pel_bi_pixels32_8_neon: 281.2 put_hevc_pel_bi_pixels48_8_c: 5787.5 put_hevc_pel_bi_pixels48_8_neon: 673.5 put_hevc_pel_bi_pixels64_8_c: 9897.0 put_hevc_pel_bi_pixels64_8_neon: 1159.5 Co-Authored-By: J. Dekker Signed-off-by: Logan Lyu --- libavcodec/aarch64/hevcdsp_epel_neon.S | 179 ++++++++++++++++++++++ libavcodec/aarch64/hevcdsp_init_aarch64.c | 10 +- 2 files changed, 187 insertions(+), 2 deletions(-) NEON8_FNASSIGN(c->put_hevc_epel, 1, 0, epel_v,); NEON8_FNASSIGN(c->put_hevc_qpel, 0, 0, pel_pixels,); NEON8_FNASSIGN(c->put_hevc_qpel, 1, 0, qpel_v,); + NEON8_FNASSIGN(c->put_hevc_epel_bi, 0, 0, pel_bi_pixels,); + NEON8_FNASSIGN(c->put_hevc_qpel_bi, 0, 0, pel_bi_pixels,); NEON8_FNASSIGN(c->put_hevc_epel_uni, 0, 0, pel_uni_pixels,); NEON8_FNASSIGN(c->put_hevc_epel_uni, 1, 0, epel_uni_v,); NEON8_FNASSIGN(c->put_hevc_qpel_uni, 0, 0, pel_uni_pixels,); diff --git a/libavcodec/aarch64/hevcdsp_epel_neon.S b/libavcodec/aarch64/hevcdsp_epel_neon.S index 708b903b00..74165273d7 100644 --- a/libavcodec/aarch64/hevcdsp_epel_neon.S +++ b/libavcodec/aarch64/hevcdsp_epel_neon.S @@ -244,6 +244,185 @@ function ff_hevc_put_hevc_pel_pixels64_8_neon, export=1 endfunc +function ff_hevc_put_hevc_pel_bi_pixels4_8_neon, export=1 + mov x10, #(MAX_PB_SIZE * 2) +1: ld1 {v0.s}[0], [x2], x3 // src + ushll v16.8h, v0.8b, #6 + ld1 {v20.4h}, [x4], x10 // src2 + sqadd v16.8h, v16.8h, v20.8h + sqrshrun v0.8b, v16.8h, #7 + st1 {v0.s}[0], [x0], x1 + subs w5, w5, #1 + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_pel_bi_pixels6_8_neon, export=1 + mov x10, #(MAX_PB_SIZE * 2) + sub x1, x1, #4 +1: ld1 {v0.8b}, [x2], x3 + ushll v16.8h, v0.8b, #6 + ld1 {v20.8h}, [x4], x10 + sqadd v16.8h, v16.8h, v20.8h + sqrshrun v0.8b, v16.8h, #7 + st1 {v0.s}[0], [x0], #4 + st1 {v0.h}[2], [x0], x1 + subs w5, w5, #1 + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_pel_bi_pixels8_8_neon, export=1 + mov x10, #(MAX_PB_SIZE * 2) +1: ld1 {v0.8b}, [x2], x3 // src + ushll v16.8h, v0.8b, #6 + ld1 {v20.8h}, [x4], x10 // src2 + sqadd v16.8h, v16.8h, v20.8h + sqrshrun v0.8b, v16.8h, #7 + subs w5, w5, #1 + st1 {v0.8b}, [x0], x1 + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_pel_bi_pixels12_8_neon, export=1 + mov x10, #(MAX_PB_SIZE * 2) + sub x1, x1, #8 +1: ld1 {v0.16b}, [x2], x3 + ushll v16.8h, v0.8b, #6 + ushll2 v17.8h, v0.16b, #6 + ld1 {v20.8h, v21.8h}, [x4], x10 + sqadd v16.8h, v16.8h, v20.8h + sqadd v17.8h, v17.8h, v21.8h + sqrshrun v0.8b, v16.8h, #7 + sqrshrun2 v0.16b, v17.8h, #7 + st1 {v0.8b}, [x0], #8 + subs w5, w5, #1 + st1 {v0.s}[2], [x0], x1 + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_pel_bi_pixels16_8_neon, export=1 + mov x10, #(MAX_PB_SIZE * 2) +1: ld1 {v0.16b}, [x2], x3 // src + ushll v16.8h, v0.8b, #6 + ushll2 v17.8h, v0.16b, #6 + ld1 {v20.8h, v21.8h}, [x4], x10 // src2 + sqadd v16.8h, v16.8h, v20.8h + sqadd v17.8h, v17.8h, v21.8h + sqrshrun v0.8b, v16.8h, #7 + sqrshrun2 v0.16b, v17.8h, #7 + subs w5, w5, #1 + st1 {v0.16b}, [x0], x1 + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_pel_bi_pixels24_8_neon, export=1 + mov x10, #(MAX_PB_SIZE * 2) +1: ld1 {v0.8b-v2.8b}, [x2], x3 // src + ushll v16.8h, v0.8b, #6 + ushll v17.8h, v1.8b, #6 + ushll v18.8h, v2.8b, #6 + ld1 {v20.8h-v22.8h}, [x4], x10 // src2 + sqadd v16.8h, v16.8h, v20.8h + sqadd v17.8h, v17.8h, v21.8h + sqadd v18.8h, v18.8h, v22.8h + sqrshrun v0.8b, v16.8h, #7 + sqrshrun v1.8b, v17.8h, #7 + sqrshrun v2.8b, v18.8h, #7 + subs w5, w5, #1 + st1 {v0.8b-v2.8b}, [x0], x1 + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_pel_bi_pixels32_8_neon, export=1 + mov x10, #(MAX_PB_SIZE * 2) +1: ld1 {v0.16b-v1.16b}, [x2], x3 // src + ushll v16.8h, v0.8b, #6 + ushll2 v17.8h, v0.16b, #6 + ushll v18.8h, v1.8b, #6 + ushll2 v19.8h, v1.16b, #6 + ld1 {v20.8h-v23.8h}, [x4], x10 // src2 + sqadd v16.8h, v16.8h, v20.8h + sqadd v17.8h, v17.8h, v21.8h + sqadd v18.8h, v18.8h, v22.8h + sqadd v19.8h, v19.8h, v23.8h + sqrshrun v0.8b, v16.8h, #7 + sqrshrun2 v0.16b, v17.8h, #7 + sqrshrun v1.8b, v18.8h, #7 + sqrshrun2 v1.16b, v19.8h, #7 + st1 {v0.16b-v1.16b}, [x0], x1 + subs w5, w5, #1 + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_pel_bi_pixels48_8_neon, export=1 + mov x10, #(MAX_PB_SIZE) +1: ld1 {v0.16b-v2.16b}, [x2], x3 // src + ushll v16.8h, v0.8b, #6 + ushll2 v17.8h, v0.16b, #6 + ushll v18.8h, v1.8b, #6 + ushll2 v19.8h, v1.16b, #6 + ushll v20.8h, v2.8b, #6 + ushll2 v21.8h, v2.16b, #6 + ld1 {v24.8h-v27.8h}, [x4], #(MAX_PB_SIZE) // src2 + sqadd v16.8h, v16.8h, v24.8h + sqadd v17.8h, v17.8h, v25.8h + sqadd v18.8h, v18.8h, v26.8h + sqadd v19.8h, v19.8h, v27.8h + ld1 {v24.8h-v25.8h}, [x4], x10 + sqadd v20.8h, v20.8h, v24.8h + sqadd v21.8h, v21.8h, v25.8h + sqrshrun v0.8b, v16.8h, #7 + sqrshrun2 v0.16b, v17.8h, #7 + sqrshrun v1.8b, v18.8h, #7 + sqrshrun2 v1.16b, v19.8h, #7 + sqrshrun v2.8b, v20.8h, #7 + sqrshrun2 v2.16b, v21.8h, #7 + subs w5, w5, #1 + st1 {v0.16b-v2.16b}, [x0], x1 + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_pel_bi_pixels64_8_neon, export=1 +1: ld1 {v0.16b, v1.16b, v2.16b, v3.16b}, [x2], x3 // src + ushll v16.8h, v0.8b, #6 + ushll2 v17.8h, v0.16b, #6 + ushll v18.8h, v1.8b, #6 + ushll2 v19.8h, v1.16b, #6 + ushll v20.8h, v2.8b, #6 + ushll2 v21.8h, v2.16b, #6 + ushll v22.8h, v3.8b, #6 + ushll2 v23.8h, v3.16b, #6 + ld1 {v24.8h, v25.8h, v26.8h, v27.8h}, [x4], #(MAX_PB_SIZE) // src2 + sqadd v16.8h, v16.8h, v24.8h + sqadd v17.8h, v17.8h, v25.8h + sqadd v18.8h, v18.8h, v26.8h + sqadd v19.8h, v19.8h, v27.8h + ld1 {v24.8h, v25.8h, v26.8h, v27.8h}, [x4], #(MAX_PB_SIZE) + sqadd v20.8h, v20.8h, v24.8h + sqadd v21.8h, v21.8h, v25.8h + sqadd v22.8h, v22.8h, v26.8h + sqadd v23.8h, v23.8h, v27.8h + sqrshrun v0.8b, v16.8h, #7 + sqrshrun2 v0.16b, v17.8h, #7 + sqrshrun v1.8b, v18.8h, #7 + sqrshrun2 v1.16b, v19.8h, #7 + sqrshrun v2.8b, v20.8h, #7 + sqrshrun2 v2.16b, v21.8h, #7 + sqrshrun v3.8b, v22.8h, #7 + sqrshrun2 v3.16b, v23.8h, #7 + st1 {v0.16b, v1.16b, v2.16b, v3.16b}, [x0], x1 + subs w5, w5, #1 + b.ne 1b + ret +endfunc + function ff_hevc_put_hevc_epel_v4_8_neon, export=1 load_epel_filterb x5, x4 sub x1, x1, x2 diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index c51488275c..cf171023e7 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -156,8 +156,12 @@ NEON8_FNPROTO(pel_pixels, (int16_t *dst, const uint8_t *src, ptrdiff_t srcstride, int height, intptr_t mx, intptr_t my, int width),); -NEON8_FNPROTO(epel_v, (int16_t *dst, - const uint8_t *src, ptrdiff_t srcstride, +NEON8_FNPROTO(pel_bi_pixels, (uint8_t *dst, ptrdiff_t dststride, + const uint8_t *_src, ptrdiff_t _srcstride, const int16_t *src2, + int height, intptr_t mx, intptr_t my, int width),); + +NEON8_FNPROTO(epel_v, (uint8_t *dst, ptrdiff_t dststride, + const uint8_t *_src, ptrdiff_t _srcstride, const int16_t *src2, int height, intptr_t mx, intptr_t my, int width),); NEON8_FNPROTO(pel_uni_pixels, (uint8_t *_dst, ptrdiff_t _dststride, @@ -324,6 +328,8 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) From patchwork Sat Nov 18 02:06:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Logan.Lyu" X-Patchwork-Id: 44703 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6a89:b0:181:818d:5e7f with SMTP id bi9csp293191pzb; Fri, 17 Nov 2023 18:07:12 -0800 (PST) X-Google-Smtp-Source: AGHT+IETbmECOQ3yaWqilRlgDdsYUBDczm2fQ8BnRnV4Gno5DT2s2BL5ApP3GFYmH4hRerZ0IJkE X-Received: by 2002:aa7:c513:0:b0:53f:6ed5:4dab with SMTP id o19-20020aa7c513000000b0053f6ed54dabmr664918edq.24.1700273232445; Fri, 17 Nov 2023 18:07:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700273232; cv=none; d=google.com; s=arc-20160816; b=iPCs+Q5Gr/qyLRx0ZUgayvYlyG3w5gx/2/bCjbTHjpaqucX07wFFFvBcIJ6IBwlgnV ixKxbjbDzBMZN5YEJ/8RofpkQgxdWV8kP2ed5vsOSJv+kbLwYGpkJkCea1udqP6UvzH0 KcnLDkIF/TmokFEmIulgHL9dAId8iTTalruwQUbKGxTnsLrCiU1wDC6e6muzlEN2tf9a pua/ZiGKyeLUrQYkX01bOg9pwNqNJSXXTLQ8+bbd83TOApiRo65HMcMMf/miRaJXNUQ6 cut4TkdnZEutBO1n9Eq3dnFFJ7HNjJxOgXz7pYOiEKTFBI7lGouy58CPW6bCknRiMR6P BfBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :organization:to:from:user-agent:mime-version:date:message-id :delivered-to; bh=f+5ILYXjPI3g/MB3fkSwDfFd/XyTsonVSyggfFn23Nw=; fh=4VBelKDE4DH3L7jF6H/1Jmu78FdN+YP76yfdJCQTJ30=; b=wlnzD2Z03wPXsbmdmWcs6FvRMviLtm/SU3AhoLrsO8GTTM7IfeS1L/ANIA+DnpjRHZ 0m3qLg8o9TFL1vGVyVqGLcikYJlneFY7ZqmnYPpYBG50nLXCb+RHfDbdPtRWtgFvPl3F B414EuJz1a37649aJ3ccEpGcHUSoyLEvRmUfoxW/8srNyx/oVKWspEoO5rVwRZbD2why CvsT7/gZl1CeGfQ1lgl1yhzxPdU7oOGBu79okOKZlCmcVmRWxWeJEgZA/+XHkN0qngOw zOJu3rUiSTQwFlaa6Cqfr1hqtI8dInsCDXfmmvOonD4TMDf+31YHBmuyfWb3vOFxputX YvKA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x18-20020aa7dad2000000b0053b32146616si1609638eds.665.2023.11.17.18.07.12; Fri, 17 Nov 2023 18:07:12 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 365E168CD3C; Sat, 18 Nov 2023 04:06:59 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-my3-01p8.yunyou.top (smtp-my3-01p8.yunyou.top [60.247.169.8]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id ADD1368CCFB for ; Sat, 18 Nov 2023 04:06:51 +0200 (EET) Received: from [192.168.15.105] (unknown [122.233.146.151]) by smtp-my-01.yunyou.top (WestCloudMail) with ESMTPA id 3CDDF141F26; Sat, 18 Nov 2023 10:06:46 +0800 (CST) Message-ID: <5969017d-81db-470d-91bd-37c5b02ae5c1@myais.com.cn> Date: Sat, 18 Nov 2023 10:06:46 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Logan.Lyu" To: ffmpeg-devel@ffmpeg.org Organization: myais Subject: [FFmpeg-devel] [PATCH 2/6] lavc/aarch64: new optimization for 8-bit hevc_epel_bi_h X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: jdek@itanimul.li Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: B/JT41wPj4lp put_hevc_epel_bi_h4_8_c: 96.0 put_hevc_epel_bi_h4_8_neon: 36.3 put_hevc_epel_bi_h6_8_c: 288.3 put_hevc_epel_bi_h6_8_neon: 59.3 put_hevc_epel_bi_h8_8_c: 358.5 put_hevc_epel_bi_h8_8_neon: 61.5 put_hevc_epel_bi_h12_8_c: 759.8 put_hevc_epel_bi_h12_8_neon: 159.5 put_hevc_epel_bi_h16_8_c: 1307.0 put_hevc_epel_bi_h16_8_neon: 182.0 put_hevc_epel_bi_h24_8_c: 2778.3 put_hevc_epel_bi_h24_8_neon: 430.5 put_hevc_epel_bi_h32_8_c: 4952.3 put_hevc_epel_bi_h32_8_neon: 679.5 put_hevc_epel_bi_h48_8_c: 11803.3 put_hevc_epel_bi_h48_8_neon: 1443.5 put_hevc_epel_bi_h64_8_c: 20654.8 put_hevc_epel_bi_h64_8_neon: 2737.0 put_hevc_qpel_bi_h4_8_c: 140.0 put_hevc_qpel_bi_h4_8_neon: 111.5 put_hevc_qpel_bi_h6_8_c: 318.0 put_hevc_qpel_bi_h6_8_neon: 85.8 put_hevc_qpel_bi_h8_8_c: 536.5 put_hevc_qpel_bi_h8_8_neon: 95.3 put_hevc_qpel_bi_h12_8_c: 1188.5 put_hevc_qpel_bi_h12_8_neon: 291.3 put_hevc_qpel_bi_h16_8_c: 2064.3 put_hevc_qpel_bi_h16_8_neon: 365.3 put_hevc_qpel_bi_h24_8_c: 4757.5 put_hevc_qpel_bi_h24_8_neon: 1010.0 put_hevc_qpel_bi_h32_8_c: 8351.8 put_hevc_qpel_bi_h32_8_neon: 2917.8 put_hevc_qpel_bi_h48_8_c: 19299.8 put_hevc_qpel_bi_h48_8_neon: 2976.8 put_hevc_qpel_bi_h64_8_c: 34182.5 put_hevc_qpel_bi_h64_8_neon: 5236.3 Co-Authored-By: J. Dekker Signed-off-by: Logan Lyu --- libavcodec/aarch64/hevcdsp_epel_neon.S | 257 ++++++++++++++++++++++ libavcodec/aarch64/hevcdsp_init_aarch64.c | 5 + 2 files changed, 262 insertions(+) NEON8_FNASSIGN(c->put_hevc_epel_uni, 1, 0, epel_uni_v,); diff --git a/libavcodec/aarch64/hevcdsp_epel_neon.S b/libavcodec/aarch64/hevcdsp_epel_neon.S index 74165273d7..070c8c9990 100644 --- a/libavcodec/aarch64/hevcdsp_epel_neon.S +++ b/libavcodec/aarch64/hevcdsp_epel_neon.S @@ -423,6 +423,263 @@ function ff_hevc_put_hevc_pel_bi_pixels64_8_neon, export=1 ret endfunc + +function ff_hevc_put_hevc_epel_bi_h4_8_neon, export=1 + load_epel_filterb x6, x7 + sub x2, x2, #1 + mov x10, #(MAX_PB_SIZE * 2) +1: ld1 {v4.8b}, [x2], x3 + ext v5.8b, v4.8b, v4.8b, #1 + ext v6.8b, v4.8b, v4.8b, #2 + ext v7.8b, v4.8b, v4.8b, #3 + calc_epelb v16, v4, v5, v6, v7 + ld1 {v20.4h}, [x4], x10 + sqadd v16.8h, v16.8h, v20.8h + sqrshrun v4.8b, v16.8h, #7 + st1 {v4.s}[0], [x0], x1 + subs w5, w5, #1 // height + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_h6_8_neon, export=1 + load_epel_filterb x6, x7 + sub w1, w1, #4 + sub x2, x2, #1 + mov x10, #(MAX_PB_SIZE * 2) +1: ld1 {v24.16b}, [x2], x3 + ext v26.16b, v24.16b, v24.16b, #1 + ext v27.16b, v24.16b, v24.16b, #2 + ext v28.16b, v24.16b, v24.16b, #3 + calc_epelb v16, v24, v26, v27, v28 + ld1 {v20.8h}, [x4], x10 + sqadd v16.8h, v16.8h, v20.8h + sqrshrun v16.8b, v16.8h, #7 + st1 {v16.s}[0], [x0], #4 + st1 {v16.h}[2], [x0], x1 + subs w5, w5, #1 // height + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_h8_8_neon, export=1 + load_epel_filterb x6, x7 + sub x2, x2, #1 + mov x10, #(MAX_PB_SIZE * 2) +1: ld1 {v24.16b}, [x2], x3 + ext v26.16b, v24.16b, v24.16b, #1 + ext v27.16b, v24.16b, v24.16b, #2 + ext v28.16b, v24.16b, v24.16b, #3 + calc_epelb v16, v24, v26 v27, v28 + ld1 {v20.8h}, [x4], x10 + sqadd v16.8h, v16.8h, v20.8h + sqrshrun v16.8b, v16.8h, #7 + st1 {v16.8b}, [x0], x1 + subs w5, w5, #1 // height + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_h12_8_neon, export=1 + load_epel_filterb x6, x7 + sub x1, x1, #8 + sub x2, x2, #1 + mov x10, #(MAX_PB_SIZE * 2) +1: ld1 {v24.16b}, [x2], x3 + ext v26.16b, v24.16b, v24.16b, #1 + ext v27.16b, v24.16b, v24.16b, #2 + ext v28.16b, v24.16b, v24.16b, #3 + calc_epelb v16, v24, v26, v27, v28 + calc_epelb2 v17, v24, v26, v27, v28 + ld1 {v20.8h, v21.8h}, [x4], x10 + sqadd v18.8h, v16.8h, v20.8h + sqadd v19.8h, v17.8h, v21.8h + sqrshrun v20.8b, v18.8h, #7 + sqrshrun v21.8b, v19.8h, #7 + st1 {v20.8b}, [x0], #8 + st1 {v21.s}[0], [x0], x1 + subs w5, w5, #1 // height + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_h16_8_neon, export=1 + load_epel_filterb x6, x7 + sub x2, x2, #1 + mov x10, #(MAX_PB_SIZE * 2) +1: ldr q24, [x2] + ldr s25, [x2, #16] + add x2, x2, x3 + ext v26.16b, v24.16b, v25.16b, #1 + ext v27.16b, v24.16b, v25.16b, #2 + ext v28.16b, v24.16b, v25.16b, #3 + calc_epelb v16, v24, v26, v27, v28 + calc_epelb2 v17, v24, v26, v27, v28 + ld1 {v24.8h, v25.8h}, [x4], x10 + sqadd v16.8h, v16.8h, v24.8h + sqadd v17.8h, v17.8h, v25.8h + sqrshrun v4.8b, v16.8h, #7 + sqrshrun2 v4.16b, v17.8h, #7 + st1 {v4.16b}, [x0], x1 + subs w5, w5, #1 // height + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_h24_8_neon, export=1 + load_epel_filterb x6, x7 + sub x2, x2, #1 + mov x10, #(MAX_PB_SIZE * 2) +1: ld1 {v24.16b, v25.16b}, [x2], x3 + ext v26.16b, v24.16b, v25.16b, #1 + ext v27.16b, v24.16b, v25.16b, #2 + ext v28.16b, v24.16b, v25.16b, #3 + calc_epelb v16, v24, v26, v27, v28 + calc_epelb2 v17, v24, v26, v27, v28 + ext v26.16b, v25.16b, v25.16b, #1 + ext v27.16b, v25.16b, v25.16b, #2 + ext v28.16b, v25.16b, v25.16b, #3 + calc_epelb v18, v25, v26, v27, v28 + ld1 {v20.8h, v21.8h, v22.8h}, [x4], x10 + sqadd v16.8h, v16.8h, v20.8h + sqadd v17.8h, v17.8h, v21.8h + sqadd v18.8h, v18.8h, v22.8h + sqrshrun v4.8b, v16.8h, #7 + sqrshrun v5.8b, v17.8h, #7 + sqrshrun v6.8b, v18.8h, #7 + st1 {v4.8b, v5.8b, v6.8b}, [x0], x1 + subs w5, w5, #1 // height + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_h32_8_neon, export=1 + load_epel_filterb x6, x7 + sub x2, x2, #1 + mov x10, #(MAX_PB_SIZE * 2) +1: ldp q24, q25, [x2] + ldr s26, [x2, #32] + add x2, x2, x3 + ext v27.16b, v24.16b, v25.16b, #1 + ext v28.16b, v24.16b, v25.16b, #2 + ext v29.16b, v24.16b, v25.16b, #3 + calc_epelb v16, v24, v27, v28, v29 + calc_epelb2 v17, v24, v27, v28, v29 + ext v27.16b, v25.16b, v26.16b, #1 + ext v28.16b, v25.16b, v26.16b, #2 + ext v29.16b, v25.16b, v26.16b, #3 + calc_epelb v18, v25, v27, v28, v29 + calc_epelb2 v19, v25, v27, v28, v29 + ld1 {v24.8h, v25.8h, v26.8h, v27.8h}, [x4], x10 + sqadd v16.8h, v16.8h, v24.8h + sqadd v17.8h, v17.8h, v25.8h + sqadd v18.8h, v18.8h, v26.8h + sqadd v19.8h, v19.8h, v27.8h + sqrshrun v4.8b, v16.8h, #7 + sqrshrun2 v4.16b, v17.8h, #7 + sqrshrun v5.8b, v18.8h, #7 + sqrshrun2 v5.16b, v19.8h, #7 + st1 {v4.16b, v5.16b}, [x0], x1 + subs w5, w5, #1 + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_h48_8_neon, export=1 + load_epel_filterb x6, x7 + sub x2, x2, #1 + mov x7, #24 + mov x10, #(MAX_PB_SIZE * 2 - 48) +1: ld1 {v24.16b, v25.16b, v26.16b}, [x2] + ldr s27, [x2, #48] + add x2, x2, x3 + ext v28.16b, v24.16b, v25.16b, #1 + ext v29.16b, v24.16b, v25.16b, #2 + ext v30.16b, v24.16b, v25.16b, #3 + calc_epelb v16, v24, v28, v29, v30 + calc_epelb2 v17, v24, v28, v29, v30 + ext v28.16b, v25.16b, v26.16b, #1 + ext v29.16b, v25.16b, v26.16b, #2 + ext v30.16b, v25.16b, v26.16b, #3 + calc_epelb v18, v25, v28, v29, v30 + calc_epelb2 v19, v25, v28, v29, v30 + ext v28.16b, v26.16b, v27.16b, #1 + ext v29.16b, v26.16b, v27.16b, #2 + ext v30.16b, v26.16b, v27.16b, #3 + calc_epelb v20, v26, v28, v29, v30 + calc_epelb2 v21, v26, v28, v29, v30 + + ld1 {v24.8h, v25.8h, v26.8h}, [x4], #48 + sqadd v16.8h, v16.8h, v24.8h + sqadd v17.8h, v17.8h, v25.8h + sqadd v18.8h, v18.8h, v26.8h + ld1 {v27.8h, v28.8h, v29.8h}, [x4], x10 + sqadd v19.8h, v19.8h, v27.8h + sqadd v20.8h, v20.8h, v28.8h + sqadd v21.8h, v21.8h, v29.8h + sqrshrun v4.8b, v16.8h, #7 + sqrshrun2 v4.16b, v17.8h, #7 + sqrshrun v5.8b, v18.8h, #7 + sqrshrun2 v5.16b, v19.8h, #7 + sqrshrun v6.8b, v20.8h, #7 + sqrshrun2 v6.16b, v21.8h, #7 + st1 {v4.16b, v5.16b, v6.16b}, [x0], x1 + subs w5, w5, #1 // height + b.ne 1b + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_h64_8_neon, export=1 + load_epel_filterb x6, x7 + sub x2, x2, #1 +1: ld1 {v24.16b, v25.16b, v26.16b, v27.16b}, [x2] + ldr s28, [x2, #64] + add x2, x2, x3 + ext v29.16b, v24.16b, v25.16b, #1 + ext v30.16b, v24.16b, v25.16b, #2 + ext v31.16b, v24.16b, v25.16b, #3 + calc_epelb v16, v24, v29, v30, v31 + calc_epelb2 v17, v24, v29, v30, v31 + ext v29.16b, v25.16b, v26.16b, #1 + ext v30.16b, v25.16b, v26.16b, #2 + ext v31.16b, v25.16b, v26.16b, #3 + calc_epelb v18, v25, v29, v30, v31 + calc_epelb2 v19, v25, v29, v30, v31 + ld1 {v4.8h, v5.8h, v6.8h, v7.8h}, [x4], #64 + sqadd v16.8h, v16.8h, v4.8h + sqadd v17.8h, v17.8h, v5.8h + sqadd v18.8h, v18.8h, v6.8h + sqadd v19.8h, v19.8h, v7.8h + sqrshrun v16.8b, v16.8h, #7 + sqrshrun2 v16.16b, v17.8h, #7 + sqrshrun v17.8b, v18.8h, #7 + sqrshrun2 v17.16b, v19.8h, #7 + + ext v29.16b, v26.16b, v27.16b, #1 + ext v30.16b, v26.16b, v27.16b, #2 + ext v31.16b, v26.16b, v27.16b, #3 + calc_epelb v20, v26, v29, v30, v31 + calc_epelb2 v21, v26, v29, v30, v31 + ext v29.16b, v27.16b, v28.16b, #1 + ext v30.16b, v27.16b, v28.16b, #2 + ext v31.16b, v27.16b, v28.16b, #3 + calc_epelb v22, v27, v29, v30, v31 + calc_epelb2 v23, v27, v29, v30, v31 + ld1 {v4.8h, v5.8h, v6.8h, v7.8h}, [x4], #64 + sqadd v20.8h, v20.8h, v4.8h + sqadd v21.8h, v21.8h, v5.8h + sqadd v22.8h, v22.8h, v6.8h + sqadd v23.8h, v23.8h, v7.8h + sqrshrun v18.8b, v20.8h, #7 + sqrshrun2 v18.16b, v21.8h, #7 + sqrshrun v19.8b, v22.8h, #7 + sqrshrun2 v19.16b, v23.8h, #7 + st1 {v16.16b, v17.16b, v18.16b, v19.16b}, [x0], x1 + subs w5, w5, #1 + b.ne 1b + ret +endfunc + function ff_hevc_put_hevc_epel_v4_8_neon, export=1 load_epel_filterb x5, x4 sub x1, x1, x2 diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index cf171023e7..a16c1a83e0 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -160,6 +160,10 @@ NEON8_FNPROTO(pel_bi_pixels, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *_src, ptrdiff_t _srcstride, const int16_t *src2, int height, intptr_t mx, intptr_t my, int width),); +NEON8_FNPROTO(epel_bi_h, (uint8_t *dst, ptrdiff_t dststride, + const uint8_t *src, ptrdiff_t srcstride, const int16_t *src2, + int height, intptr_t mx, intptr_t my, int width),); + NEON8_FNPROTO(epel_v, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *_src, ptrdiff_t _srcstride, const int16_t *src2, int height, intptr_t mx, intptr_t my, int width),); @@ -329,6 +333,7 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) NEON8_FNASSIGN(c->put_hevc_qpel, 0, 0, pel_pixels,); NEON8_FNASSIGN(c->put_hevc_qpel, 1, 0, qpel_v,); NEON8_FNASSIGN(c->put_hevc_epel_bi, 0, 0, pel_bi_pixels,); + NEON8_FNASSIGN(c->put_hevc_epel_bi, 0, 1, epel_bi_h,); NEON8_FNASSIGN(c->put_hevc_qpel_bi, 0, 0, pel_bi_pixels,); NEON8_FNASSIGN(c->put_hevc_epel_uni, 0, 0, pel_uni_pixels,); From patchwork Sat Nov 18 02:06:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Logan.Lyu" X-Patchwork-Id: 44704 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6a89:b0:181:818d:5e7f with SMTP id bi9csp293267pzb; Fri, 17 Nov 2023 18:07:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IF7dggB9QgbMZfjAG3GjRbq/9S7hPZyMMUZ4v2ZmeNj/KXb+d2gpClC/MjEEQmyiqhqJuRM X-Received: by 2002:a17:906:74c1:b0:9f2:3e76:5d39 with SMTP id z1-20020a17090674c100b009f23e765d39mr807386ejl.0.1700273242949; Fri, 17 Nov 2023 18:07:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700273242; cv=none; d=google.com; s=arc-20160816; b=tYMbADlg3LSZopqzkOlwTk+XR+ATx/eQ5hEJ+EoAjbQ6C2W+O6eV73IwYU4hq3yNPq aaX5/uEjbdIyhuTZ1SOGbAKX/KVHh9GX6w9DG77kmmL1nlnkUY3+4vW7pd+sLKj5yNjg KDVYeb5emSKm7ddBIkEUEdNho+K0W72rLsIK4lyojBVn/zl62pCss7wgXk/1V+8FPl9b 0IPC7v+9OEFC8JVHlVn5PBlb4qlBkZvwoGJI8obR4fREa1ZzabRqtl0QHJXZhg3+JnRQ nAwFs/4Rf4iY/hYm8Rmts4SM6d/QFRAJTwEyZbxbZYP1bPhRHw81ts+Z1TK7xxPOMYZf 333Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :organization:to:from:user-agent:mime-version:date:message-id :delivered-to; bh=2Pz4Lqd/2OcvxKLEX0wRtRjivN7/LWIz6Krz8rM3BiQ=; fh=4VBelKDE4DH3L7jF6H/1Jmu78FdN+YP76yfdJCQTJ30=; b=V2e5TJO2PlXBYQP2zfRtM73s6oUmSqeNtUTXwpzoaLpxB+4OveJEC12O9XL3MKlxRj fqYaw1PcWRSeigXd1TIQd75P3Ail5Gn5ZsUYIUs+gKp5/7oMK4ItgmRbzoD+MJS3QNie Ac4SAeA9TJl5eQW2DOSCRRXDosleLdchyddLTcNSYg6rIvnOxkjLlk7yIRPUR4lwuXac nbsAUkrHUHUzfJNFJgl1QgJGKX/9QhF2fCVtaIAv8BwUGC4yJZHJuEIGGdf8yFk5WndK qUhN7GMq+SyKj+AvceXNVKLlZfwu6lvbVsVMIiBONiyEQSadq/PFc0VnyteethNxgtKr 5fwA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id h23-20020a170906591700b009e5ed846305si1679045ejq.741.2023.11.17.18.07.22; Fri, 17 Nov 2023 18:07:22 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4FD6768CD33; Sat, 18 Nov 2023 04:07:05 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-my3-01p9.yunyou.top (smtp-my3-01p9.yunyou.top [60.247.169.9]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 296A568CD33 for ; Sat, 18 Nov 2023 04:06:58 +0200 (EET) Received: from [192.168.15.105] (unknown [122.233.146.151]) by smtp-my-01.yunyou.top (WestCloudMail) with ESMTPA id 7EBCE141F20; Sat, 18 Nov 2023 10:06:53 +0800 (CST) Message-ID: Date: Sat, 18 Nov 2023 10:06:53 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Logan.Lyu" To: ffmpeg-devel@ffmpeg.org Organization: myais Subject: [FFmpeg-devel] [PATCH 3/6] lavc/aarch64: new optimization for 8-bit hevc_epel_bi_v X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: jdek@itanimul.li Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: OBkS5eWt3JG0 put_hevc_epel_bi_v4_8_c: 138.4 put_hevc_epel_bi_v4_8_neon: 33.7 put_hevc_epel_bi_v6_8_c: 302.9 put_hevc_epel_bi_v6_8_neon: 46.7 put_hevc_epel_bi_v8_8_c: 408.7 put_hevc_epel_bi_v8_8_neon: 48.7 put_hevc_epel_bi_v12_8_c: 779.4 put_hevc_epel_bi_v12_8_neon: 139.7 put_hevc_epel_bi_v16_8_c: 1344.9 put_hevc_epel_bi_v16_8_neon: 160.2 put_hevc_epel_bi_v24_8_c: 2981.7 put_hevc_epel_bi_v24_8_neon: 344.9 put_hevc_epel_bi_v32_8_c: 5280.9 put_hevc_epel_bi_v32_8_neon: 618.4 put_hevc_epel_bi_v48_8_c: 12494.9 put_hevc_epel_bi_v48_8_neon: 1364.4 put_hevc_epel_bi_v64_8_c: 22127.7 put_hevc_epel_bi_v64_8_neon: 2473.7 Co-Authored-By: J. Dekker Signed-off-by: Logan Lyu --- libavcodec/aarch64/hevcdsp_epel_neon.S | 212 ++++++++++++++++++++++ libavcodec/aarch64/hevcdsp_init_aarch64.c | 5 + 2 files changed, 217 insertions(+) NEON8_FNASSIGN(c->put_hevc_epel_uni, 1, 0, epel_uni_v,); diff --git a/libavcodec/aarch64/hevcdsp_epel_neon.S b/libavcodec/aarch64/hevcdsp_epel_neon.S index 070c8c9990..54e55cd508 100644 --- a/libavcodec/aarch64/hevcdsp_epel_neon.S +++ b/libavcodec/aarch64/hevcdsp_epel_neon.S @@ -680,6 +680,218 @@ function ff_hevc_put_hevc_epel_bi_h64_8_neon, export=1 ret endfunc +function ff_hevc_put_hevc_epel_bi_v4_8_neon, export=1 + load_epel_filterb x7, x6 + sub x2, x2, x3 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.s}[0], [x2], x3 + ld1 {v17.s}[0], [x2], x3 + ld1 {v18.s}[0], [x2], x3 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().s}[0], [x2], x3 + calc_epelb v4, \src0, \src1, \src2, \src3 + ld1 {v24.4h}, [x4], x10 + sqadd v4.8h, v4.8h, v24.8h + sqrshrun v4.8b, v4.8h, #7 + subs w5, w5, #1 + st1 {v4.s}[0], [x0], x1 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_v6_8_neon, export=1 + load_epel_filterb x7, x6 + sub x2, x2, x3 + sub x1, x1, #4 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8b}, [x2], x3 + ld1 {v17.8b}, [x2], x3 + ld1 {v18.8b}, [x2], x3 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().8b}, [x2], x3 + calc_epelb v4, \src0, \src1, \src2, \src3 + ld1 {v24.8h}, [x4], x10 + sqadd v4.8h, v4.8h, v24.8h + sqrshrun v4.8b, v4.8h, #7 + st1 {v4.s}[0], [x0], #4 + subs w5, w5, #1 + st1 {v4.h}[2], [x0], x1 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_v8_8_neon, export=1 + load_epel_filterb x7, x6 + sub x2, x2, x3 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8b}, [x2], x3 + ld1 {v17.8b}, [x2], x3 + ld1 {v18.8b}, [x2], x3 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().8b}, [x2], x3 + calc_epelb v4, \src0, \src1, \src2, \src3 + ld1 {v24.8h}, [x4], x10 + sqadd v4.8h, v4.8h, v24.8h + sqrshrun v4.8b, v4.8h, #7 + subs w5, w5, #1 + st1 {v4.8b}, [x0], x1 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_v12_8_neon, export=1 + load_epel_filterb x7, x6 + sub x1, x1, #8 + sub x2, x2, x3 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.16b}, [x2], x3 + ld1 {v17.16b}, [x2], x3 + ld1 {v18.16b}, [x2], x3 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().16b}, [x2], x3 + calc_epelb v4, \src0, \src1, \src2, \src3 + calc_epelb2 v5, \src0, \src1, \src2, \src3 + ld1 {v24.8h, v25.8h}, [x4], x10 + sqadd v4.8h, v4.8h, v24.8h + sqadd v5.8h, v5.8h, v25.8h + sqrshrun v4.8b, v4.8h, #7 + sqrshrun2 v4.16b, v5.8h, #7 + st1 {v4.8b}, [x0], #8 + subs w5, w5, #1 + st1 {v4.s}[2], [x0], x1 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_v16_8_neon, export=1 + load_epel_filterb x7, x6 + sub x2, x2, x3 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.16b}, [x2], x3 + ld1 {v17.16b}, [x2], x3 + ld1 {v18.16b}, [x2], x3 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().16b}, [x2], x3 + calc_epelb v4, \src0, \src1, \src2, \src3 + calc_epelb2 v5, \src0, \src1, \src2, \src3 + ld1 {v24.8h, v25.8h}, [x4], x10 + sqadd v4.8h, v4.8h, v24.8h + sqadd v5.8h, v5.8h, v25.8h + sqrshrun v4.8b, v4.8h, #7 + sqrshrun2 v4.16b, v5.8h, #7 + st1 {v4.16b}, [x0], x1 + subs w5, w5, #1 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_v24_8_neon, export=1 + load_epel_filterb x7, x6 + sub x2, x2, x3 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8b, v17.8b, v18.8b}, [x2], x3 + ld1 {v19.8b, v20.8b, v21.8b}, [x2], x3 + ld1 {v22.8b, v23.8b, v24.8b}, [x2], x3 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11 + ld1 {\src9\().8b, \src10\().8b, \src11\().8b}, [x2], x3 + calc_epelb v4, \src0, \src3, \src6, \src9 + calc_epelb v5, \src1, \src4, \src7, \src10 + calc_epelb v6, \src2, \src5, \src8, \src11 + ld1 {v28.8h, v29.8h, v30.8h}, [x4], x10 + sqadd v4.8h, v4.8h, v28.8h + sqadd v5.8h, v5.8h, v29.8h + sqadd v6.8h, v6.8h, v30.8h + sqrshrun v4.8b, v4.8h, #7 + sqrshrun v5.8b, v5.8h, #7 + sqrshrun v6.8b, v6.8h, #7 + subs w5, w5, #1 + st1 {v4.8b, v5.8b, v6.8b}, [x0], x1 +.endm +1: calc_all12 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_v32_8_neon, export=1 + load_epel_filterb x7, x6 + sub x2, x2, x3 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.16b, v17.16b}, [x2], x3 + ld1 {v18.16b, v19.16b}, [x2], x3 + ld1 {v20.16b, v21.16b}, [x2], x3 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\src6\().16b, \src7\().16b}, [x2], x3 + calc_epelb v4, \src0, \src2, \src4, \src6 + calc_epelb2 v5, \src0, \src2, \src4, \src6 + calc_epelb v6, \src1, \src3, \src5, \src7 + calc_epelb2 v7, \src1, \src3, \src5, \src7 + ld1 {v24.8h-v27.8h}, [x4], x10 + sqadd v4.8h, v4.8h, v24.8h + sqadd v5.8h, v5.8h, v25.8h + sqadd v6.8h, v6.8h, v26.8h + sqadd v7.8h, v7.8h, v27.8h + sqrshrun v4.8b, v4.8h, #7 + sqrshrun2 v4.16b, v5.8h, #7 + sqrshrun v5.8b, v6.8h, #7 + sqrshrun2 v5.16b, v7.8h, #7 + st1 {v4.16b, v5.16b}, [x0], x1 + subs w5, w5, #1 +.endm +1: calc_all8 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_v48_8_neon, export=1 + stp x4, x5, [sp, #-64]! + stp x2, x3, [sp, #16] + stp x0, x1, [sp, #32] + stp x7, x30, [sp, #48] + bl X(ff_hevc_put_hevc_epel_bi_v24_8_neon) + ldp x4, x5, [sp] + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp, #32] + ldr x7, [sp, #48] + add sp, sp, #48 + add x0, x0, #24 + add x2, x2, #24 + add x4, x4, #48 + bl X(ff_hevc_put_hevc_epel_bi_v24_8_neon) + ldr x30, [sp, #8] + add sp, sp, #16 + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_v64_8_neon, export=1 + stp x4, x5, [sp, #-64]! + stp x2, x3, [sp, #16] + stp x0, x1, [sp, #32] + stp x7, x30, [sp, #48] + bl X(ff_hevc_put_hevc_epel_bi_v32_8_neon) + ldp x4, x5, [sp] + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp, #32] + ldr x7, [sp, #48] + add sp, sp, #48 + add x0, x0, #32 + add x2, x2, #32 + add x4, x4, #64 + bl X(ff_hevc_put_hevc_epel_bi_v32_8_neon) + ldr x30, [sp, #8] + add sp, sp, #16 + ret +endfunc + function ff_hevc_put_hevc_epel_v4_8_neon, export=1 load_epel_filterb x5, x4 sub x1, x1, x2 diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index a16c1a83e0..a0f0f072f8 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -164,6 +164,10 @@ NEON8_FNPROTO(epel_bi_h, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *src, ptrdiff_t srcstride, const int16_t *src2, int height, intptr_t mx, intptr_t my, int width),); +NEON8_FNPROTO(epel_bi_v, (uint8_t *dst, ptrdiff_t dststride, + const uint8_t *src, ptrdiff_t srcstride, const int16_t *src2, + int height, intptr_t mx, intptr_t my, int width),); + NEON8_FNPROTO(epel_v, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *_src, ptrdiff_t _srcstride, const int16_t *src2, int height, intptr_t mx, intptr_t my, int width),); @@ -334,6 +338,7 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) NEON8_FNASSIGN(c->put_hevc_qpel, 1, 0, qpel_v,); NEON8_FNASSIGN(c->put_hevc_epel_bi, 0, 0, pel_bi_pixels,); NEON8_FNASSIGN(c->put_hevc_epel_bi, 0, 1, epel_bi_h,); + NEON8_FNASSIGN(c->put_hevc_epel_bi, 1, 0, epel_bi_v,); NEON8_FNASSIGN(c->put_hevc_qpel_bi, 0, 0, pel_bi_pixels,); NEON8_FNASSIGN(c->put_hevc_epel_uni, 0, 0, pel_uni_pixels,); From patchwork Sat Nov 18 02:06:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Logan.Lyu" X-Patchwork-Id: 44701 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6a89:b0:181:818d:5e7f with SMTP id bi9csp293325pzb; Fri, 17 Nov 2023 18:07:34 -0800 (PST) X-Google-Smtp-Source: AGHT+IGZg8PmFwVcLqW8vlCnixGTTLXbnemuqLKX9zdy8DG6yBi37fcBRbBeSGFdy/Okm438ATVa X-Received: by 2002:a05:651c:198a:b0:2c5:994:ed72 with SMTP id bx10-20020a05651c198a00b002c50994ed72mr1020445ljb.21.1700273253455; Fri, 17 Nov 2023 18:07:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700273253; cv=none; d=google.com; s=arc-20160816; b=alJT/gLvlh56xnAKsb0CdOWO/5zfQHWyV6Hd4yekFaYAqSTGlDram8+aP/o2KJLbFL ZRMq1/VACLR1fRB90vlpB9ZuIqZz9WWhV9t06dF9nMdvLbt+dpbxyh1y2Df2+Dsuw3wx FGY5omDpmu1Jhw4nzl13BUK0SNHzoaEochp8kXJ0Y/YhjFT4M3X6J8cg2oVaXunExZxk iZGdTe/FT/WRu8FQ0mz3FmAXrY3geyTK9R8raTOv7EGnXTkbkEhYXugZ7hm+tH8Mi3yg +u0UQf0OCVTUuNHQi0AcdBcoRr55ARwGGUYfsKoB+EKn8jli0Jx22dywHpCU/1GyjdJ6 SEqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :organization:to:from:user-agent:mime-version:date:message-id :delivered-to; bh=22CfKNDSjKDW47BNu1+10BiiltY0S1wJ8M+eobUIv7E=; fh=4VBelKDE4DH3L7jF6H/1Jmu78FdN+YP76yfdJCQTJ30=; b=jb6kmysuRRjsiPF7xtdlCKE9FDYUFWubEKoDhgWaI5+nBpIMYRIz+cxUcu6aVkvBr5 0SxMKdbJEH5VOIwi21jvOOxEIaB6ss5yWkfH8Liq7RqpLSLE2lGZbZkdJiMZ16BhTBDy MuehR1pOQtL71FKixC1Ha0u2YYNG8eDt+hDXC4pNaSYTLks3WxLJu/PeXUwoFJx2GPbc CVu0gEbDGI5iDaFr4WeXcosqbv+ZC8h6u+ShLbbnsOxhageuUmjraUcbB/cNlt/cbeia OVA7YGKvee9fN2KLv1ax588rFek7t6LLn+dWfIlEwSxXozblRLXFiT7ov5+XC/ISlF5C T4ug== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s21-20020a170906221500b009b2dde5027asi1826928ejs.313.2023.11.17.18.07.33; Fri, 17 Nov 2023 18:07:33 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6724868CD80; Sat, 18 Nov 2023 04:07:10 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-my3-01p3.yunyou.top (smtp-my3-01p3.yunyou.top [60.247.169.3]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9C58368C96C for ; Sat, 18 Nov 2023 04:07:03 +0200 (EET) Received: from [192.168.15.105] (unknown [122.233.146.151]) by smtp-my-01.yunyou.top (WestCloudMail) with ESMTPA id C699A141F24; Sat, 18 Nov 2023 10:06:58 +0800 (CST) Message-ID: <27f6bf27-e39c-41d4-a30e-45d170ac5e46@myais.com.cn> Date: Sat, 18 Nov 2023 10:06:58 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Logan.Lyu" To: ffmpeg-devel@ffmpeg.org Organization: myais Subject: [FFmpeg-devel] [PATCH 4/6] lavc/aarch64: new optimization for 8-bit hevc_epel_bi_hv X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: jdek@itanimul.li Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: CiglOEh6TMQ+ put_hevc_epel_bi_hv4_8_c: 242.9 put_hevc_epel_bi_hv4_8_i8mm: 68.6 put_hevc_epel_bi_hv6_8_c: 402.4 put_hevc_epel_bi_hv6_8_i8mm: 135.9 put_hevc_epel_bi_hv8_8_c: 636.4 put_hevc_epel_bi_hv8_8_i8mm: 145.6 put_hevc_epel_bi_hv12_8_c: 1363.1 put_hevc_epel_bi_hv12_8_i8mm: 324.1 put_hevc_epel_bi_hv16_8_c: 2222.1 put_hevc_epel_bi_hv16_8_i8mm: 509.1 put_hevc_epel_bi_hv24_8_c: 4793.4 put_hevc_epel_bi_hv24_8_i8mm: 1091.9 put_hevc_epel_bi_hv32_8_c: 8393.9 put_hevc_epel_bi_hv32_8_i8mm: 1720.6 put_hevc_epel_bi_hv48_8_c: 19526.6 put_hevc_epel_bi_hv48_8_i8mm: 4285.9 put_hevc_epel_bi_hv64_8_c: 33915.4 put_hevc_epel_bi_hv64_8_i8mm: 6783.6 Co-Authored-By: J. Dekker Signed-off-by: Logan Lyu --- libavcodec/aarch64/hevcdsp_epel_neon.S | 331 ++++++++++++++++++++++ libavcodec/aarch64/hevcdsp_init_aarch64.c | 5 + 2 files changed, 336 insertions(+) NEON8_FNASSIGN(c->put_hevc_qpel, 0, 1, qpel_h, _i8mm); NEON8_FNASSIGN(c->put_hevc_qpel, 1, 1, qpel_hv, _i8mm); NEON8_FNASSIGN(c->put_hevc_qpel_uni, 1, 1, qpel_uni_hv, _i8mm); diff --git a/libavcodec/aarch64/hevcdsp_epel_neon.S b/libavcodec/aarch64/hevcdsp_epel_neon.S index 54e55cd508..b60091bf23 100644 --- a/libavcodec/aarch64/hevcdsp_epel_neon.S +++ b/libavcodec/aarch64/hevcdsp_epel_neon.S @@ -3212,6 +3212,337 @@ function ff_hevc_put_hevc_epel_uni_w_hv64_8_neon_i8mm, export=1 ret endfunc + +function ff_hevc_put_hevc_epel_bi_hv4_8_neon_i8mm, export=1 + add w10, w5, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3 + mov x2, x3 + add w3, w5, #3 + mov x4, x6 + mov x5, x7 + bl X(ff_hevc_put_hevc_epel_h4_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + load_epel_filterh x7, x6 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.4h}, [sp], x10 + ld1 {v17.4h}, [sp], x10 + ld1 {v18.4h}, [sp], x10 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().4h}, [sp], x10 + calc_epelh v4, \src0, \src1, \src2, \src3 + ld1 {v6.4h}, [x4], x10 + sqadd v4.4h, v4.4h, v6.4h + sqrshrun v4.8b, v4.8h, #7 + subs w5, w5, #1 + st1 {v4.s}[0], [x0], x1 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_hv6_8_neon_i8mm, export=1 + add w10, w5, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3 + mov x2, x3 + add w3, w5, #3 + mov x4, x6 + mov x5, x7 + bl X(ff_hevc_put_hevc_epel_h6_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + load_epel_filterh x7, x6 + sub x1, x1, #4 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8h}, [sp], x10 + ld1 {v17.8h}, [sp], x10 + ld1 {v18.8h}, [sp], x10 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().8h}, [sp], x10 + calc_epelh v4, \src0, \src1, \src2, \src3 + calc_epelh2 v4, v5, \src0, \src1, \src2, \src3 + ld1 {v6.8h}, [x4], x10 + sqadd v4.8h, v4.8h, v6.8h + sqrshrun v4.8b, v4.8h, #7 + st1 {v4.s}[0], [x0], #4 + subs w5, w5, #1 + st1 {v4.h}[2], [x0], x1 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_hv8_8_neon_i8mm, export=1 + add w10, w5, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3 + mov x2, x3 + add w3, w5, #3 + mov x4, x6 + mov x5, x7 + bl X(ff_hevc_put_hevc_epel_h8_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + load_epel_filterh x7, x6 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8h}, [sp], x10 + ld1 {v17.8h}, [sp], x10 + ld1 {v18.8h}, [sp], x10 +.macro calc src0, src1, src2, src3 + ld1 {\src3\().8h}, [sp], x10 + calc_epelh v4, \src0, \src1, \src2, \src3 + calc_epelh2 v4, v5, \src0, \src1, \src2, \src3 + ld1 {v6.8h}, [x4], x10 + sqadd v4.8h, v4.8h, v6.8h + sqrshrun v4.8b, v4.8h, #7 + subs w5, w5, #1 + st1 {v4.8b}, [x0], x1 +.endm +1: calc_all4 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_hv12_8_neon_i8mm, export=1 + add w10, w5, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3 + mov x2, x3 + add w3, w5, #3 + mov x4, x6 + mov x5, x7 + bl X(ff_hevc_put_hevc_epel_h12_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + load_epel_filterh x7, x6 + sub x1, x1, #8 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8h, v17.8h}, [sp], x10 + ld1 {v18.8h, v19.8h}, [sp], x10 + ld1 {v20.8h, v21.8h}, [sp], x10 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\src6\().8h, \src7\().8h}, [sp], x10 + calc_epelh v4, \src0, \src2, \src4, \src6 + calc_epelh2 v4, v5, \src0, \src2, \src4, \src6 + calc_epelh v5, \src1, \src3, \src5, \src7 + ld1 {v6.8h, v7.8h}, [x4], x10 + sqadd v4.8h, v4.8h, v6.8h + sqadd v5.8h, v5.8h, v7.8h + sqrshrun v4.8b, v4.8h, #7 + sqrshrun2 v4.16b, v5.8h, #7 + st1 {v4.8b}, [x0], #8 + subs w5, w5, #1 + st1 {v4.s}[2], [x0], x1 +.endm +1: calc_all8 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_hv16_8_neon_i8mm, export=1 + add w10, w5, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3 + mov x2, x3 + add w3, w5, #3 + mov x4, x6 + mov x5, x7 + bl X(ff_hevc_put_hevc_epel_h16_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + load_epel_filterh x7, x6 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8h, v17.8h}, [sp], x10 + ld1 {v18.8h, v19.8h}, [sp], x10 + ld1 {v20.8h, v21.8h}, [sp], x10 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\src6\().8h, \src7\().8h}, [sp], x10 + calc_epelh v4, \src0, \src2, \src4, \src6 + calc_epelh2 v4, v5, \src0, \src2, \src4, \src6 + calc_epelh v5, \src1, \src3, \src5, \src7 + calc_epelh2 v5, v6, \src1, \src3, \src5, \src7 + ld1 {v6.8h, v7.8h}, [x4], x10 + sqadd v4.8h, v4.8h, v6.8h + sqadd v5.8h, v5.8h, v7.8h + sqrshrun v4.8b, v4.8h, #7 + sqrshrun2 v4.16b, v5.8h, #7 + st1 {v4.16b}, [x0], x1 + subs w5, w5, #1 +.endm +1: calc_all8 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_hv24_8_neon_i8mm, export=1 + add w10, w5, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3 + mov x2, x3 + add w3, w5, #3 + mov x4, x6 + mov x5, x7 + bl X(ff_hevc_put_hevc_epel_h24_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + load_epel_filterh x7, x6 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8h, v17.8h, v18.8h}, [sp], x10 + ld1 {v19.8h, v20.8h, v21.8h}, [sp], x10 + ld1 {v22.8h, v23.8h, v24.8h}, [sp], x10 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11 + ld1 {\src9\().8h, \src10\().8h, \src11\().8h}, [sp], x10 + calc_epelh v1, \src0, \src3, \src6, \src9 + calc_epelh2 v1, v2, \src0, \src3, \src6, \src9 + calc_epelh v2, \src1, \src4, \src7, \src10 + calc_epelh2 v2, v3, \src1, \src4, \src7, \src10 + calc_epelh v3, \src2, \src5, \src8, \src11 + calc_epelh2 v3, v4, \src2, \src5, \src8, \src11 + ld1 {v4.8h, v5.8h, v6.8h}, [x4], x10 + sqadd v1.8h, v1.8h, v4.8h + sqadd v2.8h, v2.8h, v5.8h + sqadd v3.8h, v3.8h, v6.8h + sqrshrun v1.8b, v1.8h, #7 + sqrshrun v2.8b, v2.8h, #7 + sqrshrun v3.8b, v3.8h, #7 + subs w5, w5, #1 + st1 {v1.8b, v2.8b, v3.8b}, [x0], x1 +.endm +1: calc_all12 +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_epel_bi_hv32_8_neon_i8mm, export=1 + sub sp, sp, #16 + str d8, [sp] + add w10, w5, #3 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3 + mov x2, x3 + add w3, w5, #3 + mov x4, x6 + mov x5, x7 + bl X(ff_hevc_put_hevc_epel_h32_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + load_epel_filterh x7, x6 + mov x10, #(MAX_PB_SIZE * 2) + ld1 {v16.8h, v17.8h, v18.8h, v19.8h}, [sp], x10 + ld1 {v20.8h, v21.8h, v22.8h, v23.8h}, [sp], x10 + ld1 {v24.8h, v25.8h, v26.8h, v27.8h}, [sp], x10 +.macro calc src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11, src12, src13, src14, src15 + ld1 {\src12\().8h, \src13\().8h, \src14\().8h, \src15\().8h}, [sp], x10 + calc_epelh v1, \src0, \src4, \src8, \src12 + calc_epelh2 v1, v2, \src0, \src4, \src8, \src12 + calc_epelh v2, \src1, \src5, \src9, \src13 + calc_epelh2 v2, v3, \src1, \src5, \src9, \src13 + calc_epelh v3, \src2, \src6, \src10, \src14 + calc_epelh2 v3, v4, \src2, \src6, \src10, \src14 + calc_epelh v4, \src3, \src7, \src11, \src15 + calc_epelh2 v4, v5, \src3, \src7, \src11, \src15 + ld1 {v5.8h, v6.8h, v7.8h, v8.8h}, [x4], x10 + sqadd v1.8h, v1.8h, v5.8h + sqadd v2.8h, v2.8h, v6.8h + sqadd v3.8h, v3.8h, v7.8h + sqadd v4.8h, v4.8h, v8.8h + sqrshrun v1.8b, v1.8h, #7 + sqrshrun v2.8b, v2.8h, #7 + sqrshrun v3.8b, v3.8h, #7 + sqrshrun v4.8b, v4.8h, #7 + st1 {v1.8b, v2.8b, v3.8b, v4.8b}, [x0], x1 + subs w5, w5, #1 +.endm +1: calc_all16 +.purgem calc +2: ldr d8, [sp], #16 + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_hv48_8_neon_i8mm, export=1 + stp x6, x7, [sp, #-80]! + stp x4, x5, [sp, #16] + stp x2, x3, [sp, #32] + stp x0, x1, [sp, #48] + str x30, [sp, #64] + bl X(ff_hevc_put_hevc_epel_bi_hv24_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x2, x3, [sp, #32] + ldp x0, x1, [sp, #48] + ldp x6, x7, [sp], #64 + add x0, x0, #24 + add x2, x2, #24 + add x4, x4, #48 + bl X(ff_hevc_put_hevc_epel_bi_hv24_8_neon_i8mm) + ldr x30, [sp], #16 + ret +endfunc + +function ff_hevc_put_hevc_epel_bi_hv64_8_neon_i8mm, export=1 + stp x6, x7, [sp, #-80]! + stp x4, x5, [sp, #16] + stp x2, x3, [sp, #32] + stp x0, x1, [sp, #48] + str x30, [sp, #64] + bl X(ff_hevc_put_hevc_epel_bi_hv32_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x2, x3, [sp, #32] + ldp x0, x1, [sp, #48] + ldp x6, x7, [sp], #64 + add x0, x0, #32 + add x2, x2, #32 + add x4, x4, #64 + bl X(ff_hevc_put_hevc_epel_bi_hv32_8_neon_i8mm) + ldr x30, [sp], #16 + ret +endfunc + DISABLE_I8MM #endif diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index a0f0f072f8..da5e23575d 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -168,6 +168,10 @@ NEON8_FNPROTO(epel_bi_v, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *src, ptrdiff_t srcstride, const int16_t *src2, int height, intptr_t mx, intptr_t my, int width),); +NEON8_FNPROTO(epel_bi_hv, (uint8_t *dst, ptrdiff_t dststride, + const uint8_t *src, ptrdiff_t srcstride, const int16_t *src2, + int height, intptr_t mx, intptr_t my, int width), _i8mm); + NEON8_FNPROTO(epel_v, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *_src, ptrdiff_t _srcstride, const int16_t *src2, int height, intptr_t mx, intptr_t my, int width),); @@ -354,6 +358,7 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) NEON8_FNASSIGN(c->put_hevc_epel, 1, 1, epel_hv, _i8mm); NEON8_FNASSIGN(c->put_hevc_epel_uni, 1, 1, epel_uni_hv, _i8mm); NEON8_FNASSIGN(c->put_hevc_epel_uni_w, 0, 1, epel_uni_w_h ,_i8mm); + NEON8_FNASSIGN(c->put_hevc_epel_bi, 1, 1, epel_bi_hv, _i8mm); From patchwork Sat Nov 18 02:07:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Logan.Lyu" X-Patchwork-Id: 44705 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6a89:b0:181:818d:5e7f with SMTP id bi9csp293408pzb; Fri, 17 Nov 2023 18:07:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IGsGwd5ROrUQGIOyXftD6A7EcL3Kukky+3VdDqhVY9uaDC8y4opc6Fo9BLAquDMSCIsBKTS X-Received: by 2002:a05:651c:54c:b0:2c5:509:c080 with SMTP id q12-20020a05651c054c00b002c50509c080mr947399ljp.3.1700273264595; Fri, 17 Nov 2023 18:07:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700273264; cv=none; d=google.com; s=arc-20160816; b=Lznob/B3uB8Ow8H5Uvxe/sI9WJmP/Xck/f1FhMPqNICRD2JhhdJnWGkIyEMIz4IvMM sp6umKXkPGfQbg0XBzqsj3iggDCYvd0goWIEMlvPyHElJcXEL8IJBPYG5xwX147N5IQB svcERwg9CYmemYvzB9iIZeVwFmotQxvGPqz7yjMnQt8djXZ/yEo1CQ0+4eOB6H6Ur7C9 wwa4pHpqbcHx0ZS1/hSqozA/91pp7A+d610zcK1QFcvKPmP9KR2h5U3327tct67GMTWC pG6n6hpWZbQ9yhdLyDqgbX/HcfzeHLIXjXH8+YSTHKzm1QZmhvkg/Xrs5nbivT2cWy2m BRXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :organization:to:from:user-agent:mime-version:date:message-id :delivered-to; bh=jY/0Y5Oma9mPdtCN5H1fh6DKZbWP2EnEgfr+ttd7P4k=; fh=4VBelKDE4DH3L7jF6H/1Jmu78FdN+YP76yfdJCQTJ30=; b=HGlOhyoTP6shAtJotd7xGlfqLvRG7BVCj2v+KwScHxeIJTpysaqCRmnLLjTN37yOm7 ib9wmFf9zZNDX6dk2d4gQsOO0GMEdrBF/UgZkfbJ+gQPPPelekCBGz9GgQOjtS6gTpBj NqNCx6O+rSRP6MqXVPNnbAKEeVOKRzyeC60g1tY3+ELqWujF1dDDGNseFGxIa5ODXhjK 9U8OZkbYGVDOCrE2QxBAjzuZ/YKiXHtY0FyW68oFk7y8mXnsCtsxWeAxyrjVb61AYw4l J2abJKFmduOFrYbH3sTg5zoHtgi7+qXAGgYoQOftcLNJ2J9cHz31AxNwnQWsVxKX9zvz Kx2g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s18-20020a170906501200b009929566f00asi1587208ejj.467.2023.11.17.18.07.44; Fri, 17 Nov 2023 18:07:44 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 50F3368CD85; Sat, 18 Nov 2023 04:07:16 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-my3-01p4.yunyou.top (smtp-my3-01p4.yunyou.top [60.247.169.4]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 29B6968C96C for ; Sat, 18 Nov 2023 04:07:09 +0200 (EET) Received: from [192.168.15.105] (unknown [122.233.146.151]) by smtp-my-01.yunyou.top (WestCloudMail) with ESMTPA id 55DB7141F28; Sat, 18 Nov 2023 10:07:04 +0800 (CST) Message-ID: <9dc635d0-a01d-43a8-aeff-bec37d6143a1@myais.com.cn> Date: Sat, 18 Nov 2023 10:07:04 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Logan.Lyu" To: ffmpeg-devel@ffmpeg.org Organization: myais Subject: [FFmpeg-devel] [PATCH 5/6] lavc/aarch64: new optimization for 8-bit hevc_qpel_bi_v X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: jdek@itanimul.li Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: oaTmVa1SsgAs put_hevc_qpel_bi_v4_8_c: 166.1 put_hevc_qpel_bi_v4_8_neon: 61.9 put_hevc_qpel_bi_v6_8_c: 309.4 put_hevc_qpel_bi_v6_8_neon: 75.6 put_hevc_qpel_bi_v8_8_c: 531.1 put_hevc_qpel_bi_v8_8_neon: 78.1 put_hevc_qpel_bi_v12_8_c: 1139.9 put_hevc_qpel_bi_v12_8_neon: 238.1 put_hevc_qpel_bi_v16_8_c: 2063.6 put_hevc_qpel_bi_v16_8_neon: 308.9 put_hevc_qpel_bi_v24_8_c: 4317.1 put_hevc_qpel_bi_v24_8_neon: 629.9 put_hevc_qpel_bi_v32_8_c: 8241.9 put_hevc_qpel_bi_v32_8_neon: 1140.1 put_hevc_qpel_bi_v48_8_c: 18422.9 put_hevc_qpel_bi_v48_8_neon: 2533.9 put_hevc_qpel_bi_v64_8_c: 37508.6 put_hevc_qpel_bi_v64_8_neon: 4520.1 Co-Authored-By: J. Dekker Signed-off-by: Logan Lyu --- libavcodec/aarch64/hevcdsp_init_aarch64.c | 5 + libavcodec/aarch64/hevcdsp_qpel_neon.S | 248 ++++++++++++++++++++++ 2 files changed, 253 insertions(+) + bl X(ff_hevc_put_hevc_qpel_bi_v16_8_neon) + ldr x30, [sp, #8] + add sp, sp, #16 + ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_v64_8_neon, export=1 + b X(ff_hevc_put_hevc_qpel_bi_v32_8_neon) +endfunc function ff_hevc_put_hevc_pel_uni_pixels4_8_neon, export=1 1: diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index da5e23575d..f2cce3d221 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -251,6 +251,10 @@ NEON8_FNPROTO_PARTIAL_5(qpel_uni_w_hv, (uint8_t *_dst, ptrdiff_t _dststride, int height, int denom, int wx, int ox, intptr_t mx, intptr_t my, int width), _i8mm); +NEON8_FNPROTO(qpel_bi_v, (uint8_t *dst, ptrdiff_t dststride, + const uint8_t *src, ptrdiff_t srcstride, const int16_t *src2, + int height, intptr_t mx, intptr_t my, int width),); + #define NEON8_FNASSIGN(member, v, h, fn, ext) \ member[1][v][h] = ff_hevc_put_hevc_##fn##4_8_neon##ext; \ member[2][v][h] = ff_hevc_put_hevc_##fn##6_8_neon##ext; \ @@ -344,6 +348,7 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) NEON8_FNASSIGN(c->put_hevc_epel_bi, 0, 1, epel_bi_h,); NEON8_FNASSIGN(c->put_hevc_epel_bi, 1, 0, epel_bi_v,); NEON8_FNASSIGN(c->put_hevc_qpel_bi, 0, 0, pel_bi_pixels,); + NEON8_FNASSIGN(c->put_hevc_qpel_bi, 1, 0, qpel_bi_v,); NEON8_FNASSIGN(c->put_hevc_epel_uni, 0, 0, pel_uni_pixels,); NEON8_FNASSIGN(c->put_hevc_epel_uni, 1, 0, epel_uni_v,); NEON8_FNASSIGN(c->put_hevc_qpel_uni, 0, 0, pel_uni_pixels,); diff --git a/libavcodec/aarch64/hevcdsp_qpel_neon.S b/libavcodec/aarch64/hevcdsp_qpel_neon.S index 8e257208e4..f9422942d8 100644 --- a/libavcodec/aarch64/hevcdsp_qpel_neon.S +++ b/libavcodec/aarch64/hevcdsp_qpel_neon.S @@ -866,6 +866,254 @@ function ff_hevc_put_hevc_qpel_v64_8_neon, export=1 ret endfunc +function ff_hevc_put_hevc_qpel_bi_v4_8_neon, export=1 + load_qpel_filterb x7, x6 + sub x2, x2, x3, lsl #1 + sub x2, x2, x3 + mov x12, #(MAX_PB_SIZE * 2) + ld1 {v16.s}[0], [x2], x3 + ld1 {v17.s}[0], [x2], x3 + ld1 {v18.s}[0], [x2], x3 + ld1 {v19.s}[0], [x2], x3 + ld1 {v20.s}[0], [x2], x3 + ld1 {v21.s}[0], [x2], x3 + ld1 {v22.s}[0], [x2], x3 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().s}[0], [x2], x3 + movi v24.8h, #0 + calc_qpelb v24, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + ld1 {v25.4h}, [x4], x12 // src2 + sqadd v24.8h, v24.8h, v25.8h + sqrshrun v25.8b, v24.8h, #7 + subs w5, w5, #1 + st1 {v25.s}[0], [x0], x1 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_v6_8_neon, export=1 + load_qpel_filterb x7, x6 + sub x2, x2, x3, lsl #1 + sub x2, x2, x3 + ld1 {v16.8b}, [x2], x3 + sub x1, x1, #4 + ld1 {v17.8b}, [x2], x3 + mov x12, #(MAX_PB_SIZE * 2) + ld1 {v18.8b}, [x2], x3 + ld1 {v19.8b}, [x2], x3 + ld1 {v20.8b}, [x2], x3 + ld1 {v21.8b}, [x2], x3 + ld1 {v22.8b}, [x2], x3 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().8b}, [x2], x3 + movi v24.8h, #0 + calc_qpelb v24, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + ld1 {v25.8h}, [x4], x12 // src2 + sqadd v24.8h, v24.8h, v25.8h + sqrshrun v25.8b, v24.8h, #7 + st1 {v25.s}[0], [x0], #4 + subs w5, w5, #1 + st1 {v25.h}[2], [x0], x1 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_v8_8_neon, export=1 + load_qpel_filterb x7, x6 + sub x2, x2, x3, lsl #1 + sub x2, x2, x3 + mov x12, #(MAX_PB_SIZE * 2) + ld1 {v16.8b}, [x2], x3 + ld1 {v17.8b}, [x2], x3 + ld1 {v18.8b}, [x2], x3 + ld1 {v19.8b}, [x2], x3 + ld1 {v20.8b}, [x2], x3 + ld1 {v21.8b}, [x2], x3 + ld1 {v22.8b}, [x2], x3 + .macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().8b}, [x2], x3 + movi v24.8h, #0 + calc_qpelb v24, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + ld1 {v25.8h}, [x4], x12 // src2 + sqadd v24.8h, v24.8h, v25.8h + sqrshrun v25.8b, v24.8h, #7 + subs w5, w5, #1 + st1 {v25.8b}, [x0], x1 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_v12_8_neon, export=1 + load_qpel_filterb x7, x6 + sub x2, x2, x3, lsl #1 + sub x2, x2, x3 + sub x1, x1, #8 + ld1 {v16.16b}, [x2], x3 + mov x12, #(MAX_PB_SIZE * 2) + ld1 {v17.16b}, [x2], x3 + ld1 {v18.16b}, [x2], x3 + ld1 {v19.16b}, [x2], x3 + ld1 {v20.16b}, [x2], x3 + ld1 {v21.16b}, [x2], x3 + ld1 {v22.16b}, [x2], x3 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().16b}, [x2], x3 + movi v24.8h, #0 + movi v25.8h, #0 + calc_qpelb v24, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb2 v25, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + ld1 {v26.8h, v27.8h}, [x4], x12 // src2 + sqadd v24.8h, v24.8h, v26.8h + sqadd v25.8h, v25.8h, v27.8h + sqrshrun v26.8b, v24.8h, #7 + sqrshrun2 v26.16b, v25.8h, #7 + st1 {v26.8b}, [x0], #8 + subs w5, w5, #1 + st1 {v26.s}[2], [x0], x1 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_v16_8_neon, export=1 + load_qpel_filterb x7, x6 + sub x2, x2, x3, lsl #1 + sub x2, x2, x3 + mov x12, #(MAX_PB_SIZE * 2) + ld1 {v16.16b}, [x2], x3 + ld1 {v17.16b}, [x2], x3 + ld1 {v18.16b}, [x2], x3 + ld1 {v19.16b}, [x2], x3 + ld1 {v20.16b}, [x2], x3 + ld1 {v21.16b}, [x2], x3 + ld1 {v22.16b}, [x2], x3 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().16b}, [x2], x3 + movi v24.8h, #0 + movi v25.8h, #0 + calc_qpelb v24, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb2 v25, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + ld1 {v26.8h, v27.8h}, [x4], x12 // src2 + sqadd v24.8h, v24.8h, v26.8h + sqadd v25.8h, v25.8h, v27.8h + sqrshrun v26.8b, v24.8h, #7 + subs w5, w5, #1 + sqrshrun2 v26.16b, v25.8h, #7 + st1 {v26.16b}, [x0], x1 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_v24_8_neon, export=1 + stp x4, x5, [sp, #-64]! + stp x2, x3, [sp, #16] + stp x0, x1, [sp, #32] + stp x7, x30, [sp, #48] + bl X(ff_hevc_put_hevc_qpel_bi_v16_8_neon) + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp, #32] + ldr x7, [sp, #48] + ldp x4, x5, [sp], #48 + add x0, x0, #16 + add x2, x2, #16 + add x4, x4, #32 + bl X(ff_hevc_put_hevc_qpel_bi_v8_8_neon) + ldr x30, [sp, #8] + add sp, sp, #16 + ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_v32_8_neon, export=1 + stp d8, d9, [sp, #-64]! + stp d10, d11, [sp, #16] + stp d12, d13, [sp, #32] + stp d14, d15, [sp, #48] + sub x2, x2, x3, lsl #1 + sub x2, x2, x3 + load_qpel_filterb x7, x6 + ldr w6, [sp, #64] + mov x12, #(MAX_PB_SIZE * 2) +0: mov x8, x2 // src + ld1 {v16.16b, v17.16b}, [x8], x3 + mov w11, w5 // height + ld1 {v18.16b, v19.16b}, [x8], x3 + mov x10, x0 // dst + ld1 {v20.16b, v21.16b}, [x8], x3 + mov x9, x4 // src2 + ld1 {v22.16b, v23.16b}, [x8], x3 + ld1 {v24.16b, v25.16b}, [x8], x3 + ld1 {v26.16b, v27.16b}, [x8], x3 + ld1 {v28.16b, v29.16b}, [x8], x3 +.macro calc tmp0, tmp1, src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11, src12, src13, src14, src15 + ld1 {\tmp0\().8h, \tmp1\().8h}, [x8], x3 + movi v8.8h, #0 + movi v9.8h, #0 + movi v10.8h, #0 + movi v11.8h, #0 + calc_qpelb v8, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb2 v9, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7 + calc_qpelb v10, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15 + calc_qpelb2 v11, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15 + ld1 {v12.8h, v13.8h, v14.8h, v15.8h}, [x9], x12 // src2 + sqadd v8.8h, v8.8h, v12.8h + sqadd v9.8h, v9.8h, v13.8h + sqadd v10.8h, v10.8h, v14.8h + sqadd v11.8h, v11.8h, v15.8h + sqrshrun v12.8b, v8.8h, #7 + sqrshrun2 v12.16b, v9.8h, #7 + sqrshrun v13.8b, v10.8h, #7 + sqrshrun2 v13.16b, v11.8h, #7 + subs x11, x11, #1 + st1 {v12.16b, v13.16b}, [x10], x1 +.endm +1: calc_all2 +.purgem calc +2: add x0, x0, #32 // dst + add x2, x2, #32 // src + add x4, x4, #64 // src2 + subs w6, w6, #32 + b.ne 0b + ldp d10, d11, [sp, #16] + ldp d12, d13, [sp, #32] + ldp d14, d15, [sp, #48] + ldp d8, d9, [sp], #64 + ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_v48_8_neon, export=1 + mov x8, #32 + str x8, [sp, #-80]! + stp x4, x5, [sp, #16] + stp x2, x3, [sp, #32] + stp x0, x1, [sp, #48] + stp x7, x30, [sp, #64] + bl X(ff_hevc_put_hevc_qpel_bi_v32_8_neon) + ldp x4, x5, [sp, #16] + ldp x2, x3, [sp, #32] + ldp x0, x1, [sp, #48] + ldr x7, [sp, #64] + add sp, sp, #64 + add x0, x0, #32 + add x2, x2, #32 + add x4, x4, #64 From patchwork Sat Nov 18 02:07:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Logan.Lyu" X-Patchwork-Id: 44706 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6a89:b0:181:818d:5e7f with SMTP id bi9csp293457pzb; Fri, 17 Nov 2023 18:07:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IGiA+tqWVc5+qkSZiyz+SI3Mjt6E7wk9Iz8rXY6sek5xl5hdN5B7sBwYF2YGC8t3XEU4Uzf X-Received: by 2002:adf:ab0a:0:b0:32d:a469:a1b with SMTP id q10-20020adfab0a000000b0032da4690a1bmr477221wrc.62.1700273275838; Fri, 17 Nov 2023 18:07:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700273275; cv=none; d=google.com; s=arc-20160816; b=ExAwC4AhQLJ+2DDObj3EZdKugl52hk2tMGMp5PTKr0GTFX6KWOeWohJoyWW94RCFsz R1tQ/F5F8/H3C5LXH7FxUewI9TND80B739TozotuTVjOXGbJb/+jFP5sjs8cuHrI5/Wo 1LYXyKIurgwjCIjK20xiX675BgB7fHdp1vHoTazlvl/fIkI1YM9KlX8OXwGMHj8vPUsi QRIeIoWaBtwM+L7pJUgYhyWdzMn1CUjuCLdWcDXeITADrAvslJbAJA0Acyjs29/I6vOg Jt6SMqTUOS/wS1Ht2PqHbvUu3Jh3iMZlHvaQex/APrfvImXGkLMrS1oymbw5BEvbaNgH RvHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :organization:to:from:user-agent:mime-version:date:message-id :delivered-to; bh=A6DkNF5xNnXXyEPuQstgwugb0nB9OFvAdsQ2EeNlQZQ=; fh=4VBelKDE4DH3L7jF6H/1Jmu78FdN+YP76yfdJCQTJ30=; b=bTMrofhev8jiEUWhjsWtkHXhpb4YzH04G6ALrHSKMKZinVvp3FSfC2taA/ijb5VVXu pCd9dMU1XTvuuW2pJHKTBjqBFUNa+ppqw8ZmtL8PNdUw5hUjpdO8CVAoJWAYN+Fb+h3X Hn0dYLwLdGTc9cB7gt8GnOb6UquO+nFg3RQ53/3q/1y66+uqhZ2ASo7WcKOBfvQTXZw0 x5FkI9EeQL4r3jNoHQg5z6aFo98AS7Z1tA/2uNIKnL3eL/Ai+tftkqALta0jIAxWVCvw qBPxUj6d2/Ju29r5HcUudc13/clC0UdCQXD1Ubr35tXnWF4nbj/jEqyBp5qTp7Vz0mDS c2kg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id n15-20020a05640206cf00b005455445042bsi1610301edy.427.2023.11.17.18.07.55; Fri, 17 Nov 2023 18:07:55 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6BC2C68CD8F; Sat, 18 Nov 2023 04:07:20 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-my3-01p7.yunyou.top (smtp-my3-01p7.yunyou.top [60.247.169.7]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2158168C96C for ; Sat, 18 Nov 2023 04:07:14 +0200 (EET) Received: from [192.168.15.105] (unknown [122.233.146.151]) by smtp-my-01.yunyou.top (WestCloudMail) with ESMTPA id 657B2141F2B; Sat, 18 Nov 2023 10:07:09 +0800 (CST) Message-ID: <406b8110-558e-497c-823e-0598fa954c4e@myais.com.cn> Date: Sat, 18 Nov 2023 10:07:09 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Logan.Lyu" To: ffmpeg-devel@ffmpeg.org Organization: myais Subject: [FFmpeg-devel] [PATCH 6/6] lavc/aarch64: new optimization for 8-bit hevc_qpel_bi_hv X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: jdek@itanimul.li Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: KfQxWx/aTwZH put_hevc_qpel_bi_hv4_8_c: 433.7 put_hevc_qpel_bi_hv4_8_i8mm: 117.9 put_hevc_qpel_bi_hv6_8_c: 803.9 put_hevc_qpel_bi_hv6_8_i8mm: 252.7 put_hevc_qpel_bi_hv8_8_c: 1296.4 put_hevc_qpel_bi_hv8_8_i8mm: 316.2 put_hevc_qpel_bi_hv12_8_c: 2867.4 put_hevc_qpel_bi_hv12_8_i8mm: 669.2 put_hevc_qpel_bi_hv16_8_c: 4709.4 put_hevc_qpel_bi_hv16_8_i8mm: 929.9 put_hevc_qpel_bi_hv24_8_c: 9639.7 put_hevc_qpel_bi_hv24_8_i8mm: 2072.4 put_hevc_qpel_bi_hv32_8_c: 16663.7 put_hevc_qpel_bi_hv32_8_i8mm: 3391.4 put_hevc_qpel_bi_hv48_8_c: 36972.9 put_hevc_qpel_bi_hv48_8_i8mm: 7505.7 put_hevc_qpel_bi_hv64_8_c: 64106.4 put_hevc_qpel_bi_hv64_8_i8mm: 13145.2 Co-Authored-By: J. Dekker Signed-off-by: Logan Lyu --- libavcodec/aarch64/hevcdsp_init_aarch64.c | 5 + libavcodec/aarch64/hevcdsp_qpel_neon.S | 299 ++++++++++++++++++++++ 2 files changed, 304 insertions(+) + bl X(ff_hevc_put_hevc_qpel_h64_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + mov x6, #64 // width + b .Lqpel_bi_hv16_loop +endfunc + DISABLE_I8MM #endif // HAVE_I8MM diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index f2cce3d221..cfc2f39a6b 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -255,6 +255,10 @@ NEON8_FNPROTO(qpel_bi_v, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *src, ptrdiff_t srcstride, const int16_t *src2, int height, intptr_t mx, intptr_t my, int width),); +NEON8_FNPROTO(qpel_bi_hv, (uint8_t *dst, ptrdiff_t dststride, + const uint8_t *src, ptrdiff_t srcstride, const int16_t *src2, + int height, intptr_t mx, intptr_t my, int width), _i8mm); + #define NEON8_FNASSIGN(member, v, h, fn, ext) \ member[1][v][h] = ff_hevc_put_hevc_##fn##4_8_neon##ext; \ member[2][v][h] = ff_hevc_put_hevc_##fn##6_8_neon##ext; \ @@ -370,6 +374,7 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) NEON8_FNASSIGN(c->put_hevc_qpel_uni_w, 0, 1, qpel_uni_w_h, _i8mm); NEON8_FNASSIGN(c->put_hevc_epel_uni_w, 1, 1, epel_uni_w_hv, _i8mm); NEON8_FNASSIGN_PARTIAL_5(c->put_hevc_qpel_uni_w, 1, 1, qpel_uni_w_hv, _i8mm); + NEON8_FNASSIGN(c->put_hevc_qpel_bi, 1, 1, qpel_bi_hv, _i8mm); } } diff --git a/libavcodec/aarch64/hevcdsp_qpel_neon.S b/libavcodec/aarch64/hevcdsp_qpel_neon.S index f9422942d8..56bc352648 100644 --- a/libavcodec/aarch64/hevcdsp_qpel_neon.S +++ b/libavcodec/aarch64/hevcdsp_qpel_neon.S @@ -4210,5 +4210,304 @@ function ff_hevc_put_hevc_qpel_uni_w_hv64_8_neon_i8mm, export=1 ret endfunc +function ff_hevc_put_hevc_qpel_bi_hv4_8_neon_i8mm, export=1 + add w10, w5, #7 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + sub x1, x2, x3, lsl #1 + sub x1, x1, x3 + add x0, sp, #48 + mov x2, x3 + add w3, w5, #7 + mov x4, x6 + bl X(ff_hevc_put_hevc_qpel_h4_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + mov x9, #(MAX_PB_SIZE * 2) + load_qpel_filterh x7, x6 + ld1 {v16.4h}, [sp], x9 + ld1 {v17.4h}, [sp], x9 + ld1 {v18.4h}, [sp], x9 + ld1 {v19.4h}, [sp], x9 + ld1 {v20.4h}, [sp], x9 + ld1 {v21.4h}, [sp], x9 + ld1 {v22.4h}, [sp], x9 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().4h}, [sp], x9 + calc_qpelh v1, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sshr + ld1 {v5.4h}, [x4], x9 // src2 + saddw v1.4s, v1.4s, v5.4h + rshrn v1.4h, v1.4s, #7 + sqxtun v1.8b, v1.8h + subs w5, w5, #1 + st1 {v1.s}[0], [x0], x1 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_hv6_8_neon_i8mm, export=1 + add w10, w5, #7 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + sub x1, x2, x3, lsl #1 + sub x1, x1, x3 + add x0, sp, #48 + mov x2, x3 + add x3, x5, #7 + mov x4, x6 + bl X(ff_hevc_put_hevc_qpel_h6_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + mov x9, #(MAX_PB_SIZE * 2) + load_qpel_filterh x7, x6 + sub x1, x1, #4 + ld1 {v16.8h}, [sp], x9 + ld1 {v17.8h}, [sp], x9 + ld1 {v18.8h}, [sp], x9 + ld1 {v19.8h}, [sp], x9 + ld1 {v20.8h}, [sp], x9 + ld1 {v21.8h}, [sp], x9 + ld1 {v22.8h}, [sp], x9 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().8h}, [sp], x9 + calc_qpelh v1, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sshr + calc_qpelh2 v2, v2, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sshr + ld1 {v5.8h}, [x4], x9 // src2 + saddw v1.4s, v1.4s, v5.4h + saddw2 v2.4s, v2.4s, v5.8h + rshrn v1.4h, v1.4s, #7 + rshrn2 v1.8h, v2.4s, #7 + sqxtun v1.8b, v1.8h + st1 {v1.s}[0], [x0], #4 + subs w5, w5, #1 + st1 {v1.h}[2], [x0], x1 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_hv8_8_neon_i8mm, export=1 + add w10, w5, #7 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + sub x1, x2, x3, lsl #1 + sub x1, x1, x3 + add x0, sp, #48 + mov x2, x3 + add x3, x5, #7 + mov x4, x6 + bl X(ff_hevc_put_hevc_qpel_h8_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + mov x9, #(MAX_PB_SIZE * 2) + load_qpel_filterh x7, x6 + ld1 {v16.8h}, [sp], x9 + ld1 {v17.8h}, [sp], x9 + ld1 {v18.8h}, [sp], x9 + ld1 {v19.8h}, [sp], x9 + ld1 {v20.8h}, [sp], x9 + ld1 {v21.8h}, [sp], x9 + ld1 {v22.8h}, [sp], x9 +.macro calc tmp, src0, src1, src2, src3, src4, src5, src6, src7 + ld1 {\tmp\().8h}, [sp], x9 + calc_qpelh v1, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sshr + calc_qpelh2 v2, v2, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sshr + ld1 {v5.8h}, [x4], x9 // src2 + saddw v1.4s, v1.4s, v5.4h + saddw2 v2.4s, v2.4s, v5.8h + rshrn v1.4h, v1.4s, #7 + rshrn2 v1.8h, v2.4s, #7 + sqxtun v1.8b, v1.8h + subs w5, w5, #1 + st1 {v1.8b}, [x0], x1 +.endm +1: calc_all +.purgem calc +2: ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_hv12_8_neon_i8mm, export=1 + stp x6, x7, [sp, #-80]! + stp x4, x5, [sp, #16] + stp x2, x3, [sp, #32] + stp x0, x1, [sp, #48] + str x30, [sp, #64] + bl X(ff_hevc_put_hevc_qpel_bi_hv8_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x2, x3, [sp, #32] + ldp x0, x1, [sp, #48] + ldp x6, x7, [sp], #64 + add x4, x4, #16 + add x2, x2, #8 + add x0, x0, #8 + bl X(ff_hevc_put_hevc_qpel_bi_hv4_8_neon_i8mm) + ldr x30, [sp], #16 + ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_hv16_8_neon_i8mm, export=1 + add w10, w5, #7 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3, lsl #1 + sub x1, x1, x3 + mov x2, x3 + add w3, w5, #7 + mov x4, x6 + bl X(ff_hevc_put_hevc_qpel_h16_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + mov x6, #16 // width +.Lqpel_bi_hv16_loop: + load_qpel_filterh x7, x8 + mov x9, #(MAX_PB_SIZE * 2) + mov x10, x6 +0: mov x8, sp // src + ld1 {v16.8h, v17.8h}, [x8], x9 + mov w11, w5 // height + ld1 {v18.8h, v19.8h}, [x8], x9 + mov x12, x4 // src2 + ld1 {v20.8h, v21.8h}, [x8], x9 + mov x7, x0 // dst + ld1 {v22.8h, v23.8h}, [x8], x9 + ld1 {v24.8h, v25.8h}, [x8], x9 + ld1 {v26.8h, v27.8h}, [x8], x9 + ld1 {v28.8h, v29.8h}, [x8], x9 +.macro calc tmp0, tmp1, src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10, src11, src12, src13, src14, src15 + ld1 {\tmp0\().8h, \tmp1\().8h}, [x8], x9 + calc_qpelh v1, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sshr + calc_qpelh2 v2, v2, \src0, \src1, \src2, \src3, \src4, \src5, \src6, \src7, sshr + calc_qpelh v3, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15, sshr + calc_qpelh2 v4, v4, \src8, \src9, \src10, \src11, \src12, \src13, \src14, \src15, sshr + ld1 {v5.8h, v6.8h}, [x12], x9 // src2 + saddw v1.4s, v1.4s, v5.4h + saddw2 v2.4s, v2.4s, v5.8h + saddw v3.4s, v3.4s, v6.4h + saddw2 v4.4s, v4.4s, v6.8h + rshrn v1.4h, v1.4s, #7 + rshrn2 v1.8h, v2.4s, #7 + rshrn v2.4h, v3.4s, #7 + rshrn2 v2.8h, v4.4s, #7 + sqxtun v1.8b, v1.8h + sqxtun2 v1.16b, v2.8h + subs x11, x11, #1 + st1 {v1.16b}, [x7], x1 +.endm +1: calc_all2 +.purgem calc +2: add x0, x0, #16 + add sp, sp, #32 + subs x10, x10, #16 + add x4, x4, #32 + b.ne 0b + add w10, w5, #7 + lsl x10, x10, #7 + sub x10, x10, x6, lsl #1 // part of first line + add sp, sp, x10 // tmp_array without first line + ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_hv24_8_neon_i8mm, export=1 + stp x6, x7, [sp, #-80]! + stp x4, x5, [sp, #16] + stp x2, x3, [sp, #32] + stp x0, x1, [sp, #48] + str x30, [sp, #64] + bl X(ff_hevc_put_hevc_qpel_bi_hv16_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x2, x3, [sp, #32] + ldp x0, x1, [sp, #48] + ldp x6, x7, [sp], #64 + add x4, x4, #32 + add x2, x2, #16 + add x0, x0, #16 + bl X(ff_hevc_put_hevc_qpel_bi_hv8_8_neon_i8mm) + ldr x30, [sp], #16 + ret +endfunc + +function ff_hevc_put_hevc_qpel_bi_hv32_8_neon_i8mm, export=1 + add w10, w5, #7 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3, lsl #1 + mov x2, x3 + sub x1, x1, x3 + add w3, w5, #7 + mov x4, x6 + bl X(ff_hevc_put_hevc_qpel_h32_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + mov x6, #32 // width + b .Lqpel_bi_hv16_loop +endfunc + +function ff_hevc_put_hevc_qpel_bi_hv48_8_neon_i8mm, export=1 + add w10, w5, #7 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3, lsl #1 + mov x2, x3 + sub x1, x1, x3 + add w3, w5, #7 + mov x4, x6 + bl X(ff_hevc_put_hevc_qpel_h48_8_neon_i8mm) + ldp x4, x5, [sp, #16] + ldp x0, x1, [sp, #32] + ldp x7, x30, [sp], #48 + mov x6, #48 // width + b .Lqpel_bi_hv16_loop +endfunc + +function ff_hevc_put_hevc_qpel_bi_hv64_8_neon_i8mm, export=1 + add w10, w5, #7 + lsl x10, x10, #7 + sub sp, sp, x10 // tmp_array + stp x7, x30, [sp, #-48]! + stp x4, x5, [sp, #16] + stp x0, x1, [sp, #32] + add x0, sp, #48 + sub x1, x2, x3, lsl #1 + mov x2, x3 + sub x1, x1, x3 + add w3, w5, #7 + mov x4, x6