From patchwork Mon Mar 25 15:02:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Martin_Storsj=C3=B6?= X-Patchwork-Id: 47445 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:c889:b0:1a3:b6bb:3029 with SMTP id hb9csp1249055pzb; Mon, 25 Mar 2024 08:06:18 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUmTpWjS0ZSVEXqFOM6ptD3yNKV71njUz7+dWGbmeJw1T6UzflwhH6as7SNcZzs5MTuC8ERceDSpSb+4S9xzoQyI4H/p2AzlGkZ8g== X-Google-Smtp-Source: AGHT+IFN6vCCzFKdZV7by62/qNiOo4yoRXplR99E+WoeOAcA2nGBbmLkGfrgZws2tNlfejwMnHiv X-Received: by 2002:a50:8701:0:b0:568:b46c:c4ba with SMTP id i1-20020a508701000000b00568b46cc4bamr4527793edb.30.1711379178658; Mon, 25 Mar 2024 08:06:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1711379178; cv=none; d=google.com; s=arc-20160816; b=YXQFGYTxfCo2SgKNMKl8qzPS3gytTMSjW5GhU9ZnuePNvaC4m4lWGMAko+1ugaRfDT 0QhHoD1O9ldw4GtPjgqv4xvuOgYeVtkniZ1Bw67kav7TbsHMsY0wXNixIjrdyllQsGAf P3gdF1loIh7Z+OC47cLuJWaEoaykk0TC+RaLGNd4FcVST9vnudAaq4ruErTcpSFLe84t 5k4081TBx9CADahL9Xa8ZARh+jxdmt6rBuUaOBTA0Y82Gd1Zdzrx9EEJt9wn2w79XenU fl2bsjF18KO0vFN7y6r+JuzeY9kb0QQe6hRGXh/hvOGcAH4GPlgcR8edX2rvIAWSIHMm zvRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=sSWXMI8NE7gdVtGc1Ob/Gy5u9ewBcaAzrFV3zHpHLU8=; fh=DRs4GoYGH1Ee5ERfKSLc+LBtFKtorfi5XrD3r13LPeM=; b=S7sztLsY3rjtgKcTinqTrc7BB3xUvMTqbhPWWpRZ0eMJcM3WDpnTmY5FwRu3EDKmae 0eksKKE14fObY09YyljgodEsfT03TErwFbMdsx/trUutdij6G4lTE9IXWNkeWiB+CsrM rUw8hmU+dbXif/fEeKmorcy5gfr3dZnpoOjH90nY0UfAoA8hLXL4JzkYAYDAux22c1/H Gb1gFuET+8lUHOonqaOyixoMAMoJEFhZqBm314PBvxVgW/6XKchRzn7zlIdtddc7E0SZ 9refdvdO8QeQxH8x3c9Nmg4bfl7amR9ut9uWHsHDI1/kqbugTf93bFG72Oo9qmAnINJK Bv5w==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@martin-st.20230601.gappssmtp.com header.s=20230601 header.b=NRtVfHGs; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id u4-20020aa7db84000000b0056bbf8ca1d7si2600722edt.491.2024.03.25.08.06.18; Mon, 25 Mar 2024 08:06:18 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@martin-st.20230601.gappssmtp.com header.s=20230601 header.b=NRtVfHGs; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 33A1B68D614; Mon, 25 Mar 2024 17:03:15 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 741D568D567 for ; Mon, 25 Mar 2024 17:03:00 +0200 (EET) Received: by mail-lj1-f169.google.com with SMTP id 38308e7fff4ca-2d21cdbc85bso60877911fa.2 for ; Mon, 25 Mar 2024 08:03:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=martin-st.20230601.gappssmtp.com; s=20230601; t=1711378979; x=1711983779; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DhU/sZptSP5fBOOMxykK1agjuyo37t10vtOZoa4J0wQ=; b=NRtVfHGsvFQsgtjZM8L8EhNeEUMxQmgdUa/P98Y+aWwvaQoEOASFh+SV0BwO5IF7eT YOPjliD3sow3hGC3y/FC2QPd34G6uGY6+/0ycOrMUfYEuN7uZTdkSm4O1rxavYzOIFUe wjfPy/wmX4USInEPRP5xmFtc3z0ZYesUC/8o4YtPWq9uyE7VqGt0fBv+p98LIGBlx/oH c71kS+xmhxzmpOkrY/d5g4lPl4PkcMDKz1Y2fPrcvFWiKtbaV2u7q8hNs80YCaQGtsoq laN0T4U4Kv+WqDoX9eqKO4AniM8ZxlWzN5sxCi9kMYKB0YiTtqsRdo0nIRhV0onbTEZ9 P+CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711378979; x=1711983779; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DhU/sZptSP5fBOOMxykK1agjuyo37t10vtOZoa4J0wQ=; b=uRxwcXWLTpJSzWRJ6nT3iACTLMwMlhq+oIbYTr2PoeLRwGw2Io6SA+hXXqA/6wRCS+ ney6v9fjp6TwMcxadoAMBXNL6tvDW1wFuAihkiUSO17pK9D+Qtwwq3TBiLceq7lbcer7 PEQX0Lbm5g5U5VG5HDfe8fmxpF/LIichmXymxGU014NTREk/c6JwnR4iDEFNL4djiNz6 YdnxbxMNjPaQKJfV/q7Wz5RNCx0KFVV1S8kymT17TuhmIa0kxrkoYy7DfDoMZfRMuHHF Gg+iOlR3jQOheyh1kKdTiSjpGmd/ATZ/1gnv8H+TqwG53f0rEOtm6JMtsR2MBruFmd0e gT+Q== X-Gm-Message-State: AOJu0YxQ0p466BcuHCH5DzIMZF9IYW1HV8jccGFqBdmicEPvU74bcCBg uHuEAD16enBkdnslcn76AvMs6qfSRTsoJBia2SlmZbndG1d2KVLm3ssm+mVNA50rVOekd0+tdGG lWH5g X-Received: by 2002:a05:6512:328b:b0:513:9d80:f776 with SMTP id p11-20020a056512328b00b005139d80f776mr5134304lfe.30.1711378979522; Mon, 25 Mar 2024 08:02:59 -0700 (PDT) Received: from localhost (host-114-191.parnet.fi. [77.234.114.191]) by smtp.gmail.com with ESMTPSA id b7-20020a0565120b8700b005159d671616sm1104488lfv.134.2024.03.25.08.02.59 (version=TLS1 cipher=AES128-SHA bits=128/128); Mon, 25 Mar 2024 08:02:59 -0700 (PDT) From: =?utf-8?q?Martin_Storsj=C3=B6?= To: ffmpeg-devel@ffmpeg.org Date: Mon, 25 Mar 2024 17:02:41 +0200 Message-Id: <20240325150243.59058-20-martin@martin.st> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20240325150243.59058-1-martin@martin.st> References: <20240325150243.59058-1-martin@martin.st> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 19/21] aarch64: hevc: Produce plain neon versions of qpel_uni_hv X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Logan Lyu , "J . Dekker" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: XHOW3Nj9UIZF As the plain neon qpel_h functions process two rows at a time, we need to allocate storage for h+8 rows instead of h+7. By allocating storage for h+8 rows, incrementing the stack pointer won't end up at the right spot in the end. Store the intended final stack pointer value in a register x14 which we store on the stack. AWS Graviton 3: put_hevc_qpel_uni_hv4_8_c: 384.2 put_hevc_qpel_uni_hv4_8_neon: 127.5 put_hevc_qpel_uni_hv4_8_i8mm: 85.5 put_hevc_qpel_uni_hv6_8_c: 705.5 put_hevc_qpel_uni_hv6_8_neon: 224.5 put_hevc_qpel_uni_hv6_8_i8mm: 176.2 put_hevc_qpel_uni_hv8_8_c: 1136.5 put_hevc_qpel_uni_hv8_8_neon: 216.5 put_hevc_qpel_uni_hv8_8_i8mm: 214.0 put_hevc_qpel_uni_hv12_8_c: 2259.5 put_hevc_qpel_uni_hv12_8_neon: 498.5 put_hevc_qpel_uni_hv12_8_i8mm: 410.7 put_hevc_qpel_uni_hv16_8_c: 3824.7 put_hevc_qpel_uni_hv16_8_neon: 670.0 put_hevc_qpel_uni_hv16_8_i8mm: 603.7 put_hevc_qpel_uni_hv24_8_c: 8113.5 put_hevc_qpel_uni_hv24_8_neon: 1474.7 put_hevc_qpel_uni_hv24_8_i8mm: 1351.5 put_hevc_qpel_uni_hv32_8_c: 14744.5 put_hevc_qpel_uni_hv32_8_neon: 2599.7 put_hevc_qpel_uni_hv32_8_i8mm: 2266.0 put_hevc_qpel_uni_hv48_8_c: 32800.0 put_hevc_qpel_uni_hv48_8_neon: 5650.0 put_hevc_qpel_uni_hv48_8_i8mm: 5011.7 put_hevc_qpel_uni_hv64_8_c: 57856.2 put_hevc_qpel_uni_hv64_8_neon: 9863.5 put_hevc_qpel_uni_hv64_8_i8mm: 8767.7 --- libavcodec/aarch64/hevcdsp_init_aarch64.c | 5 + libavcodec/aarch64/hevcdsp_qpel_neon.S | 156 ++++++++++++++-------- 2 files changed, 102 insertions(+), 59 deletions(-) diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index 105c26017b..0531db027b 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -277,6 +277,10 @@ NEON8_FNPROTO(qpel_uni_v, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *src, ptrdiff_t srcstride, int height, intptr_t mx, intptr_t my, int width),); +NEON8_FNPROTO(qpel_uni_hv, (uint8_t *dst, ptrdiff_t dststride, + const uint8_t *src, ptrdiff_t srcstride, + int height, intptr_t mx, intptr_t my, int width),); + NEON8_FNPROTO(qpel_uni_hv, (uint8_t *dst, ptrdiff_t dststride, const uint8_t *src, ptrdiff_t srcstride, int height, intptr_t mx, intptr_t my, int width), _i8mm); @@ -441,6 +445,7 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) NEON8_FNASSIGN_SHARED_32(c->put_hevc_qpel_uni_w, 0, 1, qpel_uni_w_h,); NEON8_FNASSIGN(c->put_hevc_qpel, 1, 1, qpel_hv,); + NEON8_FNASSIGN(c->put_hevc_qpel_uni, 1, 1, qpel_uni_hv,); if (have_i8mm(cpu_flags)) { NEON8_FNASSIGN(c->put_hevc_epel, 0, 1, epel_h, _i8mm); diff --git a/libavcodec/aarch64/hevcdsp_qpel_neon.S b/libavcodec/aarch64/hevcdsp_qpel_neon.S index 7bffb991a7..f285ab7461 100644 --- a/libavcodec/aarch64/hevcdsp_qpel_neon.S +++ b/libavcodec/aarch64/hevcdsp_qpel_neon.S @@ -2169,7 +2169,8 @@ function hevc_put_hevc_qpel_uni_hv4_8_end_neon .endm 1: calc_all .purgem calc -2: ret +2: mov sp, x14 + ret endfunc function hevc_put_hevc_qpel_uni_hv6_8_end_neon @@ -2198,7 +2199,8 @@ function hevc_put_hevc_qpel_uni_hv6_8_end_neon .endm 1: calc_all .purgem calc -2: ret +2: mov sp, x14 + ret endfunc function hevc_put_hevc_qpel_uni_hv8_8_end_neon @@ -2225,7 +2227,8 @@ function hevc_put_hevc_qpel_uni_hv8_8_end_neon .endm 1: calc_all .purgem calc -2: ret +2: mov sp, x14 + ret endfunc function hevc_put_hevc_qpel_uni_hv12_8_end_neon @@ -2252,7 +2255,8 @@ function hevc_put_hevc_qpel_uni_hv12_8_end_neon .endm 1: calc_all2 .purgem calc -2: ret +2: mov sp, x14 + ret endfunc function hevc_put_hevc_qpel_uni_hv16_8_end_neon @@ -2286,21 +2290,17 @@ function hevc_put_hevc_qpel_uni_hv16_8_end_neon add sp, sp, #32 subs w7, w7, #16 b.ne 0b - add w10, w4, #6 - add sp, sp, x12 // discard rest of first line - lsl x10, x10, #7 - add sp, sp, x10 // tmp_array without first line + mov sp, x14 ret endfunc -#if HAVE_I8MM -ENABLE_I8MM - -function ff_hevc_put_hevc_qpel_uni_hv4_8_neon_i8mm, export=1 - add w10, w4, #7 +.macro qpel_uni_hv suffix +function ff_hevc_put_hevc_qpel_uni_hv4_8_\suffix, export=1 + add w10, w4, #8 lsl x10, x10, #7 + mov x14, sp sub sp, sp, x10 // tmp_array - str x30, [sp, #-48]! + stp x30, x14,[sp, #-48]! stp x4, x6, [sp, #16] stp x0, x1, [sp, #32] sub x1, x2, x3, lsl #1 @@ -2309,18 +2309,19 @@ function ff_hevc_put_hevc_qpel_uni_hv4_8_neon_i8mm, export=1 mov x2, x3 add x3, x4, #7 mov x4, x5 - bl X(ff_hevc_put_hevc_qpel_h4_8_neon_i8mm) + bl X(ff_hevc_put_hevc_qpel_h4_8_\suffix) ldp x4, x6, [sp, #16] ldp x0, x1, [sp, #32] - ldr x30, [sp], #48 + ldp x30, x14, [sp], #48 b hevc_put_hevc_qpel_uni_hv4_8_end_neon endfunc -function ff_hevc_put_hevc_qpel_uni_hv6_8_neon_i8mm, export=1 - add w10, w4, #7 +function ff_hevc_put_hevc_qpel_uni_hv6_8_\suffix, export=1 + add w10, w4, #8 lsl x10, x10, #7 + mov x14, sp sub sp, sp, x10 // tmp_array - str x30, [sp, #-48]! + stp x30, x14,[sp, #-48]! stp x4, x6, [sp, #16] stp x0, x1, [sp, #32] sub x1, x2, x3, lsl #1 @@ -2329,18 +2330,19 @@ function ff_hevc_put_hevc_qpel_uni_hv6_8_neon_i8mm, export=1 mov x2, x3 add w3, w4, #7 mov x4, x5 - bl X(ff_hevc_put_hevc_qpel_h6_8_neon_i8mm) + bl X(ff_hevc_put_hevc_qpel_h6_8_\suffix) ldp x4, x6, [sp, #16] ldp x0, x1, [sp, #32] - ldr x30, [sp], #48 + ldp x30, x14, [sp], #48 b hevc_put_hevc_qpel_uni_hv6_8_end_neon endfunc -function ff_hevc_put_hevc_qpel_uni_hv8_8_neon_i8mm, export=1 - add w10, w4, #7 +function ff_hevc_put_hevc_qpel_uni_hv8_8_\suffix, export=1 + add w10, w4, #8 lsl x10, x10, #7 + mov x14, sp sub sp, sp, x10 // tmp_array - str x30, [sp, #-48]! + stp x30, x14,[sp, #-48]! stp x4, x6, [sp, #16] stp x0, x1, [sp, #32] sub x1, x2, x3, lsl #1 @@ -2349,60 +2351,67 @@ function ff_hevc_put_hevc_qpel_uni_hv8_8_neon_i8mm, export=1 mov x2, x3 add w3, w4, #7 mov x4, x5 - bl X(ff_hevc_put_hevc_qpel_h8_8_neon_i8mm) + bl X(ff_hevc_put_hevc_qpel_h8_8_\suffix) ldp x4, x6, [sp, #16] ldp x0, x1, [sp, #32] - ldr x30, [sp], #48 + ldp x30, x14, [sp], #48 b hevc_put_hevc_qpel_uni_hv8_8_end_neon endfunc -function ff_hevc_put_hevc_qpel_uni_hv12_8_neon_i8mm, export=1 - add w10, w4, #7 +function ff_hevc_put_hevc_qpel_uni_hv12_8_\suffix, export=1 + add w10, w4, #8 lsl x10, x10, #7 + mov x14, sp sub sp, sp, x10 // tmp_array - stp x7, x30, [sp, #-48]! + stp x7, x30, [sp, #-64]! stp x4, x6, [sp, #16] stp x0, x1, [sp, #32] + str x14, [sp, #48] sub x1, x2, x3, lsl #1 sub x1, x1, x3 mov x2, x3 - add x0, sp, #48 + add x0, sp, #64 add w3, w4, #7 mov x4, x5 - bl X(ff_hevc_put_hevc_qpel_h12_8_neon_i8mm) + mov w6, #12 + bl X(ff_hevc_put_hevc_qpel_h12_8_\suffix) + ldr x14, [sp, #48] ldp x4, x6, [sp, #16] ldp x0, x1, [sp, #32] - ldp x7, x30, [sp], #48 + ldp x7, x30, [sp], #64 b hevc_put_hevc_qpel_uni_hv12_8_end_neon endfunc -function ff_hevc_put_hevc_qpel_uni_hv16_8_neon_i8mm, export=1 - add w10, w4, #7 +function ff_hevc_put_hevc_qpel_uni_hv16_8_\suffix, export=1 + add w10, w4, #8 lsl x10, x10, #7 + mov x14, sp sub sp, sp, x10 // tmp_array - stp x7, x30, [sp, #-48]! + stp x7, x30, [sp, #-64]! stp x4, x6, [sp, #16] stp x0, x1, [sp, #32] - add x0, sp, #48 + str x14, [sp, #48] + add x0, sp, #64 sub x1, x2, x3, lsl #1 sub x1, x1, x3 mov x2, x3 add w3, w4, #7 mov x4, x5 - bl X(ff_hevc_put_hevc_qpel_h16_8_neon_i8mm) + bl X(ff_hevc_put_hevc_qpel_h16_8_\suffix) + ldr x14, [sp, #48] ldp x4, x6, [sp, #16] ldp x0, x1, [sp, #32] - ldp x7, x30, [sp], #48 + ldp x7, x30, [sp], #64 b hevc_put_hevc_qpel_uni_hv16_8_end_neon endfunc -function ff_hevc_put_hevc_qpel_uni_hv24_8_neon_i8mm, export=1 +function ff_hevc_put_hevc_qpel_uni_hv24_8_\suffix, export=1 stp x4, x5, [sp, #-64]! stp x2, x3, [sp, #16] stp x0, x1, [sp, #32] stp x6, x30, [sp, #48] mov x7, #16 - bl X(ff_hevc_put_hevc_qpel_uni_hv16_8_neon_i8mm) + bl X(ff_hevc_put_hevc_qpel_uni_hv16_8_\suffix) ldp x2, x3, [sp, #16] add x2, x2, #16 ldp x0, x1, [sp, #32] @@ -2410,71 +2419,100 @@ function ff_hevc_put_hevc_qpel_uni_hv24_8_neon_i8mm, export=1 mov x7, #8 add x0, x0, #16 ldr x6, [sp] - bl X(ff_hevc_put_hevc_qpel_uni_hv8_8_neon_i8mm) + bl X(ff_hevc_put_hevc_qpel_uni_hv8_8_\suffix) ldr x30, [sp, #8] add sp, sp, #16 ret endfunc -function ff_hevc_put_hevc_qpel_uni_hv32_8_neon_i8mm, export=1 - add w10, w4, #7 +function ff_hevc_put_hevc_qpel_uni_hv32_8_\suffix, export=1 + add w10, w4, #8 lsl x10, x10, #7 + mov x14, sp sub sp, sp, x10 // tmp_array - stp x7, x30, [sp, #-48]! + stp x7, x30, [sp, #-64]! stp x4, x6, [sp, #16] stp x0, x1, [sp, #32] + str x14, [sp, #48] sub x1, x2, x3, lsl #1 - add x0, sp, #48 + add x0, sp, #64 sub x1, x1, x3 mov x2, x3 add w3, w4, #7 mov x4, x5 - bl X(ff_hevc_put_hevc_qpel_h32_8_neon_i8mm) + mov w6, #32 + bl X(ff_hevc_put_hevc_qpel_h32_8_\suffix) + ldr x14, [sp, #48] ldp x4, x6, [sp, #16] ldp x0, x1, [sp, #32] - ldp x7, x30, [sp], #48 + ldp x7, x30, [sp], #64 b hevc_put_hevc_qpel_uni_hv16_8_end_neon endfunc -function ff_hevc_put_hevc_qpel_uni_hv48_8_neon_i8mm, export=1 - add w10, w4, #7 +function ff_hevc_put_hevc_qpel_uni_hv48_8_\suffix, export=1 + add w10, w4, #8 lsl x10, x10, #7 + mov x14, sp sub sp, sp, x10 // tmp_array - stp x7, x30, [sp, #-48]! + stp x7, x30, [sp, #-64]! stp x4, x6, [sp, #16] stp x0, x1, [sp, #32] + str x14, [sp, #48] sub x1, x2, x3, lsl #1 sub x1, x1, x3 mov x2, x3 - add x0, sp, #48 + add x0, sp, #64 add w3, w4, #7 mov x4, x5 - bl X(ff_hevc_put_hevc_qpel_h48_8_neon_i8mm) +.ifc \suffix, neon + mov w6, #48 + bl X(ff_hevc_put_hevc_qpel_h32_8_\suffix) +.else + bl X(ff_hevc_put_hevc_qpel_h48_8_\suffix) +.endif + ldr x14, [sp, #48] ldp x4, x6, [sp, #16] ldp x0, x1, [sp, #32] - ldp x7, x30, [sp], #48 + ldp x7, x30, [sp], #64 b hevc_put_hevc_qpel_uni_hv16_8_end_neon endfunc -function ff_hevc_put_hevc_qpel_uni_hv64_8_neon_i8mm, export=1 - add w10, w4, #7 +function ff_hevc_put_hevc_qpel_uni_hv64_8_\suffix, export=1 + add w10, w4, #8 lsl x10, x10, #7 + mov x14, sp sub sp, sp, x10 // tmp_array - stp x7, x30, [sp, #-48]! + stp x7, x30, [sp, #-64]! stp x4, x6, [sp, #16] stp x0, x1, [sp, #32] - add x0, sp, #48 + str x14, [sp, #48] + add x0, sp, #64 sub x1, x2, x3, lsl #1 mov x2, x3 sub x1, x1, x3 add w3, w4, #7 mov x4, x5 - bl X(ff_hevc_put_hevc_qpel_h64_8_neon_i8mm) +.ifc \suffix, neon + mov w6, #64 + bl X(ff_hevc_put_hevc_qpel_h32_8_\suffix) +.else + bl X(ff_hevc_put_hevc_qpel_h64_8_\suffix) +.endif + ldr x14, [sp, #48] ldp x4, x6, [sp, #16] ldp x0, x1, [sp, #32] - ldp x7, x30, [sp], #48 + ldp x7, x30, [sp], #64 b hevc_put_hevc_qpel_uni_hv16_8_end_neon endfunc +.endm + +qpel_uni_hv neon + +#if HAVE_I8MM +ENABLE_I8MM + +qpel_uni_hv neon_i8mm + DISABLE_I8MM #endif