From patchwork Fri Apr 15 21:37:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Swinney, Jonathan" X-Patchwork-Id: 35332 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:671c:b0:7c:62c8:b2d1 with SMTP id q28csp468725pzh; Fri, 15 Apr 2022 14:37:23 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxx8embhJwJoRczhnhMUZi7mtup8hd1cR0AkixoqPA5kEuLWrJ/s+zmB3XwRnujk0P22kfo X-Received: by 2002:a17:906:c14c:b0:6e8:6526:7647 with SMTP id dp12-20020a170906c14c00b006e865267647mr700212ejc.257.1650058643480; Fri, 15 Apr 2022 14:37:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1650058643; cv=none; d=google.com; s=arc-20160816; b=e1gxUYN6QA7PCVw7R7uw8WtEpSczuaL0PSnkmIxkArfse3VK98rPV6F9LNnKne8CeR iNz6tlQpKIPjSETlnz1d02MKZHfaU9wlAdkdTNrcHp0u5hsVmqtmaVHlVvZ/VkMd4YRM zv1joTXhcsKYoyp+F0Bs6Zw/BH8/F3z0XNpxBrmBt4yTB0yPFIFg70aOBVJ1Nhq1hY+b VHyspEJvohr6neqLBK/z8dkWr+TqQO/nk3Hpf1wlVjwpWU5NfH86GNlInoLwMtntlFTn ijmMW61ProbqxWYVzKd6cu0VwsqAQ7ihu2BxSJg2WeHqXY7yXEvTyjRSItaM1ymAIn0c xtXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:content-language :accept-language:message-id:date:thread-index:thread-topic:to:from :dkim-signature:delivered-to; bh=pIrZhvQY+HwuAcYF2BgFqS1h0J2VT7N5aR24RMHoHyw=; b=UlwYCrb5pC/47J/tnrE5GroAtuuPqUeyN0q4puaO1DormCd6bmz+VpKJy6M9evrE10 D/WEEKj+qUYI2gpCa/Fn0XTsK7J7o+8XkNAnT20mnkp4JT4NC4mLsyFL3w15gGePca1g gfWg1aeM5iRD2F3ICTvnRyVjBTwZhIaKamjGI+RBS829eYyR+KgBBQD704AWBeivtlSM ZlVISt5NuUlQyMSBaCJ6RMLAmmqQnsdapRgs0h91x6pu9P5rpBoYi7XXcSHFwlapuv/y W9qyGCXZgRMSqMerzT6y5qEOwz+yTW+BaVvgCMREac/4Ti+XiQatrBtVyjZMWvQb41u9 lGVw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@amazon.com header.s=amazon201209 header.b=qpBnirEP; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x8-20020a170906148800b006ec0acfe293si1668816ejc.881.2022.04.15.14.37.23; Fri, 15 Apr 2022 14:37:23 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@amazon.com header.s=amazon201209 header.b=qpBnirEP; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E2D4368B4F8; Sat, 16 Apr 2022 00:37:18 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-fw-80006.amazon.com (smtp-fw-80006.amazon.com [99.78.197.217]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D030568B4D4 for ; Sat, 16 Apr 2022 00:37:10 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1650058636; x=1681594636; h=from:to:cc:subject:date:message-id: content-transfer-encoding:mime-version; bh=lEdWc2wr7nxWLCixhNPoPAIgny/cjO2FdjiQwlHHVco=; b=qpBnirEP8LNcvYJoHe6/JM1RPsd/USGtDG8ubQ8nYklkv1eC0LSjUhCI SQsf4f+ZpJ6xZ36svRY4mYDk2dS+Ke5I50p7QBIH4FKkACb7gfZop5iMj CUv/CBPsh2zi7ZHEbBtyszsGsxvsox7zYlzW2sjXzxLLiiWPScZ9WmabG Q=; X-IronPort-AV: E=Sophos;i="5.90,263,1643673600"; d="scan'208";a="80297828" Received: from pdx4-co-svc-p1-lb2-vlan3.amazon.com (HELO email-inbound-relay-pdx-2b-02ee77e7.us-west-2.amazon.com) ([10.25.36.214]) by smtp-border-fw-80006.pdx80.corp.amazon.com with ESMTP; 15 Apr 2022 21:37:08 +0000 Received: from EX13MTAUWB001.ant.amazon.com (pdx1-ws-svc-p6-lb9-vlan2.pdx.amazon.com [10.236.137.194]) by email-inbound-relay-pdx-2b-02ee77e7.us-west-2.amazon.com (Postfix) with ESMTPS id 1457340DAF; Fri, 15 Apr 2022 21:37:08 +0000 (UTC) Received: from EX13D01UWB004.ant.amazon.com (10.43.161.157) by EX13MTAUWB001.ant.amazon.com (10.43.161.207) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Fri, 15 Apr 2022 21:37:06 +0000 Received: from EX13D07UWB004.ant.amazon.com (10.43.161.196) by EX13d01UWB004.ant.amazon.com (10.43.161.157) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Fri, 15 Apr 2022 21:37:06 +0000 Received: from EX13D07UWB004.ant.amazon.com ([10.43.161.196]) by EX13D07UWB004.ant.amazon.com ([10.43.161.196]) with mapi id 15.00.1497.033; Fri, 15 Apr 2022 21:37:06 +0000 From: "Swinney, Jonathan" To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH 2/2] swscale/aarch64: add vscale specializations Thread-Index: AdhRDA+QV+DuTNdhRaigK3LBQ9rnSg== Date: Fri, 15 Apr 2022 21:37:06 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.43.160.81] MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] swscale/aarch64: add vscale specializations X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: =?utf-8?q?Martin_Storsj=C3=B6?= , "Pop, Sebastian" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: gzTnPKFKJst3 This commit adds new code paths for vscale when filterSize is 2, 4, or 8. By using specialized code with unrolling to match the filterSize we can improve performance. | (seconds) | c6g | | | | ------------| ----- | ----- | ----- | | filterSize | 2 | 4 | 8 | | original | 0.581 | 0.974 | 1.744 | | optimized | 0.399 | 0.569 | 1.052 | | improvement | 31.1% | 41.6% | 39.7% | Signed-off-by: Jonathan Swinney --- libswscale/aarch64/output.S | 147 +++++++++++++++++++++++++++++++++-- libswscale/aarch64/swscale.c | 12 +++ 2 files changed, 153 insertions(+), 6 deletions(-) diff --git a/libswscale/aarch64/output.S b/libswscale/aarch64/output.S index af71de6050..9c99c3bea9 100644 --- a/libswscale/aarch64/output.S +++ b/libswscale/aarch64/output.S @@ -21,12 +21,27 @@ #include "libavutil/aarch64/asm.S" function ff_yuv2planeX_8_neon, export=1 +// x0 - const int16_t *filter, +// x1 - int filterSize, +// x2 - const int16_t **src, +// x3 - uint8_t *dest, +// x4 - int dstW, +// x5 - const uint8_t *dither, +// x6 - int offset + ld1 {v0.8B}, [x5] // load 8x8-bit dither cbz w6, 1f // check if offsetting present ext v0.8B, v0.8B, v0.8B, #3 // honor offsetting which can be 0 or 3 only 1: uxtl v0.8H, v0.8B // extend dither to 16-bit ushll v1.4S, v0.4H, #12 // extend dither to 32-bit with left shift by 12 (part 1) ushll2 v2.4S, v0.8H, #12 // extend dither to 32-bit with left shift by 12 (part 2) + cmp w1, #8 // if filterSize == 8, branch to specialized version + b.eq 5f + cmp w1, #4 // if filterSize == 4, branch to specialized version + b.eq 7f + cmp w1, #2 // if filterSize == 2, branch to specialized version + b.eq 9f + mov x7, #0 // i = 0 2: mov v3.16B, v1.16B // initialize accumulator part 1 with dithering value mov v4.16B, v2.16B // initialize accumulator part 2 with dithering value @@ -34,16 +49,15 @@ function ff_yuv2planeX_8_neon, export=1 mov x9, x2 // srcp = src mov x10, x0 // filterp = filter 3: ldp x11, x12, [x9], #16 // get 2 pointers: src[j] and src[j+1] + ld2r {v16.8H, v17.8H}, [x10], #4 // read 2x16-bit coeff X and Y at filter[j] and filter[j+1] add x11, x11, x7, lsl #1 // &src[j ][i] add x12, x12, x7, lsl #1 // &src[j+1][i] ld1 {v5.8H}, [x11] // read 8x16-bit @ src[j ][i + {0..7}]: A,B,C,D,E,F,G,H ld1 {v6.8H}, [x12] // read 8x16-bit @ src[j+1][i + {0..7}]: I,J,K,L,M,N,O,P - ld1r {v7.8H}, [x10], #2 // read 1x16-bit coeff X at filter[j ] and duplicate across lanes - ld1r {v16.8H}, [x10], #2 // read 1x16-bit coeff Y at filter[j+1] and duplicate across lanes - smlal v3.4S, v5.4H, v7.4H // val0 += {A,B,C,D} * X - smlal2 v4.4S, v5.8H, v7.8H // val1 += {E,F,G,H} * X - smlal v3.4S, v6.4H, v16.4H // val0 += {I,J,K,L} * Y - smlal2 v4.4S, v6.8H, v16.8H // val1 += {M,N,O,P} * Y + smlal v3.4S, v5.4H, v16.4H // val0 += {A,B,C,D} * X + smlal2 v4.4S, v5.8H, v16.8H // val1 += {E,F,G,H} * X + smlal v3.4S, v6.4H, v17.4H // val0 += {I,J,K,L} * Y + smlal2 v4.4S, v6.8H, v17.8H // val1 += {M,N,O,P} * Y subs w8, w8, #2 // tmpfilterSize -= 2 b.gt 3b // loop until filterSize consumed @@ -55,4 +69,125 @@ function ff_yuv2planeX_8_neon, export=1 add x7, x7, #8 // i += 8 b.gt 2b // loop until width consumed ret + +5: // fs=8 + ldp x5, x6, [x2] // load 2 pointers: src[j ] and src[j+1] + ldp x7, x9, [x2, #16] // load 2 pointers: src[j+2] and src[j+3] + ldp x10, x11, [x2, #32] // load 2 pointers: src[j+4] and src[j+5] + ldp x12, x13, [x2, #48] // load 2 pointers: src[j+6] and src[j+7] + + // load 8x16-bit values for filter[j], where j=0..7 and replicated across lanes + ld4r {v16.8H, v17.8H, v18.8H, v19.8H}, [x0], #8 + ld4r {v20.8H, v21.8H, v22.8H, v23.8H}, [x0] +6: + mov v3.16B, v1.16B // initialize accumulator part 1 with dithering value + mov v4.16B, v2.16B // initialize accumulator part 2 with dithering value + + ld1 {v24.8H}, [x5], #16 // load 8x16-bit values for src[j + 0][i + {0..7}] + ld1 {v25.8H}, [x6], #16 // load 8x16-bit values for src[j + 1][i + {0..7}] + ld1 {v26.8H}, [x7], #16 // load 8x16-bit values for src[j + 2][i + {0..7}] + ld1 {v27.8H}, [x9], #16 // load 8x16-bit values for src[j + 3][i + {0..7}] + ld1 {v28.8H}, [x10], #16 // load 8x16-bit values for src[j + 4][i + {0..7}] + ld1 {v29.8H}, [x11], #16 // load 8x16-bit values for src[j + 5][i + {0..7}] + ld1 {v30.8H}, [x12], #16 // load 8x16-bit values for src[j + 6][i + {0..7}] + ld1 {v31.8H}, [x13], #16 // load 8x16-bit values for src[j + 7][i + {0..7}] + + smlal v3.4S, v24.4H, v16.4H // val0 += src[0][i + {0..3}] * filter[0] + smlal2 v4.4S, v24.8H, v16.8H // val1 += src[0][i + {4..7}] * filter[0] + smlal v3.4S, v25.4H, v17.4H // val0 += src[1][i + {0..3}] * filter[1] + smlal2 v4.4S, v25.8H, v17.8H // val1 += src[1][i + {4..7}] * filter[1] + smlal v3.4S, v26.4H, v18.4H // val0 += src[2][i + {0..3}] * filter[2] + smlal2 v4.4S, v26.8H, v18.8H // val1 += src[2][i + {4..7}] * filter[2] + smlal v3.4S, v27.4H, v19.4H // val0 += src[3][i + {0..3}] * filter[3] + smlal2 v4.4S, v27.8H, v19.8H // val1 += src[3][i + {4..7}] * filter[3] + smlal v3.4S, v28.4H, v20.4H // val0 += src[4][i + {0..3}] * filter[4] + smlal2 v4.4S, v28.8H, v20.8H // val1 += src[4][i + {4..7}] * filter[4] + smlal v3.4S, v29.4H, v21.4H // val0 += src[5][i + {0..3}] * filter[5] + smlal2 v4.4S, v29.8H, v21.8H // val1 += src[5][i + {4..7}] * filter[5] + smlal v3.4S, v30.4H, v22.4H // val0 += src[6][i + {0..3}] * filter[6] + smlal2 v4.4S, v30.8H, v22.8H // val1 += src[6][i + {4..7}] * filter[6] + smlal v3.4S, v31.4H, v23.4H // val0 += src[7][i + {0..3}] * filter[7] + smlal2 v4.4S, v31.8H, v23.8H // val1 += src[7][i + {4..7}] * filter[7] + + sqshrun v3.4h, v3.4s, #16 // clip16(val0>>16) + sqshrun2 v3.8h, v4.4s, #16 // clip16(val1>>16) + uqshrn v3.8b, v3.8h, #3 // clip8(val>>19) + st1 {v3.8b}, [x3], #8 // write to destination + subs w4, w4, #8 // dstW -= 8 + b.gt 6b // loop until width consumed + ret + +7: // fs=4 + ldp x5, x6, [x2] // load 2 pointers: src[j ] and src[j+1] + ldp x7, x9, [x2, #16] // load 2 pointers: src[j+2] and src[j+3] + + // load 4x16-bit values for filter[j], where j=0..3 and replicated across lanes + ld4r {v16.8H, v17.8H, v18.8H, v19.8H}, [x0] +8: + mov v3.16B, v1.16B // initialize accumulator part 1 with dithering value + mov v4.16B, v2.16B // initialize accumulator part 2 with dithering value + + ld1 {v24.8H}, [x5], #16 // load 8x16-bit values for src[j + 0][i + {0..7}] + ld1 {v25.8H}, [x6], #16 // load 8x16-bit values for src[j + 1][i + {0..7}] + ld1 {v26.8H}, [x7], #16 // load 8x16-bit values for src[j + 2][i + {0..7}] + ld1 {v27.8H}, [x9], #16 // load 8x16-bit values for src[j + 3][i + {0..7}] + + smlal v3.4S, v24.4H, v16.4H // val0 += src[0][i + {0..3}] * filter[0] + smlal2 v4.4S, v24.8H, v16.8H // val1 += src[0][i + {4..7}] * filter[0] + smlal v3.4S, v25.4H, v17.4H // val0 += src[1][i + {0..3}] * filter[1] + smlal2 v4.4S, v25.8H, v17.8H // val1 += src[1][i + {4..7}] * filter[1] + smlal v3.4S, v26.4H, v18.4H // val0 += src[2][i + {0..3}] * filter[2] + smlal2 v4.4S, v26.8H, v18.8H // val1 += src[2][i + {4..7}] * filter[2] + smlal v3.4S, v27.4H, v19.4H // val0 += src[3][i + {0..3}] * filter[3] + smlal2 v4.4S, v27.8H, v19.8H // val1 += src[3][i + {4..7}] * filter[3] + + sqshrun v3.4h, v3.4s, #16 // clip16(val0>>16) + sqshrun2 v3.8h, v4.4s, #16 // clip16(val1>>16) + uqshrn v3.8b, v3.8h, #3 // clip8(val>>19) + st1 {v3.8b}, [x3], #8 // write to destination + subs w4, w4, #8 // dstW -= 8 + b.gt 8b // loop until width consumed + ret + +9: // fs=2 + ldp x5, x6, [x2] // load 2 pointers: src[j ] and src[j+1] + + // load 2x16-bit values for filter[j], where j=0..1 and replicated across lanes + ld2r {v16.8H, v17.8H}, [x0] +10: + mov v3.16B, v1.16B // initialize accumulator part 1 with dithering value + mov v4.16B, v2.16B // initialize accumulator part 2 with dithering value + + ld1 {v24.8H}, [x5], #16 // load 8x16-bit values for src[j + 0][i + {0..7}] + ld1 {v25.8H}, [x6], #16 // load 8x16-bit values for src[j + 1][i + {0..7}] + + smlal v3.4S, v24.4H, v16.4H // val0 += src[0][i + {0..3}] * filter[0] + smlal2 v4.4S, v24.8H, v16.8H // val1 += src[0][i + {4..7}] * filter[0] + smlal v3.4S, v25.4H, v17.4H // val0 += src[1][i + {0..3}] * filter[1] + smlal2 v4.4S, v25.8H, v17.8H // val1 += src[1][i + {4..7}] * filter[1] + + sqshrun v3.4h, v3.4s, #16 // clip16(val0>>16) + sqshrun2 v3.8h, v4.4s, #16 // clip16(val1>>16) + uqshrn v3.8b, v3.8h, #3 // clip8(val>>19) + st1 {v3.8b}, [x3], #8 // write to destination + subs w4, w4, #8 // dstW -= 8 + b.gt 10b // loop until width consumed + ret +endfunc +function ff_yuv2plane1_8_neon, export=1 + ld1 {v0.8B}, [x5] // load 8x8-bit dither + cbz w6, 1f // check if offsetting present + ext v0.8B, v0.8B, v0.8B, #3 // honor offsetting which can be 0 or 3 only +1: uxtl v0.8H, v0.8B // extend dither to 16-bit + + +2: + ld1 {v5.8H}, [x0], #16 // read 8x16-bit @ src[j ][i + {0..7}]: A,B,C,D,E,F,G,H + add v3.8H, v0.8H, v5.8H + + uqshrn v3.8b, v3.8h, #7 // clip8(val>>7) + st1 {v3.8b}, [x1], #8 // write to destination + subs w2, w2, #8 // dstW -= 8 + b.gt 2b // loop until width consumed + ret endfunc diff --git a/libswscale/aarch64/swscale.c b/libswscale/aarch64/swscale.c index 2ea4ccb3a6..0cc821bf11 100644 --- a/libswscale/aarch64/swscale.c +++ b/libswscale/aarch64/swscale.c @@ -40,6 +40,12 @@ ALL_SCALE_FUNCS(neon); void ff_yuv2planeX_8_neon(const int16_t *filter, int filterSize, const int16_t **src, uint8_t *dest, int dstW, const uint8_t *dither, int offset); +void ff_yuv2plane1_8_neon( + const int16_t *src, + uint8_t *dest, + int dstW, + const uint8_t *dither, + int offset); #define ASSIGN_SCALE_FUNC2(hscalefn, filtersize, opt) do { \ if (c->srcBpc == 8 && c->dstBpc <= 14) { \ @@ -56,6 +62,11 @@ void ff_yuv2planeX_8_neon(const int16_t *filter, int filterSize, ASSIGN_SCALE_FUNC2(hscalefn, X8, opt); \ break; \ } +#define ASSIGN_VSCALE_FUNC(vscalefn, opt1) \ + switch(c->dstBpc){ \ + case 8: vscalefn = ff_yuv2plane1_8_ ## opt1; break; \ + default: break; \ + } av_cold void ff_sws_init_swscale_aarch64(SwsContext *c) { @@ -64,6 +75,7 @@ av_cold void ff_sws_init_swscale_aarch64(SwsContext *c) if (have_neon(cpu_flags)) { ASSIGN_SCALE_FUNC(c->hyScale, c->hLumFilterSize, neon); ASSIGN_SCALE_FUNC(c->hcScale, c->hChrFilterSize, neon); + ASSIGN_VSCALE_FUNC(c->yuv2plane1, neon); if (c->dstBpc == 8) { c->yuv2planeX = ff_yuv2planeX_8_neon; }