From patchwork Sun Dec 17 23:46:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Lynne X-Patchwork-Id: 45197 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1225:b0:181:818d:5e7f with SMTP id v37csp7033306pzf; Sun, 17 Dec 2023 15:47:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IFsYi7CfNByDYs0MDb1LMDX54D3mElND+RcxV1kBiaRk9vZkCu9t3LIEJLpihbXHEopGj9F X-Received: by 2002:a50:c30c:0:b0:553:59ca:4d0b with SMTP id a12-20020a50c30c000000b0055359ca4d0bmr70054edb.38.1702856822507; Sun, 17 Dec 2023 15:47:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702856822; cv=none; d=google.com; s=arc-20160816; b=OYrm+5mv3tC+LBfXb6iHj7v/4ST3ApBPxKad1izzEAh6al6cDg+ALwKiWph20t2qaW zr6SujmnDmPfZ9o26NNGw7LGiTviIXnu66GH2RVjpn8JeMXcUjPzxCR+n/0srYm/nOeA xvhM0S09nC7Y2vSC7qDQFQ7HVZexkheeWMOiNdeqFBmk+N3Aq3cv2UqoeCuURs6jtn2I TUYVafHjh9Bmbljhf2VyZExjI26ICWI8uRmtB6bA8vcw2c5wtlmLbIFsKopsnHSlAoFR YHaGsO/sCSNmZqxAgipQRpeWowWHN1IkC3CRGKql6p/T1+M2LNATFI20/o/2reyBBNGY DNow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:message-id:to:from:date:dkim-signature:delivered-to; bh=vB7/1LTrjimocF8ERUdZ9U5iGx8CqWYMcEHQsS8nygI=; fh=Q46kXK7oI5D1Jhi90JBr53c7NIaTxGaU4KPeRZyM/hI=; b=cO9+UtZhD5VgyRPQ5pPbpatj/KFHtVbPkPgZGlIkO/1fL5icXPVlKcgR3VAvXPsrFU ks9jlR8Ge25QsDDJH7mYTay38gjF+L5DzEgdMOaETghIna/PJP1uEWaRvlE759jOj7XD I6GNhOoV6LTIUehgSENeNdO9Q5DnHJJoCwxNZzugXj7gcEgZ2saqSpaYVp/WWcKBozpB sRdjZwN97QjggG7HE2FrrSK9bd6AsSpmpj1SRH6vxf4KpH6GYlyE8a5z5Jt9N4/uofa6 MmfXUrLRE5Gc0DQ8t0M9+HEu7mLoRpt2mp2mqMN8GhnLKxOLbPAs3az6pL4FipDQOZaE fmnw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@lynne.ee header.s=s1 header.b=n0wNx2yV; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=lynne.ee Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f16-20020a056402355000b0054c74a35e54si10113269edd.277.2023.12.17.15.47.01; Sun, 17 Dec 2023 15:47:02 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@lynne.ee header.s=s1 header.b=n0wNx2yV; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=lynne.ee Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 84AD268D174; Mon, 18 Dec 2023 01:46:57 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from w4.tutanota.de (w4.tutanota.de [81.3.6.165]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 08A4968CFD0 for ; Mon, 18 Dec 2023 01:46:50 +0200 (EET) Received: from tutadb.w10.tutanota.de (unknown [192.168.1.10]) by w4.tutanota.de (Postfix) with ESMTP id 2A81B10602EE for ; Sun, 17 Dec 2023 23:46:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1702856809; s=s1; d=lynne.ee; h=From:From:To:To:Subject:Subject:Content-Description:Content-ID:Content-Type:Content-Type:Content-Transfer-Encoding:Cc:Date:Date:In-Reply-To:MIME-Version:MIME-Version:Message-ID:Message-ID:Reply-To:References:Sender; bh=/T6fNMSyYLAm2g1nD31pdUxaUHp5zl+eizIcoyzX690=; b=n0wNx2yVHnfUkpHSXY624jxYctmL0gnwkfGloFc9eXJCsdGGvWW9gY3EOX0VLYq4 r+qQVsDNNnMau2i/15m6awJh65p/IEk/FreJ/+W+AicozSBy8lAzjxL/kdgPNF1Wjup ylG//lx20cgaIS3MM//r456fN0TOMXwkg6fEt+XWpiPGWadQWqRK9Jm3aS4+Dq4pUeA xlSfIzKT51CooRrBpHI4xVqFTAz+UOO2jIHRVECeEfZ893Ww4fMO83js00gIhaJxvFB oK7RPaeQOWw2+lJV99sD1dHA8/yKmp/yyezjrGJBTSLQ+tbcblGoJDZ/mEzNAdeyxpt tTpzbjB0Bw== Date: Mon, 18 Dec 2023 00:46:49 +0100 (CET) From: Lynne To: Ffmpeg Devel Message-ID: MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] x86/tx_float: add SIMD for FFT->RDFT conversion (R2C) X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: vnd9t2h7x3Tl Before: 179987 decicycles in av_tx (r2c), 1048573 runs,      3 skips After: 56458 decicycles in av_tx (r2c), 1048576 runs,      0 skips Patch attached From 1a0d97791b9a7eba8ac08e212ae60ef9961c75c5 Mon Sep 17 00:00:00 2001 From: Lynne Date: Mon, 11 Dec 2023 13:29:21 +0100 Subject: [PATCH] x86/tx_float: add SIMD for FFT->RDFT conversion (R2C) Before: 179987 decicycles in av_tx (r2c), 1048573 runs, 3 skips After: 56458 decicycles in av_tx (r2c), 1048576 runs, 0 skips --- libavutil/tx_template.c | 20 ++--- libavutil/x86/tx_float.asm | 146 ++++++++++++++++++++++++++++++++++ libavutil/x86/tx_float_init.c | 61 ++++++++++++++ tests/checkasm/av_tx.c | 7 ++ 4 files changed, 222 insertions(+), 12 deletions(-) diff --git a/libavutil/tx_template.c b/libavutil/tx_template.c index a2c27465cb..08a0243a2e 100644 --- a/libavutil/tx_template.c +++ b/libavutil/tx_template.c @@ -1647,13 +1647,10 @@ static av_cold int TX_NAME(ff_tx_rdft_init)(AVTXContext *s, *tab++ = RESCALE( (0.5 - inv) * m); *tab++ = RESCALE(-(0.5 - inv) * m); - for (int i = 0; i < len4; i++) + for (int i = 0; i < len4; i++) { *tab++ = RESCALE(cos(i*f)); - - tab = ((TXSample *)s->exp) + len4 + 8; - - for (int i = 0; i < len4; i++) *tab++ = RESCALE(cos(((len - i*4)/4.0)*f)) * (inv ? 1 : -1); + } return 0; } @@ -1665,8 +1662,7 @@ static void TX_NAME(ff_tx_rdft_ ##n)(AVTXContext *s, void *_dst, \ const int len2 = s->len >> 1; \ const int len4 = s->len >> 2; \ const TXSample *fact = (void *)s->exp; \ - const TXSample *tcos = fact + 8; \ - const TXSample *tsin = tcos + len4; \ + const TXSample *exp = fact + 8; \ TXComplex *data = inv ? _src : _dst; \ TXComplex t[3]; \ \ @@ -1688,18 +1684,18 @@ static void TX_NAME(ff_tx_rdft_ ##n)(AVTXContext *s, void *_dst, \ \ for (int i = 1; i < len4; i++) { \ /* Separate even and odd FFTs */ \ - t[0].re = MULT(fact[4], (data[i].re + data[len2 - i].re)); \ t[0].im = MULT(fact[5], (data[i].im - data[len2 - i].im)); \ - t[1].re = MULT(fact[6], (data[i].im + data[len2 - i].im)); \ + t[0].re = MULT(fact[4], (data[i].re + data[len2 - i].re)); \ t[1].im = MULT(fact[7], (data[i].re - data[len2 - i].re)); \ + t[1].re = MULT(fact[6], (data[i].im + data[len2 - i].im)); \ \ /* Apply twiddle factors to the odd FFT and add to the even FFT */ \ - CMUL(t[2].re, t[2].im, t[1].re, t[1].im, tcos[i], tsin[i]); \ + CMUL(t[2].re, t[2].im, t[1].re, t[1].im, exp[i*2 + 0], exp[i*2 + 1]); \ \ - data[ i].re = t[0].re + t[2].re; \ + data[ i].re = t[2].re + t[0].re; \ data[ i].im = t[2].im - t[0].im; \ data[len2 - i].re = t[0].re - t[2].re; \ - data[len2 - i].im = t[2].im + t[0].im; \ + data[len2 - i].im = t[0].im + t[2].im; \ } \ \ if (inv) { \ diff --git a/libavutil/x86/tx_float.asm b/libavutil/x86/tx_float.asm index e1533a8595..958815a70f 100644 --- a/libavutil/x86/tx_float.asm +++ b/libavutil/x86/tx_float.asm @@ -91,6 +91,11 @@ s16_perm: dd 0, 1, 2, 3, 1, 0, 3, 2 s15_perm: dd 0, 6, 5, 3, 2, 4, 7, 1 +rdft_perm_pos: dd 3, 2, 2, 3, 1, 0, 0, 1 +rdft_perm_neg: dd 1, 0, 0, 1, 3, 2, 2, 3 +rdft_perm_exp: dd 1, 1, 0, 0, 3, 3, 2, 2 +rdft_m11: times 2 dd 0x0, 0x0, 0x3f800000, 0x3f800000 ; 0, 0, 1, 1 + mask_mmppmmmm: dd NEG, NEG, POS, POS, NEG, NEG, NEG, NEG mask_mmmmpppm: dd NEG, NEG, NEG, NEG, POS, POS, POS, NEG mask_ppmpmmpm: dd POS, POS, NEG, POS, NEG, NEG, POS, NEG @@ -1934,3 +1939,144 @@ cglobal fft_pfa_15xM_ns_float, 4, 14, 16, 320, ctx, out, in, stride, len, lut, b PFA_15_FN avx2, 0 PFA_15_FN avx2, 1 %endif + +%macro RDFT_CONV 3 +%endmacro + +%macro RDFT_FN 3 +INIT_YMM %1 +cglobal rdft_ %+ %2 %+ _float, 4, 14, 16, 320, ctx, out, in, stride, len, lut, exp, t1, t2, t3, \ + t4, t5, btmp + ; FFT setup + mov btmpq, ctxq ; backup original context + mov t3q, [ctxq + AVTXContext.fn] ; subtransform's jump point + + mov ctxq, [ctxq + AVTXContext.sub] ; load subtransform's context + mov lutq, [ctxq + AVTXContext.map] ; load subtransform's map + movsxd lenq, dword [ctxq + AVTXContext.len] ; load subtransform's length + + mov expq, outq +.preshuf: + LOAD64_LUT m0, inq, lutq, 0, t4q, m1, m2 + movaps [outq], m0 + add outq, mmsize + add lutq, (mmsize/2) + sub lenq, (mmsize/8) + jg .preshuf + + mov outq, expq + mov inq, expq + movsxd lenq, dword [ctxq + AVTXContext.len] ; load subtransform's length + + call t3q ; call the FFT + + mov ctxq, btmpq ; restore original context + + movsxd lenq, dword [ctxq + AVTXContext.len] + mov expq, [ctxq + AVTXContext.exp] + + movsd xm0, [outq] ; data[0].reim + movhps xm0, [outq + lenq*2] ; data[len4].reim + + shufps xm1, xm0, xm0, q2301 ; data[0].imre, data[1].imre + addsubps xm2, xm1, xm0 ; t[0].imre, junk + shufps xm1, xm2, xm1, q2301 ; t[0].reim, data[1].reim + mulps xm9, xm1, [expq] ; data[01].reim + + movaps m10, [rdft_perm_neg] + movaps m11, [rdft_perm_pos] + vbroadcastf128 m12, [expq + 4*4] ; fact[5476] + movaps m13, [mask_pmmppmmp] ; +--+ + + movaps m14, [rdft_m11] ; 0.0, 0.0, 1.0, 1.0 + movaps m15, [rdft_perm_exp] + + mov inq, outq + lea t1q, [outq + lenq*4 - mmsize] + mov t2q, lenq + add outq, 8 + +.loop: + movups m8, [expq + (8 + 2)*4] + + vperm2f128 m4, m8, m8, 0x00 ; cos,sin,cos,sin x2 + vperm2f128 m6, m8, m8, 0x11 ; cos,sin,cos,sin x2 + + vpermilps m5, m4, m15 ; cos1,cos1,cos1,cos1,sin2,sin2,cos2,cos2 + vpermilps m7, m6, m15 ; cos1,cos1,cos1,cos1,sin2,sin2,cos2,cos2 + + shufpd m4, m14, m5, 1111b ; 1,1,cos1,cos1,1,1,cos2,cos2 + shufpd m5, m14, m5, 0000b ; 0,0,sin1,sin1,0,0,sin2,sin2 + shufpd m6, m14, m7, 1111b ; 1,1,cos1,cos1,1,1,cos2,cos2 + shufpd m7, m14, m7, 0000b ; 0,0,sin1,sin1,0,0,sin2,sin2 + + movups m2, [outq] + movups m3, [t1q] + + vperm2f128 m0, m2, m2, 0x00 + vperm2f128 m1, m3, m3, 0x11 + vperm2f128 m2, m2, m2, 0x11 + vperm2f128 m3, m3, m3, 0x00 + + vpermilps m0, m0, m10 ; data[0].imrereim, data[1].imrereim + vpermilps m1, m1, m11 ; data[len - 01].imrereim + vpermilps m2, m2, m10 ; data[0].imrereim, data[1].imrereim + vpermilps m3, m3, m11 ; data[len - 01].imrereim + + addsubps m0, m1 ; data[0] - data[len - 0] x2 + addsubps m2, m3 ; data[0] - data[len - 0] x2 + + mulps m0, m12 ; t[01].imre + mulps m2, m12 ; t[01].imre + + shufps m1, m0, m0, q2301 ; t[01].reim + shufps m3, m2, m2, q2301 ; t[01].reim + + mulps m1, m4 ; 1, 1, tcos, tcos x2 + mulps m0, m5 ; 0, 0, tsin, tsin x2 + mulps m3, m6 ; 1, 1, tcos, tcos x2 + mulps m2, m7 ; 0, 0, tsin, tsin x2 + + addsubps m1, m0 ; t[02].reim + addsubps m3, m2 ; t[02].reim + + shufpd m0, m1, m1, 0101b ; t[20].reim + shufpd m2, m3, m3, 0101b ; t[20].reim + + xorps m1, m13 ; +--+t[02].reim + xorps m3, m13 ; +--+t[02].reim + + addps m0, m1 ; data[0].reim, data[len2 - 0].reim x2 + addps m2, m3 ; data[0].reim, data[len2 - 0].reim x2 + + shufpd m1, m0, m2, 0000b ; high + shufpd m3, m0, m2, 1111b ; low + + vpermpd m0, m1, q3120 + vpermpd m2, m3, q0213 + + movups [outq], m0 + movups [t1q], m2 + + add expq, mmsize + sub t1q, mmsize + add outq, mmsize + + sub t2q, mmsize/2 + jg .loop + + ; Write DC, middle and tail + movhps [inq + lenq*2], xm9 + xorps xm0, xm0 + shufps xm9, xm9, xm0, q3210 + shufps xm8, xm9, xm9, q2120 + movsd [inq], xm8 + movhps [inq + lenq*4], xm8 + + RET +%endmacro + +%if ARCH_X86_64 && HAVE_AVX2_EXTERNAL +RDFT_FN avx2, r2c, 0 +;RDFT_FN avx2, c2r, 1 +%endif diff --git a/libavutil/x86/tx_float_init.c b/libavutil/x86/tx_float_init.c index d3c0beb50f..adefda4de3 100644 --- a/libavutil/x86/tx_float_init.c +++ b/libavutil/x86/tx_float_init.c @@ -52,6 +52,8 @@ TX_DECL_FN(fft_pfa_15xM_ns, avx2) TX_DECL_FN(mdct_inv, avx2) +TX_DECL_FN(rdft_r2c, avx2) + TX_DECL_FN(fft2_asm, sse3) TX_DECL_FN(fft4_fwd_asm, sse2) TX_DECL_FN(fft4_inv_asm, sse2) @@ -167,6 +169,62 @@ static av_cold int m_inv_init(AVTXContext *s, const FFTXCodelet *cd, return 0; } +static av_cold int rdft_init(AVTXContext *s, const FFTXCodelet *cd, + uint64_t flags, FFTXCodeletOptions *opts, + int len, int inv, const void *scale) +{ + int ret; + double f, m; + TXSample *tab; + uint64_t r2r = flags & AV_TX_REAL_TO_REAL; + int len4 = FFALIGN(len, 4) / 4; + FFTXCodeletOptions sub_opts = { .map_dir = FF_TX_MAP_GATHER }; + + s->scale_d = *((SCALE_TYPE *)scale); + s->scale_f = s->scale_d; + + flags &= ~(AV_TX_REAL_TO_REAL | AV_TX_REAL_TO_IMAGINARY); + flags |= FF_TX_PRESHUFFLE; /* This function handles the permute step */ + flags |= FF_TX_ASM_CALL; /* We want an assembly function, not C */ + + if ((ret = ff_tx_init_subtx(s, TX_TYPE(FFT), flags, &sub_opts, + len >> 1, inv, scale))) + return ret; + + if (!(s->exp = av_mallocz((8 + 2*len4)*sizeof(*s->exp)))) + return AVERROR(ENOMEM); + + if (!(s->tmp = av_malloc(len*sizeof(*s->tmp)))) + return AVERROR(ENOMEM); + + tab = (TXSample *)s->exp; + + f = 2*M_PI/len; + + m = (inv ? 2*s->scale_d : s->scale_d); + + *tab++ = RESCALE((inv ? 0.5 : 1.0) * m); + *tab++ = -RESCALE(inv ? 0.5*m : 1.0*m); + *tab++ = RESCALE( m); + *tab++ = RESCALE(-m); + + if (r2r) + *tab++ = 1 / s->scale_f; + else + *tab++ = RESCALE( (0.0 - 0.5) * m); + *tab++ = RESCALE( (0.5 - 0.0) * m); + + *tab++ = RESCALE(-(0.5 - inv) * m); + *tab++ = RESCALE( (0.5 - inv) * m); + + for (int i = 0; i < len4; i++) { + *tab++ = RESCALE(cos(i*f)); + *tab++ = RESCALE(cos(((len - i*4)/4.0)*f)) * (inv ? 1 : -1); + } + + return 0; +} + static av_cold int fft_pfa_init(AVTXContext *s, const FFTXCodelet *cd, uint64_t flags, @@ -303,6 +361,9 @@ const FFTXCodelet * const ff_tx_codelet_list_float_x86[] = { TX_DEF(mdct_inv, MDCT, 16, TX_LEN_UNLIMITED, 2, TX_FACTOR_ANY, 384, m_inv_init, avx2, AVX2, FF_TX_INVERSE_ONLY, AV_CPU_FLAG_AVXSLOW | AV_CPU_FLAG_SLOW_GATHER), + + TX_DEF(rdft_r2c, RDFT, 16, TX_LEN_UNLIMITED, 2, TX_FACTOR_ANY, 384, rdft_init, avx2, AVX2, + FF_TX_FORWARD_ONLY, AV_CPU_FLAG_AVXSLOW | AV_CPU_FLAG_SLOW_GATHER), #endif #endif diff --git a/tests/checkasm/av_tx.c b/tests/checkasm/av_tx.c index aa8fc6b4e9..3cea2777fd 100644 --- a/tests/checkasm/av_tx.c +++ b/tests/checkasm/av_tx.c @@ -43,6 +43,10 @@ static const int check_lens[] = { 2, 4, 8, 16, 32, 64, 120, 960, 1024, 1920, 16384, }; +static const int rdft_check_lens[] = { + 32, 1024 +}; + static AVTXContext *tx_refs[AV_TX_NB][2 /* Direction */][FF_ARRAY_ELEMS(check_lens)] = { 0 }; static int init = 0; @@ -113,6 +117,9 @@ void checkasm_check_av_tx(void) CHECK_TEMPLATE("float_imdct", AV_TX_FLOAT_MDCT, 1, float, float, check_lens, !float_near_abs_eps_array(out_ref, out_new, EPS, len)); + CHECK_TEMPLATE("float_r2c", AV_TX_FLOAT_RDFT, 0, float, float, rdft_check_lens, + !float_near_abs_eps_array(out_ref, out_new, EPS, len)); + randomize_complex(in, 16384, AVComplexDouble, SCALE_NOOP); CHECK_TEMPLATE("double_fft", AV_TX_DOUBLE_FFT, 0, AVComplexDouble, double, check_lens, !double_near_abs_eps_array(out_ref, out_new, EPS, len*2)); -- 2.43.0