diff mbox series

[FFmpeg-devel,v3] wmavoice: convert DCT-I/DST-I to lavu/tx

Message ID NdE1Sht--3-9@lynne.ee
State New
Headers show
Series [FFmpeg-devel,v3] wmavoice: convert DCT-I/DST-I to lavu/tx | expand

Checks

Context Check Description
andriy/make_x86 success Make finished
andriy/make_fate_x86 success Make fate finished

Commit Message

Lynne Sept. 1, 2023, 4:39 a.m. UTC 
Added a patch to fix scaling of R2R transforms and
improved table generation precision slightly.

Planning to push this at the end of today, as it has been
on the mailing list for over a month with two LGTMs.
diff mbox series

Patch

From 30003aa435f2dc21722b344598ae5087e0b6f8ef Mon Sep 17 00:00:00 2001
From: Lynne <dev@lynne.ee>
Date: Fri, 1 Sep 2023 06:13:51 +0200
Subject: [PATCH 11/11] lavu/tx: fix scaling of R2R transforms

Still slightly inaccurate, but it's good enough now.
---
 libavutil/tx_template.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/libavutil/tx_template.c b/libavutil/tx_template.c
index 4745b56700..6e3b3dad33 100644
--- a/libavutil/tx_template.c
+++ b/libavutil/tx_template.c
@@ -1613,6 +1613,7 @@  static av_cold int TX_NAME(ff_tx_rdft_init)(AVTXContext *s,
     int ret;
     double f, m;
     TXSample *tab;
+    uint64_t r2r = flags & AV_TX_REAL_TO_REAL;
     int len4 = FFALIGN(len, 4) / 4;
 
     s->scale_d = *((SCALE_TYPE *)scale);
@@ -1638,7 +1639,10 @@  static av_cold int TX_NAME(ff_tx_rdft_init)(AVTXContext *s,
     *tab++ = RESCALE(-m);
 
     *tab++ = RESCALE( (0.5 - 0.0) * m);
-    *tab++ = RESCALE( (0.0 - 0.5) * m);
+    if (r2r)
+        *tab++ = 1 / s->scale_f;
+    else
+        *tab++ = RESCALE( (0.0 - 0.5) * m);
     *tab++ = RESCALE( (0.5 - inv) * m);
     *tab++ = RESCALE(-(0.5 - inv) * m);
 
@@ -1804,7 +1808,7 @@  static void TX_NAME(ff_tx_rdft_ ##n)(AVTXContext *s, void *_dst,               \
     if (mode == AV_TX_REAL_TO_REAL) {                                          \
         out[len2] = tmp_dc;                                                    \
         if (mod2)                                                              \
-            out[len4 + 1] = tmp_mid;                                           \
+            out[len4 + 1] = tmp_mid * fact[5];                                 \
     } else if (mod2) {                                                         \
         out[len4] = tmp_mid;                                                   \
     }                                                                          \
-- 
2.40.1