From patchwork Fri May 12 14:27:06 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Darnley X-Patchwork-Id: 3656 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.3.129 with SMTP id 123csp357415vsd; Fri, 12 May 2017 07:34:16 -0700 (PDT) X-Received: by 10.223.151.203 with SMTP id t11mr3704244wrb.18.1494599656897; Fri, 12 May 2017 07:34:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1494599656; cv=none; d=google.com; s=arc-20160816; b=cmgjbJ+gNxe90D0RRntMCdaHVQiRcMjhzFCXp5LYcahTwm5ylln2zXaJI72kvZOoY9 v5xxfR8yqtv5l6Hx2wZIfal/SJSx92PWABgGEVOh2dLbnJpDnnrJS3c6wdu6tprYnegw iZfCaTKafIjEAs0+mTX392yFXKOT2fq97+n0yPNT/9yDjnWZs9vsgQFqXYQmRk/K3/8Y PGcctMPjpmjVE0w2RqnGYo6B8wKEUyxqseSAVS4UI2AOMRKYP+sT+cnlxOLoC2UkjOAy tCauXlp86T5PWt0f9TACd3/X43eSzBz/PN2AiNpHPrrhl0SpSVOAOVUvEBYeHO62wfE2 ZVGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=nh92SIVEJ0sHO/FeWFSkvpF90WRx0w1EDkvE//XI5LI=; b=YzEW4HLfH0erEeuQM92yOSe3Xw52CzIyXe5Ld1NhTNRFIGyUdNjnRghmgvHRk6Sj1B Q22VWiLeaidh0qZaNgoDJZ36zu0q0HfNnveU8ut6J71vEbG+lbgI3K65NLK9+eGbtktp khDIrUIon9bZm9VqxkLPr4xasxQN4eZzPPKxCIflIJY/VUIrUae8YCpa2ihHDikJ1kDS bDFKF/zWo539xFXtsn7SUSY5CobnqO4QzYY2JKq5oFciJwVnxqWywgy2no+6CXtOpja8 A34GSpuFG6OXH/Ih2Otmz2u6+JbeItQILSE9w7g3vpocqGVIScDvGPYRSxK+CTLwQdcU /PBA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@ob-encoder-com.20150623.gappssmtp.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d24si3527195wrb.106.2017.05.12.07.34.16; Fri, 12 May 2017 07:34:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@ob-encoder-com.20150623.gappssmtp.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C587F68991D; Fri, 12 May 2017 17:34:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm0-f48.google.com (mail-wm0-f48.google.com [74.125.82.48]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2F50B689753 for ; Fri, 12 May 2017 17:33:58 +0300 (EEST) Received: by mail-wm0-f48.google.com with SMTP id 142so74170733wma.1 for ; Fri, 12 May 2017 07:34:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ob-encoder-com.20150623.gappssmtp.com; s=20150623; h=sender:from:to:subject:date:message-id; bh=9kLJeiUJreo9dLK+11DwqAwcZtAQTzuAEpL+tUwaeIc=; b=BmFWb9zbDliG6fcbids/XQSThdBsyzNhsp7us6i/mgqokf0X6w844awhGfsiLEgeFd IfddrCfjCjPgmeYHPiswV7366siEBknczsL7RmgE9gttW/sEHFD+iS9ZXrvnsUeqS7+/ zdKDAR3ppJCZFm/zyGw7XIo6U8wKWj08CvOZffttosl8+l0jq/DvlnvDAGHNsjMaWCsq JnxvcrGI6zbWYTv9pxyeCpqfyghLfSRZnMU+EQj7d0sxyGcMeNskxdaRALrcerCltMCt 6uLodXLomdCauaYJMXjzGS+geDhMp000xbIXpLurZtwaaNt36Syo5o8+3MFMgXAnkmnW s49g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id; bh=9kLJeiUJreo9dLK+11DwqAwcZtAQTzuAEpL+tUwaeIc=; b=iuJcHQ2xi9JeyWfNmxq+oOrr3qzMSWSHUwbgaPLEmT4qTmYG8UwKeRYBV+BdYGSYT6 qpRhZxwCrI6UTkFXO9HtUDjmW5qOYru2fz3I/eh/1noiG+M+Zslz4zIZ9pljP94sHVWF fAVFSHdoSoo5Fy4R/YpIEdce7yIjIcGfNuMDvY8iDcoXw34p349HNxPiajKPGrz7rfCM 5/8aaG+QMqw7rapfVO/0scH9K+iXjcT6fYme++FN4ZMn/whwitGHD5ivsKXbLC2cLNLd xwNKZ5VSt9OPVyYF1sK91Z7Yvb6RyAYiFFT1cAPXtVCr7ZCicmjgkozgI/A7Inis9CNB 5eGg== X-Gm-Message-State: AODbwcDsv2SzoNR4vC6xKZXN6qlwtRWYCoC74JQ7cB59uzG8AzS6UeWE 54o7riNrKjIbaa67 X-Received: by 10.80.184.226 with SMTP id l89mr3575016ede.137.1494599233974; Fri, 12 May 2017 07:27:13 -0700 (PDT) Received: from Highwind.systemlords.lan (d51A44418.access.telenet.be. [81.164.68.24]) by smtp.gmail.com with ESMTPSA id x18sm1205986eda.39.2017.05.12.07.27.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 May 2017 07:27:13 -0700 (PDT) From: James Darnley To: FFmpeg development discussions and patches Date: Fri, 12 May 2017 16:27:06 +0200 Message-Id: <20170512142706.9994-1-jdarnley@obe.tv> X-Mailer: git-send-email 2.12.2 Subject: [FFmpeg-devel] [PATCH/WIP] avcodec/x86: move simple_idct to external assembly X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" --- For initial review and comments. I plan to drop the '2' from the filename before pushing. I haven't done it yet because I am still working on the file. I didn't make any changes with speedup in mind so I haven't done any benchmarking yet. libavcodec/x86/Makefile | 4 +- libavcodec/x86/simple_idct.c | 929 ---------------------------------------- libavcodec/x86/simple_idct2.asm | 892 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 894 insertions(+), 931 deletions(-) delete mode 100644 libavcodec/x86/simple_idct.c create mode 100644 libavcodec/x86/simple_idct2.asm diff --git a/libavcodec/x86/Makefile b/libavcodec/x86/Makefile index d4cb27fa13..af3b50f17a 100644 --- a/libavcodec/x86/Makefile +++ b/libavcodec/x86/Makefile @@ -79,7 +79,6 @@ OBJS-$(CONFIG_WEBP_DECODER) += x86/vp8dsp_init.o # GCC inline assembly optimizations # subsystems MMX-OBJS-$(CONFIG_FDCTDSP) += x86/fdct.o -MMX-OBJS-$(CONFIG_IDCTDSP) += x86/simple_idct.o MMX-OBJS-$(CONFIG_VC1DSP) += x86/vc1dsp_mmx.o # decoders/encoders @@ -128,7 +127,8 @@ YASM-OBJS-$(CONFIG_QPELDSP) += x86/qpeldsp.o \ YASM-OBJS-$(CONFIG_RV34DSP) += x86/rv34dsp.o YASM-OBJS-$(CONFIG_VC1DSP) += x86/vc1dsp_loopfilter.o \ x86/vc1dsp_mc.o -YASM-OBJS-$(CONFIG_IDCTDSP) += x86/simple_idct10.o +YASM-OBJS-$(CONFIG_IDCTDSP) += x86/simple_idct10.o \ + x86/simple_idct2.o YASM-OBJS-$(CONFIG_VIDEODSP) += x86/videodsp.o YASM-OBJS-$(CONFIG_VP3DSP) += x86/vp3dsp.o YASM-OBJS-$(CONFIG_VP8DSP) += x86/vp8dsp.o \ diff --git a/libavcodec/x86/simple_idct.c b/libavcodec/x86/simple_idct.c deleted file mode 100644 index 1155920ae8..0000000000 --- a/libavcodec/x86/simple_idct.c +++ /dev/null @@ -1,929 +0,0 @@ -/* - * Simple IDCT MMX - * - * Copyright (c) 2001, 2002 Michael Niedermayer - * - * This file is part of FFmpeg. - * - * FFmpeg is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 2.1 of the License, or (at your option) any later version. - * - * FFmpeg is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public - * License along with FFmpeg; if not, write to the Free Software - * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA - */ - -#include "libavutil/mem.h" -#include "libavutil/x86/asm.h" - -#include "libavcodec/idctdsp.h" -#include "libavcodec/x86/idctdsp.h" - -#include "idctdsp.h" -#include "simple_idct.h" - -#if HAVE_INLINE_ASM - -/* -23170.475006 -22725.260826 -21406.727617 -19265.545870 -16384.000000 -12872.826198 -8866.956905 -4520.335430 -*/ -#define C0 23170 //cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 -#define C1 22725 //cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 -#define C2 21407 //cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 -#define C3 19266 //cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 -#define C4 16383 //cos(i*M_PI/16)*sqrt(2)*(1<<14) - 0.5 -#define C5 12873 //cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 -#define C6 8867 //cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 -#define C7 4520 //cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 - -#define ROW_SHIFT 11 -#define COL_SHIFT 20 // 6 - -DECLARE_ASM_CONST(8, uint64_t, wm1010)= 0xFFFF0000FFFF0000ULL; -DECLARE_ASM_CONST(8, uint64_t, d40000)= 0x0000000000040000ULL; - -DECLARE_ALIGNED(8, static const int16_t, coeffs)[]= { - 1<<(ROW_SHIFT-1), 0, 1<<(ROW_SHIFT-1), 0, -// 1<<(COL_SHIFT-1), 0, 1<<(COL_SHIFT-1), 0, -// 0, 1<<(COL_SHIFT-1-16), 0, 1<<(COL_SHIFT-1-16), - 1<<(ROW_SHIFT-1), 1, 1<<(ROW_SHIFT-1), 0, - // the 1 = ((1<<(COL_SHIFT-1))/C4)< +; +; This file is part of FFmpeg. +; +; FFmpeg is free software; you can redistribute it and/or +; modify it under the terms of the GNU Lesser General Public +; License as published by the Free Software Foundation; either +; version 2.1 of the License, or (at your option) any later version. +; +; FFmpeg is distributed in the hope that it will be useful, +; but WITHOUT ANY WARRANTY; without even the implied warranty of +; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +; Lesser General Public License for more details. +; +; You should have received a copy of the GNU Lesser General Public +; License along with FFmpeg; if not, write to the Free Software +; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +;/ + +%include "libavutil/x86/x86util.asm" + +SECTION_RODATA + +cextern pb_80 + +wm1010: dw 0, 0xffff, 0, 0xffff +d40000: dw 0, 4, 0, 0 + +; 23170.475006 +; 22725.260826 +; 21406.727617 +; 19265.545870 +; 16384.000000 +; 12872.826198 +; 8866.956905 +; 4520.335430 + +%define C0 23170 ; cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 +%define C1 22725 ; cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 +%define C2 21407 ; cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 +%define C3 19266 ; cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 +%define C4 16383 ; cos(i*M_PI/16)*sqrt(2)*(1<<14) - 0.5 +%define C5 12873 ; cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 +%define C6 8867 ; cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 +%define C7 4520 ; cos(i*M_PI/16)*sqrt(2)*(1<<14) + 0.5 + +%define ROW_SHIFT 11 +%define COL_SHIFT 20 ; 6 + +coeffs: + dw 1 << (ROW_SHIFT - 1), 0 + dw 1 << (ROW_SHIFT - 1), 0 + dw 1 << (ROW_SHIFT - 1), 1 + dw 1 << (ROW_SHIFT - 1), 0 + + dw C4, C4, C4, C4 + dw C4, -C4, C4, -C4 + + dw C2, C6, C2, C6 + dw C6, -C2, C6, -C2 + + dw C1, C3, C1, C3 + dw C5, C7, C5, C7 + + dw C3, -C7, C3, -C7 + dw -C1, -C5, -C1, -C5 + + dw C5, -C1, C5, -C1 + dw C7, C3, C7, C3 + + dw C7, -C5, C7, -C5 + dw C3, -C1, C3, -C1 + +SECTION .text + +%macro DC_COND_IDCT 7 +movq mm0, [blockq + %1] ; R4 R0 r4 r0 +movq mm1, [blockq + %2] ; R6 R2 r6 r2 +movq mm2, [blockq + %3] ; R3 R1 r3 r1 +movq mm3, [blockq + %4] ; R7 R5 r7 r5 +movq mm4, [wm1010] +pand mm4, mm0 +por mm4, mm1 +por mm4, mm2 +por mm4, mm3 +packssdw mm4, mm4 +movd t0d, mm4 +or t0d, t0d +jz %%1 +movq mm4, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm4, mm0 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm0, mm5 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm5, [coeffs + 32] ; C6 C2 C6 C2 +pmaddwd mm5, mm1 ; C6R6+C2R2 C6r6+C2r2 +movq mm6, [coeffs + 40] ; -C2 C6 -C2 C6 +pmaddwd mm1, mm6 ; -C2R6+C6R2 -C2r6+C6r2 +movq mm7, [coeffs + 48] ; C3 C1 C3 C1 +pmaddwd mm7, mm2 ; C3R3+C1R1 C3r3+C1r1 +paddd mm4, [coeffs + 8] +movq mm6, mm4 ; C4R4+C4R0 C4r4+C4r0 +paddd mm4, mm5 ; A0 a0 +psubd mm6, mm5 ; A3 a3 +movq mm5, [coeffs + 56] ; C7 C5 C7 C5 +pmaddwd mm5, mm3 ; C7R7+C5R5 C7r7+C5r5 +paddd mm0, [coeffs + 8] +paddd mm1, mm0 ; A1 a1 +paddd mm0, mm0 +psubd mm0, mm1 ; A2 a2 +pmaddwd mm2, [coeffs + 64] ; -C7R3+C3R1 -C7r3+C3r1 +paddd mm7, mm5 ; B0 b0 +movq mm5, [coeffs + 72] ; -C5 -C1 -C5 -C1 +pmaddwd mm5, mm3 ; -C5R7-C1R5 -C5r7-C1r5 +paddd mm7, mm4 ; A0+B0 a0+b0 +paddd mm4, mm4 ; 2A0 2a0 +psubd mm4, mm7 ; A0-B0 a0-b0 +paddd mm5, mm2 ; B1 b1 +psrad mm7, %7 +psrad mm4, %7 +movq mm2, mm1 ; A1 a1 +paddd mm1, mm5 ; A1+B1 a1+b1 +psubd mm2, mm5 ; A1-B1 a1-b1 +psrad mm1, %7 +psrad mm2, %7 +packssdw mm7, mm1 ; A1+B1 a1+b1 A0+B0 a0+b0 +packssdw mm2, mm4 ; A0-B0 a0-b0 A1-B1 a1-b1 +movq [%5], mm7 +movq mm1, [blockq + %3] ; R3 R1 r3 r1 +movq mm4, [coeffs + 80] ; -C1 C5 -C1 C5 +movq [24+ %5], mm2 +pmaddwd mm4, mm1 ; -C1R3+C5R1 -C1r3+C5r1 +movq mm7, [coeffs + 88] ; C3 C7 C3 C7 +pmaddwd mm1, [coeffs + 96] ; -C5R3+C7R1 -C5r3+C7r1 +pmaddwd mm7, mm3 ; C3R7+C7R5 C3r7+C7r5 +movq mm2, mm0 ; A2 a2 +pmaddwd mm3, [coeffs + 104] ; -C1R7+C3R5 -C1r7+C3r5 +paddd mm4, mm7 ; B2 b2 +paddd mm2, mm4 ; A2+B2 a2+b2 +psubd mm0, mm4 ; a2-B2 a2-b2 +psrad mm2, %7 +psrad mm0, %7 +movq mm4, mm6 ; A3 a3 +paddd mm3, mm1 ; B3 b3 +paddd mm6, mm3 ; A3+B3 a3+b3 +psubd mm4, mm3 ; a3-B3 a3-b3 +psrad mm6, %7 +packssdw mm2, mm6 ; A3+B3 a3+b3 A2+B2 a2+b2 +movq [8+ %5], mm2 +psrad mm4, %7 +packssdw mm4, mm0 ; A2-B2 a2-b2 A3-B3 a3-b3 +movq [16+ %5], mm4 +jmp %%2 +%%1: +pslld mm0, 16 +paddd mm0, [d40000] +psrad mm0, 13 +packssdw mm0, mm0 +movq [%5], mm0 +movq [8+ %5], mm0 +movq [16+ %5], mm0 +movq [24+ %5], mm0 +%%2: +%endmacro + +%macro Z_COND_IDCT 8 +movq mm0, [blockq + %1] ; R4 R0 r4 r0 +movq mm1, [blockq + %2] ; R6 R2 r6 r2 +movq mm2, [blockq + %3] ; R3 R1 r3 r1 +movq mm3, [blockq + %4] ; R7 R5 r7 r5 +movq mm4, mm0 +por mm4, mm1 +por mm4, mm2 +por mm4, mm3 +packssdw mm4, mm4 +movd t0d, mm4 +or t0d, t0d +jz %8 +movq mm4, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm4, mm0 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm0, mm5 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm5, [coeffs + 32] ; C6 C2 C6 C2 +pmaddwd mm5, mm1 ; C6R6+C2R2 C6r6+C2r2 +movq mm6, [coeffs + 40] ; -C2 C6 -C2 C6 +pmaddwd mm1, mm6 ; -C2R6+C6R2 -C2r6+C6r2 +movq mm7, [coeffs + 48] ; C3 C1 C3 C1 +pmaddwd mm7, mm2 ; C3R3+C1R1 C3r3+C1r1 +paddd mm4, [coeffs] +movq mm6, mm4 ; C4R4+C4R0 C4r4+C4r0 +paddd mm4, mm5 ; A0 a0 +psubd mm6, mm5 ; A3 a3 +movq mm5, [coeffs + 56] ; C7 C5 C7 C5 +pmaddwd mm5, mm3 ; C7R7+C5R5 C7r7+C5r5 +paddd mm0, [coeffs] +paddd mm1, mm0 ; A1 a1 +paddd mm0, mm0 +psubd mm0, mm1 ; A2 a2 +pmaddwd mm2, [coeffs + 64] ; -C7R3+C3R1 -C7r3+C3r1 +paddd mm7, mm5 ; B0 b0 +movq mm5, [coeffs + 72] ; -C5 -C1 -C5 -C1 +pmaddwd mm5, mm3 ; -C5R7-C1R5 -C5r7-C1r5 +paddd mm7, mm4 ; A0+B0 a0+b0 +paddd mm4, mm4 ; 2A0 2a0 +psubd mm4, mm7 ; A0-B0 a0-b0 +paddd mm5, mm2 ; B1 b1 +psrad mm7, %7 +psrad mm4, %7 +movq mm2, mm1 ; A1 a1 +paddd mm1, mm5 ; A1+B1 a1+b1 +psubd mm2, mm5 ; A1-B1 a1-b1 +psrad mm1, %7 +psrad mm2, %7 +packssdw mm7, mm1 ; A1+B1 a1+b1 A0+B0 a0+b0 +packssdw mm2, mm4 ; A0-B0 a0-b0 A1-B1 a1-b1 +movq [%5], mm7 +movq mm1, [blockq + %3] ; R3 R1 r3 r1 +movq mm4, [coeffs + 80] ; -C1 C5 -C1 C5 +movq [24+ %5], mm2 +pmaddwd mm4, mm1 ; -C1R3+C5R1 -C1r3+C5r1 +movq mm7, [coeffs + 88] ; C3 C7 C3 C7 +pmaddwd mm1, [coeffs + 96] ; -C5R3+C7R1 -C5r3+C7r1 +pmaddwd mm7, mm3 ; C3R7+C7R5 C3r7+C7r5 +movq mm2, mm0 ; A2 a2 +pmaddwd mm3, [coeffs + 104] ; -C1R7+C3R5 -C1r7+C3r5 +paddd mm4, mm7 ; B2 b2 +paddd mm2, mm4 ; A2+B2 a2+b2 +psubd mm0, mm4 ; a2-B2 a2-b2 +psrad mm2, %7 +psrad mm0, %7 +movq mm4, mm6 ; A3 a3 +paddd mm3, mm1 ; B3 b3 +paddd mm6, mm3 ; A3+B3 a3+b3 +psubd mm4, mm3 ; a3-B3 a3-b3 +psrad mm6, %7 +packssdw mm2, mm6 ; A3+B3 a3+b3 A2+B2 a2+b2 +movq [8+ %5], mm2 +psrad mm4, %7 +packssdw mm4, mm0 ; A2-B2 a2-b2 A3-B3 a3-b3 +movq [16+ %5], mm4 +%endmacro + +%macro IDCT1 6 +movq mm0, %1 ; R4 R0 r4 r0 +movq mm1, %2 ; R6 R2 r6 r2 +movq mm2, %3 ; R3 R1 r3 r1 +movq mm3, %4 ; R7 R5 r7 r5 +movq mm4, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm4, mm0 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm0, mm5 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm5, [coeffs + 32] ; C6 C2 C6 C2 +pmaddwd mm5, mm1 ; C6R6+C2R2 C6r6+C2r2 +movq mm6, [coeffs + 40] ; -C2 C6 -C2 C6 +pmaddwd mm1, mm6 ; -C2R6+C6R2 -C2r6+C6r2 +movq mm6, mm4 ; C4R4+C4R0 C4r4+C4r0 +movq mm7, [coeffs + 48] ; C3 C1 C3 C1 +pmaddwd mm7, mm2 ; C3R3+C1R1 C3r3+C1r1 +paddd mm4, mm5 ; A0 a0 +psubd mm6, mm5 ; A3 a3 +movq mm5, mm0 ; -C4R4+C4R0 -C4r4+C4r0 +paddd mm0, mm1 ; A1 a1 +psubd mm5, mm1 ; A2 a2 +movq mm1, [coeffs + 56] ; C7 C5 C7 C5 +pmaddwd mm1, mm3 ; C7R7+C5R5 C7r7+C5r5 +pmaddwd mm2, [coeffs + 64] ; -C7R3+C3R1 -C7r3+C3r1 +paddd mm7, mm1 ; B0 b0 +movq mm1, [coeffs + 72] ; -C5 -C1 -C5 -C1 +pmaddwd mm1, mm3 ; -C5R7-C1R5 -C5r7-C1r5 +paddd mm7, mm4 ; A0+B0 a0+b0 +paddd mm4, mm4 ; 2A0 2a0 +psubd mm4, mm7 ; A0-B0 a0-b0 +paddd mm1, mm2 ; B1 b1 +psrad mm7, %6 +psrad mm4, %6 +movq mm2, mm0 ; A1 a1 +paddd mm0, mm1 ; A1+B1 a1+b1 +psubd mm2, mm1 ; A1-B1 a1-b1 +psrad mm0, %6 +psrad mm2, %6 +packssdw mm7, mm7 ; A0+B0 a0+b0 +movd [%5], mm7 +packssdw mm0, mm0 ; A1+B1 a1+b1 +movd [16+ %5], mm0 +packssdw mm2, mm2 ; A1-B1 a1-b1 +movd [96+ %5], mm2 +packssdw mm4, mm4 ; A0-B0 a0-b0 +movd [112+ %5], mm4 +movq mm0, %3 ; R3 R1 r3 r1 +movq mm4, [coeffs + 80] ; -C1 C5 -C1 C5 +pmaddwd mm4, mm0 ; -C1R3+C5R1 -C1r3+C5r1 +movq mm7, [coeffs + 88] ; C3 C7 C3 C7 +pmaddwd mm0, [coeffs + 96] ; -C5R3+C7R1 -C5r3+C7r1 +pmaddwd mm7, mm3 ; C3R7+C7R5 C3r7+C7r5 +movq mm2, mm5 ; A2 a2 +pmaddwd mm3, [coeffs + 104] ; -C1R7+C3R5 -C1r7+C3r5 +paddd mm4, mm7 ; B2 b2 +paddd mm2, mm4 ; A2+B2 a2+b2 +psubd mm5, mm4 ; a2-B2 a2-b2 +psrad mm2, %6 +psrad mm5, %6 +movq mm4, mm6 ; A3 a3 +paddd mm3, mm0 ; B3 b3 +paddd mm6, mm3 ; A3+B3 a3+b3 +psubd mm4, mm3 ; a3-B3 a3-b3 +psrad mm6, %6 +psrad mm4, %6 +packssdw mm2, mm2 ; A2+B2 a2+b2 +packssdw mm6, mm6 ; A3+B3 a3+b3 +movd [32+ %5], mm2 +packssdw mm4, mm4 ; A3-B3 a3-b3 +packssdw mm5, mm5 ; A2-B2 a2-b2 +movd [48+ %5], mm6 +movd [64+ %5], mm4 +movd [80+ %5], mm5 +%endmacro + +%macro IDCT2 6 +movq mm0, %1 ; R4 R0 r4 r0 +movq mm1, %2 ; R6 R2 r6 r2 +movq mm3, %4 ; R7 R5 r7 r5 +movq mm4, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm4, mm0 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm0, mm5 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm5, [coeffs + 32] ; C6 C2 C6 C2 +pmaddwd mm5, mm1 ; C6R6+C2R2 C6r6+C2r2 +movq mm6, [coeffs + 40] ; -C2 C6 -C2 C6 +pmaddwd mm1, mm6 ; -C2R6+C6R2 -C2r6+C6r2 +movq mm6, mm4 ; C4R4+C4R0 C4r4+C4r0 +paddd mm4, mm5 ; A0 a0 +psubd mm6, mm5 ; A3 a3 +movq mm5, mm0 ; -C4R4+C4R0 -C4r4+C4r0 +paddd mm0, mm1 ; A1 a1 +psubd mm5, mm1 ; A2 a2 +movq mm1, [coeffs + 56] ; C7 C5 C7 C5 +pmaddwd mm1, mm3 ; C7R7+C5R5 C7r7+C5r5 +movq mm7, [coeffs + 72] ; -C5 -C1 -C5 -C1 +pmaddwd mm7, mm3 ; -C5R7-C1R5 -C5r7-C1r5 +paddd mm1, mm4 ; A0+B0 a0+b0 +paddd mm4, mm4 ; 2A0 2a0 +psubd mm4, mm1 ; A0-B0 a0-b0 +psrad mm1, %6 +psrad mm4, %6 +movq mm2, mm0 ; A1 a1 +paddd mm0, mm7 ; A1+B1 a1+b1 +psubd mm2, mm7 ; A1-B1 a1-b1 +psrad mm0, %6 +psrad mm2, %6 +packssdw mm1, mm1 ; A0+B0 a0+b0 +movd [%5], mm1 +packssdw mm0, mm0 ; A1+B1 a1+b1 +movd [16+ %5], mm0 +packssdw mm2, mm2 ; A1-B1 a1-b1 +movd [96+ %5], mm2 +packssdw mm4, mm4 ; A0-B0 a0-b0 +movd [112+ %5], mm4 +movq mm1, [coeffs + 88] ; C3 C7 C3 C7 +pmaddwd mm1, mm3 ; C3R7+C7R5 C3r7+C7r5 +movq mm2, mm5 ; A2 a2 +pmaddwd mm3, [coeffs + 104] ; -C1R7+C3R5 -C1r7+C3r5 +paddd mm2, mm1 ; A2+B2 a2+b2 +psubd mm5, mm1 ; a2-B2 a2-b2 +psrad mm2, %6 +psrad mm5, %6 +movq mm1, mm6 ; A3 a3 +paddd mm6, mm3 ; A3+B3 a3+b3 +psubd mm1, mm3 ; a3-B3 a3-b3 +psrad mm6, %6 +psrad mm1, %6 +packssdw mm2, mm2 ; A2+B2 a2+b2 +packssdw mm6, mm6 ; A3+B3 a3+b3 +movd [32+ %5], mm2 +packssdw mm1, mm1 ; A3-B3 a3-b3 +packssdw mm5, mm5 ; A2-B2 a2-b2 +movd [48+ %5], mm6 +movd [64+ %5], mm1 +movd [80+ %5], mm5 +%endmacro + +%macro IDCT3 6 +movq mm0, %1 ; R4 R0 r4 r0 +movq mm3, %4 ; R7 R5 r7 r5 +movq mm4, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm4, mm0 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm0, mm5 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm6, mm4 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, mm0 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm1, [coeffs + 56] ; C7 C5 C7 C5 +pmaddwd mm1, mm3 ; C7R7+C5R5 C7r7+C5r5 +movq mm7, [coeffs + 72] ; -C5 -C1 -C5 -C1 +pmaddwd mm7, mm3 ; -C5R7-C1R5 -C5r7-C1r5 +paddd mm1, mm4 ; A0+B0 a0+b0 +paddd mm4, mm4 ; 2A0 2a0 +psubd mm4, mm1 ; A0-B0 a0-b0 +psrad mm1, %6 +psrad mm4, %6 +movq mm2, mm0 ; A1 a1 +paddd mm0, mm7 ; A1+B1 a1+b1 +psubd mm2, mm7 ; A1-B1 a1-b1 +psrad mm0, %6 +psrad mm2, %6 +packssdw mm1, mm1 ; A0+B0 a0+b0 +movd [%5], mm1 +packssdw mm0, mm0 ; A1+B1 a1+b1 +movd [16+ %5], mm0 +packssdw mm2, mm2 ; A1-B1 a1-b1 +movd [96+ %5], mm2 +packssdw mm4, mm4 ; A0-B0 a0-b0 +movd [112+ %5], mm4 +movq mm1, [coeffs + 88] ; C3 C7 C3 C7 +pmaddwd mm1, mm3 ; C3R7+C7R5 C3r7+C7r5 +movq mm2, mm5 ; A2 a2 +pmaddwd mm3, [coeffs + 104] ; -C1R7+C3R5 -C1r7+C3r5 +paddd mm2, mm1 ; A2+B2 a2+b2 +psubd mm5, mm1 ; a2-B2 a2-b2 +psrad mm2, %6 +psrad mm5, %6 +movq mm1, mm6 ; A3 a3 +paddd mm6, mm3 ; A3+B3 a3+b3 +psubd mm1, mm3 ; a3-B3 a3-b3 +psrad mm6, %6 +psrad mm1, %6 +packssdw mm2, mm2 ; A2+B2 a2+b2 +packssdw mm6, mm6 ; A3+B3 a3+b3 +movd [32+ %5], mm2 +packssdw mm1, mm1 ; A3-B3 a3-b3 +packssdw mm5, mm5 ; A2-B2 a2-b2 +movd [48+ %5], mm6 +movd [64+ %5], mm1 +movd [80+ %5], mm5 +%endmacro + +%macro IDCT4 6 +movq mm0, %1 ; R4 R0 r4 r0 +movq mm2, %3 ; R3 R1 r3 r1 +movq mm3, %4 ; R7 R5 r7 r5 +movq mm4, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm4, mm0 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm0, mm5 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm6, mm4 ; C4R4+C4R0 C4r4+C4r0 +movq mm7, [coeffs + 48] ; C3 C1 C3 C1 +pmaddwd mm7, mm2 ; C3R3+C1R1 C3r3+C1r1 +movq mm5, mm0 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm1, [coeffs + 56] ; C7 C5 C7 C5 +pmaddwd mm1, mm3 ; C7R7+C5R5 C7r7+C5r5 +pmaddwd mm2, [coeffs + 64] ; -C7R3+C3R1 -C7r3+C3r1 +paddd mm7, mm1 ; B0 b0 +movq mm1, [coeffs + 72] ; -C5 -C1 -C5 -C1 +pmaddwd mm1, mm3 ; -C5R7-C1R5 -C5r7-C1r5 +paddd mm7, mm4 ; A0+B0 a0+b0 +paddd mm4, mm4 ; 2A0 2a0 +psubd mm4, mm7 ; A0-B0 a0-b0 +paddd mm1, mm2 ; B1 b1 +psrad mm7, %6 +psrad mm4, %6 +movq mm2, mm0 ; A1 a1 +paddd mm0, mm1 ; A1+B1 a1+b1 +psubd mm2, mm1 ; A1-B1 a1-b1 +psrad mm0, %6 +psrad mm2, %6 +packssdw mm7, mm7 ; A0+B0 a0+b0 +movd [%5], mm7 +packssdw mm0, mm0 ; A1+B1 a1+b1 +movd [16+ %5], mm0 +packssdw mm2, mm2 ; A1-B1 a1-b1 +movd [96+ %5], mm2 +packssdw mm4, mm4 ; A0-B0 a0-b0 +movd [112+ %5], mm4 +movq mm0, %3 ; R3 R1 r3 r1 +movq mm4, [coeffs + 80] ; -C1 C5 -C1 C5 +pmaddwd mm4, mm0 ; -C1R3+C5R1 -C1r3+C5r1 +movq mm7, [coeffs + 88] ; C3 C7 C3 C7 +pmaddwd mm0, [coeffs + 96] ; -C5R3+C7R1 -C5r3+C7r1 +pmaddwd mm7, mm3 ; C3R7+C7R5 C3r7+C7r5 +movq mm2, mm5 ; A2 a2 +pmaddwd mm3, [coeffs + 104] ; -C1R7+C3R5 -C1r7+C3r5 +paddd mm4, mm7 ; B2 b2 +paddd mm2, mm4 ; A2+B2 a2+b2 +psubd mm5, mm4 ; a2-B2 a2-b2 +psrad mm2, %6 +psrad mm5, %6 +movq mm4, mm6 ; A3 a3 +paddd mm3, mm0 ; B3 b3 +paddd mm6, mm3 ; A3+B3 a3+b3 +psubd mm4, mm3 ; a3-B3 a3-b3 +psrad mm6, %6 +psrad mm4, %6 +packssdw mm2, mm2 ; A2+B2 a2+b2 +packssdw mm6, mm6 ; A3+B3 a3+b3 +movd [32+ %5], mm2 +packssdw mm4, mm4 ; A3-B3 a3-b3 +packssdw mm5, mm5 ; A2-B2 a2-b2 +movd [48+ %5], mm6 +movd [64+ %5], mm4 +movd [80+ %5], mm5 +%endmacro + +%macro IDCT5 6 +movq mm0, %1 ; R4 R0 r4 r0 +movq mm2, %3 ; R3 R1 r3 r1 +movq mm4, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm4, mm0 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm0, mm5 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm6, mm4 ; C4R4+C4R0 C4r4+C4r0 +movq mm7, [coeffs + 48] ; C3 C1 C3 C1 +pmaddwd mm7, mm2 ; C3R3+C1R1 C3r3+C1r1 +movq mm5, mm0 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm3, [coeffs + 64] +pmaddwd mm3, mm2 ; -C7R3+C3R1 -C7r3+C3r1 +paddd mm7, mm4 ; A0+B0 a0+b0 +paddd mm4, mm4 ; 2A0 2a0 +psubd mm4, mm7 ; A0-B0 a0-b0 +psrad mm7, %6 +psrad mm4, %6 +movq mm1, mm0 ; A1 a1 +paddd mm0, mm3 ; A1+B1 a1+b1 +psubd mm1, mm3 ; A1-B1 a1-b1 +psrad mm0, %6 +psrad mm1, %6 +packssdw mm7, mm7 ; A0+B0 a0+b0 +movd [%5], mm7 +packssdw mm0, mm0 ; A1+B1 a1+b1 +movd [16+ %5], mm0 +packssdw mm1, mm1 ; A1-B1 a1-b1 +movd [96+ %5], mm1 +packssdw mm4, mm4 ; A0-B0 a0-b0 +movd [112+ %5], mm4 +movq mm4, [coeffs + 80] ; -C1 C5 -C1 C5 +pmaddwd mm4, mm2 ; -C1R3+C5R1 -C1r3+C5r1 +pmaddwd mm2, [coeffs + 96] ; -C5R3+C7R1 -C5r3+C7r1 +movq mm1, mm5 ; A2 a2 +paddd mm1, mm4 ; A2+B2 a2+b2 +psubd mm5, mm4 ; a2-B2 a2-b2 +psrad mm1, %6 +psrad mm5, %6 +movq mm4, mm6 ; A3 a3 +paddd mm6, mm2 ; A3+B3 a3+b3 +psubd mm4, mm2 ; a3-B3 a3-b3 +psrad mm6, %6 +psrad mm4, %6 +packssdw mm1, mm1 ; A2+B2 a2+b2 +packssdw mm6, mm6 ; A3+B3 a3+b3 +movd [32+ %5], mm1 +packssdw mm4, mm4 ; A3-B3 a3-b3 +packssdw mm5, mm5 ; A2-B2 a2-b2 +movd [48+ %5], mm6 +movd [64+ %5], mm4 +movd [80+ %5], mm5 +%endmacro + +%macro IDCT6 6 +movq mm0, [%1] ; R4 R0 r4 r0 +movq mm1, [%2] ; R6 R2 r6 r2 +movq mm4, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm4, mm0 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm0, mm5 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm5, [coeffs + 32] ; C6 C2 C6 C2 +pmaddwd mm5, mm1 ; C6R6+C2R2 C6r6+C2r2 +movq mm6, [coeffs + 40] ; -C2 C6 -C2 C6 +pmaddwd mm1, mm6 ; -C2R6+C6R2 -C2r6+C6r2 +movq mm6, mm4 ; C4R4+C4R0 C4r4+C4r0 +paddd mm4, mm5 ; A0 a0 +psubd mm6, mm5 ; A3 a3 +movq mm5, mm0 ; -C4R4+C4R0 -C4r4+C4r0 +paddd mm0, mm1 ; A1 a1 +psubd mm5, mm1 ; A2 a2 +movq mm2, [8+ %1] ; R4 R0 r4 r0 +movq mm3, [8+ %2] ; R6 R2 r6 r2 +movq mm1, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm1, mm2 ; C4R4+C4R0 C4r4+C4r0 +movq mm7, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm2, mm7 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm7, [coeffs + 32] ; C6 C2 C6 C2 +pmaddwd mm7, mm3 ; C6R6+C2R2 C6r6+C2r2 +pmaddwd mm3, [coeffs + 40] ; -C2R6+C6R2 -C2r6+C6r2 +paddd mm7, mm1 ; A0 a0 +paddd mm1, mm1 ; 2C0 2c0 +psubd mm1, mm7 ; A3 a3 +paddd mm3, mm2 ; A1 a1 +paddd mm2, mm2 ; 2C1 2c1 +psubd mm2, mm3 ; A2 a2 +psrad mm4, %6 +psrad mm7, %6 +psrad mm3, %6 +packssdw mm4, mm7 ; A0 a0 +movq [%5], mm4 +psrad mm0, %6 +packssdw mm0, mm3 ; A1 a1 +movq [16+ %5], mm0 +movq [96+ %5], mm0 +movq [112+ %5], mm4 +psrad mm5, %6 +psrad mm6, %6 +psrad mm2, %6 +packssdw mm5, mm2 ; A2-B2 a2-b2 +movq [32+ %5], mm5 +psrad mm1, %6 +packssdw mm6, mm1 ; A3+B3 a3+b3 +movq [48+ %5], mm6 +movq [64+ %5], mm6 +movq [80+ %5], mm5 +%endmacro + +%macro IDCT7 6 +movq mm0, %1 ; R4 R0 r4 r0 +movq mm1, %2 ; R6 R2 r6 r2 +movq mm2, %3 ; R3 R1 r3 r1 +movq mm4, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm4, mm0 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm0, mm5 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm5, [coeffs + 32] ; C6 C2 C6 C2 +pmaddwd mm5, mm1 ; C6R6+C2R2 C6r6+C2r2 +movq mm6, [coeffs + 40] ; -C2 C6 -C2 C6 +pmaddwd mm1, mm6 ; -C2R6+C6R2 -C2r6+C6r2 +movq mm6, mm4 ; C4R4+C4R0 C4r4+C4r0 +movq mm7, [coeffs + 48] ; C3 C1 C3 C1 +pmaddwd mm7, mm2 ; C3R3+C1R1 C3r3+C1r1 +paddd mm4, mm5 ; A0 a0 +psubd mm6, mm5 ; A3 a3 +movq mm5, mm0 ; -C4R4+C4R0 -C4r4+C4r0 +paddd mm0, mm1 ; A1 a1 +psubd mm5, mm1 ; A2 a2 +movq mm1, [coeffs + 64] +pmaddwd mm1, mm2 ; -C7R3+C3R1 -C7r3+C3r1 +paddd mm7, mm4 ; A0+B0 a0+b0 +paddd mm4, mm4 ; 2A0 2a0 +psubd mm4, mm7 ; A0-B0 a0-b0 +psrad mm7, %6 +psrad mm4, %6 +movq mm3, mm0 ; A1 a1 +paddd mm0, mm1 ; A1+B1 a1+b1 +psubd mm3, mm1 ; A1-B1 a1-b1 +psrad mm0, %6 +psrad mm3, %6 +packssdw mm7, mm7 ; A0+B0 a0+b0 +movd [%5], mm7 +packssdw mm0, mm0 ; A1+B1 a1+b1 +movd [16+ %5], mm0 +packssdw mm3, mm3 ; A1-B1 a1-b1 +movd [96+ %5], mm3 +packssdw mm4, mm4 ; A0-B0 a0-b0 +movd [112+ %5], mm4 +movq mm4, [coeffs + 80] ; -C1 C5 -C1 C5 +pmaddwd mm4, mm2 ; -C1R3+C5R1 -C1r3+C5r1 +pmaddwd mm2, [coeffs + 96] ; -C5R3+C7R1 -C5r3+C7r1 +movq mm3, mm5 ; A2 a2 +paddd mm3, mm4 ; A2+B2 a2+b2 +psubd mm5, mm4 ; a2-B2 a2-b2 +psrad mm3, %6 +psrad mm5, %6 +movq mm4, mm6 ; A3 a3 +paddd mm6, mm2 ; A3+B3 a3+b3 +psubd mm4, mm2 ; a3-B3 a3-b3 +psrad mm6, %6 +packssdw mm3, mm3 ; A2+B2 a2+b2 +movd [32+ %5], mm3 +psrad mm4, %6 +packssdw mm6, mm6 ; A3+B3 a3+b3 +movd [48+ %5], mm6 +packssdw mm4, mm4 ; A3-B3 a3-b3 +packssdw mm5, mm5 ; A2-B2 a2-b2 +movd [64+ %5], mm4 +movd [80+ %5], mm5 +%endmacro + +%macro IDCT8 6 +movq mm0, [%1] ; R4 R0 r4 r0 +movq mm4, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm4, mm0 ; C4R4+C4R0 C4r4+C4r0 +movq mm5, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm0, mm5 ; -C4R4+C4R0 -C4r4+C4r0 +psrad mm4, %6 +psrad mm0, %6 +movq mm2, [8+ %1] ; R4 R0 r4 r0 +movq mm1, [coeffs + 16] ; C4 C4 C4 C4 +pmaddwd mm1, mm2 ; C4R4+C4R0 C4r4+C4r0 +movq mm7, [coeffs + 24] ; -C4 C4 -C4 C4 +pmaddwd mm2, mm7 ; -C4R4+C4R0 -C4r4+C4r0 +movq mm7, [coeffs + 32] ; C6 C2 C6 C2 +psrad mm1, %6 +packssdw mm4, mm1 ; A0 a0 +movq [%5], mm4 +psrad mm2, %6 +packssdw mm0, mm2 ; A1 a1 +movq [16+ %5], mm0 +movq [96+ %5], mm0 +movq [112+ %5], mm4 +movq [32+ %5], mm0 +movq [48+ %5], mm4 +movq [64+ %5], mm4 +movq [80+ %5], mm0 +%endmacro + +%macro IDCT 0 +; :: "r" (block), "r" (temp), "r" (coeffs) +; NAMED_CONSTRAINTS_ADD(wm1010,d40000) +; : "%eax" + DC_COND_IDCT 0, 8, 16, 24, rsp + 0, null, 11 + Z_COND_IDCT 32, 40, 48, 56, rsp + 32, null, 11, %%4 + Z_COND_IDCT 64, 72, 80, 88, rsp + 64, null, 11, %%2 + Z_COND_IDCT 96, 104, 112, 120, rsp + 96, null, 11, %%1 + + IDCT1 [rsp + 0], [rsp + 64], [rsp + 32], [rsp + 96], blockq + 0, 20 + IDCT1 [rsp + 8], [rsp + 72], [rsp + 40], [rsp + 104], blockq + 4, 20 + IDCT1 [rsp + 16], [rsp + 80], [rsp + 48], [rsp + 112], blockq + 8, 20 + IDCT1 [rsp + 24], [rsp + 88], [rsp + 56], [rsp + 120], blockq + 12, 20 + jmp %%9 + + ALIGN 16 + %%4: + Z_COND_IDCT 64, 72, 80, 88, rsp + 64, null, 11, %%6 + Z_COND_IDCT 96, 104, 112, 120, rsp + 96, null, 11, %%5 + + IDCT2 [rsp + 0], [rsp + 64], [rsp + 32], [rsp + 96], blockq + 0, 20 + IDCT2 [rsp + 8], [rsp + 72], [rsp + 40], [rsp + 104], blockq + 4, 20 + IDCT2 [rsp + 16], [rsp + 80], [rsp + 48], [rsp + 112], blockq + 8, 20 + IDCT2 [rsp + 24], [rsp + 88], [rsp + 56], [rsp + 120], blockq + 12, 20 + jmp %%9 + + ALIGN 16 + %%6: + Z_COND_IDCT 96, 104, 112, 120, rsp + 96, null, 11, %%7 + + IDCT3 [rsp + 0], [rsp + 64], [rsp + 32], [rsp + 96], blockq + 0, 20 + IDCT3 [rsp + 8], [rsp + 72], [rsp + 40], [rsp + 104], blockq + 4, 20 + IDCT3 [rsp + 16], [rsp + 80], [rsp + 48], [rsp + 112], blockq + 8, 20 + IDCT3 [rsp + 24], [rsp + 88], [rsp + 56], [rsp + 120], blockq + 12, 20 + jmp %%9 + + ALIGN 16 + %%2: + Z_COND_IDCT 96, 104, 112, 120, rsp + 96, null, 11, %%3 + + IDCT4 [rsp + 0], [rsp + 64], [rsp + 32], [rsp + 96], blockq + 0, 20 + IDCT4 [rsp + 8], [rsp + 72], [rsp + 40], [rsp + 104], blockq + 4, 20 + IDCT4 [rsp + 16], [rsp + 80], [rsp + 48], [rsp + 112], blockq + 8, 20 + IDCT4 [rsp + 24], [rsp + 88], [rsp + 56], [rsp + 120], blockq + 12, 20 + jmp %%9 + + ALIGN 16 + %%3: + + IDCT5 [rsp + 0], [rsp + 64], [rsp + 32], [rsp + 96], blockq + 0, 20 + IDCT5 [rsp + 8], [rsp + 72], [rsp + 40], [rsp + 104], blockq + 4, 20 + IDCT5 [rsp + 16], [rsp + 80], [rsp + 48], [rsp + 112], blockq + 8, 20 + IDCT5 [rsp + 24], [rsp + 88], [rsp + 56], [rsp + 120], blockq + 12, 20 + jmp %%9 + + ALIGN 16 + %%5: + + IDCT6 rsp + 0, rsp + 64, rsp + 32, rsp + 96, blockq + 0, 20 + IDCT6 rsp + 16, rsp + 80, rsp + 48, rsp + 112, blockq + 8, 20 + jmp %%9 + + ALIGN 16 + %%1: + + IDCT7 [rsp + 0], [rsp + 64], [rsp + 32], [rsp + 96], blockq + 0, 20 + IDCT7 [rsp + 8], [rsp + 72], [rsp + 40], [rsp + 104], blockq + 4, 20 + IDCT7 [rsp + 16], [rsp + 80], [rsp + 48], [rsp + 112], blockq + 8, 20 + IDCT7 [rsp + 24], [rsp + 88], [rsp + 56], [rsp + 120], blockq + 12, 20 + jmp %%9 + + ALIGN 16 + %%7: + + IDCT8 rsp + 0, rsp + 64, rsp + 32, rsp + 96, blockq + 0, 20 + IDCT8 rsp + 16, rsp + 80, rsp + 48, rsp + 112, blockq + 8, 20 + + %%9: +; :: "r" (block), "r" (temp), "r" (coeffs) +; NAMED_CONSTRAINTS_ADD(wm1010,d40000) +; : "%eax" +%endmacro + +%macro PUT_PIXELS_CLAMPED_HALF 1 + mova m0, [blockq+mmsize*0+%1] + mova m1, [blockq+mmsize*2+%1] +%if mmsize == 8 + mova m2, [blockq+mmsize*4+%1] + mova m3, [blockq+mmsize*6+%1] +%endif + packuswb m0, [blockq+mmsize*1+%1] + packuswb m1, [blockq+mmsize*3+%1] +%if mmsize == 8 + packuswb m2, [blockq+mmsize*5+%1] + packuswb m3, [blockq+mmsize*7+%1] + movq [pixelsq], m0 + movq [lsizeq+pixelsq], m1 + movq [2*lsizeq+pixelsq], m2 + movq [lsize3q+pixelsq], m3 +%else + movq [pixelsq], m0 + movhps [lsizeq+pixelsq], m0 + movq [2*lsizeq+pixelsq], m1 + movhps [lsize3q+pixelsq], m1 +%endif +%endmacro + +%macro ADD_PIXELS_CLAMPED 1 + mova m0, [blockq+mmsize*0+%1] + mova m1, [blockq+mmsize*1+%1] +%if mmsize == 8 + mova m5, [blockq+mmsize*2+%1] + mova m6, [blockq+mmsize*3+%1] +%endif + movq m2, [pixelsq] + movq m3, [pixelsq+lsizeq] +%if mmsize == 8 + mova m7, m2 + punpcklbw m2, m4 + punpckhbw m7, m4 + paddsw m0, m2 + paddsw m1, m7 + mova m7, m3 + punpcklbw m3, m4 + punpckhbw m7, m4 + paddsw m5, m3 + paddsw m6, m7 +%else + punpcklbw m2, m4 + punpcklbw m3, m4 + paddsw m0, m2 + paddsw m1, m3 +%endif + packuswb m0, m1 +%if mmsize == 8 + packuswb m5, m6 + movq [pixelsq], m0 + movq [pixelsq+lsizeq], m5 +%else + movq [pixelsq], m0 + movhps [pixelsq+lsizeq], m0 +%endif +%endmacro + +INIT_MMX mmx + +cglobal simple_idct, 1, 2, 8, 128, block, t0 + IDCT +RET + +cglobal simple_idct_put, 3, 5, 8, 128, pixels, lsize, block, lsize3, t0 + IDCT + lea lsize3q, [lsizeq*3] + PUT_PIXELS_CLAMPED_HALF 0 + lea pixelsq, [pixelsq+lsizeq*4] + PUT_PIXELS_CLAMPED_HALF 64 +RET + +cglobal simple_idct_add, 3, 4, 8, 128, pixels, lsize, block, t0 + IDCT + pxor m4, m4 + ADD_PIXELS_CLAMPED 0 + lea pixelsq, [pixelsq+lsizeq*2] + ADD_PIXELS_CLAMPED 32 + lea pixelsq, [pixelsq+lsizeq*2] + ADD_PIXELS_CLAMPED 64 + lea pixelsq, [pixelsq+lsizeq*2] + ADD_PIXELS_CLAMPED 96 +RET + +INIT_XMM sse2 + +cglobal simple_idct_put, 3, 5, 8, 128, pixels, lsize, block, lsize3, t0 + IDCT + lea lsize3q, [lsizeq*3] + PUT_PIXELS_CLAMPED_HALF 0 + lea pixelsq, [pixelsq+lsizeq*4] + PUT_PIXELS_CLAMPED_HALF 64 +RET + +cglobal simple_idct_add, 3, 4, 8, 128, pixels, lsize, block, t0 + IDCT + pxor m4, m4 + ADD_PIXELS_CLAMPED 0 + lea pixelsq, [pixelsq+lsizeq*2] + ADD_PIXELS_CLAMPED 32 + lea pixelsq, [pixelsq+lsizeq*2] + ADD_PIXELS_CLAMPED 64 + lea pixelsq, [pixelsq+lsizeq*2] + ADD_PIXELS_CLAMPED 96 +RET