From patchwork Sun Aug 18 01:48:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nuo Mi X-Patchwork-Id: 51062 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:b6ca:0:b0:48e:c0f8:d0de with SMTP id s10csp1228950vqj; Sat, 17 Aug 2024 18:49:06 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUaIaQPFYf+1U/Y/zxZiBFrBPKCYG1RDH7F8I0/1Lnag90SmTKMzlBNrA3i2a/8BD/mlqr5ttEAuv/iVcORZPX0@gmail.com X-Google-Smtp-Source: AGHT+IGArpFHyGVy4S3v2WGTS0wYl15k3mnk3IVLimp2bL8Z39oJqtHaA9H2lQlyIv3qOyHYKym3 X-Received: by 2002:a2e:f19:0:b0:2f3:d032:44a9 with SMTP id 38308e7fff4ca-2f3d0324836mr5001671fa.0.1723945745704; Sat, 17 Aug 2024 18:49:05 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 38308e7fff4ca-2f3b7716d63si17303961fa.457.2024.08.17.18.49.05; Sat, 17 Aug 2024 18:49:05 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@outlook.com header.s=selector1 header.b=KhtfOg+K; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 696CA68DD7D; Sun, 18 Aug 2024 04:48:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from APC01-PSA-obe.outbound.protection.outlook.com (mail-psaapc01olkn2023.outbound.protection.outlook.com [40.92.52.23]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 10C7168DD74 for ; Sun, 18 Aug 2024 04:48:44 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=j2tvptXthXwejpM0RTlzADpcgE/TYJ1cEZ99atsq2iUw7nWZUU5C6PI+4QO3e6mIxxdj1sYEPxLMiueMtikMg5U45VSB8mIMh5JB4TugIftmEIUV8x/mIG1DhOPeS6d3xxT5hZuutqCx40pR0HhU6Wx7xdfFZKeK/MPxOvfG0+HbvoFlIjALF2nW86j17lWh+74bfgbDKfAH5PcTEob1zsY9V0g9WmID6lj8QXYkHTISEP8ZNM+LXP3N9xyYczzF7duGLfBbf0PodubID2YnPllLCkwy4DVrDBKWO4ntxQhjjuiR2nXi3MLHeO4PfMeXf1IMrMlYMauHAhgWc2Ro7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P0vpY+rmyPRqfeyBKxiKOFrl3UxwD1J3TmIfercHdro=; b=CwXOsA4T2Mb9P8q8/IQL2kFL/Bal6BJeUP1fOiSdtY2RHJArcZYaRdSZPExpGCFvDz1pYTr7JRd1fPccfj05Sc8jD3LS1HNYwNr4xT4uVCchBO2U3hFfJ0zVjdP5ehS8VaoomOtaMUYth8yQxxTAD+ZMaNIM3uek+pfA0sAd+Y41Om6q0svoR3hc++hjyC4Ct/hsRKtrQi5an4/2mLLBl23KxcKQy0UQnE0INLpC2PD30HWi+GTY7hcHC6YF9ioH1igkeDjVOkfSZXURms3gV9kBDgT+gQVIEwooRxau6gjnOjDXxmxq9qEZOItLcxNwC3AzVYgO1IYWBhujz5zuWg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P0vpY+rmyPRqfeyBKxiKOFrl3UxwD1J3TmIfercHdro=; b=KhtfOg+KEsSZc9a4aX6rLz+ictpy4kDGx39A3fpywF60qt13ueUILbLNegRtBhniN6CvM1gFeE5EoxkMpFOmP2Nmq0622FzYS+ItlDlWrMSd3GIYZdpgbHY5UGAaV5TaNh50B/EG4beOHSsi6d6Opj3n0nJ9YwuC6F04pMq25Hq91Rp8UdoBKQ1GUKZiBRJ0NGwTcGyHsR7P1YnzWx48DQRqXKUmMsaHZbtVNh3QB3tLYhn1PzrMnOwh9Q7Y6+D7GrMgjXO+0QhS0EKT+HafovV9aEnCxnSAb9RTpsMzog4T3SL/RRBIpWIxN+M0xkIvGTXt1SqlIJPWpyjUhaGDqA== Received: from TYSPR06MB6433.apcprd06.prod.outlook.com (2603:1096:400:47a::6) by SEYPR06MB5208.apcprd06.prod.outlook.com (2603:1096:101:88::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7875.20; Sun, 18 Aug 2024 01:48:28 +0000 Received: from TYSPR06MB6433.apcprd06.prod.outlook.com ([fe80::81f7:9125:583a:1cca]) by TYSPR06MB6433.apcprd06.prod.outlook.com ([fe80::81f7:9125:583a:1cca%5]) with mapi id 15.20.7875.019; Sun, 18 Aug 2024 01:48:28 +0000 From: Nuo Mi To: ffmpeg-devel@ffmpeg.org Date: Sun, 18 Aug 2024 09:48:06 +0800 Message-ID: X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240818014807.47423-1-nuomi2021@gmail.com> References: <20240818014807.47423-1-nuomi2021@gmail.com> X-TMN: [hzgOfztUWuf4t+YvBEX8ATAVzrHy+Te9] X-ClientProxiedBy: TY2PR0101CA0034.apcprd01.prod.exchangelabs.com (2603:1096:404:8000::20) To TYSPR06MB6433.apcprd06.prod.outlook.com (2603:1096:400:47a::6) X-Microsoft-Original-Message-ID: <20240818014807.47423-3-nuomi2021@gmail.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 2 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: TYSPR06MB6433:EE_|SEYPR06MB5208:EE_ X-MS-Office365-Filtering-Correlation-Id: ff5361f5-fcb6-4564-acf0-08dcbf27db12 X-Microsoft-Antispam: BCL:0; ARA:14566002|5072599009|19110799003|15080799003|8060799006|461199028|3412199025|440099028|1710799026; X-Microsoft-Antispam-Message-Info: pGecB3od1bdnZQB12VBwNqJ3eb7PYCp7WnD8zfTo3TGAj5LeXYT+nmmNNQRSYq1t3oRGT1ruWVREoIKsENkIUy80Kl+koHvIFHEkw6QSOEdSrAkSYqE+SARkaoqLs8fo+x9460TZPHmT75WMr2R9VZ6ydXT2j7IXW5LBtT53JFvfNOtLxNuBA84DHZ6K9MNq4NKZKxVy0rO3Cnprv8nxkaylu1t3lqsbIkN1D2zg0sXOa9LUAf6SXlDDlPe5ycJQvf1vrt0mPHu5LwaZcUWHi3dLwDsDa2gru9ggBdALAezi3tZEGobPASl40GhJBpEdjPGlJl3qb3Vy9GvAG0lNeULro7kk4WF3Mbh9bdfWIUCzF1pqISyNKllg4w8B5ubh6dzGXG8gSIzcT/kXm7ai/WAQ4qsGGs/ypTq7R1MgSdpGS8HFbzFhUcE4cpBrSyP+mWob5734Rn606HL4OMVAqw5kcGyEE4JwGC6PSHi4vc/2r0A3K4WKtrMdewOKn9XanHyHL5nZG4pEG2noC3sLqgrT8Hcw2SmQLiK+ubggKiymRxglMp9blZVsJUdf1UJ2+fvop/XIf3etdmRn4RNHHaLcloALiuQJW7gBf3mmrLLaRnmIOrTJXj4X9OctumKCG/JRCt1R1/Fi/SMgk+bv318agImnh9sNusiqYcTW0sQeDc/uIm+/7MhNSqRuvFcgpJDknmx0Ap62LSaWu+5SsKJtiCrxZWZKqnL/ej/PsMg= X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?EHemgWnf331ggwEMiHCpFe4EU6pW?= =?utf-8?q?ZFNPKQ6IyAG8WuHAS0klfRwpARG0Fb2m4Xix/Q6MmeWkxgvHC6T7NhnZo21Al/ZFV?= =?utf-8?q?UKZ+2IJ5DSZh3EdOvTLpHTySvsMbFyp4xiYvjkMYBuIQ1z4MhjZGUjzDSRbTnvDj5?= =?utf-8?q?jwFWMbsBWsOple+T4QUA/cHNv5NJGFOsE2RlvfcVxwCDRQ80u2lxdl4g8C+Hv5BZp?= =?utf-8?q?66k6yxqvb6/BxRIVknI3Rk7y/mtJT9i3W34ImyTNPkAvSebwdhIqY/tq58oPBMwTj?= =?utf-8?q?RYZrmkv6y4rJx+mSsMN6t3Ja1D9+NrQGGKXdmY8wcm+x0lEx984xwUQ4LJY9PAYEL?= =?utf-8?q?KK1sQMw/X3cCUbHbmBwL96yIZZDG5j5v5Kz/fBKVVtMmxtczoYI5WLciA9cu96F6A?= =?utf-8?q?DprL9uF+dT1+7Nm/j/Z8+vtIbnSDu5KE7VZcWIN/KQSaFMn5XAPxqI66+HvxPOAfJ?= =?utf-8?q?yLqVwqLDMQmzMUJIQ9OJm8PxRASPgllaeuKhUdFi0Ye9xqoSSNqovaR1W5ARg5Qg2?= =?utf-8?q?zCe1y1KmGWCTyGbFUulVaupuj4zBr5An/KREDiMOmkJPc82H8DQhp18+ZzyfHijPi?= =?utf-8?q?6fwtUKy8PeGcPphpWNN+9UBEYCa2jPe9Gn2+pnCAyvdBlQU6LTyVngbW3JhFF2b/R?= =?utf-8?q?tRX7h0j0BPK+PB5ti5To07SzUFZH1W3bWZL0k2AMkjJS4Oi8iXgYEMIvl9SdJAjFo?= =?utf-8?q?gey0Aly4cOvAOOBOSNDW2OyZeDO8HivXLha6RSn5u+piekE9xIcqDIedN8Xv+20f4?= =?utf-8?q?SSZCi+Yfi3CelmHMuR8VwcTjdHJJ4p+CyxZi/kieyuOuCWmUwtvBMY9uB8j2gxada?= =?utf-8?q?gehSIyYNx2SYREsIV0+KavBBErWFl6jXX/lYTevA/hav61AYSYhit2alJ34/UApqp?= =?utf-8?q?EsaWPfaNe4YaOeipYuhN+Ub3CsAx9fctdTdpn/GjjCgmw2L/Kfho3ezqENXNUGN5+?= =?utf-8?q?nzSrz+bxxoeyIy/QDfXrhe/goSGFqeCpTewtxr8NbM8FnbGcWhODvETmJ7hyL6A3S?= =?utf-8?q?e+BlcyZWoHZLEzQlTCAptNWFpzjwbR3gGE1AcP4hxbI6verzD90/h+/I5jOs4mKs3?= =?utf-8?q?MNaFzQLUPCX1uplsNWgZiik3hoprm+3Iq7/pchBC5qK013tne6JjiuYXR4PJCVF7Z?= =?utf-8?q?V5LwjxhjTMaRLMCAL09yVDzHCmTYe/TfsIozXG8TnOj8jrHxzc+kEbWAWHXM7utTw?= =?utf-8?q?fOO5VCPL3aTve/O/v2tAsai084GMB7R5nngiJ6A=3D=3D?= X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: ff5361f5-fcb6-4564-acf0-08dcbf27db12 X-MS-Exchange-CrossTenant-AuthSource: TYSPR06MB6433.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Aug 2024 01:48:28.1669 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEYPR06MB5208 Subject: [FFmpeg-devel] [PATCH 3/4] x86/vvcdec: inter, add optical flow avx2 code X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Nuo Mi Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: pHZe8ZjyYQ+n BDoF used about 10%–25% of the CPU for some clips. Here are the FPS for one run; please ignore the negative values, as they may be due to round-to-round variation clips | before | after | delta --------------------------------------------|--------|-------|------ RitualDance_1920x1080_60_10_420_37_RA.266 | 310.0 | 363.0 | 14.60% NovosobornayaSquare_1920x1080.bin | 322.3 | 339.7 | 5.12% Tango2_3840x2160_60_10_420_27_LD.266 | 71.0 | 68.7 | -3.35% RitualDance_1920x1080_60_10_420_32_LD.266 | 250.0 | 245.3 | -1.92% Chimera_8bit_1080P_1000_frames.vvc | 359.3 | 422.7 | 15.00% BQTerrace_1920x1080_60_10_420_22_RA.vvc | 142.3 | 147.7 | 3.66% --- libavcodec/x86/vvc/Makefile | 1 + libavcodec/x86/vvc/vvc_of.asm | 380 +++++++++++++++++++++++++++++++ libavcodec/x86/vvc/vvcdsp_init.c | 21 ++ 3 files changed, 402 insertions(+) create mode 100644 libavcodec/x86/vvc/vvc_of.asm diff --git a/libavcodec/x86/vvc/Makefile b/libavcodec/x86/vvc/Makefile index 04f16bc10c..aa59aa59cf 100644 --- a/libavcodec/x86/vvc/Makefile +++ b/libavcodec/x86/vvc/Makefile @@ -6,5 +6,6 @@ OBJS-$(CONFIG_VVC_DECODER) += x86/vvc/vvcdsp_init.o \ X86ASM-OBJS-$(CONFIG_VVC_DECODER) += x86/vvc/vvc_alf.o \ x86/vvc/vvc_dmvr.o \ x86/vvc/vvc_mc.o \ + x86/vvc/vvc_of.o \ x86/vvc/vvc_sad.o \ x86/h26x/h2656_inter.o diff --git a/libavcodec/x86/vvc/vvc_of.asm b/libavcodec/x86/vvc/vvc_of.asm new file mode 100644 index 0000000000..b70073fa82 --- /dev/null +++ b/libavcodec/x86/vvc/vvc_of.asm @@ -0,0 +1,380 @@ +; /* +; * Provide AVX2 luma optical flow functions for VVC decoding +; * Copyright (c) 2024 Nuo Mi +; * +; * This file is part of FFmpeg. +; * +; * FFmpeg is free software; you can redistribute it and/or +; * modify it under the terms of the GNU Lesser General Public +; * License as published by the Free Software Foundation; either +; * version 2.1 of the License, or (at your option) any later version. +; * +; * FFmpeg is distributed in the hope that it will be useful, +; * but WITHOUT ANY WARRANTY; without even the implied warranty of +; * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +; * Lesser General Public License for more details. +; * +; * You should have received a copy of the GNU Lesser General Public +; * License along with FFmpeg; if not, write to the Free Software +; * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +; */ +%include "libavutil/x86/x86util.asm" + +%define MAX_PB_SIZE 128 +%define SRC_STRIDE (MAX_PB_SIZE * 2) +%define SRC_PS 2 ; source pixel size, sizeof(int16_t) +%define BDOF_STACK_SIZE 10 ; (4 + 1) * 2, 4 lines + the first line, *2 for h and v +%define bdof_stack_offset(line) ((line) * 2 % BDOF_STACK_SIZE * mmsize) +%define SHIFT 6 +%define SHIFT2 4 + +SECTION_RODATA 32 +pd_15 times 8 dd 15 +pd_m15 times 8 dd -15 +cextern pb_0 + +pb_shuffle_w8 times 2 db 0, 1, 0xff, 0xff, 8, 9, 0xff, 0xff, 6, 7, 0xff, 0xff, 14, 15, 0xff, 0xff +pb_shuffle_w16 times 2 db 0, 1, 0xff, 0xff, 6, 7, 0xff, 0xff, 8, 9, 0xff, 0xff, 14, 15, 0xff, 0xff +pd_perm_w16 dd 0, 2, 1, 4, 3, 6, 5, 7 +%if ARCH_X86_64 + +%if HAVE_AVX2_EXTERNAL + +SECTION .text + +INIT_YMM avx2 + +; dst = (src0 >> shift) - (src1 >> shift) +%macro DIFF 5 ; dst, src0, src1, shift, tmp + psraw %1, %2, %4 + psraw %5, %3, %4 + psubw %1, %5 +%endmacro + +%macro LOAD_GRAD_H 4 ; dst, src, off, tmp + movu %1, [%2 + %3 + 2 * SRC_PS] + movu %4, [%2 + %3] + + DIFF %1, %1, %4, SHIFT, %4 +%endmacro + +%macro SUM_GRAD 2 ;(dst/grad0, grad1) + paddw %1, %2 + psraw %1, 1 ; shift3 +%endmacro + +%macro APPLY_BDOF_MIN_BLOCK_LINE 5 ; dst, vx, vy, tmp, line_num +%define off bdof_stack_offset(%5) + pmullw %1, %2, [rsp + off + 0 * mmsize] ; vx * (gradient_h[0] - gradient_h[1]) + pmullw %4, %3, [rsp + off + 1 * mmsize] ; vy * (gradient_v[0] - gradient_v[1]) + paddw %1, [src0q + (%5 + 1) * SRC_STRIDE + SRC_PS] + paddw %4, [src1q + (%5 + 1) * SRC_STRIDE + SRC_PS] + paddsw %1, %4 ; src0[x] + src1[x] + bdof_offset + pmulhrsw %1, m11 + CLIPW %1, m9, m10 +%endmacro + +%macro SAVE_8BPC 2 ; dst, src + packuswb m%2, m%2 + vpermq m%2, m%2, q0020 + + cmp wd, 16 + je %%w16 + movq %1, xm%2 + jmp %%wend +%%w16: + movu %1, xm%2 +%%wend: +%endmacro + +%macro SAVE_16BPC 2 ; dst, src + cmp wd, 16 + je %%w16 + movu %1, xm%2 + jmp %%wend +%%w16: + movu %1, m%2 +%%wend: +%endmacro + +%macro SAVE 2 ; dst, src + cmp pixel_maxd, (1 << 8) - 1 + jne %%save_16bpc + SAVE_8BPC %1, %2 + jmp %%end +%%save_16bpc: + SAVE_16BPC %1, %2 +%%end: +%endmacro + +; [rsp + even * mmsize] are gradient_h[0] - gradient_h[1] +; [rsp + odd * mmsize] are gradient_v[0] - gradient_v[1] +%macro APPLY_BDOF_MIN_BLOCK 4 ; block_num, vx, vy, bd + vpbroadcastd m9, [pb_0] + + movd xm10, pixel_maxd + vpbroadcastw m10, xm10 + + lea tmp0d, [pixel_maxd + 1] + movd xm11, tmp0d + VPBROADCASTW m11, xm11 ;shift_4 for pmulhrsw + + APPLY_BDOF_MIN_BLOCK_LINE m6, %2, %3, m7, (%1) * 4 + 0 + SAVE [dstq + 0 * dsq], 6 + + APPLY_BDOF_MIN_BLOCK_LINE m6, %2, %3, m7, (%1) * 4 + 1 + SAVE [dstq + 1 * dsq], 6 + + APPLY_BDOF_MIN_BLOCK_LINE m6, %2, %3, m7, (%1) * 4 + 2 + SAVE [dstq + 2 * dsq], 6 + + APPLY_BDOF_MIN_BLOCK_LINE m6, %2, %3, m7, (%1) * 4 + 3 + SAVE [dstq + ds3q], 6 +%endmacro + +%macro SUM_MIN_BLOCK_W16 4 ; src/dst, shuffle, perm, tmp + pshufb %4, %1, %2 + vpermd %4, %3, %4 + paddw %1, %4 +%endmacro + +%macro SUM_MIN_BLOCK_W8 3 ; src/dst, shuffle, tmp + pshufb %3, %1, %2 + paddw %1, %3 +%endmacro + +%macro BDOF_PROF_GRAD 2 ; line_no, last_line +%assign i0 (%1 + 0) % 3 +%assign j0 (%1 + 1) % 3 +%assign k0 (%1 + 2) % 3 +%assign i1 3 + (%1 + 0) % 3 +%assign j1 3 + (%1 + 1) % 3 +%assign k1 3 + (%1 + 2) % 3 + +; we cached src0 in m0 to m2 +%define t0 m %+ i0 +%define c0 m %+ j0 +%define b0 m %+ k0 + +; we cached src1 in m3 to m5 +%define t1 m %+ i1 +%define c1 m %+ j1 +%define b1 m %+ k1 +%define ndiff t1 +%define off bdof_stack_offset(%1) + + movu b0, [src0q + (%1 + 2) * SRC_STRIDE + SRC_PS] + movu b1, [src1q + (%1 + 2) * SRC_STRIDE + SRC_PS] + + ; gradient_v[0], gradient_v[1] + DIFF m6, b0, t0, SHIFT, t0 + DIFF m7, b1, t1, SHIFT, t1 + + ; save gradient_v[0] - gradient_v[1] + psubw m10, m6, m7 + mova [rsp + off + mmsize], m10 + + ; gradient_h[0], gradient_h[1] + LOAD_GRAD_H m8, src0q, (%1 + 1) * SRC_STRIDE, t0 + LOAD_GRAD_H m9, src1q, (%1 + 1) * SRC_STRIDE, t1 + + ; save gradient_h[0] - gradient_h[1] + psubw m11, m8, m9 + mova [rsp + off], m11 + + SUM_GRAD m8, m9 ; temph + SUM_GRAD m6, m7 ; tempv + + DIFF ndiff, c1, c0, SHIFT2, t0 ; -diff + + psignw m7, ndiff, m8 ; sgxdi + psignw m9, ndiff, m6 ; sgydi + psignw m10, m8, m6 ; sgxgy + + pabsw m6, m6 ; sgy2 + pabsw m8, m8 ; sgx2 + + ; use t0, t1 as temporary buffers + cmp wd, 16 + + je %%w16 + mova t0, [pb_shuffle_w8] + SUM_MIN_BLOCK_W8 m6, t0, m11 + SUM_MIN_BLOCK_W8 m7, t0, m11 + SUM_MIN_BLOCK_W8 m8, t0, m11 + SUM_MIN_BLOCK_W8 m9, t0, m11 + SUM_MIN_BLOCK_W8 m10, t0, m11 + jmp %%wend + +%%w16: + mova t0, [pb_shuffle_w16] + mova t1, [pd_perm_w16] + SUM_MIN_BLOCK_W16 m6, t0, t1, m11 + SUM_MIN_BLOCK_W16 m7, t0, t1, m11 + SUM_MIN_BLOCK_W16 m8, t0, t1, m11 + SUM_MIN_BLOCK_W16 m9, t0, t1, m11 + SUM_MIN_BLOCK_W16 m10, t0, t1, m11 + +%%wend: + phaddw m8, m7 + phaddw m6, m9 + phaddw m8, m6 ; 2 sgx2, 2 sgxdi, sgy2, 2 sgydi, 2 sgx2, 2 sgxdi, 2 sgy2, 2 sgydi + +%if (%1) == 0 || (%2) + ; pad for top and bottom + paddw m8, m8 + paddw m10, m10 +%endif + + paddw m12, m8 + paddw m13, m10 +%endmacro + + +%macro LOG2 5; log_sum, src, cmp, shift, tmp + pcmpgtw %5, %2, %3 + pandd %5, %4 + paddw %1, %5 + + psrlw %2, %5 + psrlw %4, 1 + psrlw %3, %4 +%endmacro + +%macro LOG2 3; dst0/src0, dst1/src, offset + pextrd tmp0d, xm%1, %3 + pextrd tmp1d, xm%2, %3 + bsr tmp0d, tmp0d + bsr tmp1d, tmp1d + pinsrd xm%1, tmp0d, %3 + pinsrd xm%2, tmp1d, %3 +%endmacro + +%macro LOG2 2; dst/src, tmp + vperm2i128 m%2, m%1, m%1, 1 + LOG2 %1, %2, 0 + LOG2 %1, %2, 1 + LOG2 %1, %2, 2 + LOG2 %1, %2, 3 + vperm2i128 m%1, m%1, m%2, q0200 +%endmacro + +; %1: 2 sgx2, 2 sgxdi, sgy2, 2 sgydi, 2 sgx2, 2 sgxdi, 2 sgy2, 2 sgydi +; %2: 4 4sgxgy +%macro BDOF_VX_VY 2 ; + pmovsxwd m6, xm%1 + vextracti128 xm7, m%1, 1 + pmovsxwd m7, xm7 + punpcklqdq m8, m6, m7 ; 4 sgx2, 4 sgy2 + punpckhqdq m9, m6, m7 ; 4 sgxdi, 4 sgydi + mova m10, m8 + + LOG2 10, 11 + psignd m11, m9, m8 + pslld m11, 2 + vpsravd m11, m11, m10 + CLIPD m11, [pd_m15], [pd_15] ; 4 vx + + pxor m6, m6 + phaddw m%2, m6 + phaddw m%2, m6 + vpermq m%2, m%2, q0020 + pshufd m%2, m%2, q1120 + pmovsxwd m%2, xmm%2 ; 4 sgxgy + + pmulld m%2, m11 ; 4 vx * sgxgy + psrad m%2, 1 + + pslld m9, 2 + vextracti128 xmm9, m9, 1 ; 4 (sgydi << 2) + psubd m9, m%2 ; 4 ((sgydi << 2) - (vx * sgxgy >> 1)) + + vextracti128 xmm8, m8, 1 ; 4 sgy2 + psignd m9, m8 + + vextracti128 xmm10, m10, 1 ; 4 log2(sgy2) + vpsravd m9, m9, m10 + CLIPD m9, [pd_m15], [pd_15] ; 4 vy + + vpermq m%1, m11, q1100 + pshuflw m%1, m%1, q2200 + vpunpckldq m%1, m%1 ; 4 x 4vx + + vpermq m%2, m9, q1100 + pshuflw m%2, m%2, q2200 + vpunpckldq m%2, m%2 ; 4 x 4vy +%endmacro + + +%macro BDOF_MINI_BLOCKS 2 ; (block_num, last_block) + +%if (%1) == 0 + movu m0, [src0q + 0 * SRC_STRIDE + SRC_PS] + movu m1, [src0q + 1 * SRC_STRIDE + SRC_PS] + movu m3, [src1q + 0 * SRC_STRIDE + SRC_PS] + movu m4, [src1q + 1 * SRC_STRIDE + SRC_PS] + + pxor m12, m12 + pxor m13, m13 + + BDOF_PROF_GRAD 0, 0 +%endif + + mova m14, m12 + mova m15, m13 + + pxor m12, m12 + pxor m13, m13 + BDOF_PROF_GRAD %1 * 4 + 1, 0 + BDOF_PROF_GRAD %1 * 4 + 2, 0 + paddw m14, m12 + paddw m15, m13 + + pxor m12, m12 + pxor m13, m13 + BDOF_PROF_GRAD %1 * 4 + 3, %2 +%if (%2) == 0 + BDOF_PROF_GRAD %1 * 4 + 4, 0 +%endif + paddw m14, m12 + paddw m15, m13 + + BDOF_VX_VY 14, 15 + APPLY_BDOF_MIN_BLOCK %1, m14, m15, bd + lea dstq, [dstq + 4 * dsq] +%endmacro + +;void ff_vvc_apply_bdof_%1(uint8_t *dst, const ptrdiff_t dst_stride, int16_t *src0, int16_t *src1, +; const int w, const int h, const int int pixel_max) +%macro BDOF_AVX2 0 +cglobal vvc_apply_bdof, 7, 10, 16, BDOF_STACK_SIZE*32, dst, ds, src0, src1, w, h, pixel_max, ds3, tmp0, tmp1 + + lea ds3q, [dsq * 3] + sub src0q, SRC_STRIDE + SRC_PS + sub src1q, SRC_STRIDE + SRC_PS + + BDOF_MINI_BLOCKS 0, 0 + + cmp hd, 16 + je .h16 + BDOF_MINI_BLOCKS 1, 1 + jmp .end + +.h16: + BDOF_MINI_BLOCKS 1, 0 + BDOF_MINI_BLOCKS 2, 0 + BDOF_MINI_BLOCKS 3, 1 + +.end: + RET +%endmacro + +%macro VVC_OF_AVX2 0 + BDOF_AVX2 +%endmacro + +VVC_OF_AVX2 + +%endif ; HAVE_AVX2_EXTERNAL + +%endif ; ARCH_X86_64 diff --git a/libavcodec/x86/vvc/vvcdsp_init.c b/libavcodec/x86/vvc/vvcdsp_init.c index d5b4f4f8a5..f3e2e3a27b 100644 --- a/libavcodec/x86/vvc/vvcdsp_init.c +++ b/libavcodec/x86/vvc/vvcdsp_init.c @@ -102,6 +102,20 @@ DMVR_PROTOTYPES( 8, avx2) DMVR_PROTOTYPES(10, avx2) DMVR_PROTOTYPES(12, avx2) +void ff_vvc_apply_bdof_avx2(uint8_t *dst, ptrdiff_t dst_stride, \ + const int16_t *src0, const int16_t *src1, int w, int h, int pixel_max); \ + +#define OF_PROTOTYPES(bd, opt) \ +static void ff_vvc_apply_bdof_##bd##_##opt(uint8_t *dst, ptrdiff_t dst_stride, \ + const int16_t *src0, const int16_t *src1, int w, int h) \ +{ \ + ff_vvc_apply_bdof##_##opt(dst, dst_stride, src0, src1, w, h, (1 << bd) - 1); \ +} \ + +OF_PROTOTYPES( 8, avx2) +OF_PROTOTYPES(10, avx2) +OF_PROTOTYPES(12, avx2) + #define ALF_BPC_PROTOTYPES(bpc, opt) \ void BF(ff_vvc_alf_filter_luma, bpc, opt)(uint8_t *dst, ptrdiff_t dst_stride, \ const uint8_t *src, ptrdiff_t src_stride, ptrdiff_t width, ptrdiff_t height, \ @@ -328,6 +342,10 @@ ALF_FUNCS(16, 12, avx2) c->inter.dmvr[1][1] = ff_vvc_dmvr_hv_##bd##_avx2; \ } while (0) +#define OF_INIT(bd) do { \ + c->inter.apply_bdof = ff_vvc_apply_bdof_##bd##_avx2; \ +} while (0) + #define ALF_INIT(bd) do { \ c->alf.filter[LUMA] = ff_vvc_alf_filter_luma_##bd##_avx2; \ c->alf.filter[CHROMA] = ff_vvc_alf_filter_chroma_##bd##_avx2; \ @@ -352,6 +370,7 @@ void ff_vvc_dsp_init_x86(VVCDSPContext *const c, const int bd) ALF_INIT(8); AVG_INIT(8, avx2); MC_LINKS_AVX2(8); + OF_INIT(8); DMVR_INIT(8); SAD_INIT(); } @@ -365,6 +384,7 @@ void ff_vvc_dsp_init_x86(VVCDSPContext *const c, const int bd) AVG_INIT(10, avx2); MC_LINKS_AVX2(10); MC_LINKS_16BPC_AVX2(10); + OF_INIT(10); DMVR_INIT(10); SAD_INIT(); } @@ -378,6 +398,7 @@ void ff_vvc_dsp_init_x86(VVCDSPContext *const c, const int bd) AVG_INIT(12, avx2); MC_LINKS_AVX2(12); MC_LINKS_16BPC_AVX2(12); + OF_INIT(12); DMVR_INIT(12); SAD_INIT(); }