From patchwork Fri Jun 25 07:54:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Kelly X-Patchwork-Id: 28712 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2042:0:0:0:0 with SMTP id z2csp1228866iod; Fri, 25 Jun 2021 00:55:30 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyeEISRqcfNfxpgP4/Cq2eOPwE7hqSWpUY0wnS1qRg1srG2LUvSnbcwwzZ0RyPjwVpkknWj X-Received: by 2002:a17:906:b191:: with SMTP id w17mr9777909ejy.10.1624607730738; Fri, 25 Jun 2021 00:55:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624607730; cv=none; d=google.com; s=arc-20160816; b=ZFIFC/0RyOOq7QRfd7rsXoisjBj0/sa+PTRMr5UPxrzfZj64xy+7ELiTcfB5k5J5y5 wqJKBYOJCFpAadsrJ8gQ0O7NJEKGZkZWFuvUd1dOK3cJbs6CLHT6tTRfQ7kHLwK55+jp UbOGZ0t5hOC4rJya8nRMsHlrlMHZIBktmcXtqJNSH08M34lp37upo5QU5IJNCcxYuJlP 79WA7ySX95EfUSl2h+LfHEM0OG3I0+P7F5L+5MlVbuJLqKloOGmr87FRS+ZLRMA4hRs1 lPojsDaxejs4pvl6fc0RqhrzjLo+so6qDEVLz22jZ/w3jA8ob2MfsFxnsYGw6N6+qQm9 4p3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:from:references:mime-version :message-id:in-reply-to:date:dkim-signature:delivered-to; bh=RmFQ51Ay4CfdG5niKGhDQcU0gTVys7uQvuLIPfgUM4I=; b=I56d5mlkebddnhWogYW+Rv/XQyHjxlT48NSnlvrewCkGYdBzz4XA9yhSfRc2AJSzRe IG0FrVWGRyNQWFuFZE23bAf+5aFkd6bpSZF+X1+1yCJiDcXb42jUeTkcCODWMavE8iH7 hs8+XwqkgRRjY76zeXVjmF3At4Gv/LKPji4x5qExij+eQLLuRxfUnY1ICZtNSm1ZLpux Nj1kdoxCgfRSTL6wZ0zz9uqcc9Ejc+X+X45d3RC97Y7Um+zA9GFLJFPrSrIuNG2M6DJ/ AMJ1Qwgwv+uULxj46zd+m1OCjZ7f3zp5kIhoZEwEwxE0SzLnKG6mGoiLS32J4OJK9AF7 suGA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20161025 header.b=EbNs6UvE; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y18si5342163edd.585.2021.06.25.00.55.30; Fri, 25 Jun 2021 00:55:30 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20161025 header.b=EbNs6UvE; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4E0F5689F3A; Fri, 25 Jun 2021 10:55:26 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1733F680814 for ; Fri, 25 Jun 2021 10:55:18 +0300 (EEST) Received: by mail-qk1-f202.google.com with SMTP id t131-20020a37aa890000b02903a9f6c1e8bfso9357068qke.10 for ; Fri, 25 Jun 2021 00:55:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qTxUkiwseX3PVGeHP5vnbm0bw8+6ZSKAJDkBDD6cj7o=; b=EbNs6UvEw+kn3G3OEXEaEhRTzRnMRgZL+CeGd72xD94rTjnL+Nj7VYP56n46XqHlNQ UHbuY+K50Am8ClN4LXHUSlqrqoOd8/lbeD95PDyBC/ifZdYVimV+mxhRh7EXQnCz9lMD AKA3Ie+6iv2QHLieg+w06+vt2NzH6x26VhI1acGMZ34JyDJuanc6VmlMamQKGreMjBZw JwD4GdU0x/Os6gyAH4stznD+x+dxEwNY0hccTLIRjpXyXDoyjt2geGmHF7VdYHUDre1C +BtPACkav1jINOJtBcMXd4bLC9ydJnYA6gkN/FeNdrZ0l2TNPuo9Vu9XN2phjCjITywz JmXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qTxUkiwseX3PVGeHP5vnbm0bw8+6ZSKAJDkBDD6cj7o=; b=MFeJw4EvQ0qHuksTflHm528Y9ufb47bNyJuDNXegc1DFehNFQcOnJUVKfUEBJHuT3W p0aXoBgr/+8jdIiTlys1FS+5KUMvKup6iJjWUOn8N5Alb2tYEZk2/1s4tJy7EIfUzdmd 9i+hTvIBU1CInf1GX7jQSWuRevgzbUHaVs0eVqmF3wh0Ml9yHypTdoZrPEZ2D3BBhr6l JFHx3j+A4T0qiapkKGOokuh+9zZnJhe7b8VVw0+ZqNYleUbJi6unWY2lEDe/5F6sZZCx g1m3wa2KdnXYeJAsSu1SFkaXueghiZHRwezelrqTb4aY065vEUQo3Dp21eaTT9sO7vpG JhZA== X-Gm-Message-State: AOAM533c2y2f4Rf68DzAmCaVHq8L58VcqCeJEK1ppJuLwYLgL0tGRroS kt+48hpXoKl5dfhFAofjNKZfQ/Y8+EUPcUDSI1rdWxTQYjuUD0HnL8HWo8OxFCNE5FEI47afHz2 miWiXV/JqTTKmkLmoh+3onoX/OdGp5vtCm1rEdD1FJY+HbRbREvBAVCJkXAxeHu9bp5t2CvM= X-Received: from alankelly0.zrh.corp.google.com ([2a00:79e0:61:301:553c:d859:29bd:64e0]) (user=alankelly job=sendgmr) by 2002:ad4:5609:: with SMTP id ca9mr9729399qvb.26.1624607717282; Fri, 25 Jun 2021 00:55:17 -0700 (PDT) Date: Fri, 25 Jun 2021 09:54:29 +0200 In-Reply-To: Message-Id: <20210625075429.72269-1-alankelly@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog From: Alan Kelly To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH 1/2] libavutil/cpu: Adds fast gather detection. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Alan Kelly Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: F67eCFLaNezE Broadwell and later and Zen3 and later have fast gather instructions. --- Gather requires between 9 and 12 cycles on Haswell, 5 to 7 on Broadwell, and 2 to 5 on Skylake and newer. It is also slow on AMD before Zen 3. libavutil/cpu.h | 2 ++ libavutil/x86/cpu.c | 18 ++++++++++++++++-- libavutil/x86/cpu.h | 1 + 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/libavutil/cpu.h b/libavutil/cpu.h index b555422dae..f94eb79af1 100644 --- a/libavutil/cpu.h +++ b/libavutil/cpu.h @@ -50,6 +50,7 @@ #define AV_CPU_FLAG_FMA4 0x0800 ///< Bulldozer FMA4 functions #define AV_CPU_FLAG_CMOV 0x1000 ///< supports cmov instruction #define AV_CPU_FLAG_AVX2 0x8000 ///< AVX2 functions: requires OS support even if YMM registers aren't used +#define AV_CPU_FLAG_AVX2SLOW 0x2000000 ///< AVX2 supported but gather is slower. #define AV_CPU_FLAG_FMA3 0x10000 ///< Haswell FMA3 functions #define AV_CPU_FLAG_BMI1 0x20000 ///< Bit Manipulation Instruction Set 1 #define AV_CPU_FLAG_BMI2 0x40000 ///< Bit Manipulation Instruction Set 2 @@ -107,6 +108,7 @@ int av_cpu_count(void); * av_set_cpu_flags_mask(), then this function will behave as if AVX is not * present. */ + size_t av_cpu_max_align(void); #endif /* AVUTIL_CPU_H */ diff --git a/libavutil/x86/cpu.c b/libavutil/x86/cpu.c index bcd41a50a2..56fcde594c 100644 --- a/libavutil/x86/cpu.c +++ b/libavutil/x86/cpu.c @@ -146,8 +146,20 @@ int ff_get_cpu_flags_x86(void) if (max_std_level >= 7) { cpuid(7, eax, ebx, ecx, edx); #if HAVE_AVX2 - if ((rval & AV_CPU_FLAG_AVX) && (ebx & 0x00000020)) + if ((rval & AV_CPU_FLAG_AVX) && (ebx & 0x00000020)){ rval |= AV_CPU_FLAG_AVX2; + + cpuid(1, eax, ebx, ecx, std_caps); + family = ((eax >> 8) & 0xf) + ((eax >> 20) & 0xff); + model = ((eax >> 4) & 0xf) + ((eax >> 12) & 0xf0); + // Haswell and earlier has slow gather + if(family == 6 && model < 70) + rval |= AV_CPU_FLAG_AVX2SLOW; + // Zen 2 and earlier + if (!strncmp(vendor.c, "AuthenticAMD", 12) && family < 25) + rval |= AV_CPU_FLAG_AVX2SLOW; + } + #if HAVE_AVX512 /* F, CD, BW, DQ, VL */ if ((xcr0_lo & 0xe0) == 0xe0) { /* OPMASK/ZMM state */ if ((rval & AV_CPU_FLAG_AVX2) && (ebx & 0xd0030000) == 0xd0030000) @@ -194,8 +206,10 @@ int ff_get_cpu_flags_x86(void) functions using XMM registers are always faster on them. AV_CPU_FLAG_AVX and AV_CPU_FLAG_AVXSLOW are both set so that AVX is used unless explicitly disabled by checking AV_CPU_FLAG_AVXSLOW. */ - if ((family == 0x15 || family == 0x16) && (rval & AV_CPU_FLAG_AVX)) + if ((family == 0x15 || family == 0x16) && (rval & AV_CPU_FLAG_AVX)){ rval |= AV_CPU_FLAG_AVXSLOW; + rval |= AV_CPU_FLAG_AVX2SLOW; + } } /* XOP and FMA4 use the AVX instruction coding scheme, so they can't be diff --git a/libavutil/x86/cpu.h b/libavutil/x86/cpu.h index 937c697fa0..a42a15a997 100644 --- a/libavutil/x86/cpu.h +++ b/libavutil/x86/cpu.h @@ -78,6 +78,7 @@ #define EXTERNAL_AVX2(flags) CPUEXT_SUFFIX(flags, _EXTERNAL, AVX2) #define EXTERNAL_AVX2_FAST(flags) CPUEXT_SUFFIX_FAST2(flags, _EXTERNAL, AVX2, AVX) #define EXTERNAL_AVX2_SLOW(flags) CPUEXT_SUFFIX_SLOW2(flags, _EXTERNAL, AVX2, AVX) +#define EXTERNAL_AVX2_FAST_GATHER(flags) CPUEXT_SUFFIX_FAST(flags, _EXTERNAL, AVX2) #define EXTERNAL_AESNI(flags) CPUEXT_SUFFIX(flags, _EXTERNAL, AESNI) #define EXTERNAL_AVX512(flags) CPUEXT_SUFFIX(flags, _EXTERNAL, AVX512) From patchwork Fri Jun 25 07:59:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Kelly X-Patchwork-Id: 28713 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2042:0:0:0:0 with SMTP id z2csp1231189iod; Fri, 25 Jun 2021 00:59:59 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwXZdfzinERtNgUukiqQFsaX+ycn514pUNTHETSG//N6Mde9hYekRP1n4rIFvQ3rscJl7hy X-Received: by 2002:a17:906:2dd5:: with SMTP id h21mr9726401eji.522.1624607999004; Fri, 25 Jun 2021 00:59:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624607998; cv=none; d=google.com; s=arc-20160816; b=J5pflmGBrKA/9oyVf+xz50lILXkZwEKIbvO5etLAV848StWRxNq8c6pWJiXXnaqorK o0l5zI6842Arwe2+LSzVk5fRv/N29uS5Mm3aVrehmfUX8O/DmwlfcFyO5WF3rC1v1vMb 1Gz8wsDYpXnl+0V+bWkiC49EMHGDyrCDesgrTN494zMB+pipr7+1vHRUZTu9FiA0EToE zWPzx7dOd+M9uq4ztm277+sOxUBBPf90RrUVJiEaePdP/vVTvq6HmN59sEYGA2QaePu9 vNWBQpe+AT63vyj2dijZwxo3eeyxdGtKmN7sL/JPCcSTKekrTTtbv9Wx1D2MFHEfYkKp wxTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:from:references:mime-version :message-id:in-reply-to:date:dkim-signature:delivered-to; bh=1wi3l75uwnRphYQhiF3HyF9seAdSZ9KANYHShBD05LE=; b=HZwPF6emTJ0QUXSarO0D4SBN5NNe8JCy0SlqH/Ug1A2n2KdZ8qwyvNOgKUcHEZSQgs rcrVZHF7uIQ51dlCZPrhHMR1pkbosOGVJR4GWtmJ9qJLj5c6Ajy99JkoYjjRtW5Quta6 gmISSCU5Hs4AtqP1Z3Vu5IRZef+jemUj7n8c+ExTaZj2wjYiRPJW+kdT8Qs56sZzDhL2 Ua1dHUQ5cFhNcyjRHBLGSBJbnyckiukYz+FlGOnArGe1QTfLZo/9E7wSq9KR5dsOJ693 ED2s6erJ9+/cIkkQF2hmj+13LB+vpsIMjv30QIrgbvajAdF0TC5KirFQothpMwC94s6N Rksw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20161025 header.b=c4ePp47+; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id v16si5316257edx.140.2021.06.25.00.59.58; Fri, 25 Jun 2021 00:59:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20161025 header.b=c4ePp47+; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D07666809BA; Fri, 25 Jun 2021 10:59:55 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B643A6809BA for ; Fri, 25 Jun 2021 10:59:48 +0300 (EEST) Received: by mail-ej1-f73.google.com with SMTP id ci22-20020a170906c356b0290492ca430d87so2793642ejb.14 for ; Fri, 25 Jun 2021 00:59:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=LkhEJBcRGjEIFeV+iojhzrLWdSJEnbiyL+uasVQJwLU=; b=c4ePp47+m76MH8NwtceR5O8Mnda6NW3iRSMLdKQYpsnghIs6K8xlfDOIWF1cR8EZ86 NF8oI6h7y1hy4QhZ/uGK2AZAFkQLyj+QfntAY8t8rFX4wAgAvj4OmUvJujhhB7DEA/2j AdZ6NyWaOSG2zRc7kt05qefmRIKTBXQC3GS+SqWQogtYRCjHgv5nYwbqhmXCxL/ZdYiL uPKSGFg+cvM1DSkIyi6nSEbvZGBPrxkHnsp9ZiRl0L1eL0Rd5rP/EC9RFRl+/zfY4bhv ZzyNYb1lE5ytaRwKoy7BQ8HCbaGlhprbOTQrqkASyrGqdpzT0n18Zk326soj9Ow6rBPg ImHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=LkhEJBcRGjEIFeV+iojhzrLWdSJEnbiyL+uasVQJwLU=; b=CNiDGs1dUK1s8q2k5X6euUAuXbW5G3eRnY5K4xIho/yxkZgNtD8pVD2WXQhbXTrTbn vhkzGF6XuRK87FvfDFgs46XnrmNfYPOwA5bFTE9Mb1ZxwyraLI4qy2ue31WQwvFLYNQg dGHYyU3zEfZ4LAsuUNwOiVzNzGq+IyrUaOYWawib1GtaJio4oc4LbDpWlKOMOyIF5IWk IfhkfFWix7+o+8KIYiTTw2Hq19YiWiaYH6StpxkZSKA4dyJjlrhnpxQjQKIaN5ZvZZRC fupAc0ycSz8vOmCt8Y42ciy0TLyO6fO4kS7CSff5vjWXvVdCSw+LvPGumOCGATmmyB22 oySg== X-Gm-Message-State: AOAM530qh3iPSpglNJzlzakwe9ZGAMobh8REv1PulrzX6Xq5vh56n7KS TejLqFfIqj5OvYKGVET3SBxDpasMHfWOI0YO1vjMxwe6quymxJfhlX1oQhiJB9vXwJsmjEToTn2 StATIJzt6vjN2h1MkR1Y5Ejv6FVZOcLfIYPdGXk/IRNEa9yqOlAG5Y5W8+nlfg9332YgskOI= X-Received: from alankelly0.zrh.corp.google.com ([2a00:79e0:61:301:553c:d859:29bd:64e0]) (user=alankelly job=sendgmr) by 2002:a17:906:2b0a:: with SMTP id a10mr9429877ejg.521.1624607988031; Fri, 25 Jun 2021 00:59:48 -0700 (PDT) Date: Fri, 25 Jun 2021 09:59:43 +0200 In-Reply-To: <20210614111407.1897690-2-alankelly@google.com> Message-Id: <20210625075943.79619-1-alankelly@google.com> Mime-Version: 1.0 References: <20210614111407.1897690-2-alankelly@google.com> X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog From: Alan Kelly To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH 2/2] libswscale: Adds ff_hscale8to15_4_avx2 and ff_hscale8to15_X4_avx2 for all filter sizes. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Alan Kelly Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: u6CXG3h8jMGe These functions replace all ff_hscale8to15_*_ssse3 when avx2 is available. --- libswscale/swscale_internal.h | 2 + libswscale/utils.c | 37 +++++++++++ libswscale/x86/Makefile | 1 + libswscale/x86/scale_avx2.asm | 112 ++++++++++++++++++++++++++++++++++ libswscale/x86/swscale.c | 19 ++++++ tests/checkasm/sw_scale.c | 21 +++++-- 6 files changed, 187 insertions(+), 5 deletions(-) create mode 100644 libswscale/x86/scale_avx2.asm diff --git a/libswscale/swscale_internal.h b/libswscale/swscale_internal.h index a1de95cee0..45ef657cd4 100644 --- a/libswscale/swscale_internal.h +++ b/libswscale/swscale_internal.h @@ -1056,4 +1056,6 @@ void ff_init_vscale_pfn(SwsContext *c, yuv2planar1_fn yuv2plane1, yuv2planarX_fn //number of extra lines to process #define MAX_LINES_AHEAD 4 +//shuffle filter and filterPos for hyScale and hcScale filters in avx2 +void ff_shuffle_filter_coefficients(SwsContext *c, int* filterPos, int filterSize, int16_t *filter, int dstW); #endif /* SWSCALE_SWSCALE_INTERNAL_H */ diff --git a/libswscale/utils.c b/libswscale/utils.c index 6bac7b658d..07c4d2f741 100644 --- a/libswscale/utils.c +++ b/libswscale/utils.c @@ -267,6 +267,41 @@ static const FormatEntry format_entries[] = { [AV_PIX_FMT_X2RGB10LE] = { 1, 1 }, }; +void ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSize, int16_t *filter, int dstW){ +#if ARCH_X86_64 + int i, j, k, l; + int cpu_flags = av_get_cpu_flags(); + if (EXTERNAL_AVX2_FAST_GATHER(cpu_flags)){ + if ((c->srcBpc == 8) && (c->dstBpc <= 14)){ + if (dstW % 16 == 0){ + if (filter != NULL){ + for (i = 0; i < dstW; i += 8){ + FFSWAP(int, filterPos[i + 2], filterPos[i+4]); + FFSWAP(int, filterPos[i + 3], filterPos[i+5]); + } + if (filterSize > 4){ + int16_t *tmp2 = av_malloc(dstW * filterSize * 2); + memcpy(tmp2, filter, dstW * filterSize * 2); + for (i = 0; i < dstW; i += 16){//pixel + for (k = 0; k < filterSize / 4; ++k){//fcoeff + for (j = 0; j < 16; ++j){//inner pixel + for (l = 0; l < 4; ++l){//coeff + int from = i * filterSize + j * filterSize + k * 4 + l; + int to = (i) * filterSize + j * 4 + l + k * 64; + filter[to] = tmp2[from]; + } + } + } + } + av_free(tmp2); + } + } + } + } + } +#endif +} + int sws_isSupportedInput(enum AVPixelFormat pix_fmt) { return (unsigned)pix_fmt < FF_ARRAY_ELEMS(format_entries) ? @@ -1697,6 +1732,7 @@ av_cold int sws_init_context(SwsContext *c, SwsFilter *srcFilter, get_local_pos(c, 0, 0, 0), get_local_pos(c, 0, 0, 0))) < 0) goto fail; + ff_shuffle_filter_coefficients(c, c->hLumFilterPos, c->hLumFilterSize, c->hLumFilter, dstW); if ((ret = initFilter(&c->hChrFilter, &c->hChrFilterPos, &c->hChrFilterSize, c->chrXInc, c->chrSrcW, c->chrDstW, filterAlign, 1 << 14, @@ -1706,6 +1742,7 @@ av_cold int sws_init_context(SwsContext *c, SwsFilter *srcFilter, get_local_pos(c, c->chrSrcHSubSample, c->src_h_chr_pos, 0), get_local_pos(c, c->chrDstHSubSample, c->dst_h_chr_pos, 0))) < 0) goto fail; + ff_shuffle_filter_coefficients(c, c->hChrFilterPos, c->hChrFilterSize, c->hChrFilter, c->chrDstW); } } // initialize horizontal stuff diff --git a/libswscale/x86/Makefile b/libswscale/x86/Makefile index bfe383364e..68391494be 100644 --- a/libswscale/x86/Makefile +++ b/libswscale/x86/Makefile @@ -11,6 +11,7 @@ OBJS-$(CONFIG_XMM_CLOBBER_TEST) += x86/w64xmmtest.o X86ASM-OBJS += x86/input.o \ x86/output.o \ x86/scale.o \ + x86/scale_avx2.o \ x86/rgb_2_rgb.o \ x86/yuv_2_rgb.o \ x86/yuv2yuvX.o \ diff --git a/libswscale/x86/scale_avx2.asm b/libswscale/x86/scale_avx2.asm new file mode 100644 index 0000000000..d90fd2d791 --- /dev/null +++ b/libswscale/x86/scale_avx2.asm @@ -0,0 +1,112 @@ +;****************************************************************************** +;* x86-optimized horizontal line scaling functions +;* Copyright 2020 Google LLC +;* Copyright (c) 2011 Ronald S. Bultje +;* +;* This file is part of FFmpeg. +;* +;* FFmpeg is free software; you can redistribute it and/or +;* modify it under the terms of the GNU Lesser General Public +;* License as published by the Free Software Foundation; either +;* version 2.1 of the License, or (at your option) any later version. +;* +;* FFmpeg is distributed in the hope that it will be useful, +;* but WITHOUT ANY WARRANTY; without even the implied warranty of +;* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;* Lesser General Public License for more details. +;* +;* You should have received a copy of the GNU Lesser General Public +;* License along with FFmpeg; if not, write to the Free Software +;* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +;****************************************************************************** + +%include "libavutil/x86/x86util.asm" + +SECTION_RODATA + +swizzle: dd 0, 4, 1, 5, 2, 6, 3, 7 +four: times 8 dd 4 + +SECTION .text + +;----------------------------------------------------------------------------- +; horizontal line scaling +; +; void hscale8to15__ +; (SwsContext *c, int16_t *dst, +; int dstW, const uint8_t *src, +; const int16_t *filter, +; const int32_t *filterPos, int filterSize); +; +; Scale one horizontal line. Input is 8-bit width Filter is 14 bits. Output is +; 15 bits (in int16_t). Each output pixel is generated from $filterSize input +; pixels, the position of the first pixel is given in filterPos[nOutputPixel]. +;----------------------------------------------------------------------------- + +%macro SCALE_FUNC 1 +cglobal hscale8to15_%1, 7, 9, 15, pos0, dst, w, srcmem, filter, fltpos, fltsize, count, inner + pxor m0, m0 + movu m15, [swizzle] + mov countq, $0 +%ifidn %1, X4 + movu m14, [four] + movsxd fltsizeq, fltsized + shr fltsizeq, 2 +%endif +.loop: + movu m1, [fltposq] + movu m2, [fltposq+32] +%ifidn %1, X4 + pxor m9, m9 + pxor m10, m10 + pxor m11, m11 + pxor m12, m12 + mov innerq, $0 +.innerloop: +%endif + vpcmpeqd m13, m13 + vpgatherdd m3,[srcmemq + m1], m13 + vpcmpeqd m13, m13 + vpgatherdd m4,[srcmemq + m2], m13 + vpunpcklbw m5, m3, m0 + vpunpckhbw m6, m3, m0 + vpunpcklbw m7, m4, m0 + vpunpckhbw m8, m4, m0 + vpmaddwd m5, m5, [filterq] + vpmaddwd m6, m6, [filterq + 32] + vpmaddwd m7, m7, [filterq + 64] + vpmaddwd m8, m8, [filterq + 96] + add filterq, $80 +%ifidn %1, X4 + paddd m9, m5 + paddd m10, m6 + paddd m11, m7 + paddd m12, m8 + paddd m1, m14 + paddd m2, m14 + add innerq, $1 + cmp innerq, fltsizeq + jl .innerloop + vphaddd m5, m9, m10 + vphaddd m6, m11, m12 +%else + vphaddd m5, m5, m6 + vphaddd m6, m7, m8 +%endif + vpsrad m5, 7 + vpsrad m6, 7 + vpackssdw m5, m5, m6 + vpermd m5, m15, m5 + vmovdqu [dstq + countq * 2], m5 + add fltposq, $40 + add countq, $10 + cmp countq, wq + jl .loop +REP_RET +%endmacro + +%if ARCH_X86_64 +INIT_YMM avx2 +SCALE_FUNC 4 +SCALE_FUNC X4 +%endif diff --git a/libswscale/x86/swscale.c b/libswscale/x86/swscale.c index 0848a31461..4412ff5f92 100644 --- a/libswscale/x86/swscale.c +++ b/libswscale/x86/swscale.c @@ -276,6 +276,9 @@ SCALE_FUNCS_SSE(sse2); SCALE_FUNCS_SSE(ssse3); SCALE_FUNCS_SSE(sse4); +SCALE_FUNC(4, 8, 15, avx2); +SCALE_FUNC(X4, 8, 15, avx2); + #define VSCALEX_FUNC(size, opt) \ void ff_yuv2planeX_ ## size ## _ ## opt(const int16_t *filter, int filterSize, \ const int16_t **src, uint8_t *dest, int dstW, \ @@ -568,6 +571,22 @@ switch(c->dstBpc){ \ } #if ARCH_X86_64 +#define ASSIGN_AVX2_SCALE_FUNC(hscalefn, filtersize) \ + switch (filtersize) { \ + case 4: hscalefn = ff_hscale8to15_4_avx2; break; \ + default: hscalefn = ff_hscale8to15_X4_avx2; break; \ + break; \ + } + + if (EXTERNAL_AVX2_FAST_GATHER(cpu_flags)){ + if ((c->srcBpc == 8) && (c->dstBpc <= 14)){ + if(c->chrDstW % 16 == 0) + ASSIGN_AVX2_SCALE_FUNC(c->hcScale, c->hChrFilterSize); + if(c->dstW % 16 == 0) + ASSIGN_AVX2_SCALE_FUNC(c->hyScale, c->hLumFilterSize); + } + } + if (EXTERNAL_AVX2_FAST(cpu_flags)) { switch (c->dstFormat) { case AV_PIX_FMT_NV12: diff --git a/tests/checkasm/sw_scale.c b/tests/checkasm/sw_scale.c index 3ac0f9082f..177f9df3c4 100644 --- a/tests/checkasm/sw_scale.c +++ b/tests/checkasm/sw_scale.c @@ -135,13 +135,13 @@ static void check_yuv2yuvX(void) } #undef SRC_PIXELS -#define SRC_PIXELS 128 +#define SRC_PIXELS 512 static void check_hscale(void) { #define MAX_FILTER_WIDTH 40 -#define FILTER_SIZES 5 - static const int filter_sizes[FILTER_SIZES] = { 4, 8, 16, 32, 40 }; +#define FILTER_SIZES 6 + static const int filter_sizes[FILTER_SIZES] = { 4, 8, 12, 16, 32, 40 }; #define HSCALE_PAIRS 2 static const int hscale_pairs[HSCALE_PAIRS][2] = { @@ -160,6 +160,8 @@ static void check_hscale(void) // padded LOCAL_ALIGNED_32(int16_t, filter, [SRC_PIXELS * MAX_FILTER_WIDTH + MAX_FILTER_WIDTH]); LOCAL_ALIGNED_32(int32_t, filterPos, [SRC_PIXELS]); + LOCAL_ALIGNED_32(int16_t, filterAvx2, [SRC_PIXELS * MAX_FILTER_WIDTH + MAX_FILTER_WIDTH]); + LOCAL_ALIGNED_32(int32_t, filterPosAvx, [SRC_PIXELS]); // The dst parameter here is either int16_t or int32_t but we use void* to // just cover both cases. @@ -167,6 +169,8 @@ static void check_hscale(void) const uint8_t *src, const int16_t *filter, const int32_t *filterPos, int filterSize); + int cpu_flags = av_get_cpu_flags(); + ctx = sws_alloc_context(); if (sws_init_context(ctx, NULL, NULL) < 0) fail(); @@ -180,9 +184,11 @@ static void check_hscale(void) ctx->srcBpc = hscale_pairs[hpi][0]; ctx->dstBpc = hscale_pairs[hpi][1]; ctx->hLumFilterSize = ctx->hChrFilterSize = width; + ctx->dstW = ctx->chrDstW = SRC_PIXELS; for (i = 0; i < SRC_PIXELS; i++) { filterPos[i] = i; + filterPosAvx[i] = i; // These filter cofficients are chosen to try break two corner // cases, namely: @@ -210,6 +216,11 @@ static void check_hscale(void) filter[SRC_PIXELS * width + i] = rnd(); } + memcpy(filterAvx2, filter, sizeof(uint16_t) * (SRC_PIXELS * MAX_FILTER_WIDTH + MAX_FILTER_WIDTH)); + if (cpu_flags & AV_CPU_FLAG_AVX2){ + ff_shuffle_filter_coefficients(ctx, filterPosAvx, width, filterAvx2, SRC_PIXELS); + } + ff_getSwsFunc(ctx); if (check_func(ctx->hcScale, "hscale_%d_to_%d_width%d", ctx->srcBpc, ctx->dstBpc + 1, width)) { @@ -217,10 +228,10 @@ static void check_hscale(void) memset(dst1, 0, SRC_PIXELS * sizeof(dst1[0])); call_ref(NULL, dst0, SRC_PIXELS, src, filter, filterPos, width); - call_new(NULL, dst1, SRC_PIXELS, src, filter, filterPos, width); + call_new(NULL, dst1, SRC_PIXELS, src, filterAvx2, filterPosAvx, width); if (memcmp(dst0, dst1, SRC_PIXELS * sizeof(dst0[0]))) fail(); - bench_new(NULL, dst0, SRC_PIXELS, src, filter, filterPos, width); + bench_new(NULL, dst0, SRC_PIXELS, src, filter, filterPosAvx, width); } } }